OVM server after upgraded, virtual machine ASM diskgroup couldn't mpuntd
Knowledge Review completed
|  |
|
Hello Bin-Bin,
It seem to me that the reason why ALPHAPROD_DATA can not be mounted is due to the following error. -- alert_+ASM.log Sun Jun 29 15:45:30 2014 ORA-15085: ASM disk "" has inconsistent sector size. alert_+ASM.log Sun Jun 29 16:12:00 2014 ORA-15085: ASM disk "" has inconsistent sector size. alert_+ASM.log Sun Jun 29 16:19:28 2014 ORA-15085: ASM disk "" has inconsistent sector size. -- This indicates that the sector size for the underlying device mapped to /dev/oracleasm/disks/MFGBLOB_DATA has changed to 4096 which may have been 512 byte before. Action Plan ------------------ 1. Can you try to mount ALPHAPROD_DATA in the following way ? $ sqlplus / as sysasm SQL> alter system set asm_diskstring='/dev/oracleasm/disks/*' ; SQL> alter diskgroup ALPHAPROD_DATA mount ; <<< If ALPHAPROD_DATA can be mounted in this way, you can startup the databases running on this server. 2. But you will need to find out why the secor size has been changed for the underlying device mapped to /dev/oracleasm/disks/MFGBLOB_DATA. Show us the following information $ cat /proc/partitions $ ls -l /sys/block/* Regards, Eun Heo | |
| Oracle Support | - 3+ months ago | | [ODM Proposed Solution Justif] |
|
Hello Bin-Bin,
It seem to me that the reason why ALPHAPROD_DATA can not be mounted is due to the following error. -- alert_+ASM.log Sun Jun 29 15:45:30 2014 ORA-15085: ASM disk "" has inconsistent sector size. alert_+ASM.log Sun Jun 29 16:12:00 2014 ORA-15085: ASM disk "" has inconsistent sector size. alert_+ASM.log Sun Jun 29 16:19:28 2014 ORA-15085: ASM disk "" has inconsistent sector size. -- This indicates that the sector size for the underlying device mapped to /dev/oracleasm/disks/MFGBLOB_DATA has changed to 4096 which may have been 512 byte before. Action Plan ------------------ 1. Can you try to mount ALPHAPROD_DATA in the following way ? $ sqlplus / as sysasm SQL> alter system set asm_diskstring='/dev/oracleasm/disks/*' ; SQL> alter diskgroup ALPHAPROD_DATA mount ; <<< If ALPHAPROD_DATA can be mounted in this way, you can startup the databases running on this server. 2. But you will need to find out why the secor size has been changed for the underlying device mapped to /dev/oracleasm/disks/MFGBLOB_DATA. Show us the following information $ cat /proc/partitions $ ls -l /sys/block/* Regards, Eun Heo | |
| BLIU@KEIHIN-NA.COM | - 3+ months ago | | [Update from Customer] |
|
Everything is good. I am going to close this SR.
Thank you for all your help. Bin-Bin | |
| Oracle Support | - 3+ months ago | | [Information] |
|
lowering to SEV2 per customer update and waiting for update
| |
| Oracle Support | - 4+ months ago | | [Information] |
|
Hi Bin-Bin,
Sure. Do you want me to lower this SR to a SEV2? Then you can update on Monday, is that ok? Thanks George Working Hours: Mon-Fri 09:00am to 5:30 PM Mountain Time Zone | |
| BLIU@KEIHIN-NA.COM | - 4+ months ago | | [Update from Customer] |
|
i am going to lower severity level. we are shutdown until next Monday(7/7/2014).
I'd like to keep this SR until next Monday. They are production system. Can I do that? Bin-Bin | |
| Oracle Support | - 4+ months ago | | [Information] |
|
Hi Bin-Bin,
It appears so. How would you like to proceed with this SR? Thanks George Working Hours: Mon-Fri 09:00am to 5:30 PM Mountain Time Zone | |
| BLIU@KEIHIN-NA.COM | - 4+ months ago | | [Update from Customer] |
|
i can mount diskgroup, and no error. so i am good, right?
Thanks! Bin-Bin | |
| Oracle Support | - 4+ months ago | | [Information] |
|
Hi Bin-Bin,
Are you able to mount the diskgroups? Any errors in the ASM alert.log? If not, then it would seem ok. Thanks George Working Hours: Mon-Fri 09:00am to 5:30 PM Mountain Time Zone | |
| Oracle Support | - 4+ months ago | | [Information] |
|
Hi Bin-Bin,
I have been assigned this SR and I am currently reviewing. I will provide an update as soon as possible. Thanks George Working Hours: Mon-Fri 09:00am to 5:30 PM Mountain Time Zone | |
| BLIU@KEIHIN-NA.COM | - 4+ months ago | | [Update from Customer] |
|
i upgraded the second without problem. i set ORACLEASM_USE_LOGICAL_BLOCK_SIZE to true.
the first server which i had problem yesterday, i changed ORACLEASM_USE_LOGICAL_BLOCK_SIZE to true. but i cannot change asm_diskstring to MFGBLOB_DATA. sm_diskstring is still '/dev/oracleasm/disks/* '. do you think is it ok? i don't have any problem after rebooted. Bin-Bin | |
| Oracle Support | - 4+ months ago | | [Information] |
|
Hello Bin-Bin,
Changing ORACLEASM_USE_LOGICAL_BLOCK_SIZE to true before reboot won't do any harm in your env, so you can change it before system reboot ( or upgrade ). There is no way of assinging the sr to a particular analyst on creating a sr as assigning procedure is all automatic. I will keep this sr in my queue as Sev1 for the time being. If you have a similar issue for another server and update this sr , the sr will get transferred to the next available analyst if I am not working. This way, you can get a help from other analysts . Thanks for your kind words, anyway. Regards, Eun Heo | |
| BLIU@KEIHIN-NA.COM | - 4+ months ago | | [Update from Customer] |
|
if i have issue in the future, can i assign my SR to you? if it is not severity 1, can i do that?
if i can, how to do it? Bin-Bin | |
| BLIU@KEIHIN-NA.COM | - 4+ months ago | | [Update from Customer] |
|
can i change "ORACLEASM_USE_LOGICAL_BLOCK_SIZE" to true before i reboot, even i din't know i will have problem or not?
Bin-Bin | |
| Oracle Support | - 4+ months ago | | [Information] |
|
Hello Bin-Bin,
I am actually based in Sydney - Australia. Please find my working hours below and start working 2PM in PST timezone. Working Hours: Monday – Friday 07:00-15:30 AEST / 21:00-05:30 GMT / 14:00-22:30 PST Please check ASM alert.log for the following error if you see the same issue. - alert_+ASM.log Sun Jun 29 15:45:30 2014 ORA-15085: ASM disk "" has inconsistent sector size. If this is the case for another server, you can do one of the solution provided. 1] Change asm_diskstring to /dev/oracleasm/disks/* OR 2] Reboot the server after changing the following parameter in /etc/sysconfig/oracleasm file. ORACLEASM_USE_LOGICAL_BLOCK_SIZE=true Regards, Eun | |
| BLIU@KEIHIN-NA.COM | - 4+ months ago | | [Update from Customer] |
|
i am in EST time. what's yours?
| |
| BLIU@KEIHIN-NA.COM | - 4+ months ago | | [Update from Customer] |
|
that is great, so this server is ready.
i am going to upgrade another server tomorrow. i am going to do the same work like this one tomorrow. if you can keep it open until i done the upgrade tomorrow. it will be the best. my time is 10:23PM, how about yours? you are very helpful, if it is possible, i want you be my contact. Thanks! Bin-Bin | |
| Oracle Support | - 4+ months ago | | [Information] |
|
Hello Bin-Bin,
Our previous posting seemed to be crossing. Yes, "/dev/oracleasm/disks/*" value is ok for your env as ASMlib is not mandatory for using ASM. If you really want to go back to "ORCL:MFGBLOB_DATA" for asm_diskstring, please implement the solution described in note 1500460.1 , i.e ORACLEASM_USE_LOGICAL_BLOCK_SIZE=true and restart everything ( reboot is better ). Please advise if you need further clarification on this issue. Regards, Eun Heo | |
| Oracle Support | - 4+ months ago | | [Information] |
|
Hello Bin-Bin,
Thanks for all the information. I don't see any issue for your current env. With the current configuration - i.e asm_diskstring=/dev/oracleasm/disks/*, you don't need to change ORACLEASM_USE_LOGICAL_BLOCK_SIZE to true. I think your current issue is understood by Oracle and I have provided reasonable reason for that. I wonder if you still need this SR to be worked around the clock 24x7 ? If not, I can hold this SR in my queue, to be worked as a Sev1 non-24x7 during my normal working hours Regards Eun Heo Global Product Support Working Hours: Monday – Friday 07:00-15:30 AEST / 21:00-05:30 GMT / 14:00-22:30 PST | |
| BLIU@KEIHIN-NA.COM | - 4+ months ago | | [Update from Customer] |
|
asm_diskstring was "ORCL:MFGBLOB_DATA".
it is "/dev/oracleasm/disks/*" right now. is it ok? Bin-Bin | |
| BLIU@KEIHIN-NA.COM | - 4+ months ago | | [Update from Customer] |
|
so you think i should change ORACLEASM_USE_LOGICAL_BLOCK_SIZE from false to true?
if i don't change it, will cause any problem? Bin-Bin | |
| BLIU@KEIHIN-NA.COM | - 4+ months ago | | [Update from Customer] |
|
[oracle@alpha ~]$ asmcmd lsdg
State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name MOUNTED EXTERN N 512 4096 1048576 953673 167982 0 167982 0 N ALPHAPROD_DATA/ [oracle@alpha ~]$ SQL> show parameter asm NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ asm_diskgroups string ALPHAPROD_DATA asm_diskstring string /dev/oracleasm/disks/* asm_power_limit integer 1 asm_preferred_read_failure_groups string SQL> [root@alpha ~]# cd /sys/devices/vbd-51744/block/xvdc/queue [root@alpha queue]# cat * 1 0 0 0 0 512 cat: iosched: Is a directory 1 512 128 0 128 32 4096 4096 0 128 0 4096 128 0 1 0 noop [deadline] cfq 0 [root@alpha queue]# | |
| Oracle Support | - 4+ months ago | | [Information] |
|
Hello Bin-Bin,
If you could startup the running databases on this server in conjunction with ASM, you will be fine. At the moment, ASMlib has been bypassed with asm_diskstring=/dev/oracleasm/disks/*' if this values is still set in ASM side. Actually, the following parameter in /etc/sysconfig/oracleasm should be set to "true" to use logical sector size ( 512 ) instead of physical ( 4096 ) -- ORACLEASM_USE_LOGICAL_BLOCK_SIZE=true -- Can you show me the following output ? 1. asmcmd lsdg 2. Connect to ASM SQL> show parameter asm 3. $ cd /sys/devices/vbd-51744/block/xvdc/queue $ cat * Regards, Eun Heo | |
| BLIU@KEIHIN-NA.COM | - 4+ months ago | | [Update from Customer] |
|
You know what, after i mounted ALPHAPROD_DATA, i rebooted server, and there is no problem at all.
and, my oracleasm-support is the one on document. the ORACLEASM_USE_LOGICAL_BLOCK_SIZE=false is default. do you think i am fine now? Bin-Bin [root@alpha ~]# yum list installed | grep oracleasm-support oracleasm-support.x86_64 2.1.8-1.el6 @public_ol6_latest [root@alpha ~]# # ORACLEASM_USE_LOGICAL_BLOCK_SIZE: 'true' means use the logical block size # reported by the underlying disk instead of the physical. The default # is 'false' ORACLEASM_USE_LOGICAL_BLOCK_SIZE=false | |
| BLIU@KEIHIN-NA.COM | - 4+ months ago | | [Update from Customer] |
|
there is no SAN device on this machine.
| |
| BLIU@KEIHIN-NA.COM | - 4+ months ago | | [Update from Customer] |
|
please keep this SR open until i upgrade ASMlib.
thanks! Bin-Bin | |
| Oracle Support | - 4+ months ago | | [Information] |
|
Hello Bin-Bin,
The main reason for the current issue is because the sector size for the underlying device mapped to MFGBLOB_DATA has been changed from 512 to 4096 on OS level as part of VM upgrade. From kfed_DH.out, ALPHAPROD_DATA was created with sector size - 512 byte but I saw secor size - 4096 from the previous asm_info_html after VM upgrade. 1] kfed_DH.out -------------------- /dev/oracleasm/disks/MFGBLOB_DATA kfbh.endian: 1 ; 0x000: 0x01 kfbh.hard: 130 ; 0x001: 0x82 kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD kfbh.datfmt: 1 ; 0x003: 0x01 kfbh.block.blk: 0 ; 0x004: blk=0 kfbh.block.obj: 2147483648 ; 0x008: disk=0 kfbh.check: 3262415636 ; 0x00c: 0xc2748314 kfbh.fcn.base: 0 ; 0x010: 0x00000000 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 kfdhdb.driver.provstr:ORCLDISKMFGBLOB_DATA ; 0x000: length=20 kfdhdb.driver.reserved[0]: 1111967309 ; 0x008: 0x4247464d kfdhdb.driver.reserved[1]: 1598181196 ; 0x00c: 0x5f424f4c kfdhdb.driver.reserved[2]: 1096040772 ; 0x010: 0x41544144 kfdhdb.driver.reserved[3]: 0 ; 0x014: 0x00000000 kfdhdb.driver.reserved[4]: 0 ; 0x018: 0x00000000 kfdhdb.driver.reserved[5]: 0 ; 0x01c: 0x00000000 kfdhdb.compat: 186646528 ; 0x020: 0x0b200000 kfdhdb.dsknum: 0 ; 0x024: 0x0000 kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER kfdhdb.dskname: MFGBLOB_DATA ; 0x028: length=12 <<<<<<<<<<<--------------- Disk name kfdhdb.grpname: ALPHAPROD_DATA ; 0x048: length=14 <<<<<<<<<<<<-------------- Diskgroup name kfdhdb.fgname: MFGBLOB_DATA ; 0x068: length=12 .. kfdhdb.secsize: 512 ; 0x0b8: 0x0200 <<<<<<<<<<<<-------------- Sector size kfdhdb.blksize: 4096 ; 0x0ba: 0x1000 .. -------------------------------------------------------------------------------------------------------------------------------- You have 2 choices here. 1. Use asm_diskstring=/dev/oracleasm/disks/* in which case ASM will pick up logical sector size ( 512 ) instead of physical size ( 4096 ). 2. Or please download the latest ASMlib and configure ASMlib to use logical sector size described in the following note . - Alert: After SAN Firmware Upgrade, ASM Diskgroups ( Using ASMLIB) Cannot Be Mounted Due To ORA-15085: ASM disk "" has inconsistent sector size. ( Doc ID 1500460.1 ) I think your current issue is understood and I have provided reasonable reason and solution for that. I wonder if you still need this SR to be worked around the clock 24x7 ? If not, I can hold this SR in my queue, to be worked as a Sev1 non-24x7 during my normal working hours Regards Eun Heo Global Product Support Working Hours: Monday – Friday 07:00-15:30 AEST / 21:00-05:30 GMT / 14:00-22:30 PST | |
| BLIU@KEIHIN-NA.COM | - 4+ months ago | | [Update from Customer] |
|
[root@alpha sys]# cd block
[root@alpha block]# ls -l total 0 lrwxrwxrwx 1 root root 0 Jun 29 18:30 dm-0 -> ../devices/virtual/block/dm-0 lrwxrwxrwx 1 root root 0 Jun 29 18:30 dm-1 -> ../devices/virtual/block/dm-1 lrwxrwxrwx 1 root root 0 Jun 29 14:03 dm-2 -> ../devices/virtual/block/dm-2 lrwxrwxrwx 1 root root 0 Jun 29 14:03 dm-3 -> ../devices/virtual/block/dm-3 lrwxrwxrwx 1 root root 0 Jun 29 14:03 dm-4 -> ../devices/virtual/block/dm-4 lrwxrwxrwx 1 root root 0 Jun 29 14:03 dm-5 -> ../devices/virtual/block/dm-5 lrwxrwxrwx 1 root root 0 Jun 29 14:03 dm-6 -> ../devices/virtual/block/dm-6 lrwxrwxrwx 1 root root 0 Jun 29 14:03 loop0 -> ../devices/virtual/block/loop0 lrwxrwxrwx 1 root root 0 Jun 29 14:03 loop1 -> ../devices/virtual/block/loop1 lrwxrwxrwx 1 root root 0 Jun 29 14:03 loop2 -> ../devices/virtual/block/loop2 lrwxrwxrwx 1 root root 0 Jun 29 14:03 loop3 -> ../devices/virtual/block/loop3 lrwxrwxrwx 1 root root 0 Jun 29 14:03 loop4 -> ../devices/virtual/block/loop4 lrwxrwxrwx 1 root root 0 Jun 29 14:03 loop5 -> ../devices/virtual/block/loop5 lrwxrwxrwx 1 root root 0 Jun 29 14:03 loop6 -> ../devices/virtual/block/loop6 lrwxrwxrwx 1 root root 0 Jun 29 14:03 loop7 -> ../devices/virtual/block/loop7 lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram0 -> ../devices/virtual/block/ram0 lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram1 -> ../devices/virtual/block/ram1 lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram10 -> ../devices/virtual/block/ram10 lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram11 -> ../devices/virtual/block/ram11 lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram12 -> ../devices/virtual/block/ram12 lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram13 -> ../devices/virtual/block/ram13 lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram14 -> ../devices/virtual/block/ram14 lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram15 -> ../devices/virtual/block/ram15 lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram2 -> ../devices/virtual/block/ram2 lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram3 -> ../devices/virtual/block/ram3 lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram4 -> ../devices/virtual/block/ram4 lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram5 -> ../devices/virtual/block/ram5 lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram6 -> ../devices/virtual/block/ram6 lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram7 -> ../devices/virtual/block/ram7 lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram8 -> ../devices/virtual/block/ram8 lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram9 -> ../devices/virtual/block/ram9 lrwxrwxrwx 1 root root 0 Jun 29 18:30 xvda -> ../devices/vbd-51712/block/xvda lrwxrwxrwx 1 root root 0 Jun 29 18:30 xvdc -> ../devices/vbd-51744/block/xvdc lrwxrwxrwx 1 root root 0 Jun 29 18:30 xvdd -> ../devices/vbd-51760/block/xvdd lrwxrwxrwx 1 root root 0 Jun 29 14:03 zram0 -> ../devices/virtual/block/zram0 [root@alpha block]# cd ../devices/vbd-51744/block/xvdc [root@alpha xvdc]# ls -l total 0 -r--r--r-- 1 root root 4096 Jun 29 18:45 alignment_offset lrwxrwxrwx 1 root root 0 Jun 29 18:45 bdi -> ../../../virtual/bdi/202:32 -r--r--r-- 1 root root 4096 Jun 29 18:45 capability -r--r--r-- 1 root root 4096 Jun 29 18:45 dev lrwxrwxrwx 1 root root 0 Jun 29 18:45 device -> ../../../vbd-51744 -r--r--r-- 1 root root 4096 Jun 29 18:45 discard_alignment -r--r--r-- 1 root root 4096 Jun 29 18:45 ext_range drwxr-xr-x 2 root root 0 Jun 29 18:45 holders -r--r--r-- 1 root root 4096 Jun 29 18:45 inflight drwxr-xr-x 2 root root 0 Jun 29 18:45 power drwxr-xr-x 3 root root 0 Jun 29 14:03 queue -r--r--r-- 1 root root 4096 Jun 29 18:45 range -r--r--r-- 1 root root 4096 Jun 29 18:45 removable -r--r--r-- 1 root root 4096 Jun 29 18:45 ro -r--r--r-- 1 root root 4096 Jun 29 18:45 size drwxr-xr-x 2 root root 0 Jun 29 18:45 slaves -r--r--r-- 1 root root 4096 Jun 29 18:45 stat lrwxrwxrwx 1 root root 0 Jun 29 18:45 subsystem -> ../../../../class/block drwxr-xr-x 2 root root 0 Jun 29 18:45 trace -rw-r--r-- 1 root root 4096 Jun 29 18:45 uevent drwxr-xr-x 5 root root 0 Jun 29 16:34 xvdc1 [root@alpha xvdc]# pwd /sys/devices/vbd-51744/block/xvdc [root@alpha xvdc]# cat * 0 cat: bdi: Is a directory 10 202:32 cat: device: Is a director | |
| Oracle Support | - 4+ months ago | | [Information] |
|
Hello Bin-Bin,
I don't see any trace of mfgblob_data diskgroup from the uploaded ASM alert.log which traced back to 25-Dec-2013 as below. Maybe mfgblob_data diskgroup has never existed on the server, even though the disk name is MFGBLOB_DATA. ------------ alert_+ASM.log Wed Dec 25 09:55:36 2013 SUCCESS: diskgroup ALPHAPROD_DATA was mounted alert_+ASM.log Wed Dec 25 09:55:48 2013 SUCCESS: diskgroup ALPHAPROD_DATA was dismounted alert_+ASM.log Wed Dec 25 09:55:51 2013 SUCCESS: diskgroup ALPHAPROD_DATA was mounted alert_+ASM.log Sat Dec 28 09:59:19 2013 SUCCESS: diskgroup ALPHAPROD_DATA was dismounted alert_+ASM.log Sat Dec 28 11:29:26 2013 SUCCESS: diskgroup ALPHAPROD_DATA was mounted alert_+ASM.log Sat Dec 28 11:33:20 2013 SUCCESS: diskgroup ALPHAPROD_DATA was dismounted alert_+ASM.log Sat Dec 28 11:35:24 2013 SUCCESS: diskgroup ALPHAPROD_DATA was mounted alert_+ASM.log Sun Feb 09 09:15:10 2014 SUCCESS: diskgroup ALPHAPROD_DATA was dismounted alert_+ASM.log Sun Feb 09 10:22:12 2014 SUCCESS: diskgroup ALPHAPROD_DATA was mounted alert_+ASM.log Mon Jun 16 16:47:30 2014 SUCCESS: diskgroup ALPHAPROD_DATA was dismounted alert_+ASM.log Mon Jun 16 16:54:54 2014 SUCCESS: diskgroup ALPHAPROD_DATA was mounted alert_+ASM.log Sun Jun 29 09:06:42 2014 SUCCESS: diskgroup ALPHAPROD_DATA was dismounted ---------------- I wonder if you could startup all databases running on the server after mounting ALPHAPROD_DATA diskgroup. Can you also upload the following information ? 1. /dev/oracleasm/disks/MFGBLOB_DATA is mapped to /dev/xvdc. $ cd /sys/block/xvdc $ ls -l $ cat * 2. Another fresh asm_info.html after mounting ALPHAPROD_DATA Provide the following sql output (asm_info.html) after connecting to ASM instance. ================================================================== spool asm_info.html set pagesize 1000 set linesize 220 set feedback off col bytes format 999,999,999,999 col space format 999,999,999,999 col gn format 999 col name format a25 col au format 99999999 col state format a12 col type format a12 col total_mb format 999,999,999 col free_mb format 999,999,999 col od format 999 col compatibility format a12 col dn format 999 col mount_status format a12 col header_status format a12 col mode_status format a12 col mode format a12 col failgroup format a25 col label format a20 col path format a50 col path1 format a40 col path2 format a40 col path3 format a40 col bytes_read format 999,999,999,999,999 col bytes_written format 999,999,999,999,999 col cold_bytes_read format 999,999,999,999,999 col cold_bytes_written format 999,999,999,999,999 alter session set nls_date_format='DD-MON-YYYY HH24:MI:SS' ; select to_char(sysdate, 'DD-MON-YYYY HH24:MI:SS' ) current_time from dual; select group_number gn, name, allocation_unit_size au, state, type, total_mb, free_mb, offline_disks od, compatibility from v$asm_diskgroup; select group_number gn,disk_number dn, mount_status, header_status,mode_status,state, total_mb, free_mb,name, failgroup, label, path from v$asm_disk order by group_number, disk_number; select inst_id, power_kfgmg, sofar_kfgmg, work_kfgmg, rate_kfgmg, time_kfgmg, file_kfgmg from x$kfgmg; break on g_n skip 1 break on failgroup skip 1 compute sum of t_mb f_mb on failgroup compute count of failgroup on failgroup select g.group_number g_n,g.disk_number d_n,g.name , g.path , g.total_mb t_mb,g.free_mb f_mb,g.failgroup from v$asm_disk g order by g_n, failgroup, d_n; select inst_id, power_kfgmg, sofar_kfgmg, work_kfgmg, rate_kfgmg, time_kfgmg, file_kfgmg from x$kfgmg; SET MARKUP HTML ON set echo on select 'THIS ASM REPORT WAS GENERATED AT: ==)> ' , sysdate " " from dual; select 'HOSTNAME ASSOCIATED WITH THIS ASM INSTANCE: ==)> ' , MACHINE " " from v$session where program like '%SMON%'; select * from v$asm_diskgroup; SELECT * FROM V$ASM_DISK ORDER BY GROUP_NUMBER,DISK_NUMBER; select inst_id, power_kfgmg, sofar_kfgmg, work_kfgmg, rate_kfgmg, time_kfgmg, file_kfgmg from x$kfgmg; SELECT * FROM V$ASM_CLIENT; select * from V$ASM_ATTRIBUTE; select * from v$asm_operation; select * from v$version; show parameter show sga spool off exit ================================================================== Regards, Eun Heo | |
| BLIU@KEIHIN-NA.COM | - 4+ months ago | | [Update from Customer] |
|
Upload to TDS successful for the file kfed_FS.out.
| |
| BLIU@KEIHIN-NA.COM | - 4+ months ago | | [Update from Customer] |
|
Upload to TDS successful for the file kfed_DH.out.
| |
| BLIU@KEIHIN-NA.COM | - 4+ months ago | | [Update from Customer] |
|
Upload to TDS successful for the file kfed_BK.out.
| |
| BLIU@KEIHIN-NA.COM | - 4+ months ago | | [Update from Customer] |
|
Upload to TDS successful for the file kfed_4MBK.out.
| |
| BLIU@KEIHIN-NA.COM | - 4+ months ago | | [Update from Customer] |
|
are you saying the diskgroup should be alphaprod_data, not mfgblob_data?
[oracle@alpha ~]$ cat /proc/partitions major minor #blocks name 202 0 492830720 xvda 202 1 512000 xvda1 202 2 492317696 xvda2 202 32 1169304832 xvdc 202 33 976561152 xvdc1 202 48 584960000 xvdd 202 49 581053440 xvdd1 252 0 52428800 dm-0 252 1 27885568 dm-1 252 2 883859456 dm-2 252 3 26214400 dm-3 252 4 10485760 dm-4 252 5 41943040 dm-5 252 6 26214400 dm-6 [oracle@alpha ~]$ ls -l /sys/block/* lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/dm-0 -> ../devices/virtual/block/dm-0 lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/dm-1 -> ../devices/virtual/block/dm-1 lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/dm-2 -> ../devices/virtual/block/dm-2 lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/dm-3 -> ../devices/virtual/block/dm-3 lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/dm-4 -> ../devices/virtual/block/dm-4 lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/dm-5 -> ../devices/virtual/block/dm-5 lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/dm-6 -> ../devices/virtual/block/dm-6 lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/loop0 -> ../devices/virtual/block/loop0 lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/loop1 -> ../devices/virtual/block/loop1 lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/loop2 -> ../devices/virtual/block/loop2 lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/loop3 -> ../devices/virtual/block/loop3 lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/loop4 -> ../devices/virtual/block/loop4 lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/loop5 -> ../devices/virtual/block/loop5 lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/loop6 -> ../devices/virtual/block/loop6 lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/loop7 -> ../devices/virtual/block/loop7 lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram0 -> ../devices/virtual/block/ram0 lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram1 -> ../devices/virtual/block/ram1 lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram10 -> ../devices/virtual/block/ram10 lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram11 -> ../devices/virtual/block/ram11 lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram12 -> ../devices/virtual/block/ram12 lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram13 -> ../devices/virtual/block/ram13 lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram14 -> ../devices/virtual/block/ram14 lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram15 -> ../devices/virtual/block/ram15 lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram2 -> ../devices/virtual/block/ram2 lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram3 -> ../devices/virtual/block/ram3 lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram4 -> ../devices/virtual/block/ram4 lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram5 -> ../devices/virtual/block/ram5 lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram6 -> ../devices/virtual/block/ram6 lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram7 -> ../devices/virtual/block/ram7 lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram8 -> ../devices/virtual/block/ram8 lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram9 -> ../devices/virtual/block/ram9 lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/xvda -> ../devices/vbd-51712/block/xvda lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/xvdc -> ../devices/vbd-51744/block/xvdc lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/xvdd -> ../devices/vbd-51760/block/xvdd lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/zram0 -> ../devices/virtual/block/zram0 [oracle@alpha ~]$ | |
| BLIU@KEIHIN-NA.COM | - 4+ months ago | | [Update from Customer] |
|
i can mount alphaprod_data. how about mfgblob_data
do the same thing? thanks! Bin-Bin | |
| Oracle Support | - 4+ months ago | | [ODM Proposed Solution(s)] |
|
Hello Bin-Bin,
It seem to me that the reason why ALPHAPROD_DATA can not be mounted is due to the following error. -- alert_+ASM.log Sun Jun 29 15:45:30 2014 ORA-15085: ASM disk "" has inconsistent sector size. alert_+ASM.log Sun Jun 29 16:12:00 2014 ORA-15085: ASM disk "" has inconsistent sector size. alert_+ASM.log Sun Jun 29 16:19:28 2014 ORA-15085: ASM disk "" has inconsistent sector size. -- This indicates that the sector size for the underlying device mapped to /dev/oracleasm/disks/MFGBLOB_DATA has changed to 4096 which may have been 512 byte before. Action Plan ------------------ 1. Can you try to mount ALPHAPROD_DATA in the following way ? $ sqlplus / as sysasm SQL> alter system set asm_diskstring='/dev/oracleasm/disks/*' ; SQL> alter diskgroup ALPHAPROD_DATA mount ; <<< If ALPHAPROD_DATA can be mounted in this way, you can startup the databases running on this server. 2. But you will need to find out why the secor size has been changed for the underlying device mapped to /dev/oracleasm/disks/MFGBLOB_DATA. Show us the following information $ cat /proc/partitions $ ls -l /sys/block/* Regards, Eun Heo | |
| Oracle Support | - 4+ months ago | | [ODM Cause Justification] |
|
alert_+ASM.log Sun Jun 29 15:45:30 2014 ORA-15085: ASM disk "" has inconsistent sector size.
alert_+ASM.log Sun Jun 29 16:12:00 2014 ORA-15085: ASM disk "" has inconsistent sector size. alert_+ASM.log Sun Jun 29 16:19:28 2014 ORA-15085: ASM disk "" has inconsistent sector size. | |
| Oracle Support | - 4+ months ago | | [ODM Cause Determination] |
|
o It seem that sector size has been changed to 4096 from 512
| |
| BLIU@KEIHIN-NA.COM | - 4+ months ago | | [Update from Customer] |
|
Upload to TDS successful for the file asm_info.html.
| |
| BLIU@KEIHIN-NA.COM | - 4+ months ago | | [Update from Customer] |
|
Upload to TDS successful for the file sosreport-bliu.3-9255522561-20140629173734-6f24.tar.xz.
| |
| BLIU@KEIHIN-NA.COM | - 4+ months ago | | [Update from Customer] |
|
yes, i see one.
[oracle@alpha disks]$ ls -l total 0 brw-rw---- 1 oracle oinstall 202, 33 Jun 29 15:42 MFGBLOB_DATA [oracle@alpha disks]$ pwd /dev/oracleasm/disks [oracle@alpha disks]$ | |
| BLIU@KEIHIN-NA.COM | - 4+ months ago | | [Update from Customer] |
|
i will send you another sosreport, and the output of your sql script.
please check the early message, it is about ALPHAPROD_DATA diskgroup. someone changed the asm_diskgroups, i will changed it back. SQL> show parameter asm NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ asm_diskgroups string ALPHAPROD_DATA asm_diskstring string ORCL:MFGBLOB_DATA asm_power_limit integer 1 asm_preferred_read_failure_groups string Bin-Bin | |
| Oracle Support | - 4+ months ago | | [Information] |
|
Hello Bin-Bin,
The following error from ASMalert.log indicates that Oracle ASM can not discover ASM disks in /dev/oracleasm/disks/* . 1] alert_+ASM.log ----------------------- asm_diskstring = "ORCL:MFGBLOB_DATA" asm_diskgroups = "MFGBLOB_DATA" .. Sun Jun 29 16:35:43 2014 ASM Health Checker found 1 new failures GMON dismounting group 1 at 4 for pid 17, osid 11746 ERROR: diskgroup MFGBLOB_DATA was not mounted ORA-15032: not all alterations performed ORA-15017: diskgroup "MFGBLOB_DATA" cannot be mounted ORA-15063: ASM discovered an insufficient number of disks for diskgroup "MFGBLOB_DATA" -------------------------------------------- Question - Do you see any of ASM disks in /dev/oracleasm/disks/* directory ? And I'd like to see the following information to investigate the current issue further Action Plan ------------------- 1. Provide the following sql output (asm_info.html) after connecting to ASM instance. ================================================================== spool asm_info.html set pagesize 1000 set linesize 220 set feedback off col bytes format 999,999,999,999 col space format 999,999,999,999 col gn format 999 col name format a25 col au format 99999999 col state format a12 col type format a12 col total_mb format 999,999,999 col free_mb format 999,999,999 col od format 999 col compatibility format a12 col dn format 999 col mount_status format a12 col header_status format a12 col mode_status format a12 col mode format a12 col failgroup format a25 col label format a20 col path format a50 col path1 format a40 col path2 format a40 col path3 format a40 col bytes_read format 999,999,999,999,999 col bytes_written format 999,999,999,999,999 col cold_bytes_read format 999,999,999,999,999 col cold_bytes_written format 999,999,999,999,999 alter session set nls_date_format='DD-MON-YYYY HH24:MI:SS' ; select to_char(sysdate, 'DD-MON-YYYY HH24:MI:SS' ) current_time from dual; select group_number gn, name, allocation_unit_size au, state, type, total_mb, free_mb, offline_disks od, compatibility from v$asm_diskgroup; select group_number gn,disk_number dn, mount_status, header_status,mode_status,state, total_mb, free_mb,name, failgroup, label, path from v$asm_disk order by group_number, disk_number; select inst_id, power_kfgmg, sofar_kfgmg, work_kfgmg, rate_kfgmg, time_kfgmg, file_kfgmg from x$kfgmg; break on g_n skip 1 break on failgroup skip 1 compute sum of t_mb f_mb on failgroup compute count of failgroup on failgroup select g.group_number g_n,g.disk_number d_n,g.name , g.path , g.total_mb t_mb,g.free_mb f_mb,g.failgroup from v$asm_disk g order by g_n, failgroup, d_n; select inst_id, power_kfgmg, sofar_kfgmg, work_kfgmg, rate_kfgmg, time_kfgmg, file_kfgmg from x$kfgmg; SET MARKUP HTML ON set echo on select 'THIS ASM REPORT WAS GENERATED AT: ==)> ' , sysdate " " from dual; select 'HOSTNAME ASSOCIATED WITH THIS ASM INSTANCE: ==)> ' , MACHINE " " from v$session where program like '%SMON%'; select * from v$asm_diskgroup; SELECT * FROM V$ASM_DISK ORDER BY GROUP_NUMBER,DISK_NUMBER; select inst_id, power_kfgmg, sofar_kfgmg, work_kfgmg, rate_kfgmg, time_kfgmg, file_kfgmg from x$kfgmg; SELECT * FROM V$ASM_CLIENT; select * from V$ASM_ATTRIBUTE; select * from v$asm_operation; select * from v$version; show parameter show sga spool off exit ================================================================== 2. Upload ASM device information on the server in the following way below. Download ASMCHECK zip file ( including AsmCheck.sh and AsmCheck.sql ) from note 1346190.1 and place them in any directory where grid OS user can read & write and run it in the following way. As root OS user, # chmod 755 AsmCheck.sh # export ORACLE_HOME=<GRID_HOME> <<< Specify GRID_HOME from your env. # export ORACLE_SID=<ASM#> <<< Specify ASM instance number from your env. # ./AsmCheck.sh 'GRID_OS_USER' > `hostname` Example> If GRID_OS User on the serveris grid, # ./AsmCheck.sh grid > `hostname` NOTE 1: Some errors will be on the screen if the specified OS command and or directory does not exist, please ignore them. NOTE 2: The following shell script can be run regardless of whether ASMlib is being used on the server. NOTE 3: The following shell script can be run without running ASM instance on the server. NOTE 4: Copy and paste the output of AsmCheck.sh on the sr NOTE 5: Upload the following files after running AsmCheck.sh ********************************************************************************** 1] file generated with `hostname` in the directory where AsmCheck.sh runs. 2] kfed_DH.out/kfed_FS.out/kfed_BK.out/kfed_4MBK.out ********************************************************************************* Regards, Eun Heo | |
| Oracle Support | - 4+ months ago | | [ODM Issue Verification] |
|
1] alert_+ASM.log
----------------------- asm_diskstring = "ORCL:MFGBLOB_DATA" asm_diskgroups = "MFGBLOB_DATA" .. Sun Jun 29 16:35:43 2014 ASM Health Checker found 1 new failures GMON dismounting group 1 at 4 for pid 17, osid 11746 ERROR: diskgroup MFGBLOB_DATA was not mounted ORA-15032: not all alterations performed ORA-15017: diskgroup "MFGBLOB_DATA" cannot be mounted ORA-15063: ASM discovered an insufficient number of disks for diskgroup "MFGBLOB_DATA" -------------------------------------------- | |
| Oracle Support | - 4+ months ago | | [ODM Issue Clarification] |
|
o MFGBLOB_DATA diskgroup can not be mounted due to the following error
----------- ORA-15017: diskgroup "MFGBLOB_DATA" cannot be mounted ORA-15063: ASM discovered an insufficient number of disks for diskgroup "MFGBLOB_DATA" ------------- | |
| Oracle Support | - 4+ months ago | | [First Contact] |
|
First Phone Contact Comments
-------------------------------------------------------- Hello Bin-Bin, This SR has been transferred to Oracle ASM team, My name is Eun Heo. I have been assigned this new Sev 1 SR and am reviewing the SR and any attachments. By default, this SR was created as 24x7. Do you need this SR to be worked around the clock 24x7? If not, I can hold this SR in my queue, to be worked as a Sev1 non-24x7 during my normal working hours. Regards Eun Heo Global Product Support Working Hours: Monday – Friday 07:00-15:30 AEST / 21:00-05:30 GMT / 14:00-22:30 PST | |
| BLIU@KEIHIN-NA.COM | - 4+ months ago | | [Update from Customer] |
|
Upload to TDS successful for the file sosreport-bliu.3-9255522561-20140629170915-382d.tar.xz.
| |
| BLIU@KEIHIN-NA.COM | - 4+ months ago | | [Update from Customer] |
|
Upload to TDS successful for the file virtual_machine.rar.
| |
| BLIU@KEIHIN-NA.COM | - 4+ months ago | | [Customer Problem Description] |
|
Severity 1 Contact Information
--------------------------------------------------- ** Customer's Management 24x7 contact name: Bin-Bin Liu ** Customer's Management 24x7 contact number: +1.317.471.7002 ** Customer's Management E-mail Address: bliu@keihin-na.com ** Primary Customer 24x7 contact name: Bin-Bin Liu ** Primary Customer 24x7 contact number: +1.317.471.7002 ** Primary Customer E-mail Address: bliu@keihin-na.com ** Primary Customer Contact Method: Phone ** Secondary Customer 24x7 contact name: Bin-Bin Liu ** Secondary Customer 24x7 contact number: +1.317-471-7002 ** Alternate 24x7 Contact Number: +1.317-471-7002 ** Secondary Customer E-mail Address: bliu@keihin-na.com Customer Problem Description --------------------------------------------------- Problem Summary --------------------------------------------------- OVM server after upgraded, virtual machine ASM diskgroup couldn't mpuntd Problem Description --------------------------------------------------- this is emergency. this is my production system. i followed the oracle menu upgraded my VM server to 3.2.8, there was no problem on VM server. when i start my virtual machine, i got the following messages. I'll attach all information to you. Bin-Bin Wed Dec 25 09:55:36 2013 SQL> ALTER DISKGROUP ALL MOUNT /* asm agent call crs *//* {0:0:95} */ SQL> ALTER DISKGROUP ALL ENABLE VOLUME ALL /* asm agent *//* {0:0:95} */ Wed Dec 25 09:55:36 2013 ALTER SYSTEM SET local_listener='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=10.18.1.126)(PORT=1521))))' SCOPE=MEMORY SID='+ASM'; NOTE: Loaded library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so Wed Dec 25 09:55:36 2013 SQL> CREATE DISKGROUP ALPHAPROD_DATA EXTERNAL REDUNDANCY DISK 'ORCL:MFGBLOB_DATA' ATTRIBUTE 'compatible.asm'='11.2.0.0.0' /* ASMCA */ NOTE: Assigning number (1,0) to disk (ORCL:MFGBLOB_DATA) NOTE: initializing header on grp 1 disk MFGBLOB_DATA NOTE: initiating PST update: grp = 1 GMON updating group 1 at 1 for pid 18, osid 6891 NOTE: group ALPHAPROD_DATA: initial PST location: disk 0000 (PST copy 0) NOTE: PST update grp = 1 completed successfully NOTE: cache registered group ALPHAPROD_DATA number=1 incarn=0x2c6de211 NOTE: cache began mount (first) of group ALPHAPROD_DATA number=1 incarn=0x2c6de211 NOTE: cache opening disk 0 of grp 1: MFGBLOB_DATA label:MFGBLOB_DATA NOTE: cache creating group 1/0x2C6DE211 (ALPHAPROD_DATA) NOTE: cache mounting group 1/0x2C6DE211 (ALPHAPROD_DATA) succeeded NOTE: allocating F1X0 on grp 1 disk MFGBLOB_DATA NOTE: Created Used Space Directory for 1 threads NOTE: diskgroup must now be re-mounted prior to first use NOTE: cache dismounting (clean) group 1/0x2C6DE211 (ALPHAPROD_DATA) NOTE: messaging CKPT to quiesce pins Unix process pid: 6891, image: oracle@knaalpha.kna.local (TNS V1-V3) NOTE: lgwr not being msg'd to dismount NOTE: cache dismounted group 1/0x2C6DE211 (ALPHAPROD_DATA) GMON dismounting group 1 at 2 for pid 18, osid 6891 GMON dismount Error Codes --------------------------------------------------- Problem Category/Subcategory --------------------------------------------------- Database Storage (ASM, ACFS, DNFS, Compression, Corruption, Partitioning, LOBS, Tablespaces, Files)/ASM_ACFS: Generic problems Uploaded Files --------------------------------------------------- Template Question Responses --------------------------------------------------- 1) ### RECENT CHANGES ### 2) ### SHARING ORACLE_HOME ### 3) ### DATABASE ORACLE_HOME ### 4) ### MULTIPATHING INFORMATION ### 5) ### STORAGE CONFIGURATION ### 6) Describe how this problem is impacting your business. Include relevant information such as critical events (i.e., upgrade or project milestones), dates (i.e., go live dates), number of users affected, financial impact, etc. |
No comments:
Post a Comment