Wednesday, October 29, 2014

OVM server after upgraded, virtual machine ASM diskgroup couldn't mount


OVM server after upgraded, virtual machine ASM diskgroup couldn't mpuntd


Knowledge Review completed



Hello Bin-Bin,

It seem to me that the reason why ALPHAPROD_DATA can not be mounted is due to the following error.

--
alert_+ASM.log Sun Jun 29 15:45:30 2014 ORA-15085: ASM disk "" has inconsistent sector size.
alert_+ASM.log Sun Jun 29 16:12:00 2014 ORA-15085: ASM disk "" has inconsistent sector size.
alert_+ASM.log Sun Jun 29 16:19:28 2014 ORA-15085: ASM disk "" has inconsistent sector size.
--

This indicates that the sector size for the underlying device mapped to /dev/oracleasm/disks/MFGBLOB_DATA has changed to 4096 which may have been 512 byte before.

Action Plan
------------------
1. Can you try to mount ALPHAPROD_DATA in the following way ?
$ sqlplus / as sysasm
SQL> alter system set asm_diskstring='/dev/oracleasm/disks/*' ;
SQL> alter diskgroup ALPHAPROD_DATA mount ;

<<< If ALPHAPROD_DATA can be mounted in this way, you can startup the databases running on this server.

2. But you will need to find out why the secor size has been changed for the underlying device mapped to /dev/oracleasm/disks/MFGBLOB_DATA.
Show us the following information

$ cat /proc/partitions
$ ls -l /sys/block/*

Regards,
Eun Heo 

Oracle Support- 3+ months ago[ODM Proposed Solution Justif]
Hello Bin-Bin,

It seem to me that the reason why ALPHAPROD_DATA can not be mounted is due to the following error.

--
alert_+ASM.log Sun Jun 29 15:45:30 2014 ORA-15085: ASM disk "" has inconsistent sector size.
alert_+ASM.log Sun Jun 29 16:12:00 2014 ORA-15085: ASM disk "" has inconsistent sector size.
alert_+ASM.log Sun Jun 29 16:19:28 2014 ORA-15085: ASM disk "" has inconsistent sector size.
--

This indicates that the sector size for the underlying device mapped to /dev/oracleasm/disks/MFGBLOB_DATA has changed to 4096 which may have been 512 byte before.

Action Plan
------------------
1. Can you try to mount ALPHAPROD_DATA in the following way ?
$ sqlplus / as sysasm
SQL> alter system set asm_diskstring='/dev/oracleasm/disks/*' ;
SQL> alter diskgroup ALPHAPROD_DATA mount ;

<<< If ALPHAPROD_DATA can be mounted in this way, you can startup the databases running on this server.

2. But you will need to find out why the secor size has been changed for the underlying device mapped to /dev/oracleasm/disks/MFGBLOB_DATA.
Show us the following information

$ cat /proc/partitions
$ ls -l /sys/block/*

Regards,
Eun Heo 

BLIU@KEIHIN-NA.COM- 3+ months ago[Update from Customer]
Everything is good. I am going to close this SR.
Thank you for all your help.

Bin-Bin

Oracle Support- 3+ months ago[Information]
lowering to SEV2 per customer update and waiting for update 

Oracle Support- 4+ months ago[Information]
Hi Bin-Bin,

Sure.
Do you want me to lower this SR to a SEV2?
Then you can update on Monday, is that ok?

Thanks
George

Working Hours: Mon-Fri
09:00am to 5:30 PM Mountain Time Zone 

BLIU@KEIHIN-NA.COM- 4+ months ago[Update from Customer]
i am going to lower severity level. we are shutdown until next Monday(7/7/2014).
I'd like to keep this SR until next Monday. They are production system.
Can I do that?

Bin-Bin

Oracle Support- 4+ months ago[Information]
Hi Bin-Bin,

It appears so.
How would you like to proceed with this SR?

Thanks
George

Working Hours: Mon-Fri
09:00am to 5:30 PM Mountain Time Zone 

BLIU@KEIHIN-NA.COM- 4+ months ago[Update from Customer]
i can mount diskgroup, and no error. so i am good, right?

Thanks!

Bin-Bin

Oracle Support- 4+ months ago[Information]
Hi Bin-Bin,

Are you able to mount the diskgroups?
Any errors in the ASM alert.log?
If not, then it would seem ok.

Thanks
George

Working Hours: Mon-Fri
09:00am to 5:30 PM Mountain Time Zone 

Oracle Support- 4+ months ago[Information]
Hi Bin-Bin,

I have been assigned this SR and I am currently reviewing.
I will provide an update as soon as possible.

Thanks
George

Working Hours: Mon-Fri
09:00am to 5:30 PM Mountain Time Zone 

BLIU@KEIHIN-NA.COM- 4+ months ago[Update from Customer]
i upgraded the second without problem. i set ORACLEASM_USE_LOGICAL_BLOCK_SIZE to true.

the first server which i had problem yesterday, i changed ORACLEASM_USE_LOGICAL_BLOCK_SIZE to true.
but i cannot change asm_diskstring to MFGBLOB_DATA. sm_diskstring is still '/dev/oracleasm/disks/* '.
do you think is it ok? i don't have any problem after rebooted.

Bin-Bin

Oracle Support- 4+ months ago[Information]
Hello Bin-Bin,

Changing ORACLEASM_USE_LOGICAL_BLOCK_SIZE to true before reboot won't do any harm in your env, so you can change it before system reboot ( or upgrade ).

There is no way of assinging the sr to a particular analyst on creating a sr as assigning procedure is all automatic.

I will keep this sr in my queue as Sev1 for the time being. If you have a similar issue for another server and update this sr , the sr will get transferred to the next available analyst if I am not working.

This way, you can get a help from other analysts .

Thanks for your kind words, anyway.


Regards,
Eun Heo 

BLIU@KEIHIN-NA.COM- 4+ months ago[Update from Customer]
if i have issue in the future, can i assign my SR to you? if it is not severity 1, can i do that?
if i can, how to do it?

Bin-Bin

BLIU@KEIHIN-NA.COM- 4+ months ago[Update from Customer]
can i change "ORACLEASM_USE_LOGICAL_BLOCK_SIZE" to true before i reboot, even i din't know i will have problem or not?

Bin-Bin

Oracle Support- 4+ months ago[Information]
Hello Bin-Bin,

I am actually based in Sydney - Australia. Please find my working hours below and start working 2PM in PST timezone.
Working Hours: Monday – Friday 07:00-15:30 AEST / 21:00-05:30 GMT / 14:00-22:30 PST

Please check ASM alert.log for the following error if you see the same issue.
- alert_+ASM.log Sun Jun 29 15:45:30 2014 ORA-15085: ASM disk "" has inconsistent sector size.

If this is the case for another server, you can do one of the solution provided.
1] Change asm_diskstring to /dev/oracleasm/disks/*

OR
2] Reboot the server after changing the following parameter in /etc/sysconfig/oracleasm file.
ORACLEASM_USE_LOGICAL_BLOCK_SIZE=true


Regards,
Eun 

BLIU@KEIHIN-NA.COM- 4+ months ago[Update from Customer]
i am in EST time. what's yours?

BLIU@KEIHIN-NA.COM- 4+ months ago[Update from Customer]
that is great, so this server is ready.
i am going to upgrade another server tomorrow. i am going to do the same work like this one tomorrow.
if you can keep it open until i done the upgrade tomorrow.
it will be the best. my time is 10:23PM, how about yours?
you are very helpful, if it is possible, i want you be my contact.

Thanks!

Bin-Bin

Oracle Support- 4+ months ago[Information]
Hello Bin-Bin,

Our previous posting seemed to be crossing.

Yes, "/dev/oracleasm/disks/*" value is ok for your env as ASMlib is not mandatory for using ASM.

If you really want to go back to "ORCL:MFGBLOB_DATA" for asm_diskstring, please implement the solution described in note 1500460.1 , i.e ORACLEASM_USE_LOGICAL_BLOCK_SIZE=true and restart everything ( reboot is better ).

Please advise if you need further clarification on this issue.

Regards,
Eun Heo 

Oracle Support- 4+ months ago[Information]
Hello Bin-Bin,

Thanks for all the information.

I don't see any issue for your current env.

With the current configuration - i.e asm_diskstring=/dev/oracleasm/disks/*, you don't need to change ORACLEASM_USE_LOGICAL_BLOCK_SIZE to true.

I think your current issue is understood by Oracle and I have provided reasonable reason for that.

I wonder if you still need this SR to be worked around the clock 24x7 ? If not, I can hold this SR in my queue, to be worked as a Sev1 non-24x7 during my normal working hours

Regards
Eun Heo
Global Product Support
Working Hours: Monday – Friday 07:00-15:30 AEST / 21:00-05:30 GMT / 14:00-22:30 PST 

BLIU@KEIHIN-NA.COM- 4+ months ago[Update from Customer]
asm_diskstring was "ORCL:MFGBLOB_DATA".
it is "/dev/oracleasm/disks/*" right now.
is it ok?

Bin-Bin

BLIU@KEIHIN-NA.COM- 4+ months ago[Update from Customer]
so you think i should change ORACLEASM_USE_LOGICAL_BLOCK_SIZE from false to true?
if i don't change it, will cause any problem?

Bin-Bin

BLIU@KEIHIN-NA.COM- 4+ months ago[Update from Customer]
[oracle@alpha ~]$ asmcmd lsdg
State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED EXTERN N 512 4096 1048576 953673 167982 0 167982 0 N ALPHAPROD_DATA/
[oracle@alpha ~]$

SQL> show parameter asm

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
asm_diskgroups string ALPHAPROD_DATA
asm_diskstring string /dev/oracleasm/disks/*
asm_power_limit integer 1
asm_preferred_read_failure_groups string
SQL>

[root@alpha ~]# cd /sys/devices/vbd-51744/block/xvdc/queue
[root@alpha queue]# cat *
1
0
0
0
0
512
cat: iosched: Is a directory
1
512
128
0
128
32
4096
4096
0
128
0
4096
128
0
1
0
noop [deadline] cfq
0
[root@alpha queue]#

Oracle Support- 4+ months ago[Information]
Hello Bin-Bin,

If you could startup the running databases on this server in conjunction with ASM, you will be fine.

At the moment, ASMlib has been bypassed with asm_diskstring=/dev/oracleasm/disks/*' if this values is still set in ASM side.

Actually, the following parameter in /etc/sysconfig/oracleasm should be set to "true" to use logical sector size ( 512 ) instead of physical ( 4096 )
--
ORACLEASM_USE_LOGICAL_BLOCK_SIZE=true
--

Can you show me the following output ?

1. asmcmd lsdg
2. Connect to ASM
SQL> show parameter asm

3. $ cd /sys/devices/vbd-51744/block/xvdc/queue
$ cat *

Regards,
Eun Heo 

BLIU@KEIHIN-NA.COM- 4+ months ago[Update from Customer]
You know what, after i mounted ALPHAPROD_DATA, i rebooted server, and there is no problem at all.
and, my oracleasm-support is the one on document. the ORACLEASM_USE_LOGICAL_BLOCK_SIZE=false is default.
do you think i am fine now?

Bin-Bin

[root@alpha ~]# yum list installed | grep oracleasm-support
oracleasm-support.x86_64 2.1.8-1.el6 @public_ol6_latest
[root@alpha ~]#


# ORACLEASM_USE_LOGICAL_BLOCK_SIZE: 'true' means use the logical block size
# reported by the underlying disk instead of the physical. The default
# is 'false'
ORACLEASM_USE_LOGICAL_BLOCK_SIZE=false 

BLIU@KEIHIN-NA.COM- 4+ months ago[Update from Customer]
there is no SAN device on this machine.

BLIU@KEIHIN-NA.COM- 4+ months ago[Update from Customer]
please keep this SR open until i upgrade ASMlib.

thanks!

Bin-Bin

Oracle Support- 4+ months ago[Information]
Hello Bin-Bin,

The main reason for the current issue is because the sector size for the underlying device mapped to MFGBLOB_DATA has been changed from 512 to 4096 on OS level as part of VM upgrade.

From kfed_DH.out, ALPHAPROD_DATA was created with sector size - 512 byte but I saw secor size - 4096 from the previous asm_info_html after VM upgrade.

1] kfed_DH.out
--------------------
/dev/oracleasm/disks/MFGBLOB_DATA
kfbh.endian: 1 ; 0x000: 0x01
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 2147483648 ; 0x008: disk=0
kfbh.check: 3262415636 ; 0x00c: 0xc2748314
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr:ORCLDISKMFGBLOB_DATA ; 0x000: length=20
kfdhdb.driver.reserved[0]: 1111967309 ; 0x008: 0x4247464d
kfdhdb.driver.reserved[1]: 1598181196 ; 0x00c: 0x5f424f4c
kfdhdb.driver.reserved[2]: 1096040772 ; 0x010: 0x41544144
kfdhdb.driver.reserved[3]: 0 ; 0x014: 0x00000000
kfdhdb.driver.reserved[4]: 0 ; 0x018: 0x00000000
kfdhdb.driver.reserved[5]: 0 ; 0x01c: 0x00000000
kfdhdb.compat: 186646528 ; 0x020: 0x0b200000
kfdhdb.dsknum: 0 ; 0x024: 0x0000
kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL
kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname: MFGBLOB_DATA ; 0x028: length=12 <<<<<<<<<<<--------------- Disk name
kfdhdb.grpname: ALPHAPROD_DATA ; 0x048: length=14 <<<<<<<<<<<<-------------- Diskgroup name
kfdhdb.fgname: MFGBLOB_DATA ; 0x068: length=12
..
kfdhdb.secsize: 512 ; 0x0b8: 0x0200 <<<<<<<<<<<<-------------- Sector size
kfdhdb.blksize: 4096 ; 0x0ba: 0x1000
..
--------------------------------------------------------------------------------------------------------------------------------

You have 2 choices here.

1. Use asm_diskstring=/dev/oracleasm/disks/* in which case ASM will pick up logical sector size ( 512 ) instead of physical size ( 4096 ).

2. Or please download the latest ASMlib and configure ASMlib to use logical sector size described in the following note .
- Alert: After SAN Firmware Upgrade, ASM Diskgroups ( Using ASMLIB) Cannot Be Mounted Due To ORA-15085: ASM disk "" has inconsistent sector size. ( Doc ID 1500460.1 )


I think your current issue is understood and I have provided reasonable reason and solution for that.

I wonder if you still need this SR to be worked around the clock 24x7 ? If not, I can hold this SR in my queue, to be worked as a Sev1 non-24x7 during my normal working hours

Regards
Eun Heo
Global Product Support
Working Hours: Monday – Friday 07:00-15:30 AEST / 21:00-05:30 GMT / 14:00-22:30 PST 

BLIU@KEIHIN-NA.COM- 4+ months ago[Update from Customer]
[root@alpha sys]# cd block
[root@alpha block]# ls -l
total 0
lrwxrwxrwx 1 root root 0 Jun 29 18:30 dm-0 -> ../devices/virtual/block/dm-0
lrwxrwxrwx 1 root root 0 Jun 29 18:30 dm-1 -> ../devices/virtual/block/dm-1
lrwxrwxrwx 1 root root 0 Jun 29 14:03 dm-2 -> ../devices/virtual/block/dm-2
lrwxrwxrwx 1 root root 0 Jun 29 14:03 dm-3 -> ../devices/virtual/block/dm-3
lrwxrwxrwx 1 root root 0 Jun 29 14:03 dm-4 -> ../devices/virtual/block/dm-4
lrwxrwxrwx 1 root root 0 Jun 29 14:03 dm-5 -> ../devices/virtual/block/dm-5
lrwxrwxrwx 1 root root 0 Jun 29 14:03 dm-6 -> ../devices/virtual/block/dm-6
lrwxrwxrwx 1 root root 0 Jun 29 14:03 loop0 -> ../devices/virtual/block/loop0
lrwxrwxrwx 1 root root 0 Jun 29 14:03 loop1 -> ../devices/virtual/block/loop1
lrwxrwxrwx 1 root root 0 Jun 29 14:03 loop2 -> ../devices/virtual/block/loop2
lrwxrwxrwx 1 root root 0 Jun 29 14:03 loop3 -> ../devices/virtual/block/loop3
lrwxrwxrwx 1 root root 0 Jun 29 14:03 loop4 -> ../devices/virtual/block/loop4
lrwxrwxrwx 1 root root 0 Jun 29 14:03 loop5 -> ../devices/virtual/block/loop5
lrwxrwxrwx 1 root root 0 Jun 29 14:03 loop6 -> ../devices/virtual/block/loop6
lrwxrwxrwx 1 root root 0 Jun 29 14:03 loop7 -> ../devices/virtual/block/loop7
lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram0 -> ../devices/virtual/block/ram0
lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram1 -> ../devices/virtual/block/ram1
lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram10 -> ../devices/virtual/block/ram10
lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram11 -> ../devices/virtual/block/ram11
lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram12 -> ../devices/virtual/block/ram12
lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram13 -> ../devices/virtual/block/ram13
lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram14 -> ../devices/virtual/block/ram14
lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram15 -> ../devices/virtual/block/ram15
lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram2 -> ../devices/virtual/block/ram2
lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram3 -> ../devices/virtual/block/ram3
lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram4 -> ../devices/virtual/block/ram4
lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram5 -> ../devices/virtual/block/ram5
lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram6 -> ../devices/virtual/block/ram6
lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram7 -> ../devices/virtual/block/ram7
lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram8 -> ../devices/virtual/block/ram8
lrwxrwxrwx 1 root root 0 Jun 29 18:30 ram9 -> ../devices/virtual/block/ram9
lrwxrwxrwx 1 root root 0 Jun 29 18:30 xvda -> ../devices/vbd-51712/block/xvda
lrwxrwxrwx 1 root root 0 Jun 29 18:30 xvdc -> ../devices/vbd-51744/block/xvdc
lrwxrwxrwx 1 root root 0 Jun 29 18:30 xvdd -> ../devices/vbd-51760/block/xvdd
lrwxrwxrwx 1 root root 0 Jun 29 14:03 zram0 -> ../devices/virtual/block/zram0
[root@alpha block]# cd ../devices/vbd-51744/block/xvdc
[root@alpha xvdc]# ls -l
total 0
-r--r--r-- 1 root root 4096 Jun 29 18:45 alignment_offset
lrwxrwxrwx 1 root root 0 Jun 29 18:45 bdi -> ../../../virtual/bdi/202:32
-r--r--r-- 1 root root 4096 Jun 29 18:45 capability
-r--r--r-- 1 root root 4096 Jun 29 18:45 dev
lrwxrwxrwx 1 root root 0 Jun 29 18:45 device -> ../../../vbd-51744
-r--r--r-- 1 root root 4096 Jun 29 18:45 discard_alignment
-r--r--r-- 1 root root 4096 Jun 29 18:45 ext_range
drwxr-xr-x 2 root root 0 Jun 29 18:45 holders
-r--r--r-- 1 root root 4096 Jun 29 18:45 inflight
drwxr-xr-x 2 root root 0 Jun 29 18:45 power
drwxr-xr-x 3 root root 0 Jun 29 14:03 queue
-r--r--r-- 1 root root 4096 Jun 29 18:45 range
-r--r--r-- 1 root root 4096 Jun 29 18:45 removable
-r--r--r-- 1 root root 4096 Jun 29 18:45 ro
-r--r--r-- 1 root root 4096 Jun 29 18:45 size
drwxr-xr-x 2 root root 0 Jun 29 18:45 slaves
-r--r--r-- 1 root root 4096 Jun 29 18:45 stat
lrwxrwxrwx 1 root root 0 Jun 29 18:45 subsystem -> ../../../../class/block
drwxr-xr-x 2 root root 0 Jun 29 18:45 trace
-rw-r--r-- 1 root root 4096 Jun 29 18:45 uevent
drwxr-xr-x 5 root root 0 Jun 29 16:34 xvdc1
[root@alpha xvdc]# pwd
/sys/devices/vbd-51744/block/xvdc
[root@alpha xvdc]# cat *
0
cat: bdi: Is a directory
10
202:32
cat: device: Is a director

Oracle Support- 4+ months ago[Information]
Hello Bin-Bin,

I don't see any trace of mfgblob_data diskgroup from the uploaded ASM alert.log which traced back to 25-Dec-2013 as below.
Maybe mfgblob_data diskgroup has never existed on the server, even though the disk name is MFGBLOB_DATA.
------------
alert_+ASM.log Wed Dec 25 09:55:36 2013 SUCCESS: diskgroup ALPHAPROD_DATA was mounted
alert_+ASM.log Wed Dec 25 09:55:48 2013 SUCCESS: diskgroup ALPHAPROD_DATA was dismounted
alert_+ASM.log Wed Dec 25 09:55:51 2013 SUCCESS: diskgroup ALPHAPROD_DATA was mounted
alert_+ASM.log Sat Dec 28 09:59:19 2013 SUCCESS: diskgroup ALPHAPROD_DATA was dismounted
alert_+ASM.log Sat Dec 28 11:29:26 2013 SUCCESS: diskgroup ALPHAPROD_DATA was mounted
alert_+ASM.log Sat Dec 28 11:33:20 2013 SUCCESS: diskgroup ALPHAPROD_DATA was dismounted
alert_+ASM.log Sat Dec 28 11:35:24 2013 SUCCESS: diskgroup ALPHAPROD_DATA was mounted
alert_+ASM.log Sun Feb 09 09:15:10 2014 SUCCESS: diskgroup ALPHAPROD_DATA was dismounted
alert_+ASM.log Sun Feb 09 10:22:12 2014 SUCCESS: diskgroup ALPHAPROD_DATA was mounted
alert_+ASM.log Mon Jun 16 16:47:30 2014 SUCCESS: diskgroup ALPHAPROD_DATA was dismounted
alert_+ASM.log Mon Jun 16 16:54:54 2014 SUCCESS: diskgroup ALPHAPROD_DATA was mounted
alert_+ASM.log Sun Jun 29 09:06:42 2014 SUCCESS: diskgroup ALPHAPROD_DATA was dismounted
----------------


I wonder if you could startup all databases running on the server after mounting ALPHAPROD_DATA diskgroup.

Can you also upload the following information ?

1. /dev/oracleasm/disks/MFGBLOB_DATA is mapped to /dev/xvdc.

$ cd /sys/block/xvdc
$ ls -l
$ cat *

2. Another fresh asm_info.html after mounting ALPHAPROD_DATA

Provide the following sql output (asm_info.html) after connecting to ASM instance.
==================================================================
spool asm_info.html
set pagesize 1000
set linesize 220
set feedback off
col bytes format 999,999,999,999
col space format 999,999,999,999
col gn format 999
col name format a25
col au format 99999999
col state format a12
col type format a12
col total_mb format 999,999,999
col free_mb format 999,999,999
col od format 999
col compatibility format a12
col dn format 999
col mount_status format a12
col header_status format a12
col mode_status format a12
col mode format a12
col failgroup format a25
col label format a20
col path format a50
col path1 format a40
col path2 format a40
col path3 format a40
col bytes_read format 999,999,999,999,999
col bytes_written format 999,999,999,999,999
col cold_bytes_read format 999,999,999,999,999
col cold_bytes_written format 999,999,999,999,999

alter session set nls_date_format='DD-MON-YYYY HH24:MI:SS' ;

select to_char(sysdate, 'DD-MON-YYYY HH24:MI:SS' ) current_time from dual;
select group_number gn, name, allocation_unit_size au, state, type, total_mb, free_mb, offline_disks od, compatibility from v$asm_diskgroup;
select group_number gn,disk_number dn, mount_status, header_status,mode_status,state, total_mb, free_mb,name, failgroup, label, path from v$asm_disk order by group_number, disk_number;
select inst_id, power_kfgmg, sofar_kfgmg, work_kfgmg, rate_kfgmg, time_kfgmg, file_kfgmg from x$kfgmg;

break on g_n skip 1
break on failgroup skip 1
compute sum of t_mb f_mb on failgroup
compute count of failgroup on failgroup

select g.group_number g_n,g.disk_number d_n,g.name , g.path , g.total_mb t_mb,g.free_mb f_mb,g.failgroup from v$asm_disk g order by g_n, failgroup, d_n;
select inst_id, power_kfgmg, sofar_kfgmg, work_kfgmg, rate_kfgmg, time_kfgmg, file_kfgmg from x$kfgmg;

SET MARKUP HTML ON
set echo on


select 'THIS ASM REPORT WAS GENERATED AT: ==)> ' , sysdate " " from dual;


select 'HOSTNAME ASSOCIATED WITH THIS ASM INSTANCE: ==)> ' , MACHINE " " from v$session where program like '%SMON%';

select * from v$asm_diskgroup;

SELECT * FROM V$ASM_DISK ORDER BY GROUP_NUMBER,DISK_NUMBER;
select inst_id, power_kfgmg, sofar_kfgmg, work_kfgmg, rate_kfgmg, time_kfgmg, file_kfgmg from x$kfgmg;
SELECT * FROM V$ASM_CLIENT;

select * from V$ASM_ATTRIBUTE;

select * from v$asm_operation;

select * from v$version;

show parameter

show sga

spool off

exit

==================================================================



Regards,
Eun Heo 

BLIU@KEIHIN-NA.COM- 4+ months ago[Update from Customer]
Upload to TDS successful for the file kfed_FS.out.

BLIU@KEIHIN-NA.COM- 4+ months ago[Update from Customer]
Upload to TDS successful for the file kfed_DH.out.

BLIU@KEIHIN-NA.COM- 4+ months ago[Update from Customer]
Upload to TDS successful for the file kfed_BK.out.

BLIU@KEIHIN-NA.COM- 4+ months ago[Update from Customer]
Upload to TDS successful for the file kfed_4MBK.out.

BLIU@KEIHIN-NA.COM- 4+ months ago[Update from Customer]
are you saying the diskgroup should be alphaprod_data, not mfgblob_data?

[oracle@alpha ~]$ cat /proc/partitions
major minor #blocks name

202 0 492830720 xvda
202 1 512000 xvda1
202 2 492317696 xvda2
202 32 1169304832 xvdc
202 33 976561152 xvdc1
202 48 584960000 xvdd
202 49 581053440 xvdd1
252 0 52428800 dm-0
252 1 27885568 dm-1
252 2 883859456 dm-2
252 3 26214400 dm-3
252 4 10485760 dm-4
252 5 41943040 dm-5
252 6 26214400 dm-6
[oracle@alpha ~]$ ls -l /sys/block/*
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/dm-0 -> ../devices/virtual/block/dm-0
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/dm-1 -> ../devices/virtual/block/dm-1
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/dm-2 -> ../devices/virtual/block/dm-2
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/dm-3 -> ../devices/virtual/block/dm-3
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/dm-4 -> ../devices/virtual/block/dm-4
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/dm-5 -> ../devices/virtual/block/dm-5
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/dm-6 -> ../devices/virtual/block/dm-6
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/loop0 -> ../devices/virtual/block/loop0
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/loop1 -> ../devices/virtual/block/loop1
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/loop2 -> ../devices/virtual/block/loop2
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/loop3 -> ../devices/virtual/block/loop3
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/loop4 -> ../devices/virtual/block/loop4
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/loop5 -> ../devices/virtual/block/loop5
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/loop6 -> ../devices/virtual/block/loop6
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/loop7 -> ../devices/virtual/block/loop7
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram0 -> ../devices/virtual/block/ram0
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram1 -> ../devices/virtual/block/ram1
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram10 -> ../devices/virtual/block/ram10
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram11 -> ../devices/virtual/block/ram11
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram12 -> ../devices/virtual/block/ram12
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram13 -> ../devices/virtual/block/ram13
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram14 -> ../devices/virtual/block/ram14
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram15 -> ../devices/virtual/block/ram15
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram2 -> ../devices/virtual/block/ram2
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram3 -> ../devices/virtual/block/ram3
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram4 -> ../devices/virtual/block/ram4
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram5 -> ../devices/virtual/block/ram5
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram6 -> ../devices/virtual/block/ram6
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram7 -> ../devices/virtual/block/ram7
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram8 -> ../devices/virtual/block/ram8
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/ram9 -> ../devices/virtual/block/ram9
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/xvda -> ../devices/vbd-51712/block/xvda
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/xvdc -> ../devices/vbd-51744/block/xvdc
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/xvdd -> ../devices/vbd-51760/block/xvdd
lrwxrwxrwx 1 root root 0 Jun 29 14:03 /sys/block/zram0 -> ../devices/virtual/block/zram0
[oracle@alpha ~]$ 

BLIU@KEIHIN-NA.COM- 4+ months ago[Update from Customer]
i can mount alphaprod_data. how about mfgblob_data
do the same thing?

thanks!

Bin-Bin

Oracle Support- 4+ months ago[ODM Proposed Solution(s)]
Hello Bin-Bin,

It seem to me that the reason why ALPHAPROD_DATA can not be mounted is due to the following error.

--
alert_+ASM.log Sun Jun 29 15:45:30 2014 ORA-15085: ASM disk "" has inconsistent sector size.
alert_+ASM.log Sun Jun 29 16:12:00 2014 ORA-15085: ASM disk "" has inconsistent sector size.
alert_+ASM.log Sun Jun 29 16:19:28 2014 ORA-15085: ASM disk "" has inconsistent sector size.
--

This indicates that the sector size for the underlying device mapped to /dev/oracleasm/disks/MFGBLOB_DATA has changed to 4096 which may have been 512 byte before.

Action Plan
------------------
1. Can you try to mount ALPHAPROD_DATA in the following way ?
$ sqlplus / as sysasm
SQL> alter system set asm_diskstring='/dev/oracleasm/disks/*' ;
SQL> alter diskgroup ALPHAPROD_DATA mount ;

<<< If ALPHAPROD_DATA can be mounted in this way, you can startup the databases running on this server.

2. But you will need to find out why the secor size has been changed for the underlying device mapped to /dev/oracleasm/disks/MFGBLOB_DATA.
Show us the following information

$ cat /proc/partitions
$ ls -l /sys/block/*

Regards,
Eun Heo 

Oracle Support- 4+ months ago[ODM Cause Justification]
alert_+ASM.log Sun Jun 29 15:45:30 2014 ORA-15085: ASM disk "" has inconsistent sector size.
alert_+ASM.log Sun Jun 29 16:12:00 2014 ORA-15085: ASM disk "" has inconsistent sector size.
alert_+ASM.log Sun Jun 29 16:19:28 2014 ORA-15085: ASM disk "" has inconsistent sector size. 

Oracle Support- 4+ months ago[ODM Cause Determination]
o It seem that sector size has been changed to 4096 from 512 

BLIU@KEIHIN-NA.COM- 4+ months ago[Update from Customer]
Upload to TDS successful for the file asm_info.html.

BLIU@KEIHIN-NA.COM- 4+ months ago[Update from Customer]
Upload to TDS successful for the file sosreport-bliu.3-9255522561-20140629173734-6f24.tar.xz.

BLIU@KEIHIN-NA.COM- 4+ months ago[Update from Customer]
yes, i see one.

[oracle@alpha disks]$ ls -l
total 0
brw-rw---- 1 oracle oinstall 202, 33 Jun 29 15:42 MFGBLOB_DATA
[oracle@alpha disks]$ pwd
/dev/oracleasm/disks
[oracle@alpha disks]$ 

BLIU@KEIHIN-NA.COM- 4+ months ago[Update from Customer]
i will send you another sosreport, and the output of your sql script.
please check the early message, it is about ALPHAPROD_DATA diskgroup.
someone changed the asm_diskgroups, i will changed it back.

SQL> show parameter asm

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
asm_diskgroups string ALPHAPROD_DATA
asm_diskstring string ORCL:MFGBLOB_DATA
asm_power_limit integer 1
asm_preferred_read_failure_groups string


Bin-Bin

Oracle Support- 4+ months ago[Information]
Hello Bin-Bin,

The following error from ASMalert.log indicates that Oracle ASM can not discover ASM disks in /dev/oracleasm/disks/* .

1] alert_+ASM.log
-----------------------
asm_diskstring = "ORCL:MFGBLOB_DATA"
asm_diskgroups = "MFGBLOB_DATA"
..
Sun Jun 29 16:35:43 2014
ASM Health Checker found 1 new failures
GMON dismounting group 1 at 4 for pid 17, osid 11746
ERROR: diskgroup MFGBLOB_DATA was not mounted
ORA-15032: not all alterations performed
ORA-15017: diskgroup "MFGBLOB_DATA" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "MFGBLOB_DATA"
--------------------------------------------

Question - Do you see any of ASM disks in /dev/oracleasm/disks/* directory ?


And I'd like to see the following information to investigate the current issue further

Action Plan
-------------------
1. Provide the following sql output (asm_info.html) after connecting to ASM instance.
==================================================================
spool asm_info.html
set pagesize 1000
set linesize 220
set feedback off
col bytes format 999,999,999,999
col space format 999,999,999,999
col gn format 999
col name format a25
col au format 99999999
col state format a12
col type format a12
col total_mb format 999,999,999
col free_mb format 999,999,999
col od format 999
col compatibility format a12
col dn format 999
col mount_status format a12
col header_status format a12
col mode_status format a12
col mode format a12
col failgroup format a25
col label format a20
col path format a50
col path1 format a40
col path2 format a40
col path3 format a40
col bytes_read format 999,999,999,999,999
col bytes_written format 999,999,999,999,999
col cold_bytes_read format 999,999,999,999,999
col cold_bytes_written format 999,999,999,999,999

alter session set nls_date_format='DD-MON-YYYY HH24:MI:SS' ;

select to_char(sysdate, 'DD-MON-YYYY HH24:MI:SS' ) current_time from dual;
select group_number gn, name, allocation_unit_size au, state, type, total_mb, free_mb, offline_disks od, compatibility from v$asm_diskgroup;
select group_number gn,disk_number dn, mount_status, header_status,mode_status,state, total_mb, free_mb,name, failgroup, label, path from v$asm_disk order by group_number, disk_number;
select inst_id, power_kfgmg, sofar_kfgmg, work_kfgmg, rate_kfgmg, time_kfgmg, file_kfgmg from x$kfgmg;

break on g_n skip 1
break on failgroup skip 1
compute sum of t_mb f_mb on failgroup
compute count of failgroup on failgroup

select g.group_number g_n,g.disk_number d_n,g.name , g.path , g.total_mb t_mb,g.free_mb f_mb,g.failgroup from v$asm_disk g order by g_n, failgroup, d_n;
select inst_id, power_kfgmg, sofar_kfgmg, work_kfgmg, rate_kfgmg, time_kfgmg, file_kfgmg from x$kfgmg;

SET MARKUP HTML ON
set echo on


select 'THIS ASM REPORT WAS GENERATED AT: ==)> ' , sysdate " " from dual;


select 'HOSTNAME ASSOCIATED WITH THIS ASM INSTANCE: ==)> ' , MACHINE " " from v$session where program like '%SMON%';

select * from v$asm_diskgroup;

SELECT * FROM V$ASM_DISK ORDER BY GROUP_NUMBER,DISK_NUMBER;
select inst_id, power_kfgmg, sofar_kfgmg, work_kfgmg, rate_kfgmg, time_kfgmg, file_kfgmg from x$kfgmg;
SELECT * FROM V$ASM_CLIENT;

select * from V$ASM_ATTRIBUTE;

select * from v$asm_operation;

select * from v$version;

show parameter

show sga

spool off

exit

==================================================================



2. Upload ASM device information on the server in the following way below.

Download ASMCHECK zip file ( including AsmCheck.sh and AsmCheck.sql ) from note 1346190.1 and place them in any directory where grid OS user can read & write and run it in the following way.

As root OS user,
# chmod 755 AsmCheck.sh
# export ORACLE_HOME=<GRID_HOME> <<< Specify GRID_HOME from your env.
# export ORACLE_SID=<ASM#> <<< Specify ASM instance number from your env.
# ./AsmCheck.sh 'GRID_OS_USER' > `hostname`
Example> If GRID_OS User on the serveris grid, # ./AsmCheck.sh grid > `hostname`

NOTE 1: Some errors will be on the screen if the specified OS command and or directory does not exist, please ignore them.
NOTE 2: The following shell script can be run regardless of whether ASMlib is being used on the server.
NOTE 3: The following shell script can be run without running ASM instance on the server.
NOTE 4: Copy and paste the output of AsmCheck.sh on the sr
NOTE 5: Upload the following files after running AsmCheck.sh
**********************************************************************************
1] file generated with `hostname` in the directory where AsmCheck.sh runs.
2] kfed_DH.out/kfed_FS.out/kfed_BK.out/kfed_4MBK.out
*********************************************************************************



Regards,
Eun Heo 

Oracle Support- 4+ months ago[ODM Issue Verification]
1] alert_+ASM.log
-----------------------
asm_diskstring = "ORCL:MFGBLOB_DATA"
asm_diskgroups = "MFGBLOB_DATA"
..
Sun Jun 29 16:35:43 2014
ASM Health Checker found 1 new failures
GMON dismounting group 1 at 4 for pid 17, osid 11746
ERROR: diskgroup MFGBLOB_DATA was not mounted
ORA-15032: not all alterations performed
ORA-15017: diskgroup "MFGBLOB_DATA" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "MFGBLOB_DATA"
-------------------------------------------- 

Oracle Support- 4+ months ago[ODM Issue Clarification]
o MFGBLOB_DATA diskgroup can not be mounted due to the following error
-----------
ORA-15017: diskgroup "MFGBLOB_DATA" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "MFGBLOB_DATA"
------------- 

Oracle Support- 4+ months ago[First Contact]
First Phone Contact Comments
--------------------------------------------------------
Hello Bin-Bin,


This SR has been transferred to Oracle ASM team, My name is Eun Heo. I have been assigned this new Sev 1 SR and am reviewing the SR and any attachments.

By default, this SR was created as 24x7. Do you need this SR to be worked around the clock 24x7?

If not, I can hold this SR in my queue, to be worked as a Sev1 non-24x7 during my normal working hours.

Regards
Eun Heo
Global Product Support
Working Hours: Monday – Friday 07:00-15:30 AEST / 21:00-05:30 GMT / 14:00-22:30 PST 

BLIU@KEIHIN-NA.COM- 4+ months ago[Update from Customer]
Upload to TDS successful for the file sosreport-bliu.3-9255522561-20140629170915-382d.tar.xz.

BLIU@KEIHIN-NA.COM- 4+ months ago[Update from Customer]
Upload to TDS successful for the file virtual_machine.rar.

BLIU@KEIHIN-NA.COM- 4+ months ago[Customer Problem Description]
Severity 1 Contact Information
---------------------------------------------------
** Customer's Management 24x7 contact name: Bin-Bin Liu
** Customer's Management 24x7 contact number: +1.317.471.7002
** Customer's Management E-mail Address: bliu@keihin-na.com

** Primary Customer 24x7 contact name: Bin-Bin Liu
** Primary Customer 24x7 contact number: +1.317.471.7002
** Primary Customer E-mail Address: bliu@keihin-na.com
** Primary Customer Contact Method: Phone

** Secondary Customer 24x7 contact name: Bin-Bin Liu
** Secondary Customer 24x7 contact number: +1.317-471-7002
** Alternate 24x7 Contact Number: +1.317-471-7002
** Secondary Customer E-mail Address: bliu@keihin-na.com

Customer Problem Description
---------------------------------------------------

Problem Summary
---------------------------------------------------
OVM server after upgraded, virtual machine ASM diskgroup couldn't mpuntd

Problem Description
---------------------------------------------------
this is emergency. this is my production system.
i followed the oracle menu upgraded my VM server to 3.2.8, there was no problem on VM server.
when i start my virtual machine, i got the following messages.
I'll attach all information to you.

Bin-Bin

Wed Dec 25 09:55:36 2013
SQL> ALTER DISKGROUP ALL MOUNT /* asm agent call crs *//* {0:0:95} */
SQL> ALTER DISKGROUP ALL ENABLE VOLUME ALL /* asm agent *//* {0:0:95} */
Wed Dec 25 09:55:36 2013
ALTER SYSTEM SET local_listener='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=10.18.1.126)(PORT=1521))))' SCOPE=MEMORY SID='+ASM';
NOTE: Loaded library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so
Wed Dec 25 09:55:36 2013
SQL> CREATE DISKGROUP ALPHAPROD_DATA EXTERNAL REDUNDANCY DISK 'ORCL:MFGBLOB_DATA' ATTRIBUTE 'compatible.asm'='11.2.0.0.0' /* ASMCA */
NOTE: Assigning number (1,0) to disk (ORCL:MFGBLOB_DATA)
NOTE: initializing header on grp 1 disk MFGBLOB_DATA
NOTE: initiating PST update: grp = 1
GMON updating group 1 at 1 for pid 18, osid 6891
NOTE: group ALPHAPROD_DATA: initial PST location: disk 0000 (PST copy 0)
NOTE: PST update grp = 1 completed successfully
NOTE: cache registered group ALPHAPROD_DATA number=1 incarn=0x2c6de211
NOTE: cache began mount (first) of group ALPHAPROD_DATA number=1 incarn=0x2c6de211
NOTE: cache opening disk 0 of grp 1: MFGBLOB_DATA label:MFGBLOB_DATA
NOTE: cache creating group 1/0x2C6DE211 (ALPHAPROD_DATA)
NOTE: cache mounting group 1/0x2C6DE211 (ALPHAPROD_DATA) succeeded
NOTE: allocating F1X0 on grp 1 disk MFGBLOB_DATA
NOTE: Created Used Space Directory for 1 threads
NOTE: diskgroup must now be re-mounted prior to first use
NOTE: cache dismounting (clean) group 1/0x2C6DE211 (ALPHAPROD_DATA)
NOTE: messaging CKPT to quiesce pins Unix process pid: 6891, image: oracle@knaalpha.kna.local (TNS V1-V3)
NOTE: lgwr not being msg'd to dismount
NOTE: cache dismounted group 1/0x2C6DE211 (ALPHAPROD_DATA)
GMON dismounting group 1 at 2 for pid 18, osid 6891
GMON dismount

Error Codes
---------------------------------------------------


Problem Category/Subcategory
---------------------------------------------------
Database Storage (ASM, ACFS, DNFS, Compression, Corruption, Partitioning, LOBS, Tablespaces, Files)/ASM_ACFS: Generic problems

Uploaded Files
---------------------------------------------------


Template Question Responses
---------------------------------------------------
1) ### RECENT CHANGES ###


2) ### SHARING ORACLE_HOME ###


3) ### DATABASE ORACLE_HOME ###


4) ### MULTIPATHING INFORMATION ###


5) ### STORAGE CONFIGURATION ###


6) Describe how this problem is impacting your business. Include relevant information such as critical events (i.e., upgrade or project milestones), dates (i.e., go live dates), number of users affected, financial impact, etc.
at No comments:
Subscribe to: Posts (Atom)

  How to Change Instance Type & Security Group of EC2 in AWS By David Taylor Updated April 29, 2023 EC2 stands for Elastic Compute Cloud...