Friday, August 9, 2019

Make a RAID Volume Bootable Using the LSI WebBIOS Configuration Utility

Make a RAID Volume Bootable Using the LSI WebBIOS Configuration Utility

1. Reset or power on the server.
For example, to reset the server:
■ From the local server, press the Power button (approximately 1 second) on the
front panel of the server to power off the server, then press the Power button
again to power on the server.
■ From the Oracle ILOM web interface, select Host Management > Power
Control, then select Reset from the Select Action list box.
■ From the Oracle ILOM CLI, type: reset /System
The power-on self-test (POST) sequence begins.

2. While the BIOS is running the power-on self-tests (POST), and upon seeing the
prompt Press <Ctrl><H> for WebBIOS..., immediately press the Ctrl+H
key combination to access the LSI MegaRAID utility.
The Adapter Selection screen appears.

Launch Oracle System Assistant at Startup

Launch Oracle System Assistant at Startup


1. Verify that the server is in standby power mode.
2. Verify that a monitor, keyboard, and mouse are attached to the server, either locally or through a remote KVM session. For details, see “Launch a Remote System Console Redirection Session” on page 40.
3. Power on the server. Boot messages appear on the monitor. 4. When prompted, press the F9 function key. You can also press CTRL-O on a serial keyboard.

ORA-12571: TNS:packet writer failure – One of the hardest problems I’ve had to resolve

ORA-12571: TNS:packet writer failure

It is a very complicated problem,
mostly related to getting information correctly back and forth across the network.

- a firewall is getting in the way
- host name resolution failing
- multiple process colliding on the same port number
- somewhere in the infrastructure there is a timeout, so (for example) a network router may drop inactive connections
- anti-virus software (typically includes a software firewall).
- somewhere in the infrastructure there is a timeout, so (for example) a network router may drop inactive connections

It is "something is stopping us from successfully sending information between the db and the client across sqlnet."

ORA-12571: TNS:packet writer failure. This is one of the hardest problems I’ve come across in my career and take us a whopping 16 months to resolve. The issue wasn’t really causing any serious problems on our website, but it was a very annoying problem that we spent a ton of time resolving. I want anyone else experiencing this problem to see the solution, so I posted it on Experts Exchange and this blog. Below is the subsequent question I posted on Experts Exchange, and 16 months later the actual solution.

Question/Problem

We have multiple web servers that are load balanced. Each web server connects to our Oracle Database Server through a firewall and load balancer. Everything was working correctly when our Oracle Database Server was Solaris 8 running Oracle 9i.
Since we migrated to Oracle Enterprise Linux 64-bit 5.4 (RHEL 5) running Oracle 11G update 2 (11.2.0.1.0), we are now seeing ORA-12571: TNS:packet writer failure at Oracle.DataAccess.Client.OracleException.HandleErrorHelper messages. The new Linux Oracle Database Server is on the same subnet as the Solaris Oracle 9i Server, and the path the network traffic takes is identical.
What is particularly strange about it is that it happens most (if not exclusively) at times where our website traffic is low. For example, on our busiest day of the month website traffic wise, we went without a TNS:packet writer failure error for 17 hours (7:00am until midnight). Yet as soon as we went into the early hours of the morning (where are website traffic is low or lower), those errors came back.
When we get one of those errors, the website visitor can just wait 1 -3 secs and refresh the page and everything works correctly.
Here’s some other info:
Web servers are running IIS 6, Oracle ODP Net 10.2.0.1.0. Web server connection string (excluding username, p/w, server, port) is ;Persist Security Info=False;Connection Timeout=30;Connection Lifetime=120;Enlist=False;Pooling=True;Max Pool Size=25;Min Pool Size=5;Incr Pool Size=1;Decr Pool Size=1
On the Database Server
listener.ora connect_timeout=10 – no other listener parameters explicitly set.
sqlnet.ora – no parameters explicitly set
Network port stats on Server and switch do not show dropped packets or anything leading us to believe there is a network problem. Also, like I say, those TNS:packet writer failure messages occur during times of less website traffic.
Does anyone have any idea why we are getting those TNS:packet writer failure messages?

Answer/Solution

We finally discovered the cause of the problem. What was happening is that in periods of inactivity the database connections from the web servers had no activity and were being severed after 1hr by the Cisco Firewall and/or Cisco ACE 4710 load balancer (both have 1hr inactivity timeout thresholds by default).
To get around this problem, we added “SQLNET.EXPIRE_TIME= 10” to the sqlnet.ora and reloaded the listeners. This is a dead connection detection parameter that will tell the database server to check that the connections are still alive. How this helps is that it checks the database connections with the web servers every 10 mins and thus the firewall and/or load balancer will thus see activity and thus NOT severe the connections due to inactivity.
This was actually implemented on our old Oracle database Server (Solaris 8/Oracle 9i) but NOT implemented on the new Linux/Oracle 11G Server. And unfortunately was overlooked until now.

Wednesday, August 7, 2019

Oracle 11gR2 - Active Data Guard

Oracle 11gR2 - Active Data Guard

Active Data Guard allows a standby database to be opened for read-only access whilst redo is still being applied. For some applications Active Data Guard can represent a more efficient use of Oracle licenses on the standby database. However, this benefit is offset to a certain extent by the fact that Active Data Guard is available on Enterprise Edition only and is cost option which must be licensed on both the primary and standby database.

Several of my customers are currently using Active Data Guard; in general they are very happy with it. A few others have discovered that it is very easy to inadvertently enable Active Data Guard. This is not desirable or advisable as Oracle have instigated licence audits with a large number of UK customers over the past couple of years.
To determine whether a standby database is using Active Data Guard use the following query:

SELECT database_role, open_mode FROM v$database;


For example: 

SQL> SELECT database_role, open_mode FROM v$database;

DATABASE_ROLE    OPEN_MODE
---------------- --------------------
PHYSICAL STANDBY READ ONLY WITH APPLY
If you start a database in SQL*Plus using the STARTUP command and then invoke managed recovery, the Active Data Guard will be enabled. For example:

[oracle@server14]$ sqlplus / as sysdba
SQL> STARTUP
ORACLE instance started.
Total System Global Area 6497189888 bytes
Fixed Size 2238672 bytes
Variable Size 3372222256 bytes
Database Buffers 3103784960 bytes
Redo Buffers 18944000 bytes
Database mounted
Database opened

SQL> SELECT database_role, open_mode FROM v$database;
DATABASE_ROLE    OPEN_MODE
---------------- --------------------
PHYSICAL STANDBY READ ONLY

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE
USING CURRENT LOGFILE WITH SESSION SHUTDOWN;

SQL> SELECT database_role, open_mode FROM v$database;

DATABASE_ROLE    OPEN_MODE
---------------- --------------------
PHYSICAL STANDBY READ ONLY WITH APPLY

However, if the database is started in SQL*Plus using the STARTUP MOUNT  command and then managed recovery is invoked, Active Data Guard will not be  enabled. 

[oracle@server14]$ sqlplus / as sysdba
SQL> STARTUP MOUNT
ORACLE instance started.
Total System Global Area 6497189888 bytes
Fixed Size 2238672 bytes
Variable Size 3372222256 bytes
Database Buffers 3103784960 bytes
Redo Buffers 18944000 bytes
Database mounted
Database opened

SQL> SELECT database_role, open_mode FROM v$database;

DATABASE_ROLE    OPEN_MODE
---------------- --------------------
PHYSICAL STANDBY MOUNTED

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE
USING CURRENT LOGFILE WITH SESSION SHUTDOWN;

SQL> SELECT database_role, open_mode FROM v$database;

DATABASE_ROLE    OPEN_MODE
---------------- --------------------
PHYSICAL STANDBY MOUNTED

In the database has been started in SQL*Plus using STARTUP MOUNT and the  database is subsequently opened read only, then invoking managed recovery  will enable Active Data Guard. For example: 

[oracle@server14]$ sqlplus / as sysdba
SQL> STARTUP MOUNT
ORACLE instance started.
Total System Global Area 6497189888 bytes
Fixed Size 2238672 bytes
Variable Size 3372222256 bytes
Database Buffers 3103784960 bytes
Redo Buffers 18944000 bytes
Database mounted
Database opened

SQL> SELECT database_role, open_mode FROM v$database;

DATABASE_ROLE    OPEN_MODE
---------------- --------------------
PHYSICAL STANDBY MOUNTED

SQL> ALTER DATABASE OPEN READ ONLY;

SQL> SELECT database_role, open_mode FROM v$database;

DATABASE_ROLE    OPEN_MODE
---------------- --------------------
PHYSICAL STANDBY READ ONLY

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE
USING CURRENT LOGFILE WITH SESSION SHUTDOWN;

SQL> SELECT database_role, open_mode FROM v$database;

DATABASE_ROLE    OPEN_MODE
---------------- --------------------
PHYSICAL STANDBY READ ONLY WITH APPLY

Of course not all databases are started using SQL*Plus. 
If you start the database using SRVCTL then the default open mode can be specified in the OCR.
You can check the default open mode for a database using SRVCTL CONFIG DATABASE.
For example if the database is called PROD:

[oracle@server14]$ srvctl config database -d PROD
Database unique name: PROD
Database name: PROD
Oracle home: /u01/app/oracle/product/11.2.0/dbhome_1
Oracle user: oracle
Spfile: +DATA1/PROD/spfilePROD.ora
Domain:Start options: open
Stop options: immediate
Database role: PHYSICAL_STANDBY
Management policy: AUTOMATIC
Server pools: PROD
Disk Groups: DATA1, FRA1
Mount point paths:
Services:
Type: SINGLE
Database is administrator managed

In the above example, if the PROD database is started using SRVCTL then the  database will be opened in read-only mode.
For example:

[oracle@server14]$ srvctl start database -d PROD
[oracle@server14]$ sqlplus / as sysdba

SQL> SELECT database_role, open_mode FROM v$database;

DATABASE_ROLE    OPEN_MODE
---------------- --------------------
PHYSICAL STANDBY READ ONLY

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE
USING CURRENT LOGFILE WITH SESSION SHUTDOWN;

SQL> SELECT database_role, open_mode FROM v$database;

DATABASE_ROLE    OPEN_MODE
---------------- --------------------
PHYSICAL STANDBY READ ONLY WITH APPLY

The default start mode can be modified in the OCR using the SRVCTL MODIFY  DATABASE command. 

For example:

[oracle@server14]$ srvctl modify database -d PROD -s mount
The database configuration is updated as follows:

[oracle@server14]$ srvctl config database -d PROD
Database unique name: PROD
Database name: PROD
Oracle home: /u01/app/oracle/product/11.2.0/dbhome_1
Oracle user: oracle
Spfile: +DATA1/PROD/spfilePROD.ora
Domain:
Start options: mount
Stop options: immediate
Database role: PHYSICAL_STANDBY
Management policy: AUTOMATIC
Server pools: PROD
Disk Groups: DATA1, FRA1
Mount point paths:
Services:
Type: SINGLE
Database is administrator managed


When the default start mode is set to mount, Active Data Guard will not  be enabled when managed recovery is invoked. For example: 

[oracle@server14]$ srvctl start database -d PROD
[oracle@server14]$ sqlplus / as sysdba

SQL> SELECT database_role, open_mode FROM v$database;

DATABASE_ROLE    OPEN_MODE
---------------- --------------------
PHYSICAL STANDBY MOUNTED

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE
USING CURRENT LOGFILE WITH SESSION SHUTDOWN;

SQL> SELECT database_role, open_mode FROM v$database;

DATABASE_ROLE    OPEN_MODE
---------------- --------------------
PHYSICAL STANDBY MOUNTED


You can also specify the start mode as a parameter to the SRVCTL START  DATABASE command 
For example:

[oracle@server14] srvctl start database -d PROD -o open
[oracle@server14] srvctl start database -d PROD -o mount




Take care when performing a switchover or switchback that the OCR is updated as part of the procedure. 

Read-Only Standby and Active Data Guard

Once a standby database is configured, it can be opened in read-only mode to allow query access. This is often used to offload reporting to the standby server, thereby freeing up resources on the primary server. When open in read-only mode, archive log shipping continues, but managed recovery is stopped, so the standby database becomes increasingly out of date until managed recovery is resumed.

Switchover and failover in oracle 11g Data - Guard

Switchover and failover in oracle 11g Data - Guard

Switchover
Allows the primary database to switch roles with one of its standby databases. There is no data loss during a switchover. After a switchover, each database continues to participate in the Data Guard configuration with its new role.

Failover
Changes a standby database to the primary role in response to a primary database failure. If the primary database was not operating in either maximum protection mode or maximum availability mode before the failure, some data loss may occur. If Flashback Database is enabled on the primary database, it can be reinstated as a standby for the new primary database once the reason for the failure is corrected. verify the  primary database can be switched to the standby role.

Preparing for a Role Transition

        1)        Verify that there are no redo transport errors or redo gaps at the standby database by querying the V$ARCHIVE_DEST_STATUS view on the primary database.

SELECT STATUS, GAP_STATUS FROM V$ARCHIVE_DEST_STATUS WHERE DEST_ID = 2;

STATUS GAP_STATUS
--------- ------------------------
VALID NO GAP

       2)       Ensure temporary files exist on the standby database that match the temporary files on the primary database.

        3)       Remove any delay in applying redo that may be in effect on the standby database that will become the new primary database.

        4)       Before performing a switchover to a physical standby database that is in real-time query mode, consider bringing all instances of that standby database to the mounted but not open state to achieve the fastest possible role transition and to cleanly terminate any user sessions connected to the physical standby database prior to the role transition

Switchover Steps

        1)       Check Primary and standby databases are ready to  switchover

On Primary database
SELECT SWITCHOVER_STATUS FROM V$DATABASE;

SWITCHOVER_STATUS
 -----------------
 TO STANDBY
 1 row selected

·          A value of TO STANDBY or SESSIONS ACTIVE indicates that the primary database can be switched to the standby role. If neither of these values is returned, a switchover is not possible because redo transport is either misconfigured or is not functioning properly.

On Secondary Database

SELECT SWITCHOVER_STATUS FROM V$DATABASE;

SWITCHOVER_STATUS
-----------------
TO_PRIMARY
1 row selected

·          A value of TO PRIMARY or SESSIONS ACTIVE indicates that the standby database is ready to be switched to the primary role. If neither of these values is returned, verify that Redo Apply is active and that redo transport is configured and working properly.

         2)       Initiate the switchover on the primary database.

ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY WITH SESSION SHUTDOWN;
 
         3) Shut down and then mount the former primary database.

SHUTDOWN ABORT;
STARTUP MOUNT;

   4)  Switch the target physical standby database role to the primary role.

  ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WITH SESSION SHUTDOWN;

   5)    Open the new primary database.

ALTER DATABASE OPEN;

  6)    Start Redo Apply on the new physical standby database.

 ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT FROM SESSION;

Steps for failover

        1)       If the primary database can be mounted, it may be possible to flush any unsent archived and current redo from the primary database to the standby database. If this operation is successful, a zero data loss failover is possible even if the primary database is not in a zero data loss data protection mode.

ALTER SYSTEM FLUSH REDO TO target_db_name;
       
         2)    Verify that the standby database has the most recently archived redo log file for each primary database redo thread.

   SELECT UNIQUE THREAD# AS THREAD, MAX(SEQUENCE#) OVER (PARTITION BY thread#) AS LAST from V$ARCHIVED_LOG;

   THREAD       LAST
---------- ----------
         1       3555

       3)       If possible, copy the most recently archived redo log file for each primary database redo thread to the standby database if it does not exist there, and register it. This must be done for each redo thread.

     ALTER DATABASE REGISTER PHYSICAL LOGFILE 'filename ';
         
          4)       Query the V$ARCHIVE_GAP view on the target standby database to determine if there are any redo gaps on the target standby database.
SELECT THREAD#, LOW_SEQUENCE#, HIGH_SEQUENCE# FROM V$ARCHIVE_GAP;
THREAD#    LOW_SEQUENCE# HIGH_SEQUENCE#
---------- ------------- --------------
         1            3555          3555
                                                  
         5)       Stop Redo Apply.

ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;
           
         6)       Finish applying all received redo data.

ALTER DATABASE RECOVER MANAGED STANDBY DATABASE FINISH;
           
         7)       Verify that the target standby database is ready to become a primary database.

SELECT SWITCHOVER_STATUS FROM V$DATABASE;

SWITCHOVER_STATUS
-----------------
TO_PRIMARY

         8)       Switch the physical standby database to the primary role.

ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WITH SESSION SHUTDOWN;
         
         9)       Open the new primary database.

ALTER DATABASE OPEN;

Convert Physical Standby database into snapshot standby database .

We can open  standby database in read-write mode .When switched back into standby mode, all changes made whilst in read-write mode are lost is know as Snapshot standby database .

Priversly This is achieved using flashback database, but from 11g standby database does not need to have flashback database explicitly enabled to take advantage
of this feature, thought it works just the same if it is.

How To Set Up Physical Standby Database You Can Check Here 
Steps


            1)      Bring database in mount state

SHUTDOWN IMMEDIATE;
STARTUP MOUNT;

2)      Disable  recovery  on standby

ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;

3)      Convert standby database to flashback

ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;

4)      Open database

ALTER DATABASE OPEN;

5)      Check  database status



Select NAME, OPEN_MODE, DATABASE_ROLE from v$database;
NAME          OPEN_MODE            DATABASE_ROLE
--------------             ----------           ----------------
ORCL_STBY        READ WRITE    SNAPSHOT STANDBY

SELECT flashback_on FROM v$database;

FLASHBACK_ON
------------------
RESTORE POINT ONLY



6)      To convert it back to the physical standby, losing all the changes made since the conversion to snapshot standby, issue the following commands.

SHUTDOWN IMMEDIATE;
STARTUP MOUNT;
ALTER DATABASE CONVERT TO PHYSICAL STANDBY;
SHUTDOWN IMMEDIATE;
STARTUP NOMOUNT;
ALTER DATABASE MOUNT STANDBY DATABASE;
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT;
SELECT flashback_on FROM v$database;

Select NAME, OPEN_MODE, DATABASE_ROLE from v$database;
NAME                  OPEN_MODE            DATABASE_ROLE
--------------             ----------                     ----------------
ORCL_STBY        READ WRITE           PYSICAL STANDBY

FLASHBACK_ON
------------------
NO


Active Data Guard :  While the standby is open read-only  on the same time the physical standby database can be  in recovery mode  is called  ACTIVE  DATA GUARD.
Advantage
This allows you to use this standby as a real-time reporting database or even to backup the primary data, also as a result it does not have any impact on RTO or RPO.
The following operations are disallowed
  •          Any Data Manipulation Language (DML) except for select statements
  •            Any Data Definition Language (DDL)
  •            Access of local sequences
  •            DMLs on local temporary tables
·    
Note :-However, this benefit is offset to a certain extent by the fact that Active Data Guard is available on Enterprise Edition only and is cost option which must be licensed on both the primary and standby database.

Steps To create Active DATA Guard .

    1) Check the status of the Primary database and  Physical standby database and the latest sequence generated in the primary database.

Primary

select status,instance_name,database_role from v$instance,v$database;

STATUS       INSTANCE_NAME    DATABASE_ROLE
------------ ---------------- ----------------
OPEN         ORCL           PRIMARY

 select max(sequence#) from v$archived_log;

MAX(SEQUENCE#)
--------------
3333


Standby

select status,instance_name,database_role from v$database,v$instance;

STATUS   INSTANCE_NAME DATABASE_ROLE
-------- ------------- ---------------------
MOUNTED  ORCL_STBY         PHYSICAL STANDBY

SQL> select max(sequence#) from v$archived_log where applied='YES';

MAX(SEQUENCE#)
--------------
3333


)     Check if the Managed Recovery Process (MRP) is active on the physcial standby database.

select process,status,sequence# from v$managed_standby;

2)     Cancel the MRP on the physical standby database and open the standby database  in            READ-ONLY mode

alter database recover managed standby database cancel;
alter database open read only;

select status,instance_name,database_role,open_mode from v$database,v$instance;

STATUS INSTANCE_NAME  DATABASE_ROLE    OPEN_MODE
------ -------------- ---------------- ---------------
OPEN   ORCL_STBY            PHYSICAL STANDBY READ ONLY

   3)    start the MRP on the physical standby database.

alter database recover managed standby database disconnectfrom session;

                                                                                                                                 
4)       Database is now Active Data Guard



 SELECT database_role, open_mode FROM v$database;

DATABASE_ROLE    OPEN_MODE
---------------- --------------------
PHYSICAL STANDBY READ ONLY WITH APPLY

  DAILY CHECKLIST Oracle Database instance is running or not select name,open_mode from V$database; Note: Check the Oracle databases are ru...