Wednesday, December 8, 2010
Configure 2 Node RAC on LINUX with NAS
REQUIREMENT FOR 2 NODE RAC
STEP1
==========================================================================
os (ORACLE ENTERPRISE LINUX 4)
required rpms
below is the example to install rpm
[root@inam1 ~]# rpm -ivh compat-gcc-32-3.2.3-47.3.src.rpm
warning: compat-gcc-32-3.2.3-47.3.src.rpm: V3 DSA signature: NOKEY, key ID db42a60e 1:compat-gcc-32 ########################################### [100%]
[root@inam1 ~]#
binutils-2.15.90.0.1.1-32.5 gcc-3.3.3-43.24
gcc-c++-3.3.3-43.24
glibc-2.3.3-98.28
gnome-libs-1.4.1.7-671.1
compat-libstdc++-296-2.96-132.7.2.i386
libstdc++-3.3.3-43.24
libstdc++-devel-3.3.3-43.24
make-3.80-184.1
pdksh-5.2.14-780.1
sysstat-5.0.1-35.1
xscreensaver-4.16-2.6
libaio-0.3.98
STEP2
=============================================================================================
network cards
Add the below enteries to /etc/hosts on both RAC NODES as root user
127.0.0.1 localhost.localdomain localhost
########PUBLIC IPS ##########################
192.168.3.1 prod1.technorack.com prod1
192.168.3.2 prod2.technorack.com prod2
192.168.1.10 nas.technorack.com nas
########VIRTUAL IPS ##########################
192.168.3.11 prod1-vip.technorack.com prod1-vip
192.168.3.12 prod2-vip.technorack.com prod2-vip
########PRIVATE IPS ##########################
10.1.1.1 prod1-priv.technorack.com prod1-priv
10.1.1.2 prod2-priv.technorack.com prod2-priv
STEP3
=============================================================================================
Add the following lines to the /etc/sysctl.conf file on both the nodes are root user
kernel.shmall = 2097152
kernel.shmmax = 536870912
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default=262144
net.core.wmem_default=262144
net.core.rmem_max=262144
net.core.wmem_max=262144
And run the below command to effect the prameters on fly
/sbin/sysctl -p
STEP4
=============================================================================================
Add the following lines to the /etc/security/limits.conf file on both the nodes as root user
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
STEP5
=============================================================================================
Add the following line to the /etc/pam.d/login file, if it does not already exist.
session required pam_limits.so
STEP6
=============================================================================================
Set the hangcheck kernel module parameters by adding the following line to the /etc/modprobe.conf file.on both the nodes.
options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180
To load the module immediately, execute "modprobe -v hangcheck-timer".
sTEP7
=============================================================================================(with root)
groupadd oinstallgroupadd dba
useradd -g oinstall -G dba -d /home/oracle oracle passwd oracle
sTEP8
=============================================================================================
(with root on both nodes)
chown -R oracle:oinstall /opt
(with oracle on both nodes)
mkdir -p /opt/app/oracle/product/10.2.0/crsmkdir -p /opt/app/oracle/product/10.2.0/asm mkdir -p /opt/app/oracle/product/10.2.0/dbmkdir -p /opt/app/oracle/oraInventory
sTEP9
=============================================================================================
set bash profile
[oracle@inam1 ~]$ cat .bash_profile
# .bash_profile
# Get the aliases and functionsif [ -f ~/.bashrc ]; then . ~/.bashrcfi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATHunset USERNAME
ORACLE_BASE=/opt/app/oracle; export ORACLE_BASEORACLE_HOME=$ORACLE_BASE/product/10.2.0/db; export ORACLE_HOMEORACLE_SID=prod2; export ORACLE_SIDCRS_HOME=$ORACLE_BASE/product/10.2.0/crs; export CRS_HOME ORACLE_TERM=xterm; export ORACLE_TERMPATH=/usr/sbin:$PATH; export PATHPATH=$ORACLE_HOME/bin:$CRS_HOME/bin:$PATH; export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATHCLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH
if [ $USER = "oracle" ]; then if [ $SHELL = "/bin/ksh" ]; then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fifi
sTEP10
==============================================================================================
configure user equivalence
run the below commands on both the nodes as oracle
[oracle@prod1 db]$ ssh-keygen -t rsa (overwrite:Y)
[oracle@prod1 db]$ ssh-keygen -t dsa (overwrite:Y)
Run the below command on only first node
[oracle@prod1 ~]$ cd .ssh
[oracle@prod1 .ssh]$ touch authorized_keys
[oracle@prod1 .ssh]$ cat /home/oracle/.ssh/id_rsa.pub >> authorized_keys[oracle@prod1 .ssh]$ cat /home/oracle/.ssh/id_dsa.pub >> authorized_keys
ssh prod2 cat /home/oracle/.ssh/id_rsa.pub >> authorized_keys ( enter password:oracle when asked )
ssh prod2 cat /home/oracle/.ssh/id_dsa.pub >> authorized_keys ( same )
[oracle@prod1 .ssh]$ scp authorized_keys oracle@prod2:/home/oracle/.ssh/ ( enter password:oracle )
[oracle@prod1 .ssh]$ ( when u switch to prod2 from prod1 it should not ask for password )
[oracle@prod1 .ssh]$ exit ( back to prod1 )
=================================================================================
check with the below on both nodes
ssh prod1.technorack.com date
ssh prod1 date
ssh prod2.technorack.com date
ssh prod2 date
ssh prod1-priv.technorack.com date
ssh prod1-priv date
ssh prod2-priv.technorack.com date
ssh prod2-priv date
=========================
Time syncronization
1node can be ahead of 2nd server 5 mins.
to set the time with root use the below command
]#date -s "24 MAY 2010 17:15:50"
ssh prod1 date
ssh prod2 date
=====================================================
do this as root on both the nodes
Edit the /etc/sysconfig/rawdevices file, adding the following lines.
/dev/raw/raw1 /dev/sdb1/dev/raw/raw2 /dev/sdb2
Restart the rawdevices service using the following command.
]# service rawdevices restart
]# chkconfig rawdevices on
]# ln -s /dev/raw/raw1 /ocr
]# ln -s /dev/raw/raw2 /votingdisk
Run the following commands and add them the /etc/rc.local file.
chown oracle:oinstall /dev/raw/raw1chown oracle:oinstall /dev/raw/raw2chmod 600 /dev/raw/raw1chmod 600 /dev/raw/raw2fdisk
ln -s /dev/raw/
==============================================================
/etc/init.d/oracleasm configure ( on both nodes )
only required if asm is used
This will configure the on-boot properties of the Oracle ASM librarydriver. The following questions will determine whether the driver isloaded on boot and what permissions it will have. The current valueswill be shown in brackets ('[]'). Hitting without typing ananswer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface []: oracle
Default group to own the driver interface []: oinstall
Start Oracle ASM library driver on boot (y/n) [n]: y
Fix permissions of Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: [ OK ]
Loading module "oracleasm": [ OK ]
Mounting ASMlib driver filesystem: [ OK ]
Scanning system for ASM disks: [ OK ]
==================================================================================================================
/etc/init.d/oracleasm enable
[root@prod1 ~]# /etc/init.d/oracleasm listdisks
[root@prod1 ~]# /etc/init.d/oracleasm deletedisk vol1
[root@prod1 ~]# /etc/init.d/oracleasm deletedisk vol2
[root@prod1 ~]# /etc/init.d/oracleasm scandisks
[root@prod1 ~]# /etc/init.d/oracleasm createdisk vol1 /dev/sdb3Marking disk "/dev/sdb3" as an ASM disk: [ OK ]
[root@prod1 ~]# /etc/init.d/oracleasm createdisk vol2 /dev/sdb4Marking disk "/dev/sdb4" as an ASM disk: [ OK ]
run the below command on both the nodes
[root@prod1 ~]# /etc/init.d/oracleasm scandisksScanning system for ASM disks: [ OK ]
[root@prod1 ~]# /etc/init.d/oracleasm listdisksVOL1VOL2
======================================================================
configure netca
'/dev/oracleasm/disks/'
chown oracle:oinstall /dev/raw/raw1chown oracle:oinstall /dev/raw/raw2
chmod 600 /dev/raw/raw1chmod 600 /dev/raw/raw2
The shared disks are now configured.
======================================================
know issues crs is getting hanged at adding demons for 90seconds after running root.sh script,then workaround used is (with root)
chown oracle:oinstall /dev/raw/raw1chown oracle:oinstall /dev/raw/raw2chmod 600 /dev/raw/raw1chmod 600 /dev/raw/raw2
=======================================================
fdisk /dev/sdb
cluvfy stage –pre crsinst –n prod1,prod2 -verbose
Package existence check failed for "compat-gcc-c++-7.3-2.96.128".Check failed on nodes: prod2,prod1Package existence check failed for "compat-libstdc++-7.3-2.96.128".Check failed on nodes: prod2,prod1Package existence check failed for "compat-libstdc++-devel-7.3-2.96.128".Check failed on nodes: prod2,prod1Package existence check failed for "openmotif-2.2.3".
STEP1
==========================================================================
os (ORACLE ENTERPRISE LINUX 4)
required rpms
below is the example to install rpm
[root@inam1 ~]# rpm -ivh compat-gcc-32-3.2.3-47.3.src.rpm
warning: compat-gcc-32-3.2.3-47.3.src.rpm: V3 DSA signature: NOKEY, key ID db42a60e 1:compat-gcc-32 ########################################### [100%]
[root@inam1 ~]#
binutils-2.15.90.0.1.1-32.5 gcc-3.3.3-43.24
gcc-c++-3.3.3-43.24
glibc-2.3.3-98.28
gnome-libs-1.4.1.7-671.1
compat-libstdc++-296-2.96-132.7.2.i386
libstdc++-3.3.3-43.24
libstdc++-devel-3.3.3-43.24
make-3.80-184.1
pdksh-5.2.14-780.1
sysstat-5.0.1-35.1
xscreensaver-4.16-2.6
libaio-0.3.98
STEP2
=============================================================================================
network cards
Add the below enteries to /etc/hosts on both RAC NODES as root user
127.0.0.1 localhost.localdomain localhost
########PUBLIC IPS ##########################
192.168.3.1 prod1.technorack.com prod1
192.168.3.2 prod2.technorack.com prod2
192.168.1.10 nas.technorack.com nas
########VIRTUAL IPS ##########################
192.168.3.11 prod1-vip.technorack.com prod1-vip
192.168.3.12 prod2-vip.technorack.com prod2-vip
########PRIVATE IPS ##########################
10.1.1.1 prod1-priv.technorack.com prod1-priv
10.1.1.2 prod2-priv.technorack.com prod2-priv
STEP3
=============================================================================================
Add the following lines to the /etc/sysctl.conf file on both the nodes are root user
kernel.shmall = 2097152
kernel.shmmax = 536870912
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default=262144
net.core.wmem_default=262144
net.core.rmem_max=262144
net.core.wmem_max=262144
And run the below command to effect the prameters on fly
/sbin/sysctl -p
STEP4
=============================================================================================
Add the following lines to the /etc/security/limits.conf file on both the nodes as root user
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
STEP5
=============================================================================================
Add the following line to the /etc/pam.d/login file, if it does not already exist.
session required pam_limits.so
STEP6
=============================================================================================
Set the hangcheck kernel module parameters by adding the following line to the /etc/modprobe.conf file.on both the nodes.
options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180
To load the module immediately, execute "modprobe -v hangcheck-timer".
sTEP7
=============================================================================================(with root)
groupadd oinstallgroupadd dba
useradd -g oinstall -G dba -d /home/oracle oracle passwd oracle
sTEP8
=============================================================================================
(with root on both nodes)
chown -R oracle:oinstall /opt
(with oracle on both nodes)
mkdir -p /opt/app/oracle/product/10.2.0/crsmkdir -p /opt/app/oracle/product/10.2.0/asm mkdir -p /opt/app/oracle/product/10.2.0/dbmkdir -p /opt/app/oracle/oraInventory
sTEP9
=============================================================================================
set bash profile
[oracle@inam1 ~]$ cat .bash_profile
# .bash_profile
# Get the aliases and functionsif [ -f ~/.bashrc ]; then . ~/.bashrcfi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATHunset USERNAME
ORACLE_BASE=/opt/app/oracle; export ORACLE_BASEORACLE_HOME=$ORACLE_BASE/product/10.2.0/db; export ORACLE_HOMEORACLE_SID=prod2; export ORACLE_SIDCRS_HOME=$ORACLE_BASE/product/10.2.0/crs; export CRS_HOME ORACLE_TERM=xterm; export ORACLE_TERMPATH=/usr/sbin:$PATH; export PATHPATH=$ORACLE_HOME/bin:$CRS_HOME/bin:$PATH; export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATHCLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH
if [ $USER = "oracle" ]; then if [ $SHELL = "/bin/ksh" ]; then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fifi
sTEP10
==============================================================================================
configure user equivalence
run the below commands on both the nodes as oracle
[oracle@prod1 db]$ ssh-keygen -t rsa (overwrite:Y)
[oracle@prod1 db]$ ssh-keygen -t dsa (overwrite:Y)
Run the below command on only first node
[oracle@prod1 ~]$ cd .ssh
[oracle@prod1 .ssh]$ touch authorized_keys
[oracle@prod1 .ssh]$ cat /home/oracle/.ssh/id_rsa.pub >> authorized_keys[oracle@prod1 .ssh]$ cat /home/oracle/.ssh/id_dsa.pub >> authorized_keys
ssh prod2 cat /home/oracle/.ssh/id_rsa.pub >> authorized_keys ( enter password:oracle when asked )
ssh prod2 cat /home/oracle/.ssh/id_dsa.pub >> authorized_keys ( same )
[oracle@prod1 .ssh]$ scp authorized_keys oracle@prod2:/home/oracle/.ssh/ ( enter password:oracle )
[oracle@prod1 .ssh]$ ( when u switch to prod2 from prod1 it should not ask for password )
[oracle@prod1 .ssh]$ exit ( back to prod1 )
=================================================================================
check with the below on both nodes
ssh prod1.technorack.com date
ssh prod1 date
ssh prod2.technorack.com date
ssh prod2 date
ssh prod1-priv.technorack.com date
ssh prod1-priv date
ssh prod2-priv.technorack.com date
ssh prod2-priv date
=========================
Time syncronization
1node can be ahead of 2nd server 5 mins.
to set the time with root use the below command
]#date -s "24 MAY 2010 17:15:50"
ssh prod1 date
ssh prod2 date
=====================================================
do this as root on both the nodes
Edit the /etc/sysconfig/rawdevices file, adding the following lines.
/dev/raw/raw1 /dev/sdb1/dev/raw/raw2 /dev/sdb2
Restart the rawdevices service using the following command.
]# service rawdevices restart
]# chkconfig rawdevices on
]# ln -s /dev/raw/raw1 /ocr
]# ln -s /dev/raw/raw2 /votingdisk
Run the following commands and add them the /etc/rc.local file.
chown oracle:oinstall /dev/raw/raw1chown oracle:oinstall /dev/raw/raw2chmod 600 /dev/raw/raw1chmod 600 /dev/raw/raw2fdisk
ln -s /dev/raw/
==============================================================
/etc/init.d/oracleasm configure ( on both nodes )
only required if asm is used
This will configure the on-boot properties of the Oracle ASM librarydriver. The following questions will determine whether the driver isloaded on boot and what permissions it will have. The current valueswill be shown in brackets ('[]'). Hitting
Default user to own the driver interface []: oracle
Default group to own the driver interface []: oinstall
Start Oracle ASM library driver on boot (y/n) [n]: y
Fix permissions of Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: [ OK ]
Loading module "oracleasm": [ OK ]
Mounting ASMlib driver filesystem: [ OK ]
Scanning system for ASM disks: [ OK ]
==================================================================================================================
/etc/init.d/oracleasm enable
[root@prod1 ~]# /etc/init.d/oracleasm listdisks
[root@prod1 ~]# /etc/init.d/oracleasm deletedisk vol1
[root@prod1 ~]# /etc/init.d/oracleasm deletedisk vol2
[root@prod1 ~]# /etc/init.d/oracleasm scandisks
[root@prod1 ~]# /etc/init.d/oracleasm createdisk vol1 /dev/sdb3Marking disk "/dev/sdb3" as an ASM disk: [ OK ]
[root@prod1 ~]# /etc/init.d/oracleasm createdisk vol2 /dev/sdb4Marking disk "/dev/sdb4" as an ASM disk: [ OK ]
run the below command on both the nodes
[root@prod1 ~]# /etc/init.d/oracleasm scandisksScanning system for ASM disks: [ OK ]
[root@prod1 ~]# /etc/init.d/oracleasm listdisksVOL1VOL2
======================================================================
configure netca
'/dev/oracleasm/disks/'
chown oracle:oinstall /dev/raw/raw1chown oracle:oinstall /dev/raw/raw2
chmod 600 /dev/raw/raw1chmod 600 /dev/raw/raw2
The shared disks are now configured.
======================================================
know issues crs is getting hanged at adding demons for 90seconds after running root.sh script,then workaround used is (with root)
chown oracle:oinstall /dev/raw/raw1chown oracle:oinstall /dev/raw/raw2chmod 600 /dev/raw/raw1chmod 600 /dev/raw/raw2
=======================================================
fdisk /dev/sdb
cluvfy stage –pre crsinst –n prod1,prod2 -verbose
Package existence check failed for "compat-gcc-c++-7.3-2.96.128".Check failed on nodes: prod2,prod1Package existence check failed for "compat-libstdc++-7.3-2.96.128".Check failed on nodes: prod2,prod1Package existence check failed for "compat-libstdc++-devel-7.3-2.96.128".Check failed on nodes: prod2,prod1Package existence check failed for "openmotif-2.2.3".
How to bring db into archivelog
First check if it is enabled or disabled
sql > archive log list
1)Shut the database
sql> shut immediate
2)start in mount stage
sql>startup mount
3) Issue the alter command
sql> alter database archivelog
4)to check if it is enabled give
sql > archive log list
5)Bring it in open stage
sql> alter database open
6)to force a log switchto check if it is generating archives give
sql > alter system switch logfile
sql > archive log list
1)Shut the database
sql> shut immediate
2)start in mount stage
sql>startup mount
3) Issue the alter command
sql> alter database archivelog
4)to check if it is enabled give
sql > archive log list
5)Bring it in open stage
sql> alter database open
6)to force a log switchto check if it is generating archives give
sql > alter system switch logfile
Subscribe to:
Comments (Atom)