Basic Planning

 

◎ Equipment Selection

1. Servers: Two Dell R620 units. Each features an 8-core CPU, 64GB RAM, a 600GB 15,000 RPM SAS hard drive, dual power supplies, one HBA card, and one cable connecting the HBA card to the shared storage.

2. Storage: One Dell MD3200 unit. It has dual controllers and twelve 600GB 15,000 RPM SAS hard drives. To achieve maximum availability, RAID 10 is used.

3. Switches: Two Huawei/H3C switches, model H3C S5048E. Note: Gigabit Ethernet ports are preferred for all network connections.

4. Network Cables: Several 2-3 meter pre-manufactured Cat6 cables. — We previously encountered an issue where the provided cables were insufficient, so we asked the IDC facility for a self-made Cat6 cable. The poor quality resulted in intermittent heartbeat checks between RAC nodes.

5. Auxiliary Equipment: One VPN device and one KVM over IP device.

(1) A simple and cost-effective VPN solution: DI-8200 Internet Behavior Management Authentication Router. Priced at over 1,000 RMB, it is affordable and stable, making it ideal for configuring VPN access to log into the system and manage various servers.

(2) KVM over IP model ATEN CN8000. When remote login to the system is unavailable, you can call IDC technicians to connect this device to the faulty machine. You can then perform various operations via a browser, such as reinstalling the OS, viewing screen information, or pressing Ctrl+Alt+Del to restart. This provides the same effect as physically going to the server room to connect a monitor and keyboard. I guarantee that calling the IDC technicians to connect this device is faster than driving to the server room to handle the issue on-site.

 

◎ Network Planning

1. NIC Bonding: Two network cards are bonded together, utilizing all four network interfaces on the servers. This effectively utilizes bandwidth while enhancing availability.

2. IP Address Allocation: The application connection network address is 172.16.208.0/24, and the inter-node heartbeat network address is 192.168.208.0/24.

3. Network Connection and Management

(1) The heartbeat network and application network are physically separated, each connected to its own switch.

(2) The servers do not have public IP addresses; system login and management must be performed via VPN dial-up.

 

◎ Operating System and Required Tools

1. Server Operating System: CentOS 5.9 64-bit, custom installed with X Window System and GNOME selected. Since Oracle installation requires a graphical user interface, VNC Server is an excellent choice. Ensure VNC Server is installed before proceeding with the formal deployment. A customized system partition layout is as follows:

This table is wide — scroll sideways to see it all

[root@rac70 ~]# df -hFilesystem           Size Used Avail Use% Mounted on /dev/sda6             29G   16G   12G 60% / /dev/sda9             362G 215G 129G 63% /u01 /dev/sda8             15G 165M   14G   2% /tmp /dev/sda7             19G   15G 3.9G 79% /home /dev/sda5            29G 2.5G   25G 10% /usr /dev/sda3             29G 321M   27G   2% /var /dev/sda1             99M   19M   75M 21% /boot tmpfs                 32G   18G   15G 55% /dev/shm

2. Oracle Software:

(1) Cluster Management Software: linux.x64_11gR2_grid.zip, downloaded from the Oracle website (requires registration with a non-Chinese mainland email address).

(2) Database Software: linux.x64_11gR2_database, downloaded from the Oracle website (requires registration with a non-Chinese mainland email address).

(3) Storage Management Software: mdstoragemanager, obtained via supplier-provided optical media or downloaded from the Dell website.

(4) Oracle ASM-related software: oracleasm-2.6.18-308.el5-2.0.5-1.el5.x86_64.rpm, oracleasmlib-2.0.4-1.el5.x86_64.rpm, and oracleasm-support-2.1.7-1.el5.x86_64.rpm. These three packages must match the operating system kernel version exactly; otherwise, subsequent operations will fail.

 

◎ Partition Usage and File Systems

1. The Oracle database and cluster software are installed on the local disk’s /u01 partition.

(1) Cluster software path: /u01/app/grid

(2) Database software path: /u01/app/oracle/product/11.2.0

(3) The base paths for both the cluster software and database software are set to /u01/app/oracle. Since the installation path (ORACLE_HOME) cannot be the same directory as BASE, this configuration is somewhat confusing and awkward. Fortunately, this perplexing setup functions correctly. It is hoped that future projects can plan such configurations more standardly in advance. Opening an installed environment and navigating to the ORACLE_BASE directory reveals the following subdirectories:

This table is wide — scroll sideways to see it all

[root@db40 oracle]# lltotal 20 drwxr-x— 3 oracle oinstall 4096 Mar 2 23:52 admin drwxr-x— 6 oracle oinstall 4096 Mar 2 23:52 cfgtoollogs drwxr-xr-x 2 oracle oinstall 4096 Mar 2 23:20 checkpoints drwxrwxr-x 5 oracle oinstall 4096 Mar 2 23:52 diag drwxr-xr-x 3 oracle oinstall 4096 Mar 2 23:12 product

Navigating into the diag directory reveals three subdirectories:

This table is wide — scroll sideways to see it all

[root@db40 diag]# lltotal 12 drwxr-x— 3 oracle oinstall 4096 Mar 2 21:50 asm drwxr-x— 3 oracle oinstall 4096 Mar 2 23:52 rdbms drwxr-xr-x 3 oracle oinstall 4096 Mar 2 21:53 tnslsnr

As you can see, alert logs for the ASM file system, database, and listener can all be found here. For example, the ASM alert log is located at /u01/app/oracle/diag/asm/+asm/+ASM2/alert/log.xml.

2. Data storage utilizes Oracle’s own ASM (Automatic Storage Management), which is divided into three ASM disk groups: OCR, FLASH, and DATA (OCR occupies the least space, approximately 500MB; FLASH is next, allocated approximately 300GB; the remainder is allocated to DATA). Specifically:

◆ OCR stores cluster registry information

This table is wide — scroll sideways to see it all

ASMCMD> pwd+OCR/db1-scan ASMCMD> ls ASMPARAMETERFILE/ OCRFILE/

◆ FLASH stores archived logs and default RMAN backup sets

This table is wide — scroll sideways to see it all

ASMCMD> cd FLASHASMCMD> ls ZYZF/ ASMCMD> cd ZYZF ASMCMD> ls ARCHIVELOG/ BACKUPSET

◆ DATA storage holds Oracle database metadata and user data.

This table is wide — scroll sideways to see it all

ASMCMD> cd DATA ASMCMD> ls ZYZF/ ASMCMD> cd ZYZF ASMCMD> ls CONTROLFILE/ DATAFILE/ ONLINELOG/ PARAMETERFILE/ TEMPFILE/ control01.ctl control02.ctl ………Output omitted for brevity……………. redo07.log redo08.log spfilezyzf.ora

3. The Dell MD3200 storage management software uses the default installation path /opt/dell/mdstoragesoftware.

4. The ASM management software is installed via an RPM package. The installation files are scattered everywhere, as one might expect: for example, configuration files are located in /etc/sysconfig/oracleasm, device names are in the directory /dev/oracleasm/disks, and executable files are here /etc/init.d/oracleasm.

 

◎ Topology Diagram

 

Execution Steps

• Prepare the runtime environment;
• Prepare and configure shared storage;
• Prepare database storage space;
• Install cluster tools (Grid Infrastructure);
• Install and create the Oracle database.

Special Note: Since this document was written retrospectively to provide more detailed descriptions, screenshots may have been taken from different runtime environments. Therefore, if there are discrepancies between the IP addresses shown in the images and those described in the text, please do not question them excessively.

 

(I). Prepare the Runtime Environment

The runtime environment here primarily includes the remote desktop environment (I used VNC service), dependency tools required for software installation, creation of relevant accounts and directories, network connection settings, etc.

 

◎ Network Configuration (Required on each node)

◆ Modify the file /etc/modprobe.conf so that its content is as follows:

This table is wide — scroll sideways to see it all

alias scsi_hostadapter mpt2sas alias eth0 tg3 alias eth1 tg3 alias eth2 tg3 alias eth3 tg3 alias scsi_hostadapter1 megaraid_sas alias scsi_hostadapter2 ahci alias scsi_hostadapter3 usb-storage # Begin Dell MD Modification options scsi_dh_rdac blacklist=”DELL:MD3000,DELL:MD3000i” # End Dell MD Modification alias bond0 bonding options bond0 miimon=100 mode=0 alias bond1 bonding options bond1 miimon=100 mode=0

◆ Modify the network interface files so that their contents are as follows (for convenience of description, several files are presented together):

This table is wide — scroll sideways to see it all

[root@rac70 network-scripts]# more ifcfg-eth0 # Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet PCIe DEVICE=eth0 #HWADDR=E0:DB:55:20:E7:10 ONBOOT=yes TYPE=Ethernet USERCTL=no MASTER=bond0 SLAVE=yes TYPE=Ethernet   [root@rac70 network-scripts]# more ifcfg-eth1 # Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet PCIe DEVICE=eth1 BOOTPROTO=none #HWADDR=e0:db:55:20:e7:11 ONBOOT=yes TYPE=Ethernet USERCTL=no MASTER=bond0 SLAVE=yes TYPE=Ethernet   [root@rac70 network-scripts]# more ifcfg-eth2 # Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet PCIe DEVICE=eth2 BOOTPROTO=none #HWADDR=e0:db:55:20:e7:12 ONBOOT=yes TYPE=Ethernet USERCTL=no MASTER=bond1 SLAVE=yes TYPE=Ethernet   [root@rac70 network-scripts]# more ifcfg-eth3 # Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet PCIe DEVICE=eth3 #HWADDR=E0:DB:55:20:E7:13 ONBOOT=yes HOTPLUG=no BOOTPROTO=none TYPE=Ethernet MASTER=bond1 SLAVE=yes TYPE=Ethernet

◆ Create a new network configuration file with the following complete content:

This table is wide — scroll sideways to see it all

[root@rac70 network-scripts]# more ifcfg-bond0# Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet PCIe DEVICE=bond0 BOOTPROTO=none BROADCAST=172.16.208.255 IPADDR=172.16.208.70 NETMASK=255.255.255.0 NETWORK=172.16.208.0 ONBOOT=yes GATEWAY=172.16.208.201 TYPE=Ethernet   [root@rac70 network-scripts]# more ifcfg-bond1 # Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet PCIe DEVICE=bond1 BOOTPROTO=none ONBOOT=yes DHCP_HOSTNAME=oracle70 TYPE=Ethernet USERCTL=no IPADDR=192.168.208.70 NETMASK=255.255.255.0

◆ Append the following line to the /etc/rc.local file:

This table is wide — scroll sideways to see it all

modprobe bonding miimon=100 mode=0

After completing the configuration, restart the system and verify the correctness of the configuration and connectivity. The output for a correct configuration is shown in the figure below:

 

◎ Configure the VNC service (must be configured on each RAC node)

1. Ensure that the vncserver software is installed. To test, run the command `vncserver` from any directory. If the command is not found, install it by running `yum install vncserver`.

2. Execute `vncserver` to start the service. If this is the first time starting it, you will need to set a VNC password. It is recommended to set the same password as the system root password (please use a complex password!).

3. Modify the file `/root/.vnc/xstart` so that its content is:

This table is wide — scroll sideways to see it all

#!/bin/sh  # Uncomment the following two lines for normal desktop: unset SESSION_MANAGER exec /etc/X11/xinit/xinitrc   [ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup [ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources xsetroot -solid grey vncconfig -iconic & xterm -geometry 80×24+10+10 -ls -title “$VNCDESKTOP Desktop” & #twm & gnome start&

4. Restart the system, then start the vncserver. On your Windows machine, launch a VNC client and connect remotely to see if the remote server’s graphical desktop appears.

 

◎ Create the accounts, directories, dependent software, etc., required for installation. To save time and avoid errors, I have written these steps into a rough script that only needs to be executed once. The script content is as follows:

This table is wide — scroll sideways to see it all

[root@db40 oracle_rac]# more oracle_rac_rep.sh#!/bin/bash #install depending packages             # yum -y install binutils compat-libstdc++-33 elfutils-libelf elfutils-libelf-devel glibc glibc-common glibc-devel gcc gcc-c++ libaio-devel libaio libgcc libstdc++ libstdc++-devel make sysstat unixODBC unixODBC-devel pdksh numactl-devel glibc-headers   #add groups,user and create dir       # /usr/sbin/groupadd -g 501 oinstall /usr/sbin/groupadd -g 502 dba /usr/sbin/groupadd -g 504 asmadmin /usr/sbin/groupadd -g 506 asmdba /usr/sbin/groupadd -g 507 asmoper   useradd -u 1000 -g oinstall -G dba,asmdba oracle useradd -u 1006 -g oinstall -G asmadmin,asmdba,asmoper grid   mkdir /u01/app/ chown -R grid:oinstall /u01/app/ chmod -R 775 /u01/app/   mkdir -p /u01/app/oraInventory chown -R grid:oinstall /u01/app/oraInventory chmod -R 775 /u01/app/oraInventory   mkdir -p /u01/app/grid mkdir -p /u01/app/oracle   chown -R grid:oinstall /u01/app/grid chown -R oracle:oinstall /u01/app/oracle   chmod -R 775 /u01/app/oracle chmod -R 775 /u01/app/grid   #modify sysctl.conf                         # cat >> /etc/sysctl.conf <<done fs.file-max = 6815744 kernel.shmall = 2097152 #kernel.shmmax = 536870912 kernel.shmmni = 4096 kernel.sem = 250 32000 100 128 net.ipv4.ip_local_port_range = 9000 65500 net.core.rmem_default = 262144 net.core.rmem_max = 4194304 net.core.wmem_default = 262144 net.core.wmem_max = 1048576 fs.aio-max-nr = 1048576 done   sysctl -p   #modify /etc/security/limits.conf           # cat >> /etc/security/limits.conf << done grid soft nproc 2047 grid hard nproc 16384 grid soft nofile 1024 grid hard nofile 65536 oracle soft nproc 2047 oracle hard nproc 16384 oracle soft nofile 1024 oracle hard nofile 65536 done   #modify /etc/pam.d/login                       # echo “session   required     pam_limits.so”>>/etc/pam.d/login   # setting user oracle env                        # cat >> /home/oracle/.bash_profile <<done export ORACLE_SID=zyzf1 export ORACLE_UNQNAME=zyzf1 export ORACLE_base=/u01/app/oracle export ORACLE_HOME=/u01/app/oracle/product/11.2.0 export PATH=$ORACLE_HOME/bin:$PATH done   #setting user grid env                            # cat >> /home/grid/.bash_profile <<done export ORACLE_SID=+ASM1 export ORACLE_BASE=/u01/app/oracle export ORACLE_HOME=/u01/app/grid export PATH=$ORACLE_HOME/bin:$PATH done

 

◎ Set hostnames and IP mappings (perform on each node)

Modify the /etc/hosts file so that its content is:

This table is wide — scroll sideways to see it all

[root@db40 oracle_rac]# more /etc/hosts# Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 db40 localhost.localdomain localhost ::1 localhost6.localdomain6 localhost6   # Pubic 172.16.5.40 db40 172.16.5.50 db50 # Private 10.16.5.40 db40-priv 10.16.5.50 db50-priv # Virtual 172.16.5.41 db40-vip 172.16.5.51 db50-vip # SCAN 172.16.5.55 db1-scan   #dghost 172.16.5.60 datadg

 

(II). Prepare and configure shared storage

 

◆ Ensure physical connections between the server and storage devices are established: Verify this by checking device indicator lights. Under normal conditions, the HBA card indicators and the indicators at the storage device connection points should display green.

 

◆ Install PowerVault storage management software on the hosts. Note: This must be installed on every RAC node; otherwise, nodes without the installation will not be able to discover shared disks. The specific process is as follows:

◎ Connect to the server via remote VNC

◎ Navigate to the software directory and execute the command ./md_launcher_rhel_x86_64.bin

The installation window will pop up;

Click “Install MD Storage Software” with the mouse;

Simply accept the default settings;

Select SAS (I am using an MD3200, which matches perfectly), and the subsequent installation steps are straightforward, so they will not be detailed here. After installation, an icon will appear on the desktop. Clicking the icon allows you to manage the storage.

If you encounter the following error, please modify the /etc/hosts file to resolve it.

◎ Configure shared storage (execute on only one node)

1. Double-click the icon to launch the storage management tool.

2. Select Manual, then select In-band Management.

3. Enter the IP address of the host connecting to the disk array here.

4. Create a disk group, select RAID level as RAID10, and select all 12 disks. This process takes a considerable amount of time.

5. Map the virtual disk to the host. The purpose is to ensure that any host connected to the array can access the virtual disk group.

6. Enter an easily identifiable name.

7. Right-click the newly created host group name to bring up the host definition.

8. Enter the hostname (based on /etc/hosts).

9. Select the connection port. Be careful here; if incorrect, the host will not recognize the disk group.

10. Click the “Add” button with the mouse.

The interface after defining the first host looks like this:

Use the same method to define the remaining hosts.

11. Define attachment mapping, i.e., allocate accessible disk space to the host.

12. Click Attach Image.

13. Select the virtual disk (created earlier during RAID creation), then click Add with the mouse.

◎ Check whether the RAC node servers recognize the virtual disks created in the above steps (check on each node to ensure correctness). Execute `fdisk -l` on the host.

If they are not recognized, simply restart the system.

◎ Host disk partitioning (only needs to be performed on one host; check partition status on other hosts afterwards)

Since the capacity is greater than 2TB, partitioning must be done using parted. According to the plan, we need to create 6 partitions for storing OCR, FLASH AREA, and data files respectively.

◆ Before partitioning, use `multipath -ll` to view the names of the disk device files to ensure partitioning is performed on the correct devices.

◆ Enter the /dev/mapper directory to check if files named mpath* exist.

<img class="attachment-medium" src="https://www.yunwei123.com/wp-content/uploads/2016/03/10037_1409244388pObp.png" alt="10037_1409244388pObp" width="379" height="81"

I created a total of six Oracle ASM disks. The benefit of this approach is that it distributes data as evenly as possible across different disk areas. Subsequently, two ASM disks are merged into a single disk group to enhance availability and expand storage capacity.

 

If any errors occur during the execution process, you can troubleshoot by examining the log file `/var/log/oracleasm`. After completion, verify the setup using the following methods. During one installation, I encountered an issue where one node lacked the `/dev/oracleasm` directory. The log at `/var/log/oracleasm` showed: `oracleasm-read-label: Unable to open device “/dev/sdb1”: No such file or directory`. After considerable troubleshooting, I discovered the root cause was that `/etc/init.d/oracleasm enable` had not been executed.

 

◎ Verify whether the disks created by oracleasm were successful.

• Check the system directory to ensure at least six files exist.

(2) View the ASM disk groups logically. Remember to query each RAC host individually.

This table is wide — scroll sideways to see it all

[root@db50 ~]# /etc/init.d/oracleasm scandisksScanning the system for Oracle ASMLib disks: [ OK ] [root@db50 ~]# /etc/init.d/oracleasm listdisks DATA_VOL1 DATA_VOL2 FLASH_VOL1 FLASH_VOL2 OCR_VOL1 OCR_VOL2

 

 

(IV) Install Cluster Software (Grid Infrastructure)

 

◎ Grid Software Installation (The installation process only needs to be initiated on one node. However, some scripts must be run on every node simultaneously. Please follow the installation prompts carefully).

 

Connect to the server via VNC, open a terminal, execute `xhost +`, then navigate to the source installation directory and run the command `./runInstaller`. If successful, the installation interface shown below will appear:

Select Typical Installation.

The SCAN name must match exactly with `/etc/hosts`. Special attention: Ensure this name is unique within the same subnet. If deploying multiple Oracle RAC clusters in the same subnet, duplicate SCAN names will prevent the installation from proceeding!!!

Add the remaining hosts (by default, only one host is displayed initially).

Ensure you enter the details exactly as defined in `/etc/hosts`.

After adding them, the hostnames should appear in the list.

Verify SSH connectivity by clicking the “SSH Connectivity…” button. It is crucial to configure passwordless SSH authentication here; otherwise, the next step cannot proceed (I spent a significant amount of time resolving this issue).

Note: You do not necessarily need to set up SSH manually. You can simply click the “Setup” button to handle it automatically, which saves considerable effort.

Select the installation directories and storage locations (note that the base locations must be different).

Create ASM disk groups. Since the disk array is configured with RAID 10, select “External” redundancy level below.

These names look familiar, right? They correspond to the disks created earlier using `oracleasm createdisk`. Here, check two items to merge them into a single ASM disk group. If no candidate disks are found here, click the “Change Discovery Path” button to perform a manual search.

It is best to enter the full path. Fortunately, wildcards are supported here. The disk group created here is used to store ASM voting disks and other information.

Proceeding further with validation, an error message regarding NTP suddenly appeared. To resolve this, stop the `ntpd` service and delete the configuration file `/etc/ntp.conf`. This resolved the issue.

At this stage, issues are rare. Simply click the “Finish” button to extract files and proceed with the installation. At a certain point, you will need to run two scripts as the root user: `/u01/app/oraInventory/orainstRoot.sh` and `/u01/app/grid/root.sh`. The latter script takes a long time to complete and is prone to errors.

 

◆ The first server executed successfully on the first attempt, but an error occurred when executing on the second server:

This table is wide — scroll sideways to see it all

ASM failed to start. Check /u01/app/oracle/cfgtoollogs/asmca/asmca-14030210PM2304.log for details.  Configuration of ASM failed, see logs for details Did not succssfully configure and start ASM CRS-2500: Cannot stop resource ‘ora.crsd’ as it is not running CRS-4000: Command Stop failed, or completed with errors. Command return code of 1 (256) from command: /u01/app/grid/bin/crsctl stop resource ora.crsd -init Stop of resource “ora.crsd -init” failed

A comparison check revealed inconsistencies in the directory ownership between the two servers. On the server where the script ran successfully, the directories under `/dev/oracleasm/` were owned by `grid:oinstall`, whereas on the problematic host, they were owned by `root:root`. To resolve this, execute `/etc/init.d/oracleasm` to set the correct owner and group, ensuring file ownership is accurate. Then, run `./roothas.pl -delete -force -verbose` to remove the failed configuration, followed by executing `/u01/app/grid/root.sh` to complete the installation. Note: The two failures shown in the installation interface are due to the private IP being configured as public; these can be ignored.

 

◎ Verifying Installation Correctness

 

• Check ASM processes.
• Log in as the `grid` user and execute `asmcmd`. At the prompt, use `ls` to view directories or files. If content is present, it indicates the expected state.

• Inspect the installation directory to verify that the relevant directories and files have been generated.

 

(5) Oracle Installation and Database Creation

 

◎ Installing Database Software (Perform operations on one node. During installation, some scripts must be executed on every node simultaneously. Please follow the installation prompts carefully.)

 

Similar to the Grid Infrastructure installation, connect to a node’s system via VNC, execute `xhost +`, and then open a new terminal. Switch to the `oracle` user by running `su – oracle`. Navigate to the directory containing the installation source files and execute the `./runInstaller` script. If successful, the following installation interface will appear:

First, install the software; create the database separately later.

As with the Grid installation, SSH equivalence verification is required first.

After successful verification, subsequent steps are less prone to errors and are largely similar to a single-instance installation.

 

◎ Creating the Database (Perform operations on one node)

 

Execute `dbca` as the `oracle` user.

After several “Next” steps, we choose the Custom Database option.

Ensure all nodes are selected.

Create the required ASM disk groups (one group was already created during the Grid installation). Execute `asmca` as the `grid` user.

Manually input and select specific items.

It takes a bit of patience while it processes. Hey! It appeared just as I finished typing this.

Once created, modifications are still possible. However, once the database goes live, such changes cannot be made easily. Below is the summary output after creating the current environment.

Select the location for storing database files. Obviously, choose ASM. If you do not perform this disk group creation process, you will not be able to proceed if you select ASM for the database file storage location later.

Select the one with the largest capacity (OCR has already been occupied by previous operations, and FLASH is reserved for later use).

When proceeding to the next step, you need to enter an ASM password. This is the password set during the Grid installation two days ago; remember it well.

The archive area also uses ASM, utilizing the remaining ASM disk group FLASH left over from earlier. Nothing is wasted.

Do it yourself, choose the larger option.

Make sure to check the box for archiving (of course, it can be enabled after the instance starts running).

No more screenshots for the next few steps. From here, just wait patiently.

 

The deployment process is long and tests your patience.

 

Subsequent Steps and Precautions

 

◆ After creating the Oracle RAC database, some default system tablespaces are too small and will reach over 90% usage quickly. It is best to increase their size immediately after creation.

◆ The Oracle RAC listener listens on virtual IPs. Attempting to connect to the actual physical IP addresses will never succeed.

◆ Oracle RAC heartbeat checks generate significant data communication, requiring high network quality and speed. In one running Oracle RAC cluster, a missing network cable led to the use of inferior Category 6 cables provided by the data center. This caused intermittent heartbeat detection issues, resulting in instance shutdowns.

◆ For user tablespaces with fixed sizes, it is better to expand capacity by creating multiple data files.

◆ The default redo log files are also very small. You need to create additional log groups and increase their capacity.

◆ Once the Oracle RAC is running, ensure robust backups are in place. Proficiency in using RMAN will bring many benefits.

◆ If conditions permit, set up Data Guard across different data centers.

◆ Improper multipath configuration may cause data paths to alternate switching back and forth. The direct manifestation is slow data access. Useful information can be obtained by querying the alert log.

◆ Alert log files are extremely useful; monitor them frequently!

◆ Modify `/etc/sysconfig/ntp` and add the line `OPTIONS=”-x -u ntp:ntp -p /var/run/ntpd.pid”`, then restart the ntpd service.

◆ SSH equivalence must be bidirectional, meaning passwordless login must work from any server to any other server.

◆ Remove the hostname from the first line of `/etc/hosts`; additionally, the name in `/etc/sysconfig/network` must match `/etc/hosts`.

◆ When configuring Oracleasm, pay close attention to the owner and group names. Otherwise, permission issues may arise. Use `Oracleasm configure -i` to specify the interface as `grid` and the group as `oinstall`. This is because the `oracle` user also belongs to the `oinstall` group.

 

Writing these notes took considerable time. I hope they are helpful to those who need them. There are some documents online, but most are based on virtual environments, failing to capture the characteristics of ASM in real-world scenarios.