Configuring Automatic Storage Management on Redhat Linux

Redundancy
Once the decision is made to use ASM, the next step is to decide how the disk drives should be mirrored. ASM calls this the redundancy plan. ASM provides three levels of redundancy.
1. High
2. Normal
3. External

High
With high redundancy, ASM keeps three copies of every file. It provides the highest level of protection, but it also requires the most storage. For example, if you determine that you need 1TB of usable storage, you will need to purchase 3TB of raw storage.

High redundancy might be used in situations where disk failures are common or if the failures cause a significant impact to business.

Normal
With normal redundancy, ASM keeps 2 copies of every file. This is essentially RAID 10. Critical files, such as control files, are always high redundancy.
You would use normal redundancy when you want to use RAID 10, but you want ASM to handle both the striping and the mirroring. It is especially useful in a non-SAN environment.

External
With external redundancy, ASM assumes the storage is already mirrored. Typically the drives are mirrored on a SAN and then striped within ASM.
The primary difference between normal and external redundancy is how failures are handled. If the disk drives are mirrored on the SAN, drive replacements are transparent to ASM. If the disk drives are mirrored with ASM, the administrators will need to remove the drive from ASM, replace it, and then add it back to ASM.
Recommendations
1. If the system is on a SAN, mirror the disk drives on the SAN and use external redundancy in ASM. SANs tend to have powerful fault tolerance routines, so it is easy to swap out bad drives.
2. If the system is not on a SAN, mount non-mirrored drives on the server and mirror within ASM. ASM calls this normal redundancy.
3. Always try to configure a RECOVER disk group in production.
4. Disk groups should always consist of disk drives that the same size and type.
5. Never carve up a disk and present a piece to ASM. Always present the entire disk drive or nothing at all.

Oracle Managed Files
Oracle managed files are not a requirement with ASM, nor is ASM a requirement with OMF, but they complement each other very well. Consequently, when using ASM, it is generally assumed that OMF is also being used.
OMF is configured by setting a handful of init.ora parameters. Once these parameters are set, OMF will take control of file placement. The database administrator still has the option to place files somewhere else, but if the file-name is omitted, OMF will choose where to place the file.

Here are the parameters that control OMF:
* db_create_file_dest - location for primary data files
* db_create_online_log_dest_1 - location for redo logs
* db_create_online_log_dest_2 - location for mirrored redo logs
* db_recovery_file_dest - location for flash_recovery_area

This section outlines the steps required to prepare storage for ASM.
The initial steps are specific to each operating system. Once the storage is prepared, however, the management within ASM is the same, regardless of the platform.

Oracle provides a library, called ASMLIB, for handling disk drives on Linux.
The steps to configure disk drives for use on Linux are:
1. Format the disk drives if they are not already formatted.
2. Determine the correct kernel
3. Download ASMLIB
4. Install ASMLIB
5. Configure devices with ASMLIB
The steps for configuring ASMLIB are described in the Metalink document, 394953.1.

Format Disk Drives
Disk drives need to be formatted before they can be used. Linux uses the fdisk command to format drives.
Here is an example format of the /dev/sdb disk drive.
% fdisk /dev/sdb
The number of cylinders for this disk is set to 121534.
There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with:
1. Software that runs at boot time (e.g., old versions of LILO).
2. Booting and partitioning software from other OSs (e.g. DOS FDISK, OS/2 FDISK).
Command (m for help): p
Disk /dev/sdb: 999.6 GB, 999653638144 bytes
255 heads, 63 sectors/track, 121534 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System

Command (m for help): n
Command action
E extended
P primary partition (1-4)
P
Partition number (1=-4): 1
First cylinder (1-121534, default 1): 1
Last cylinder or +size or +sizeM or +sizeK (1-121534, default 121534): 121534
Command (m for help): p
Disk /dev/sdb: 999.6 GB, 999653638144 bytes
255 heads, 63 sectors/track, 121534 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 1 121534 976221823 83 Linux
Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Determine the Correct Kernel

Run "uname -r" to determine the kernel version of the server. There is a separate ASMLIB distribution for each kernel version. It is important to load the correct version, otherwise ASMLIB will not work.
For our system, the kernel version is 2.6.18-8.el5PAE. We will need the el5PAE ASMLIB distribution.

Download ASMLIB

The ASMLIB files can be located on metalink by following the instructions in document 394953.1.

Install ASMLIB

1. Download the RPMs from Metalink :
%u2022 oracleasm-2.6.18-8.el5PAE-2.0.4-1.el5.i686.rpm
%u2022 oracleasmlib-2.0.3-1.el5.i386.rpm
%u2022 oracleasm-support-2.0.4-1.el5.i386.rpm
2. Install the RPMs:
%u2022 rpm -Uvh \
%u2022 oracleasm-2.6.18-8.el5PAE-2.0.4-1.el5.i686.rpm \
%u2022 oracleasmlib-2.0.3-1.el5.i386.rpm \
%u2022 oracleasm-support-2.0.4-1.el5.i386.rpm
3. Configure the ASM library driver:
%u2022 /etc/init.d/oracleasm configure
%uF02D Default user: oracle
%uF02D Default group: dba
%uF02D Start Oracle ASM library driver on boot: y
%uF02D Fix permissions of Oracle ASM disks on boot (y/n): y
4. Mark the disks as ASM disks:
%u2022 /etc/init.d/oracleasm createdisk DISK1 /dev/sdb1
%u2022 /etc/init.d/oracleasm createdisk DISK2 /dev/sdc1
5. Verify that ASMLIB has marked the disks:
%u2022 # /etc/init.d/oracleasm listdisks
%u2022 DISK1
%u2022 DISK2

6. Configure ASM
%u2022 Dbca
%uF02D Choose "Configure Automatic Storage Management"
%uF02D It will require you to run "localconfig add" as root
%u2022 change discovery path to /dev/oracleasm/disks/*
%u2022 - create a disk group
http://www.oracle.com/technology/pub/articles/smiley_10gdb_install.html#asm

Add Disks To Disk Groups

The steps to add disks to a disk group are the same for every operating system.
Disk groups are created using the Database Configuration Assistant, DBCA. It is also possible to configure disk groups using SQL*Plus, but the DBCA is recommended.
1. The first step is to start the DBCA. The DBCA is located on the start menu in Windows. On UNIX, make sure X-Windows is working and type "dbca".
2. The first screen will be displayed. Click on "Next".

(screenshot will go here soon!)

3. The DBCA will ask what you want to configure. Select "Configure Automatic Storage Management" and click on "Next".
If this is the first time to configure ASM, it will ask some basic configuration questions. It will configure an ASM instance, and start the instance. The defaults should be accepted for all of the parameters. The only parameter that requires input is the SYS password. Once the instance is configured and started, you can proceed to add disk drives.
It may also ask you to run "localconfig add" as root. This is necessary to start up ASM when the system reboots.
(screenshot will go here soon!)

Creating Disk Groups
1. When you create a disk group, it will ask for the redundancy. Choose High, Normal, or External.
Note: It is critical that you choose a proper redundancy level. If the disks are mirrored outside ASM, choose external. Do not choose external if the disk drives are not mirrored by Linux or by the SAN. If you do, you will lose your database if there is a hard disk failure. If the disk drives are not mirrored by the operating system or by the SAN, choose normal redundancy.
2. Enter a Disk Group name, and disk drives. The steps are not listed in this section, since they are essentially the same as adding disk drives.
The best practice is to have 2 disk groups. RMAN disk backups should not be sent to ASM drives.
%u2022 DATA - primary storage
%u2022 RECOVER - archived and mirrored redo logs, but no backups
%u2022 /mnt/hostname/SID - RMAN disk backups
3. If there are not enough disk drives to have 2 disk groups, create a DATA disk group and keep the archived redo logs on the file system.
%u2022 DATA - primary storage
%u2022 /app/oracle/flash_recovery_area - archived and mirrored redo logs, but no backups
%u2022 /mnt/hostname/SID - RMAN disk backups
4. If the primary storage consists of more than one type of disk drive, create a DATA disk group for each one.
%u2022 DATA - primary storage
%u2022 DATA2 - tier 2 storage, etc.
%u2022 RECOVER - archived and mirrored redo logs, but no backups
%u2022 /mnt/hostname/SID - RMAN disk backups

Adding to Disk Groups
This section describes the process for adding disks to an existing disk group.
1. The first step is to start up the DBCA. Once the DBCA comes up, click on "Next".
2. Select "Configure Automatic Storage Management", and click on "Next". The ASM configuration screen will be displayed.
3. Select the disk group that you want to expand and click on "Add Disks".

(screenshot will go here soon!)

If your disk drives have been properly identified to ASM, you should see candidate disks. In this example, we have one candidate disk. We are going to add it to the DATA disk group.
4. Select the disk(s) you want to add. If you are adding more than one disk, it is better to add them all at the same time. Each time you add a disk, ASM must rebalance the data. If you add four disks, it will perform one rebalance operation. If you add them one at a time, it will execute the rebalance operation four different times.

5. Click on "OK" once you have chosen the disks to add. It will pause for a few seconds while it adds the disks to the disk group.

(screenshot will go here soon!)

Once it is finished, it will display the main ASM configuration page.
At this point, the new disk drive has been added to the disk group. It will continue rebalancing the data for several hours. The progress can be monitored by the v$asm_operation view in the +ASM instance. It can also be monitored in OEM.
Note: I have not seen a significant impact to rebalancing, but if there are concerns that it is impacting the database, it can be turned off. If it is not completing fast enough, it can be sped up.
alter diskgroup data rebalance power 0-11;
%u2022 Rebalance power of zero turns off rebalancing.
%u2022 Rebalance power of 1 is the default.
%u2022 Increasing the rebalance power makes it rebalance faster.

(screenshot will go here soon!)