Adventures in ASM

by Jared Still

Oracle 10g has ventured into somewhat new territory, that of storage management. If you remember “raw iron,” you will understand why I say this is “somewhat” new territory, (but that’s another story).

With ASM, Oracle intends to move further toward its goal of bringing as much of the server housekeeping into the database as possible. This is to allow you to control as much of the server environment as possible from Oracle’s Enterprise Management console, or if you wish, the command line.

Oracle’s claims for ASM include the following:
Automatic load balancing

Software data redundancy

Software RAID 1 with double or triple mirrors
Elimination of fragmentation

This seems like a bit of a stretch, as fragmentation may also be eliminated with the use of Locally Managed Tablespaces, a feature introduced in Oracle 8i.

Elimination of file management

When an ASM diskgroup is created with multiple disks, it is essentially a variation on RAID 0, or striping. The difference is that traditional RAID 0 creates a single volume that is already striped, whereas Oracle datafiles created on an ASM diskgroup will be striped across the disk at the time of file creation. In addition to automated load balancing, ASM allows the addition of new disks into a disk, with no intervention by the DBA to redistribute datafiles across the disk — ASM does it automatically.

ASM’s main purpose in life seems to be to manage disk resources for RAC, and, looking toward the future, RAC in a grid environment, but it will work in single instance implementations as well.

This article will show you how you can set up ASM and learn how to use it. Even if all you have is a small Linux box to work with, you can still set up ASM and learn how to use it. ASM will also work on Windows, of course, but this article will not cover this in any depth. Finally, not all of the aforementioned features will not be tested at this time; it will be enough for now to get it configured and working.

Flavors of Oracle Storage Management

ASM on Linux can be used with different types of I/O. Previously, the choices have been to use raw or traditional management on a file system; now, there are two additional methods to manage Oracle Storage:

ASM with UNIX I/O: Tablespaces are created on raw character devices managed by ASM using standard Linux I/O routines.
ASM with ASMLib I/O: Tablespaces are created on raw block devices managed by ASM using ASMLib.

Configuring ASM

First, a word about disk partitioning. It will be assumed that you already know how to partition disks on your system, or have someone that can do so for you. That is a bit beyond the scope of this article, and could be a rather lengthy article in itself. Also, keep in mind that any configuration of raw disks and the ASMLib need to be done while logged in as root.

ASM with UNIX I/O

Let’s start by creating ASM disks using standard UNIX I/O. If you are familiar with some of the commercial UNIX variants, you may be surprised to learn that Linux does not use raw devices by default.

A Linux raw device must be bound to the corresponding block device by the raw driver. This is covered in MetaLink note 249992.1.

Here are the partitions being used in this test system, and the raw devices to which they will be bound:

/dev/hda3	/dev/raw/raw3
/dev/hda5	/dev/raw/raw5
/dev/hda6	/dev/raw/raw6

Two disk groups, DGRAID and DGFAIL, will be created. DGRAID is a volume already created as RAID (at least we are pretending that is so in our test; it is actually a single partition on a disk), while DGFAIL will be created from two partitions that will be software mirrored by Oracle. Many more disks could be used, but this is sufficient to get started with ASM.

Edit the file /etc/sysconfig/rawdevices, and add the block and raw devices names.

# raw device bindings
# format:  <rawdev> <major> <minor>
#          <rawdev> <blockdev>
# example: /dev/raw/raw1 /dev/sda1
#          /dev/raw/raw2 8 5
/dev/raw/raw3 /dev/hda3
/dev/raw/raw5 /dev/hda5
/dev/raw/raw6 /dev/hda6

As long as you have configured the /etc/sysconfig/rawdevices file, the raw device bindings will be recreated upon reboot.

The location and format of this file may vary between different brands of Linux. On Suse, for instance, the file is /etc/raw and the format is a bit different:

# raw<n>:<blockdev> 
raw3:hda3

The ownership of the new raw devices will now need to be transferred to the Oracle user, and Oracle dba group, which is usually dba.

chown oracle:dba /dev/raw/raw3; chmod 660 /dev/raw/raw3
chown oracle:dba /dev/raw/raw5; chmod 660 /dev/raw/raw5
chown oracle:dba /dev/raw/raw6; chmod 660 /dev/raw/raw6

Oracle recommends permissions of 660, but for you, 640 or 600 may be more appropriate.

The raw devices may now be bound via the service command; alternatively, you must use the raw command on Suse. Either of these works with RedHat:

service rawdevices restart

Another method is to use the raw command directly:

raw /dev/raw/raw3 /dev/hda3
raw /dev/raw/raw5 /dev/hda5
raw /dev/raw/raw6 /dev/hda6

Create the ASM Instance and Diskgroups

First create the password file:

orapwd file=$ORACLE_HOME/dbs/orapw+ASM password=sys

The following parameters were put into used to create the +ASM instance:

*.background_dump_dest='/u01/app/oracle/admin/asm/bdump'
*.core_dump_dest='/u01/app/oracle/admin/asm/cdump'
*.instance_type='asm'
*.large_pool_size=12M
*.remote_login_passwordfile='SHARED'
*.user_dump_dest='/u01/app/oracle/admin/asm/udump'

The ASM instance was then started, and the disk string and disk group parameters modified, and an SPFILE created. Note that the commands are issued to the ASM instance that is started, but that has no database files.

alter system set asm_diskstring = '/dev/raw/raw3’,'dev/raw/raw5’,’/dev/raw/raw6’ ;
alter system set asm_diskgroups = 'DGRAID',’DGFAIL’;
create spfile from pfile;

At this point, the diskgroups may be created:

create diskgroup dgfail normal redundancy
failgroup ctl1 disk '/dev/raw/raw5'
failgroup ctl2 disk '/dev/raw/raw6';

create diskgroup dgraid external redundancy
disk '/dev/raw/raw3';

When this is completed, you are ready to create tablespaces for any and all 10g database instances you have on that machine, but be sure to first logon to the database instance. You cannot create tablespaces in an ASM instance. The syntax is nearly the same as for standard file I/O:

create tablespace iotest datafile
'+DGRAID/iotest.dbf' size 500m reuse
extent management local uniform size 256k
segment space management auto
/

create tablespace iosource datafile
'+DGRAID/iosource_1.dbf' size 500m reuse
extent management local uniform size 256k
segment space management auto
/

The filename specification is the name of the diskgroup you wish to use, preceded by a “+,” and followed by a normal filename specification. Refer to the SQL reference manual for further details on file naming with ASM.

ASM with ASMLib I/O

Metalink Note 275315.1 provides information on configuring ASMLib I/O. The document states the following regarding ASMLib:

“This library is provided to enable ASM I/O to Linux disks without the limitations of the standard UNIX I/O API.”

Some preliminary testing (admittedly, on a single machine so far) indicates that the of ASMLib I/O may cost a significant I/O penalty. You will need to test this on your own system to determine if this is so for you.

Creating ASM disks with ASMLib is significantly different from doing so with standard Linux I/O. Raw devices are not required, as ASMLib works with block devices.

Installing ASMLib

The instructions for downloading and installing ASMLib are found in Metalink note 275315.1. In a nutshell, you need to download the correct version of ASMLib for your Linux kernel, install it, configure the disks with the oracleasm utility, add them to your ASM instance, and then create tablespaces. The URL for downloading the library is in the Metalink note, and at the end of this article.

It is important that you download a version of ASMLib that has been compiled for your specific Linux kernel. If the versions do not match, you will not be able to install it.

The following command was used to install the ASMLib on the test server:

rpm -Uvh --force \
oracleasm-support-1.0.3-1.i386.rpm \
oracleasmlib-1.0.0-1.i386.rpm \
oracleasm-2.4.21-EL-1.0.3-1.i686.rpm

Now run the configuration utility is run with the oracleasm utility:

[root@sherlock root]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current value
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface []: yes
Default group to own the driver interface []:
[root@sherlock root]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current value
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Fix permissions of Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration [ OK ]
Creating /dev/oracleasm mount point [ OK ]
Loading module "oracleasm" [ OK ]
Mounting ASMlib driver filesystem [ OK ]
Scanning system for ASM disks [ OK ]
[root@sherlock root]#

And now create the ASM disks. The partitions to be used are the same as used earlier to create ASM disks with Linux I/O.

[root]# /etc/init.d/oracleasm createdisk DGRAID /dev/hda3
  Marking disk "/dev/hda3’ as an ASM disk                    [OK]

[root]# /etc/init.d/oracleasm createdisk DGFAIL1 /dev/hda5
  Marking disk "/dev/hda5" as an ASM disk                    [OK]

[root]# /etc/init.d/oracleasm createdisk DGFAIL2 /dev/hda6
  Marking disk "/dev/hda6" as an ASM disk                    [OK]

There’s a chance that you may see an error message such as this:

[root]# /etc/init.d/oracleasm createdisk DGFAIL1 /dev/hda5
 Marking disk "/dev/hda5" as an ASM disk                    [FAILED]

It is very possible that ASM is reading header information from the disk that marks it as already belonging to an ASM disk group. This can happen if the disk or partition was previously used in ASM with Linux I/O, as happened on my test server.

The solution to this problem (provided by Martin Berg) was to write over the first 64k of the disk, erasing the old ASM header.

Note: This is a very dangerous command! Do not use this unless you are 100 percent sure you are writing to the correct partition, and that you have backups!

I do not recommend that you use this command, but it did work for the test server on which ASM was installed.

dd if=/dev/zero of=/dev/hda5 bs=1024 count=64

This was done for all partitions to be used by ASM, after which they could be configured without error.

The Metalink note referred to earlier provides further details on oracleasm.

Create the ASM Instance and Diskgroups

You can create the instance as detailed in the earlier section, or the same instance may be used.

As before, two disk groups, DGRAID and DGFAIL, will be produced. The syntax for creating these will be slightly different than what was shown previously.

alter system set asm_diskstring = 
'ORCL:DGRAID','ORCL:DGFAIL1','ORCL:DGFAIL2' scope=both;

alter system set asm_diskgroups = 'DGRAID','DGFAIL' scope=both;

At this point the diskgroups may be created:

drop diskgroup dgfail including contents;
drop diskgroup dgraid including contents;
create diskgroup dgfail normal redundancy
failgroup ctl1 disk '/dev/raw/raw5'
failgroup ctl2 disk '/dev/raw/raw6';

create diskgroup dgraid external redundancy
disk '/dev/raw/raw3';

When this is completed, you are ready to create tablespaces. The syntax is identical to that syntax used to create tablespaces previously with UNIX I/O ASM. Be sure to logon to the database instance to do this.

create tablespace iotest datafile
'+DGRAID/iotest.dbf' size 500m reuse
extent management local uniform size 256k
segment space management auto
/

create tablespace iosource datafile
'+DGRAID/iosource_1.dbf' size 500m reuse
extent management local uniform size 256k
segment space management auto
/

ASM with Files

It is possible to create an ASM instance and ASM disks and diskgroups, even though you don’t have any available disks or partitions to work with. This requires setting an undocumented parameter, and as such, is not supported by Oracle. You should only use it on a database used for learning to configure ASM — say, on your laptop.

Set the following initialization parameter in the ASM instance to allow ASM to use something other than a raw disk:

alter system set _asm_allow_only_raw_disks = FALSE;

Following are the steps I used to create an ASM instance and diskgroups, using text files on a Windows 2000 laptop. An Oracle 10g database instance had already been created.

Setup the init.ora file C:\oracle\product\10.1\ts30\database\init_asm10g.ora for the ASM instance:

instance_type = ASM
_asm_allow_only_raw_disks=FALSE
remote_login_passwordfile = EXCLUSIVE
asm_diskgroups = dgrp1
asm_diskstring = 'C:\ORACLE\ORADATA\TS30\ASM\DGRP*'

Create the password file:

orapwd file=C:\oracle\product\10.1\ts30\database\PWDasm10g.ora force=y

Create the ASM instance:

oradim -NEW -ASMSID asm10g -PFILE
C:\oracle\product\10.1\ts30\database\init_asm10g.ora -SPFILE -SYSPWD sys -
STARTMODE manual -SHUTMODE immediate

Create the files used as ASM disks:

perl dc.pl

And here is the Perl script used:

my $s = '0' x 2**20;

open(DC1,">C:/oracle/oradata/ts30/asm/dgrp1a")  || die "cannot create file 
- $!\n";
open(DC2,">C:/oracle/oradata/ts30/asm/dgrp1b")  || die "cannot create file 
- $!\n";

for (my $i=1; $i<=1024; $i++) {
        print DC1 $s;
        print DC2 $s;
}

This script will create two, 1GB files. You may need to modify the path to the files.

Create the diskgroup while logged into the ASM instance as sysdba:

create diskgroup dgrp1
  failgroup fauxcontroller1
          disk 'C:\oracle\oradata\ts30\asm\dgrp1a'
failgroup fauxcontroller2
          disk 'C:\oracle\oradata\ts30\asm\dgrp1b'
/

You may now log on to the RDBMS instance and create tablespaces using the newly created diskgroups. As stated before, this should be used for demonstration purposes only.

Bugs

There were a few problems I encountered while learning to set up ASM. Most were due to incomplete documentation, or at least a lack of complete documentation all in one place, but one problem was apparently a bug.

The initialization parameters asm_diskgroups and asm_diskgroup_string are for use in an ASM instance, not in an RDBMS instance.

At one point I mistakenly ran a script in an RDBMS instance that should have been run in the ASM instance, and the following bit of SQL was run:

alter system set asm_diskgroups = 'DGRAID','DGFAIL' scope=both;

This is not supposed to work except in an ASM instance. When run in an RDBMS instance, an error should be raised warning the DBA that the parameter is invalid for this type of instance.

On the platform tested, this statement was allowed to complete and was written to the SPFILE, making it impossible to restart the database without restoring the SPFILE.

This problem is reproducible, at least as of version 10.1.0.2 on Linux.

[oracle@sherlock dbs]$ sqlplus '/ as sysdba'

SQL*Plus: Release 10.1.0.2.0 - Production on Thu Dec 2 20:22:21 2004

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area 167772160 bytes
Fixed Size                   777936 bytes
Variable Size             112206128 bytes
Database Buffers           54525952 bytes
Redo Buffers                 262144 bytes
Database mounted.
Database opened.
SQL> create spfile from pfile;

File created.

SQL> shutdown
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Production
With the Partitioning, OLAP and Data Mining options
[oracle@sherlock dbs]$ sqlplus '/ as sysdba'

SQL*Plus: Release 10.1.0.2.0 - Production on Thu Dec 2 20:23:09 2004

Connected to an idle instance.

SQL> startup
ORACLE instance started.

System altered.

SQL*Plus: Release 10.1.0.2.0 - Production on Thu Dec 2 20:23:51 2004

Connected to an idle instance.

SQL> startup
ORA-15021: parameter "asm_diskgroups" is not valid in RDBMS instance
SQL>

At that point, it was necessary to log on as SYSDBA to the smashed database instance, create a temporary PFILE, edit it, and recreate the SPFILE.

export ORACLE_SID=ts11
. oraenv 
sqlplus "/ as sysdba"
create pfile='/home/xxx/temp.ora' from spfile ;
create spfile from pfile='/home/xxx/temp.ora' ;

To Use or Not to Use

Should you use ASM? This is a new feature of Oracle, and holds much promise. As with any new feature, it should be thoroughly tested to ensure that it will work in your environment, including testing all of the claimed abilities of ASM to ensure both that they work as advertised, and to discover unforeseen problems.

References

Automatic Storage Management demo:

http://www.oracle.com/technology/obe/obe10gdb/manage/asm/asm.htm

MetaLink Notes:

249992.1 — ASM setup instructions
265633.1 — ASM best practices
275315.1 — ASMLib setup
266028.1 — ASM on files on Linux

Read the Best Practices paper for details on how ASM stores data.

OTN ASMLib Download:

http://www.oracle.com/technology/tech/linux/asmlib/index.html

Acknowledgements

Thanks to Martin Berg for help in re-using disks previously used for ASM.

Special thanks to Michael Möller of Miracle A/S for help with proofreading.

Jared Still is a DBA at RadiSys, an embedded solution provider. Jared has been an IT professional for 22 years, the last 10 of which have been as an Oracle DBA. He is the author of several Oracle articles and is co-author of Perl for Oracle DBAs.

Contributors : Jared Still
Last modified 2005-04-19 08:26 AM

DBAzine.com

Sections

Personal tools

Menu

Who Are You?

Adventures in ASM

Flavors of Oracle Storage Management

Configuring ASM

ASM with UNIX I/O

Create the ASM Instance and Diskgroups

ASM with ASMLib I/O

Installing ASMLib

Create the ASM Instance and Diskgroups

ASM with Files

Bugs

To Use or Not to Use

References

Acknowledgements