The Evolution of RAC

Oracle Parallel Server (OPS) has been around for many years. Now, Oracle has taken the general idea of OPS and improved upon it, offering a new product, which they call Cache Fusion. That term describes the ability of OPS databases to allow multiple instances to share data buffer storage between the instances while still accessing a single Oracle database.

The architecture of Oracle Parallel Server was originally designed for Massive Parallel Processors (MPPs). The idea behind OPS is that a single database can have many separate instances, each with its own System Global Area (SGA) and memory regions, accessing a single database.

An OPS configuration requires that the instances be able to share the data buffer cache. To allow the instances to communicate, Oracle created the Integrated Distributed Lock Manager (IDLM), which transfers data buffer blocks between instances. This is the Cache Fusion component. In OPS, data blocks are quickly transferred between instances, thereby fusing the multiple data buffers into a single cache. The concept of Cache Fusion is essentially advertising the fact that Oracle's IDLM can rapidly "ping" data blocks between the many instances, allowing for these instances to share data buffers in an efficient way.

Comparing Real Application Clusters to Oracle Parallel Server

It came is no surprise to many industry professionals when Oracle chose to discontinue their Oracle Parallel Server product. The OPS product allowed for a single database to be shared by many Oracle instances, and this architecture was quite good for massively parallel types of applications in which the data could be segregated onto these multiple Oracle instances. However, one serious shortcoming of the OPS architecture was the requirement that all data blocks be available to all instances. Hence a cumbersome process called an integrated distributed lock manager, or IDLM, had to constantly be at work in order to "ping" data blocks back and forth between the many instances in an OPS configuration.

This problem of the IDLM was overcome by re-architecting the OPS product and re-introducing the product under a new name, Real Application Clusters, or RAC. RAC employs a new technology called Cache Fusion whereby the data block buffers of all instances within the parallel server configuration reside in a single shared RAM memory region. By having all data blocks instantly available to all database instances, the problem of IDLM pinging is overcome, and the systems can run faster and far more reliablly than they could within an OPS architecture.

It's important to note that Oracle is touting Real Application Clusters, a generic OLTP solution for high availability. This is an important departure from their recommendation for Oracle Parallel Server, which was mostly used by organizations with massively parallel systems that required continuous availability.

It remains to be seen if Oracle will get the widespread adoption of RAC that they've been hoping for in the marketplace. Betsy Burton of the Gartner Group noted that adoption of Oracle9i Real Application Clusters has been quite slow, and she predicts that by the year 2006, only about 10 percent of Oracle users will be utilizing RAC within their production applications.

Does this mean that only 10 percent of Oracle customers require continuous availability? Clearly, the answer is no; many companies are choosing alternatives to using Real Application Clusters to provide continuous availability. One of the most common alternatives is to write their own replicated databases and come up with methods whereby when a failure occurs in one database, all transactions can be automatically redirected and restarted on another database engine.

It's also important to point out that OPS and RAC are only designed to protect against catastrophic instance failure. Should any one instance (or the hardware associated with that instance) fail, Oracle's Transparent Application Failover will cake and then redirect any in-flight transactions to the surviving nodes. Of course, the same approach can be achieved by having replicated databases distributed and having customized Web server code to redirect fail transactions.

It's also important to note that there are serious limitations of Oracle's Transaction Application Failover tool. The most serious limitation is that Oracle TAF does not support restarting of any DML statements including INSERT, UPDATE, and DELETE. For those customers using Oracle PL/SQL packages, all package states are lost, thereby requiring all PL/SQL stored procedures to be restarted from the beginning. The Oracle TAF product also does not support ALTER SESSION statements and does not support global temporary tables failover.

Here's an example of an actual TNSNAMES.ORA file for a Real Application Clusters system. Note in this example that you can specify a failover tide off from one of the two Oracle failover modes, and that you also have a re-try parameter.

prod.world =
  (DESCRIPTION_LIST =
    (FAILOVER = TRUE)
    (LOAD_BALANCE = FALSE)
    (DESCRIPTION =
    (ADDRESS =
      (PROTOCOL = TCP)
      (HOST = arkum)(PORT = 1521))
      (CONNECT_DATA =
        (SERVICE_NAME = ARK1)
        (SERVER = DEDICATED)
        (FAILOVER_MODE = 
           (TYPE = SELECT)
           (METHOD=PRECONNECT)
           (BACKUP=bkup)
        )
      )
    )
    (DESCRIPTION =
    (ADDRESS =
      (PROTOCOL = TCP)
      (HOST = diamond)(PORT = 1523))
      (CONNECT_DATA =
        (SERVICE_NAME = DIA3)
        (SERVER = DEDICATED)
      )
    )
  )

The fact that the continuously available solution employs a re-try parameter is very scary to many continuous availability professionals. Consumers are demanding systems that will automatically and reliably restart any in-flight transactions that might be running during the system failure, and the idea of delayed retries are onerous to people who are counting on continuous availability.

Another important note is that the RAC solution requires downtime to upgrade the Oracle software. Oracle is currently working to create a rolling update technology, but for now, RAC systems must be stopped when Oracle upgrades are applied.

• RAC is far simpler and more reliable than OPS (because of cache fusion).
• RAC and OPS only protect against instance failure. A failure of disk or the live cache (RAM) will cause catastrophic failure.

Inside Transparent Application Failover

Once Oracle developed the RAC technology to prevent instance outages, they had to develop a method for database connections to restart on the surviving instance. This technology was dubbed "transparent application failover," and to understand how it works we must take a close look inside the internal methods of TAF.

TAF has several failover types:

1 - SELECT Failover

When SELECT failover is used, Net8 keeps track of any SELECT statements issued in the current transaction. SELECT failover keeps track of how many rows have been fetched back to the client for each cursor associated with a SELECT statement. If connection to the instance is lost, Net8 establishes a connection to a backup instance, re-executes the SELECT statements, and positions the cursors so the client can continue fetching rows as if nothing had happened. SELECT failover can be useful for reporting applications, but that's as sophisticated as TAF gets.

2 - SESSION Failover

When the connection to an instance is lost, SESSION failover results only in the establishment of a new connection to a backup instance. Any work in progress is lost.

Within this failover type, Oracle offers two sub-methods:

• BASIC failover - In this approach, Oracle connects to backup instance only after primary connection fails.
• PRECONNECT failover - In this approach, Oracle connects to backup database and primary database. This offers faster failover, but it does so at the expense and added overhead for duplicating the Oracle connections.

The limitations of TAF are as follows:

• The effect of any ALTER SESSION statements will be lost.
• Global temporary tables will be lost.
• Any PL/SQL package states will be lost.
• Transactions involving INSERT, UPDATE, or DELETE statements cannot be handled automatically by TAF.

By default, when a TAF-initiated failover occurs, Net8 will make only one attempt to connect to the backup instance.

Using the RETRY and DELAY parameters, you can change that behavior so that Net8 makes multiple attempts to connect to the backup database.

In the following, we see 20 retries at 30-second intervals:

 (FAILOVER_MODE = 
           (TYPE = SELECT)
           (METHOD=PRECONNECT)
           (BACKUP=bkup)
           (RETRIES=20)(DELAY=30)
        )  

Monitoring Transaction Application Failover in Oracle

With the widespread acceptance of Real Application Clusters with Transparent Application Failover, the Oracle engine has made significant enhancements to the internal v$ views in order to allow the Oracle administrator to keep track of what's going on within re-connected transactions.

The biggest enhancement is to the v$ process view. Several new columns have been added to the dollar process view to allow the DBA to see exactly what's going on within the Oracle database.

• FAILOVER_TYPE - indicates the type of failover.
• NONE
• SESSION
• SELECT
• FAILOVER_METHOD - indicates the method used to establish the backup connection.
• NONE
• BASIC
• PRECONNECT
• FAILED_OVER - Indicates whether or not a session has failed over to the backup connection.
• YES
• NO

Here we see a sample an Oracle query to display the new TAF columns.

SQL> SELECT username, sid, serial#, 
     failover_type, failover_method, failed_over
     FROM v$session; 

  USERNAME   SID   SERIAL#    FAILOVER_TYPE   FAILOVER_M    FAI 
               1         1    NONE            NONE          NO 
               2         1    NONE            NONE          NO 
               3         1    NONE            NONE          NO 
               4         1    NONE            NONE          NO 
               5         1    NONE            NONE          NO 
               6         1    NONE            NONE          NO 
   SYSTEM      7       688    SELECT          PRECONNECT    YES 
               9       110    NONE            NONE          NO 
              10       109    NONE            NONE          NO 
              11       110    NONE            NONE          NO 
              12         1    NONE            NONE          NO 
              15        84    NONE            NONE          NO 
              16      1729    NONE            NONE          NO 

Alternatives to RAC for Continuous Availibility

As we've already noted, here are several alternatives to using Oracle's Real Application Clusters for continuous availability and transaction failover. Most Oracle companies in the real world will choose one of the following approaches:

Cross database links -- using a cross database link approach, all Oracle are encapsulated inside stored procedures. Within each stored procedure, distributed transactions are set up such that transactions are automatically sent to all of the replicated databases in the form of a two-phase COMMIT transaction. Doing updates as a two-phase COMMIT transaction ensures that all of the databases are either updated or rolled-back as a single unit within the Oracle database. Many companies of been very successful with this approach and they can ensure 100-percent reliability that all transactions are synchronized in real and distributed in real time to all of the different systems.

Using undo log replication -- the undo log replication technique first became popular in the year 2001 when Quest software introduced their Shareplex product. The Shareplex product reads directly from Oracle's log buffer structure, and builds SQL statements that are immediately fed to all of the replicated systems. While this product worked quite reliably, Oracle Corp. made it clear that this product would not be officially supported by Oracle because Oracle reserves the right to change the structure in future releases of how information was stored within the RAM region of law buffer. However, later in 2001, the release of Oracle9i logical databases codified this concept, and now in Oracle9i, customers can confidently use redo log-based replication to keep many Oracle databases synchronized in real time.

When building your own replicated database:

• Place all SQL inside stored procedures.
• Create back-and-forth database links for updates.
• Have the stored procedures call both databases as a two-phase commit.

Custom Failover Approaches

• Customized WebServers are written with Apache extensions
• Replicated databases with WebServers connecting based on load
• Slow response from Oracle causes WebServer to re-connect to another database and re-start transaction
• Requires knowledge of Apache code

Conclusion

Oracle RAC and TAF have been the hallmark of Oracle's continuously available solutions and are the primary toll behind Oracle's "Unbreakable" claim. Once installed and configured, RAC and TAF provide true 24x7 availability, but the installation, setup and testing costs can be cumbersome.

Hence, many companies are using alternatives to RAC and TAF, specifically self-replicating Oracle databases using shareplex and cross-database link technology.

--

Donald K. Burleson is one of the world’s top Oracle Database experts with more than 20 years of full-time DBA experience. He specializes in creating database architectures for very large online databases and he has worked with some of the world’s most powerful and complex systems. A former Adjunct Professor, Don Burleson has written 15 books, published more than 100 articles in national magazines, serves as Editor-in-Chief of Oracle Internals and edits for Rampant TechPress. Don is a popular lecturer and teacher and is a frequent speaker at Oracle Openworld and other international database conferences. Don's Web sites include DBA-Oracle, Remote-DBA, Oracle-training, remote support and remote DBA.