The Woeful CIO

Imagine a haggard CIO being interviewed about a recent failure in his production applications. The lament … “Last year I spent millions of dollars on a storage replication solution to protect my data in case of a disaster. Yesterday my business lost millions of dollars due to some improper database system maintenance. The recovery action took hours to diagnose and prepare, and even longer to execute. We’ll probably lose several accounts permanently, and will never be able to recoup the losses. The storage replication solution offered no protection from the outage. I wish I had some tools that could reduce or eliminate the downtime for ‘local’ outages as well as protect for disaster situations.”

The Problem

Companies have invested millions of dollars in their mainframe database applications. These applications allow trains and planes to move, packages to be shipped, financial transactions to occur, and manufacturing to proceed. If these applications are unavailable for any reason, the company is not receiving the expected gain on their investment. Further, some industries receive fines if certain transactions are not processed in a timely fashion. Recent legislation requires all publicly traded institutions to maintain a high-availability application plan, including recovery from total site disaster.

On August 14, 2003, the North Eastern United States experienced a massive power blackout. Independent surveys taken since then indicate that over two thirds of all respondents lost at least one full business day due to the blackout. The cost of the downtime ranged from $50,000 per hour to over $1,000,000 per hour.

The challenge facing a mainframe database application user is to maintain the recoverability of the database while not adversely impacting availability. There are several techniques to protect the database and ensure recovery to a consistent point. The techniques range from periodic dumps of storage onto transportable media, to synchronous I/O mirroring to an alternate site.

Most companies have at least a basic “Disaster Recovery” (DR) plan for mainframe database applications. In recent years, companies have begun to address the larger issue of “Business Continuity,” recognizing that ensuring application availability requires more than just a “DR plan.”

This enlargement of scope exposes the chief limitation of “hardware” solutions for disaster recovery — they only address a total-site outage. There is no provision for the much more common situation in which an application recovery is required for a non-disaster event.

The Hardware Solutions

Generally, the “hardware replication solutions” take two forms:

• “Backups” are based on getting a consistent copy of storage volumes at a Point in Time, and making a copy available at a remote site (either dumping to tape and shipping the tapes to a vault, or electronically transmitting the “Backup” dataset to a remote set of volumes. Typical data loss in this scenario is 24 hours.
• “Mirrors” are based on storage devices or processes that replicate database updates to an alternate site as they occur. Typical data loss in this scenario is almost 0.

There are several limitations to the solutions, and they can grow quite expensive to implement and maintain. Minimally, the customer will have to double their storage footprint, and some solutions call for up to six copies of production data in various states. However, several very large database customers have implemented some form of hardware replication in the name of “Disaster Recovery.”

These solutions can allow for a simplified process at time of disaster — the user performs some operations at the remote site to render the “backup” or “mirror” available to a processor, and restarts the database applications. Processing can resume, usually within a few hours of arrival at the remote site. On the surface, these solutions appear attractive to a customer willing to spend the extra money for replicating data. The user can control data loss and reduce downtime, and simplify the DR in the process. The hardware solutions are very effective tools — in the event of site-wide disaster.

The Need for Software Solutions

There is a glaring exposure in the hardware replication solutions. They only address a total site outage. For any other type of database failure, they offer no help. For some types of disaster, the mirror doesn’t even afford protection from a site outage (many of the customers impacted by the August 13, 2003 power outage saw both of their sites affected due to the broad scale nature of the outage). What can cause a database outage?

• Some outage events are planned:
• Application database maintenance
• Data migration
• Schema change implementation
• Hardware (processor, storage), operating system or DBMS maintenance
• Disaster recovery preparation

Some events are unplanned (and happen at inopportune times):

• Site disasters (floods, power outages, storms, fire, and so on
• Hardware failures
• Operating system failures
• DBMS Failures
• Operations errors
• Batch cycle errors
• Improper data feeds
• User errors
• Application software errors
• Fallback from application change migrations

Of all the potential outages listed previously, only one is protected in a storage replication environment (site disaster). This is a very rare event. All of the other risks, which are much more likely to happen, are not protected by a storage solution.

To completely protect the database environment, a variety of tools are needed to support database copy, log processing, and recovery management activities. These tools can reduce or eliminate the downtime for both planned and unplanned events. Using the right software-based recovery tools, the customer can perform the following activities:

• Produce a “clean” database image copy with minimal outage.
• Extract the effects of a bad update transaction with no outage.
• Recover a complex application with innovative log processing.
• Manage the log environment and prepare for optimum recovery, including recovery for dropped DB2 objects.
• Define application recovery groups and simulate or estimate recovery.
• Leverage any investment in Intelligent Storage Devices for Backup and Recovery.
• Prepare for automated database Recovery, with coordination between DBMS applications (e.g., IMS and DB2), to any point-in-time, for both local and disaster recovery events.
• Leverage the “recovery” tools in day-to-day operations (e.g., data migration, log reporting, reduced resource utilization).

In short, the proper database recovery tool kit will protect the application database for both local and disaster recovery, allowing for recoverability while ensuring the highest level of availability. The acquisition of such tools can be cost justified based on their impact on reducing downtime and their ability to improve daily operations.

In today’s high-availability, complex e-business world, it is imperative to protect the corporate data asset. If the database is unavailable, the millions of dollars invested in information technology are not returning a benefit to the business, and IT becomes a cost burden instead of a business asset. Software based recovery tools are not a luxury or an insurance policy — they are a key component of mission critical application database support.