Introduction
I would describe myself as a fairly good tuner. Not because tuning ever came easy to me, I have just spent a lot of time working with the optimizer. I still have troubles rewriting complex SQL statements from time to time but I will admit that 20 years of experience makes it easier on me than it used to be.

One of the areas I am strong at is using the proper indexes to influence SQL statement access paths. We know that an index can influence the optimizer to change an access path from a tablescan to a table lookup using an index ROWID. But indexes can also influence the type of join used as well as the order of the tables being joined.

Let's take a look at the benefits virtual indexes have and we'll finish this blog by reviewing how indexes can affect simple access paths, join methods and join orders.

Virtual Indexes
I use virtual indexes in both 9i and 10G. I like them because I can quickly do "what if" scenarios. I can create the index and present it to the optimizer without waiting for the index to become populated and stored on disk. This feature allows me to reduce the amount of time I spend tuning, which is always a good thing.

I also use virtual indexes when I am training myself and others. Most production environments do have test counterparts, but there are times when I need to teach many different indexing concepts and waiting for indexes to build often becomes a detriment to the learning process.

9I OEM Virtual Index Wizard
The Virtual Index Wizard feature is provided in the Tuning Pack option of 9I OEM. Although there are a couple of ways to activate the wizard, let's take a look at the one that I most often use. Since we are going to be naviating through the SQL Analyze tool, we might as well review some of the other features it provides.

SQL Analyze
9I OEM provides the SQL Analyze tool to view access path information. SQL Analyze is only available if you have purchased the tuning pack option for 9I OEM. This screenshot shows me navigating through 9I OEM's menus to activate SQL Analyze.

When you activate SQL Analyze, the first step the utility performs is to display a listing of all of the top resource consuming SQL statements that are contained in the library cache. If the statement has been flushed from cache, you won't find it in this display.

The tool lists all of the databases on the left side of the panel that you worked with in the past. As you can see, I am a pretty active user of SQL Analyze. I can double-click on any statement in the top SQL listing to view its access path information. I can also use the menu system at the top of the screen to create a blank worksheet for new queries that I can enter manually. I found the statement I wanted from the listing and double-clicked on it to activate the SQL tuning panel.

The panel displays the SQL statement, and like our other utilities, allows us to step through the access path the query is taking. I can click on the SQL drop down menu at the top of the screen to ask Oracle to explain the statement using the various optimizer modes that are available.

SQL Analyze Tuning Tools
SQL Analyze provide us with tools that facilitate the SQL tuning process. The tools menu at the top of the screen allows me to choose from three different wizards. We'll review the Virtual Index Wizard in just a moment.

The Hint Wizard allows us to choose hints from a drop-down menu system, while the SQL Tuning Wizard activates an intelligent advisor that provides us with SQL tuning recommendations. Once you have run the SQL Tuning Wizard a few times, it's pretty easy to see that it was the precursor to some of the more advanced utilities that are now available in Oracle 10G.

Virtual Index Wizard
The Virtual Index Wizard allows me to create virtual indexes on tables that my query is accessing. The key word in that last sentence is "virtual". Oracle does not permanently create the index. The intent of the wizard is to allow users to determine the potential performance impact a new index would have on the statement being analyzed. It is then up to the user to permanently create the index.

If I click on the Virtual Index Wizard menu selection, 9I OEM responds by displaying the Virtual Index Wizard introduction panel. When I click on next, the wizard displays a panel that allows me to select the columns for my virtual index. Clicking next again displays a panel that allows me to attempt to force Oracle to use the new index or allow it to choose the new index on its own. My personal preference is to allow Oracle to choose the index on its own without providing any additional prodding.

Clicking Next again displays the projected cost improvement the new index will have on the statement being analyzed. I used a full table scan on a rather large table as an example. Based on the display, we can be pretty sure that the new index will have a positive affect on our statement's performance. The last panel also displays a button that allows us to view the before and after access paths.

The Virtual Index Wizard does a fairly good job of estimating the impact that new indexes will have on a SQL statement. It is an excellent tool to use when you are starting your tuning education as well as evaluating access paths in Oracle 9I databases.

I was surprised when I found that this feature was not contained in 10G Grid Control. I continue to firmly believe that using a tool to quickly build a virtual index, determine if the optimizer will use it and then display a prediction of the index's performance boost (or degradation) is very beneficial to the tuning process. I used the virtual index wizard dozens of times and will continue to use it when I tune 9I (and earlier) databases.

Virtual Indexes in Oracle10G
Although Grid Control doesn't provide the same graphical interface to create virtual indexes as it did in 9I OEM, all is not lost. We can revert to the tried-and-true SQL*PLUS command line to build virtual indexes using the "NOSEGMENT" option. The only feature it won't provide us is the forecast of the statement's performance.

The format of this blog page doesn't provide me with enough space to show you the output from my SQL*PLUS demo. Not without a whole lot of editing anyway. So, I decided that the best option would be to create a separate HTML page for the 10G virtual index demo output.

The demo will show you:

• How to create virtual indexes in 10G
• How to configure your session to allow the optimizer to choose a virtual index as an access path
• The affect the "_use_nosegment_indexes" parameter has on access path selection
• The error messages returned if you attempt to manipulate virtual indexes
• Information stored in the data dictionary pertaining to virtual indexes

Here's a link to the 10G SQL*PLUS Virtual Index demo!

Now that we understand how to create virtual indexes, lets' take a look at how any index can influence simple access paths, join methods and join orders.

Index Affects on Access Paths
Let's use a nested loop join as a quick and somewhat easy example. In a nested loop join, one of the tables in the join is known as the outer table while the other table (the table being probed) is known as the inner table. Oracle reads a row from the outer table and uses the columns in the join condition to probe the inner table.

Take a look at the SQL statement and graphical access path display below:

select a.employee_id, a.last_name, b.department_id, b.department_name,
c.street_address, c.postal_code, c.city, c.state_province
from hr.employees a, hr.departments b, hr.locations c
where a.department_id=b.department_id
and b.location_id=c.location_id
and a.employee_id = 174
order by a.last_name;

We are joining three tables together HR.EMPLOYEES, HR.DEPARTMENTS and HR.LOCATIONS. Notice that we have two different nested loop join operations. Oracle only joins two tables together at a time. It then creates an intermediate result set and uses that result set to join to the next table. It can use any type of join method available for any of the join operations.

The indexes we have available to us are:

• EMP_EMP_ID_PK - index on the EMPLOYEE_ID column of the HR.EMPLOYEES table
• DEPT_ID_PK - index on the DEPARTMENT_ID column of the HR.DEPARTMENTS table
• LOC_ID_PK - index on the LOCATION_ID of the HR. LOCATIONS table

The optimizer has chosen to join the HR.EMPLOYEES AND HR.DEPARTMENTS tables first using the nested loop join method. It is on this join condition - A.DEPARTMENT_ID = B.DEPARTMENT_ID. HR.EMPLOYEES is known as the outer table and HR.DEPARTMENTS is classified as the inner table of the join.

The optimizer looked at the local predicate "A.EMPLOYEE_ID=174" and checked to see if there was an index that it could use to improve data access performance. It found the EMP_EMP_ID_PK index that is defined on the EMPLOYEE_ID column. It will access the index first and retrieve the ROWID for the row that has an EMPLOYEE_ID of 174. It will use the ROWID to probe the HR.EMPLOYEES table.

As stated previously, the optimizer has chosen the nested loop join operation. It will access the HR.EMPLOYEES table using the EMP_EMPID_PK index, retrieve the value for DEPARTMENT_ID and use it to probe the HR.DEPARTMENTS table looking for a match (A.DEPARTMENT_ID = B.DEPARTMENT_ID). If it finds a DEPARTMENT_ID value in the HR.DEPARTMENTS table that matches, Oracle will use that row in future operations. If it does not match, Oracle will discard it.

Notice that the optimizer has chosen to use the DEPT_ID_PK index that is built on the HR.DEPARTMENTS' DEPARTMENT_ID column. Since we are probing this table using the values we found for DEPARTMENT_ID from the HR.EMPLOYEES table, Oracle will use the index on the HR.DEPARTMENTS' DEPARTMENT_ID column.

The B.LOCATION_ID = C.LOCATION_ID is the join condition that is used in the second join operation. The intermediate result set is used as the outer table and the HR.LOCATIONS table is used as the inner table. The same type of probe occurs. Since we are looking for matches using the LOCATION_ID contained in the intermediate result set, the optimizer will choose the LOC_ID_PK index.

Here's how indexes can influence the type of join used and the order of the tables being joined:

• If we didn't have indexes on either of the first two tables being joined, the optimizer would most likely have used a hash operation instead of a nested loop join to join the tables together. The optimizer knows that the hash join operation is best used for joins that return larger result sets. It could also have influenced the join order of the tables. In the example, the optimizer could have chosen to join the HR.DEPARTMENTS table to the HR.LOCATIONS table first. It would have then used the intermediate result set created to probe the HR.EMPLOYEES table.
• The more indexed columns we have on a table, the greater the chance that the optimizer will use it as the outer table of the nested loop join. It can filter out more rows before it sends it to the join operation. The earlier it can filter rows out, the better.
• If we have an index on both join columns (i.e. indexes on the DEPARTMENT_ID columns for both the HR.DEPARTMENTS and HR.EMPLOYEES tables), Oracle will use the DEPARTMENT_ID index on the inner table but not the outer. Why is that? Oracle doesn't know if it has a DEPARTMENT_ID VALUE in the inner table that matches a DEPARTMENT_ID value in the outer table unless it reads each and every row of the outer table to do the match. Think about it. It knows that since it has to read every row from the outer table to perform the match, the only indexes it can use are the indexes built on local predicates (i.e. "A.EMPLOYEE_ID=174").
• If we have an index on the join column for one table and not the other, the optimizer will often favor the table that has the index on the join column as the inner table of the join. It knows it can't use an index on the join column if it is the outer table but it can use the index if it is the inner.

This is just one quick example to show you the impact that indexes have on access path selection. The key to success is to continuously experiment with the optimizer. One of the features that can speed up your experimentation is to use virtual indexes. Remember, this feature uses an "undocumented" parameter in 10G.

I would like to thank super-DBA Mark Shore for re-igniting my interest in both 9I and 10G virtual indexes. Marks has a natural talent for this profession which I often wish I had.

Thanks for Reading,

Chris Foot
Director of Operations
Remote DBA Experts
Oracle Ace

The key to being a successful Oracle DBA is that you must be willing to accept the fact that with each new database release, you will have to adjust the way you administer your environments. In my profession, I work with many DBAs from companies across the globe. The most successful administrators I have met are the ones that intelligently leverage the new features that are available in the releases of the database they are administering.

Running Commands and Scripts Against Multiple Targets
Being involved in an IT audit, like death and taxes, is inevitable. If you work as a technician for any length of time, you will be working with an auditor and some point. Since we provide remote database services, our organization is audited numerous times per year. Sox audits, security audits, HIPPA audits - you name the audit and we have been involved in it. We have an office that is left open full-time for onsite auditors.

Auditors often ask you to run scripts on the operating systems and databases your unit is responsible for supporting. In the past, you would review the scripts and then distribute them to team members to run on the environments they were responsible for supporting. 10G Grid Control makes this task a whole lot simpler. Grid provides you with the capability of running host commands and scripts against multiple host targets at the same time. Virtually any command or script that can be executed on a single host can also be executed on multiple hosts using 10G Grid Control R2. Here's a blog that shows you how to use 10G Grid Control R2 to run operating system scripts against multiple targets.

10G Grid Control R2 also provides an easy-to-use panel that allows administrators to run SQL scripts against multiple database targets simultaneously. Just like its host command counterpart, all you need to do is code up a SQL statement, choose a set of database targets, run the SQL against the multiple targets and review the output. The benefits that this feature provides is only constrained by the DBAs imagination. Here's a link to a blog that will show you how to run scripts against multiple database targets.

Restore Points
How many times have database recoveries been performed because of incorrect changes made to database data? A user deletes or updates "one too many rows" by mistake and the result is a time-consuming and error prone process to restore and recover the database to a point-in-time before the error occurred.

Let's go back to my days as an Oracle instructor…. A common question in the Oracle DBA backup and recovery class was "why can't I just roll the database back to remove unwanted changes instead of restoring the database from a backup and applying the redo logs to roll forward to a point-in-time before the error occurred?" The question was invariably followed by "don't you think that would be easier?" The answer was always the same "yes, it would be easier, but the Oracle database doesn't have that capability."

That was before Flashback Database came along. Oracle's Flashback Database feature provided a new tool in the DBA's recovery toolbox. Flashback Database allows the DBA to "roll back" a table, set of tables or the entire database to a previous point-in-time. Administrators often used a timestamp during the execution of flashback operations.

10G R2 enhances Flashback Database by allowing administrators to associate a user-defined name with a point-in-time. The user-defined name, called a restore point, can be used in place of a SCN or timestamp when executing a FLASHBACK TABLE or FLASHBACK DATABASE statement.

A SQL statement is used to create the restore point which means it can be embedded in application programs as we as executed on an as-needed basis in SQL*PLUS. The example below shows the creation of a restore point:

CREATE RESTORE POINT batch_daily_cycle_complete;

Now if I execute the following statement:

UPDATE cti.employee_salary_table SET
salary = 200000 WHERE empl_lname = 'FOOT';

My boss can execute this statement to correct the 'mistake' (although I prefer to call it a fully justifiable pay adjustment):

FLASHBACK TABLE cti.employee_salary_table TO batch_daily_cycle_complete

10GR2 also provides guaranteed restore points which ensure that the database can be flashed back to the point-in-time they were created. Guaranteed restore points save disk space because only the flashback logs required to meet the guaranteed restore point need to be retained by the database. The statement below creates a guaranteed restore point:

CREATE RESTORE POINT batch_daily_cycle_complete GUARANTEE FLASHBACK DATABASE;

Automatic Undo Retention
In Oracle9i, administrators had their choice of continuing to manage rollback segments on their own (manual undo management) or configuring the database to manage its own before image data (automatic undo management). Oracle refers to system managed before image segments as undo segments.

You didn't have to be an Oracle expert to know that manual rollback segments were "somewhat troublesome." Out of space conditions, contention, poor performance and the perennial favorite "snap shot too old" errors had been plaguing Oracle database administrators for over a decade. Oracle finally decided that the database could probably do a better job of managing before images of data than we could.

But implementing automatic undo retention didn't necessarily guarantee users a trouble free undo environment. There really is only one parameter that administrators can tweak in a system that uses automatic undo. The UNDO_RETENTION parameter specifies the amount of time in seconds that Oracle attempts to keep undo data available. Setting this parameter to the appropriate value could be described as more of an art than a science.

Set it too low and you are wasting disk space. In addition, you aren't taking advantage of being able to flashback your data to as far back as the disk space allocated to the undo tablespace allows. Set it too high and you are in danger of running out of freespace in the undo tablespace.

10G R2 comes to the rescue! The database now collects undo usage statistics, identifies the amount of disk space allocated to the undo tablespace and uses that information to tune the undo retention period to provide maximum undo data retention. Administrators can determine the current retention time period by querying the TUNED_UNDORETENTION column of the V$UNDOSTAT view.

RMAN Automatically Creates Temporary Datafiles During Restore
Ever perform a restore and forget to recreate the temporary tablespace's data files? I have. The database comes on-line but the first time a user's sort overflows from memory to disk, an error code is returned. Recreating a database's temporary files is one of those activities that are often overlooked. 10G R2 RMAN will automatically recreate temporary datafiles belonging to locally managed temporary tablespaces during the recovery operation.

Enterprise Manager Statistics Management Support
A new link titled 'Manage Optimizer' statistics has been added to the database administration home page in Enterprise Manager. Clicking on the link displays the new Manage Optimizer Statistics home page. The new home page provides wizards and templates that facilitate and automate optimizer statistics management. In 10GR2 administrators are able to use Enterprise Manager to gather, restore, delete, lock and unlock statistics

Why would you want to lock and unlock statistics? Let me describe one real-life example from a recent project. We have a warehouse system that contains an extremely volatile table. Describing it as volatile is like stating the Titanic sprung a small leak. Hundreds of thousands of rows are added and removed around the clock.

The level of activity is based on the particular business process taking place. At one particular time, the table can contain hundreds of thousands of rows and at other times it can contain dozens. OK, so when do you generate statistics? If you run DBMS_STATS at the same time each day, in most cases you would think you would get a predictable set of statistics generated. Not so in this table's case, sometimes you get hundreds of rows and some times you get hundreds of thousands.

If you are unlucky and generate statistics on the table when it has hundreds of rows, access paths suffer when the table grows to hundreds of thousands. So, we ran DBMS_STATS when we knew the table contained numerous rows and never generated statistics on it again. Problem is that we can no longer easily use the SCHEMA option for DBMS_STATS. Freezing statistics on this table will allow us to use the SCHEMA option and not generate statistics for selected tables.

DML Error Logging
What I like about SQL*LOADER is its ability to continue processing through load errors. If the record being loaded is rejected because of a unique constraint violation or discarded because it does not meet some user-defined criteria, SQL*LOADER places the record into a discard or reject file and keeps on running until it has reached a user-specified maximum number of rejects or discards.

The loader's log file will show how many records were loaded, rejected or discarded. I can look at the messages and review the discard or reject files, fix the problem and attempt to reload them again.

In 10G R2, this same type of processing has been applied to bulk DML operations. Users are able to specify whether they want to log errors or abort the entire DML statement, set the amount of detail information logged and the maximum error threshold. So instead of the entire DML statement blowing up and rolling back, Oracle will log the errors to the error table and continue processing until it exceeds the maximum number of errors. Just like SQL*Loader!

Oracle 10G R2's DBMS_ERRLOG package is used to create the error log output table and link it to the table being updated. The package's specification is provided below:

DBMS_ERRLOG.CREATE_ERROR_LOG (
dml_table_name IN VARCHAR2,
err_log_table_name IN VARCHAR2 := NULL,
err_log_table_owner IN VARCHAR2 := NULL,
err_log_table_space IN VARCHAR2 := NULL,
skip_unsupported IN BOOLEAN := FALSE);

Most of the columns are pretty self explanatory: table name being updated, error log table name, owner of error log table and the error log table's tablespace. If the SKIP_UNSIPPORTED is set to TRUE, column types that are not supported by error logging will be skipped over and not added to the error logging table. If it is set to FALSE, an unsupported column type will cause the procedure to fail.

Here's a quick example:

Let's create or table that will be updated:
SQL> r
1 CREATE TABLE foot.emp_table
2 (empno number(4), ename varchar2(10), job varchar2(8))
3* TABLESPACE users;

Table created.

Add a primary key:
SQL> ALTER TABLE foot. emp_table ADD PRIMARY KEY(empno)
2 USING INDEX TABLESPACE users;

Table altered.;

Load some rows:

SQL> INSERT INTO foot.emp_table VALUES
2 (7499, 'ALLEN', 'SALESMAN');

1 row created.

SQL> INSERT INTO foot.emp_table VALUES
2 (7521, 'WARD', 'SALESMAN');

1 row created.

SQL> INSERT INTO foot.emp_table VALUES
2 (7566, 'JONES', 'MANAGER');

Let's cause a unique constraint violation:

SQL> insert into foot.emp_table select * from foot.emp_table;
insert into foot.emp_table select * from foot.emp_table
*
ERROR at line 1:
ORA-00001: unique constraint (FOOT.SYS_C009069) violated

Create the 10G R2 error logging table:

exec dbms_errlog.CREATE_ERROR_LOG ('emp_table','dml_errors_emp_table')

Let's create a table that we can update and change one row to see if we can get one row to load and 2 to fail and be placed into the DML_ERRORS_EMP_TABLE:

SQL> create table foot.emp_load_table tablespace users as select * from foot.emp_table
Table created.

Update one row to change the EMPNO column's value to avoid the primary key violation:

SQL> update foot.emp_load_table set empno=123 where empno=7499;
1 row updated.

Rerun the statement specifying the new 10G R2 error logging syntax. Use our new load input table so that our one changed row will be loaded and two will be rejected and placed into the DML_ERRORS_EMP_TABLE:

SQL> insert into foot.emp_table select * from foot.emp_table
LOG ERRORS INTO dml_errors_emp_table('test_load_20050718')
REJECT LIMIT UNLIMITED;

1 row created.

OK, we loaded one. What happened to our other two? Let's see what our DML_ERRORS_EMP_TABLE contains.

First, let's describe the DML_ERRORS_EMP_TABLE:

SQL> DESC foot.dml_errors_emp_table

ORA_ERR_NUMBER$ NUMBER
ORA_ERR_MESG$ VARCHAR2(2000)
ORA_ERR_ROWID$ ROWID
ORA_ERR_OPTYP$ VARCHAR2(2)
ORA_ERR_TAG$ VARCHAR2(2000)
EMPNO VARCHAR2(4000)
ENAME VARCHAR2(4000)
JOB VARCHAR2(4000)

Our error logging table contains an incrementing error counter, the error message, ROWID, error type, tag (contains our users specified name from above -'test_load_20050718') and the three columns of our table.

Let's select from the table. I have truncated the ORA_ERR_MESG$ table for readability sake:

SQL> SELECT ora_err_number$, ora_error_mesg$, emp_no FROM foot.dml_errors_emp_table;

ORA_ERR_NUMBER$ ORA_ERR_MESG$ EMP_NO
--------------- -------------------------------------------------- ------
1 ORA-00001: unique constraint (FOOT.SYS_C009069) violated…..7521
2 ORA-00001: unique constraint (FOOT.SYS_C009069) violated….7566

We loaded one row and rejected two rows due to primary key constraint violations.

Conclusion
I hope you enjoyed this series on 10G features.

Favorite DBA Conversation #1
Any discussion that ends in "I hope you're not too busy because we need it now." I really dislike the conversation that ends in "we needed it yesterday." If you needed it yesterday you should have stopped in to discuss this request a week before yesterday. We all know these types of requests because we have all gotten them. One of your customers pops in and states:

"We need a complete refresh of XYZ database from production. Oh, and its an ERP application database and we don't have enough disk space in test to hold everything so you'll need to figure out how to reduce its disk size requirements AND make sure the application still works.

I know the box only has 1 CPU and 250 MEG of memory, but can you make sure we still get good performance? We're testing some application coding changes and we need to get performance benchmarks. We'll compare the stats from the test box to the ones we get from our 22 CPU production box. It may not be an "apples to apples" but I think the boxes are close enough to compare the performance stats.

I know its Friday afternoon at 3:00 and even though we knew about this refresh 14 weeks ago, we didn't think we needed to tell you until 1 day before we wanted it. So, we would like to have the system ready by Monday."

Over the last 20 years I have received dozens of these last minute requests. Requests that have made me cancel plans, postpone trips and generate numerous apologies to friends and family. The work never bothered me, but the timing of the requests sometimes did. It also reduced the quality of service I provided to my other customers. When you have DBAs jumping from one non-planned request to another, it affects their ability to work on planned requests as well as perform proactive administrative duties.

Fix #1: You minimize these requests by creating, advertising and enforcing DBA work request SLA agreements. Agreements that clearly state lead times per type of request. The more widespread the problem is at your organization, the more advertising you must do. It really becomes a sales pitch on the importance of a DBA unit being able to forecast workloads. The unit can't provide quality service to their customers if they are being affected by work requests that could have been planned for - but weren't.

The mantra should be "all of our customers suffer when we receive un-planned work requests." I'm not afraid of telling application manager A that she can't get her planned requests done because application manager B just gave us 32 emergency requests. Kind of like putting them both in a jar and shaking it up a bit.

Fix #2: 10G Grid Control allows you to clone both the Oracle Home software installation as well as the Oracle database. Grid will automatically adjust the specific home properties (IP address, host name, listener settings) during the cloning process. 10G DBCA templates allow you to clone a new database from existing databases. In addition, DBCA allows you to clone just the source database file structures or the structures and the data.

You can also use some of the 10G Data Pump features to tailor the new database to your user's specifications. 10G Data Pump provides numerous options that:

• Specifiy what objects you want to transfer from one database to another
• Transfer a subset of data from those objects using various filtering options
• Remap objects during the migration process. You can remap schema objects from one user to another, from one tablespace to another

You already know I'm a huge fan of 10G Data Pump. You need to leverage the features this tool provides to customize schema and data transfers from one database to another. You shouldn't be writing scripts when you don't need to. The only way you will realize the full potential of 10G Data Pump is to READ THE UTILITIES manual and see how powerful 10G Data Pump really is. It will make your life easier.

Favorite DBA Conversation #2
You come in to a meeting and you're confronted by an application manager who tells you that you need to tune your database because his ace developer "Sir Lawrence of LongSQL" can't get his 32 page SQL statement to achieve sub-second response times.

You know the SQL statement that I'm talking about. It's the one that contains 15 inner and outer joins, 12 subselects, and…….(feel free to add any other type of complex SQL Syntax here).

Note from database administrators to application developers:
Just because you can code a 32 page SQL statement, it doesn't mean you should. We all understand how powerful the SQL language is, but do you really need to try and use every feature available in a single statement? There's a difference between leveraging the power of the SQL language and attempting to perform too much work in a single SQL statement. Generating long, complex SQL statements that no one can debug does NOT classify you as a supercoder. It does classify you as something else, but I can't say that publicly.

Favorite DBA Conversation #3
Favorite DBA Conversation #3 combines components of conversations #1 and #2. It occurs when you are approached by a customer who says they have developed this "small" application that goes live in two weeks and it isn't performing "as well as we think it should." This means that you have to review the SQL from 23 new application programs in those two weeks.

Fix #1: Reinforce the need for the DBA team to be involved in all application development projects. This can be done by creating a series of application design review meetings. A dedicated an entire blog on this subject.

Fix#2: You will need to start a top down performance analysis project that focuses on tuning the most poorly performing queries first. If you don't have access to the 10G Grid Control sissy GUI tools, you can use the V$ views to retrieve the top resource consuming statements. You can use the V$SQLAREA view for Oracle9I and below. 10G offers the V$SQLSTATS view that provides some distinct advantages over its previous counterparts.

10G V$SQLSTATS Performance View
Before we discuss the new V$SQLSTATS view, let me deviate for a minute to provide you with some quick SQL tuning advice. V$SQLAREA is one of my favorite SQL tuning views. Until V$SQLSTATS came along…

If I want to perform a traditional "top down" tuning approach and tune the highest resource consuming SQL, I'll use the statements below to identify the top resource consuming queries.

The following query identifies the SQL responsible for the most disk reads:

SELECT disk_reads, executions, disk_reads/executions, address, sql_text FROM v$sqlarea WHERE disk_reads > 5000 ORDER BY disk_reads;

The following query identifies the SQL responsible for the most buffer hits:

SELECT buffer_gets, executions, buffer_gets/executions, address, sql_text FROM v$sqlarea WHERE buffer_gets > 10000 ORDER BY buffer_gets;

You can create a more readable report in SQLPLUS by inserting report breaks between the output lines. To generate the report breaks in SQLPLUS, issue the following statement before running the query:

BREAK ON disk_reads SKIP 2 --- for the disk read report and
BREAK ON buffer_gets SKIP 2 --- for the buffer get report

The first query returns SQL statements responsible for generating disk reads greater than 5,000 while the second query returns SQL statements responsible for generating buffer reads greater than 10,000. I used these numbers just as an example but I sometimes use them as a starting point. I'll then adjust them up or down accordingly, based on their output. The numbers also depend on the system I'm reviewing. I'll use different numbers for OLTP environments than I would for data warehouses.

You'll notice that I divide the number of disk and buffer reads by the number of statement executions. If a statement is generating 1,000,000 disk reads but is executed 500,000 times, it probably doesn't need tuning.

Heavy disk reads per statement execution usually means a lack of proper indexing, poor selection criteria, etc.. Heavy buffer reads sometimes means the exact opposite - indexes are being used when they shouldn't be.

But I'm personally most interested in workload, that's why I most often use the buffer cache hits in my initial queries.

But the SQLTEXT column in V$SQLAREA does not provide the entire text of the SQL statement. That's why I include the address column in the report. I can use that value to dump the entire SQL statement from V$SQLTEXT using the statement below (where xxxxxxxx is the value in the address column from the V$SQLAREA reports above):

select sql_text from v$sqltext where address = 'xxxxxxxxx' order by piece;

Oracle 10G R2 provides a new view called V$SQLSTATS that contains a combination of columns that appear in V$SQL and V$SQLAREA. The benefits that V$SQLSTATS provides are as follows:

• Since V$SQLSTATS contains the entire text of the SQL statement AND its associated performance statistics, we are no longer required to access both the V$SQLTEXT and V$SQLAREA to obtain the information we need.
• Oracle states that V$SQLSTATS is faster and more scalable.

The data in V$SQLAREA has a tendency to get its contents flushed out just when you need to get additional information from it. The V$SQLSTATS view provides users with a longer access window. That's one of the key benefits to this view.

Next Up
The series continues.

Ease of Installation
I remember the good old days. When I began my career, the Oracle installers were just screen painters that forced you to use the TAB key to navigate back and forth between selections and screens.

Installations used to take hours and experiencing problem after problem was commonplace. Successfully installing Oracle became a rite of passage as a DBA. Four hours after you started the installation (and fighting through errors, warnings and placing a half dozen CDs in the reader), you would finish up and think to yourself "I am an installation GENIUS!". You next thought would be "I hope I don't have to go through this again for a while."

With each new release, the installer became easier to use, installation speeds increased and the number of installation problems decreased. In a few years, I'll be telling new DBAs "You don't know how good you have it. I remember when installations used to take three days... We were real DBAs back then. We even had to log in at NIGHT TIME and take the data OFFLINE to do work." Darn kids don't know how good they have it nowadays….

We do DOZENS of Oracle installations a month here at Remote DBA Experts. One of the benefits of working here is the sheer thrill of working on virtually every operating system that Oracle makes a product on. Name the O/S and hardware combination and we can truthfully say "been there and installed an Oracle product on it."

When our first set of 10G CDs arrived a couple of years ago, the DBA opened up the package and said "Hey, where's the rest of them? I think they forgot to send me all of the CDs I need to do the install." We checked MetaLink and found that Oracle10G only needs a couple of CDs. One main CD and another called "The Companion CD" which really should be called "The Companion CD that is required for all installs" since it pretty much has to be installed when you install Oracle.

The team here really likes the fact that the installer now performs pre-installation requirement checks before laying down the software. The installer itself now determines if the O/S environment is configured to correctly run the Oracle software. Having the installer verify that you have performed "your pre-installation" duties is very beneficial.

10G Data Pump
I'm a big fan of 10G Data Pump. Once you become accustomed to the new environment, it's a pretty good utility. There is a learning curve - like the first time you try to restart a Data Pump job by just rerunning the script again. You'll quickly find out that Data Pump has a command for that. At least it provides you with the "Master table exists" error message to point you to your mistake.

Here's a quick list of some of the features I like:

• The utility IS faster than Export and Import. Maybe not as much faster as Oracle says it is - but it is faster. I have personally seen Exports and Imports run for hours - and days.
  
  
• I like being able to use parallelism to improve performance.
  
  
• You can stop and restart jobs. How great is that? The master table knows where you left off so you don't have to spend all of that time trying to code a new Import statement that won't duplicate rows in tables that were successfully loaded. This is really helpful.
  
  
• How many times did you have to "pick a number, any number" when you had to determine how much space you would need for your Export Dump file? Data Pump provides an option that estimates the output dump file size that would be created by a Data Pump Export.
  
  
• One of my favorite features is the network transfer option that allows administrators to attach to any source database that can be accessed from the network, transfer the data and write it to the target database. No dumpfile required! Administrators use the NETWORK_LINK parameter to specify a pre-created database link that points to the source database.
  
  
• You can use the Data Pump Flashback feature to go back in time and export data how it looked in the past. This helps when you get a visit from one of your developers stating "I think I might have updated 47 production tables by mistake, but I won't know until I see the data." Use the flashback feature during the Data Pump Export and use the Data Pump Import utility to pump the data into a new schema for the developer to review.
  
  
• You can use the EXCLUDE parameter to exclude specific objects using custom syntax. For example, EXCLUDE=INDEX:"LIKE 'DEPT%'" would exclude indexes that start with the name "DEPT".

The more I thought about the Data Pump features I liked, the longer I thought this blog would be. So here are links to three blogs I wrote on Data Pump.

• Data Pump Part 1 Features and Benefits - A general overview of the 10G Data Pump Utility.
  
  
• Data Pump Design and Architecture - We dig deeper into inner-workings of the Data Pump utility.
  
  
• Data Pump Export - We continue the Data Pump discussion by learning how to use 10G Data Pump Export to "pump" data out of an Oracle database.
  
  
• Data Pump Import - The series finale on Data Pump Import.

10G Automatic Alerting
Oracle10G provides early warning mechanisms to alert administrators of possible error conditions. The database itself collects numerous metrics that were previously collected by 9i's Oracle Enterprise Manager. Administrators are able to select from dozens of server-generated alerts. Server generated alerts are activated by a new background process called MMON. MMON is able to access the SGA directly and perform the metrics calculations and threshold monitoring.

10G Grid Control enhances these capabilities by providing administrators with a GUI interface to administer alerts and configure notification mechanisms. Administrators are able to use Grid Control or the standalone version, Database Control to set up e-mail and pager notifications. In addition, all server-generated alerts are displayed on the Database Control/Grid Control home pages.

As readers of this blog know, I'm a huge fan of 10G Grid Control. Although the database alerts can be set up in the database using the archaic command line interface, if I can do it using Grid Control with a few clicks of the mouse, that's the tool I will use. And to think I used to call 9i OEM "a Sissy GUI Tool".

Setting up 10G's "early warning system" places you in the proactive DBA category as opposed to reactive. The term we us to describe a pure reactive DBA is "firefighter". That's the DBA or DBA team that runs from one fire to the next. I will acknowledge that we are ALL firefighters from time to time. At one of my previous jobs, I spent six months working with another DBA trying to beat an absolutely atrocious third-party vendor package into submission.

You all know the type of third-party vendor package I'm talking about. Someone from your management food chain shows up in your cube and states "Hey, even though you rated this vendor an absolute 0 in all technical categories, and flatly stated that we should not consider them as a viable alternative, we chose them anyway. The Senior VP from the business unit liked the color of their screens. I'm sure you'll be able to get it working."

You then get on 72 conference calls with the vendor after the package crushes a 6 CPU box with 10 concurrent users. Most of the conversations are with the vendor's lead technical experts who are as arrogant as they are inexperienced. Their continued mantra is "you don't understand how Oracle works, it's your fault, you don't understand how Oracle works, it's your fault….". This mantra never stops even though you show them poor SQL coding, transaction looping problems, etc…

But I digress... During my career as an Oracle instructor, one of the most common recommendations I used to provide to my Oracle students was the benefits of database monitoring. My catch phrase was "We monitor by day so we don't get called at night." By the end of the DBA I class, all I would have to say is "We monitor by day…" and the class would respond accordingly. I would then award them with a set of scripts that allowed them to do database connect checks and track datafile free space.

10G Grid Control does all that (and more) for you automatically. The set of checks range the spectrum, from agent unable to connect to target (that means either the database, server or agent broke), to tablespace freespace and alert log errors. As I stated previously, 10G provides dozens of alert choices and is your one-stop-shop for proactive database monitoring. That's a good thing. I would rather wade through dozens of alerts I don't want to activate than spend ANY time coding an alert that wasn't provided by the toolset.

One of my previous blogs titled "Configuring 10G OEM Grid Control's Automatic Alert Notification System" will help you set up proactive performance monitoring in 10G. The title says it all. It provides everything you need to activate 10G Grid Control's early warning system.

Next Up
The 10 Thinks I like About 10G Continues. Could be more than 10 maybe...

We've spent a lot of time discussing SQL statement tuning. We started off this 15 blog series with a discussion on system triage and finished with Oracle access path identification and tuning. I also dedicated a dozen blogs or so to 10G Grid Control performance monitoring and analysis tools. With the abundance of long-winded information I have made available on this blog, I thought it might be beneficial to condense some of it into a review of some of my favorite tuning toolsets.

My Number One Tuning Tool of All Time - 10G Grid Control
If we are administering a 10G database, our administration tool of choice is either 10G Database Control (non-grid, standalone version), or 10G Grid Control to administer it. The advanced alerting, monitoring and administration features make this tool our preferred method of administration. The advanced performance monitoring and analysis features make it my number one tool for performance problem determination.

The future of Oracle database tuning will be administrators interpreting and implementing the recommendations generated by the intelligent advisors and ADDM. It is a foregone conclusion that Oracle will continue to improve upon its performance monitoring and analysis toolsets. Self tuning features are no longer options that are "nice to have", they are requirements for Oracle's competitive survival. SQL Server is continuing to scale, moving into areas that were once dominated by UNIX big-iron machines running Oracle databases. Oracle must compete with SQL Server's ease of use or it will undoubtedly lose market share.

As the intelligence of the advisors and ADDM increases, the need to possess an in-depth knowledge of Oracle and the usage of detailed diagnostics to improve database performance will decrease. And you heard it here first folks, I also think that reading SQL traces and statistics dumps will be a thing of the past. My crystal ball tells me that its just a matter of time until Oracle's SQL advisors make SQL traces and statistics dumps less and less important until they become totally unnecessary.

The new breed of top tuners will be the administrators who focus on how to use the toolsets and interpret their output. Not the tuners who spend the majority of time digging down into the dark, inner workings of the Oracle software. I'm not saying that knowing how the database works is immaterial. I am stating that this intimate knowledge will become less and less important as the tools mature. It's only a matter of time until tuning is relegated to a minor sub-task that can be scheduled between other activities.

Here's a listing of my favorite 10G Grid Control tools.

Host Performance Home Page
If someone tells me "the whole system is slow", the first thing I'm going to do is review the host platform's key performance indicators. 10G Grid Control's Host Performance panel provides me with exactly the information I'm looking for - current CPU, memory and disk resource utilization.

Each resource's graphical display contains links that allow me to view more detailed statistical information. Each of the drill down panels for CPU, Memory and Disk contains a drop down menu list that I can use to view current as well as historical performance statistics for the last 24 hours, 7 days and 31 days. The main performance panel also displays information on the top resource consuming processes that are currently running on the host. Once again, a drop down menu lets me switch between top CPU and top memory consumption reports.

For a more in-depth discussion on host performance tuning, please refer to my blog titled "Host Performance Monitoring Using 10G Enterprise Manager Grid Control".

10G Grid Control Database Performance Home Page
If I want to review database performance, I will access the 10G Grid Control Database Performance Home Page, which is available in both 10G Database Control and Grid Control.

The database performance home page allows me to review performance historically, identify what sessions are dominating "finite system resources", activate advisors and drill down into the top resource consumers. I can then quickly review the SQL they are executing. Does it get any better than that?

I challenge anyone to tell me that they would be able to monitor database performance more efficiently using archaic command line toolsets. By the time you have identified the problem, I'll have already contacted the user running the SQL and created a profile to improve its performance. If I have Grid Control's alerting system configured (which I always do), Grid Control's database performance threshold alert would have already notified me that a problem was occurring. By the time you got that call from an irate user, I would have had it solved. If you want to learn how to configure 10G Grid Control's performance alerting feature, please turn to my blog titled "Configuring 10G OEM Grid Control's Automatic Alert Notification System".

To learn more about 10G Grid Control's database performance monitoring and analysis features, please turn to my blog titled "Database Performance Monitoring Using 10G OEM Grid Control".

10G Grid Control Advisors
OK, I'll admit it. I'm hooked on the advisors. At first I was very wary of their recommendations, but the longer I worked with them, the better I liked them. I have lots of experience tuning Oracle, but if an advisor is going to make my job easier (and let me spend less time debugging a performance problem), I'm all for it.

For an overview of 10G Grid Control's advisory features, please refer to my blog that is aptly titled "An Overview of 10G Advisors".

Here's a listing of blogs that provide more specific information on the individual advisors. Please note that each title below also links to a blog on that advisor.

• 10G SQL Access Advisor - The SQL Access Advisor recommends a set of materialized views and indexes based on a supplied workload input. The tool can also help administrators optimize materialized views to take advantage of fast refresh and query rewrite capabilities. In addition, the SQL Access Advisor may recommend dropping indexes and materialized views that aren't being used. The SQL Advisor ranks and groups the recommendations based on the positive impact they will have if implemented.
  
  
• SQL Tuning Advisor - The SQL Tuning Advisor, as its name implies, provides SQL tuning recommendations and includes information justifying why it generated those recommendations. The recommendations may include collecting statistics on objects, new index creation, restructuring the SQL statement or the creation of a SQL profile to create a more optimal access path. I have a more detailed blog the SQL Tuning Advisor titled "SQL Access Advisor Revisited".
  
  
• Segment Advisor - Although I don't use this advisor as much as the SQL Tuning and SQL Access advisors, I do use it on a regular basis. The 10G segment advisor identifies segments that have become fragmented as a result of update and delete operations. The 10G R2 Segment Advisor has been enhanced to identify tables that suffer from excessive row chaining and row migrations. Oracle describes these objects as being sparsely populated. Not only do sparsely populated objects waste space but they can also contribute to SQL performance problems.

Automatic Database Diagnostic Monitor (ADDM)
ADDM is a "recommendation engine" that uses performance information stored in the Automatic Workload Repository (AWR) as input. Here's a blog that provides a detailed description of the Automatic Database Diagnostic Monitor. By default, AWR snapshots occur every 60 minutes. For more information on the AWR snapshots, please turn to my blog titled "Working with Automatic Workload Repository Performance Snapshots".

To learn how to tailor the 10G AWR snapshot environment to meet your specific needs, please refer to my blog titled "Configuring and Administering Automatic Workload Repository Performance Snapshots".

After the AWR information snapshot is executed, the Automatic Database Diagnostic Monitor is triggered to analyze the information contained in the AWR for the period pertaining to the last two Snapshots. ADDM's output includes a plethora of reports, charts, graphs, heartbeats and related visual aids. In general, ADDM provides users with a top-down analysis of performance bottlenecks and their associated resolutions. But you can also execute ADDM manually to generate recommendations for current, as well as historical, time periods.

Statistical information, by itself, is often useless. If you have nothing to compare a measurement to, how will you know if its good or not so good? 10G Grid Control allows two sets of snapshots to be compared to one another. The results of the comparison are displayed graphically using horizontal bars to indicate differences between the two measurements. To learn how to perform snapshot comparison, please refer to my blog titled "You Can't Improve What You Can't Compare".

Conclusion
Take some advice from your friendly ex-Oracle instructor. LEARN HOW TO USE THESE TOOLS. I can't emphasis that statement strongly enough. As I stated, ADDM and the intelligent advisors may not currently be a total replacement for DBA experience and expertise, but sooner or later, they most definitely will be. Don't get left behind - because its only a matter of time.

Learning how to tune requires that you spend time with the optimizer experimenting with different environmental settings and database objects that affect access paths. It also requires that you have a general understanding of the basics of optimization.

I dedicated an entire blog on educational resources that will help you build a firm foundation of Oracle optimization knowledge. It's critical that we understand as much as we can about optimization before we begin our experimentation. But the key to success is to spend dedicated time experimenting. I'm not professing that you spend the rest of your career becoming "at one" with the optimizer. But if you don't want to be at a total loss each time something runs long, you need to experiment to learn how to tune.

Environmental Documentation and SQL Statement Selection
Before we begin, we'll need to document some of the key parameters that affect optimization and gather some information on the data objects we will be using in our experimentation. Please refer to my blog titled "Access Path Scientific Analysis Part I" for more information on the importance of documentation as well as the documentation process itself. In a previous blog, I provided a few hints and tips on identifying what types of statements and access paths would be most beneficial to analyze.

Access Path Identification Tools
Here are the tools that I most often use to review Oracle access paths:

• SQL Scratchpad - I like to see graphical representations of access paths. Even though I have lots of experience reading SQL Trace output and the multitude of variations of access path displays, I like seeing a graphical representation. The problem is that Scratchpad must be installed on your client (comes embedded with 9I OEM) and it doesn't like accessing 10G databases. If I am tuning a 9I database and I have access to SQL Scratchpad, that is the tool I am going to use to display access path information. For more information on SQL Scratchpad and other 9I OEM tools, please turn to my blog titled "Access Path Identification Part V".
  
  
• SQL*PLUS Autotrace - If I want to quickly display the statement's access path and associated run-time performance statistics, I will use SQL*PLUS Autotrace. Here's a blog containing a few hints and tips on Autotrace.
  
  
• Explain Plan - If I don't want to run the query, and I am not concerned about the access path changing during SQL statement execution (read next bullet), I will used the tried and true "explain plan into" clause to ask Oracle to dump the access path information into a table. I'll run a statement to retrieve the data from the plan table and format the output. Please read my blogs titled "Access Path Identification Part II" and "Access Path Identification Part III" for more information.
  
  
• SQL Trace - If I want to be absolutely sure that the access path the other tools are predicting the statement will take matches what access path is chosen during runtime, I'll run SQL Trace. It is one of the only tools you can use to determine the path chosen during execution. How can the access path change during execution? Read this blog to find out! The blog contains an example of a production problem that was caused by an access path changing during runtime. For lots of information on activating SQL Trace, please turn to my blog titled "Access Path Identification Part IV". SQL Trace is the tool I will turn to if I want to analyze a set of statements that belong to a specific online transaction or batch job. The output quickly tells me what the top resource consuming statements are.
  
  
• V$SQLPLAN - V$SQLPLAN contains access path information the SQL statement takes during execution. I'll run the statement, review the access path taken during execution, make my tuning change, alter the text of the statement look different to the optimizer, retrieve the access path information again and compare the before and after results. My blog titled "Access Path Identification Part II" provides all of the information you need when you are using the V$SQLPLAN table to retrieve information on access paths taken during execution.
  
  
• Regardless of which tool I select, I will use the output to graphically display the statement's access path. Although it is time consuming, graphically displaying a statement's access path allows me to more clearly understand the access path and simplifies the tuning process. This blog will show you how to graphically display a statement's access path.

Another Graphical Display I find Useful
During my career as an Oracle instructor, I was often asked in the SQL tuning classes how I personally documented a query I was attempting to tune. This is the display that I showed them. Let's take a look at the contents, starting from the top of the page:

• SQL statement text - the full text of the SQL statement.
  
  
• Columns in the SELECT clause - a listing of the columns that are in the SELECT clause are displayed under the table they belong to.
  
  
• Columns in the WHERE clause - a listing of the columns used in the WHERE clause are displayed under the table they belong to. If the columns are tables used in a join operation, an arrow is used to designate which of the other table's columns they are joined to. If I am interested in cardinality and data skew, I would document that information directly below the column name. For more information on determining column cardinality and data skew, please turn to this blog.
  
  
• Indexed columns - The bottom of the page contains all of the indexes that are on columns in the WHERE and SELECT clauses. This allows me to easily determine if I need to create additional indexes are begin analyzing why an index isn't being used.

Performing the Scientific Analysis
Before we begin, we need to understand the effect that preloading the buffers has on SQL statement performance. If you run the query and return the results, you will be loading the buffer cache with data. As a result, subsequent runs of the same statement could be faster. This is because your statement is retrieving data from memory while the initial runs were forced to retrieve the data from disk. You'll need to run the statement twice, make your change and run the statement twice again to ensure that buffers don't impact your timings. You can then compare the output of the second execution of each test. 10G provides a SQL statement to flush the buffer cache.

If you are building indexes during your testing, don't forget to generate statistics on the new objects. You don't want the lack of object information to affect the optimization process.

In the discussion that follows, I'll be providing you with some examples of changes to make during your scientific analysis. You will run the SQL statement, record the execution time and statistics and document the access path. You will then make the suggested alteration, review the access path to see if it has changed and run the statement again to record the new performance statistics and execution time. Please note that in the examples below, I'm not going to remind you each time to run the query to evaluate the performance statistics and execution times.

Index Testing
Let's use the same sample query I used in my graphical display in this discussion:

select a.employee_id, a.last_name, b.department_id, b.department_name,
c.street_address, c.postal_code, c.city, c.state_province
from hr.employees a, hr.departments b, hr.locations c
where a.department_id=b.department_id
and b.location_id=c.location_id
and a.employee_id = 174
order by a.last_name;

There is a myriad of different changes we can make. Here's a couple of examples to get you started:

• Drop all of the indexes on the tables and review the access path. We know it will use table scans, but what join method did it choose, how about the join order?
  
  
• Build an index on one of the join columns (i.e. dept.department_id) and review the access path. Determine if it chose the index, what join method was used and the order in which the tables were joined.
  
  
• Drop that index and build an index on the other column in the join. Identify if it used the index, join method and join order.
  
  
• Build an index on employee_id, department_id and last_name. Identify if it used the index, join method and join order. Did you get an index only access path?
  
  
• Drop the index you just created and change the order putting employee_id between department_id and last_name. What was the access path?
  
  
• Change the WHERE clause to look for rows based on the last_name and first_name columns. Remove the a.employee_id = 174 clause. Build an index on the first_name and last_name columns. What happened?
  
  
• Drop the index built on the first_name and last_name columns and build two indexes, one on first_name and the other on last_name. Did the optimizer use both indexes? Which one did it choose?
  
  
• Add local predicates to the other tables and identify the access path changes.
  
  
• Identify a column in one of the tables that has low cardinality. Change the WHERE clause to access that low cardinality column. If you don't have a low cardinality column available, its pretty simple to add a column and update it with a few repeating values to create a low cardinality column. Build a bitmap index on the low cardinality column. Did the optimizer choose the bitmap index? What happens if you use a hint?
  
  
• Build a bitmap index on a column that has high cardinality (i.e. employee_id). If it doesn't use the bitmap index, use a hint to influence the optimizer to choose it. Drop the bitmap index and build a B-tree index. Does the optimizer use that index without the hint? Compare before and after execution performance statistics and timing.
  
  
• Identify columns that have skewed data. Build an index on the skewed column. Change the query to search for a value that occurs many times. Compare that execution to a search on a value that appears few times. Analyze the table and build a histogram. Re-execute the tests and compare the results.

Parameters that Affect Optimization
We also need to learn how different parameter settings affect optimization. We can start by adjusting a few of the basic parameters that have the best chance of changing the access path for a given statement. Most of the parameters can be changed in your session by using the ALTER SESSION statement. Using the ALTER SESSION statement will allow you to change the parameter for your session only. You won't have to worry about affecting other folks executing statements in the same database that you are using for your testing.

Here's a blog that provides a listing of parameters we can use during our initial testing. Please note that this is not an all inclusive list. It is a listing of parameters that I feel have the greatest chance to influence the optimizer to choose a different access path.

You need to execute the statement, change a parameter and note its affects on the statement's access path, performance statistics and runtimes. If you are going to test changing values for different parameters, don't forget to change the one you just tested back to its original setting. That way you won't have multiple changes impacting your test results. You can also test combinations of changes. Just be aware of the changes you are making.

Hints
Administrators embed hints in a SQL statement to influence the optimizer to choose a particular access path. Using hints will allow us to evaluate the affect that different access paths have on SQL statement performance. We will run the statement without any modification, review the access path and performance statistics, use a hint to (hopefully) change the access path, run the statement again and compare the before and after results. Since hints can be embedded in virtually any SQL statement, they will provide us with an easy mechanism to learn more about access paths. I have devoted a previous blog to hints. The blog also contains a demo document that will show you how to embed hints and the affects the hints have on access paths. One of the key tests is to determine the impact that operation cardinality has on SQL statement performance. You can test this by using the ORDERED hint to ask the optimizer to change the join order for statements that join more than two tables together.

Conclusion
The tests above will hopefully stimulate your creativity and allow you to generate your own test cases. As we become more experienced we'll need to review subselects, views, complex selectivity issues (and the list goes on and on…) Oracle performance tuning is truly a wonderfully complex subject. The important thing is that we start the process. That's the only way we'll become database performance scientists!

Thanks for Reading,

Chris Foot
Oracle Ace

• Let's Get Technical! - Using Deductive Reasoning and Communication Skills to Identify and Solve Performance Problems
  The first blog of this series provides some helpful hints on what questions to ask during the initial stages of application performance problem analysis.
  
  
• System Triage Part II - Host Performance Analysis Using Grid Control and Host Commands
  Using operating system commands and 10G Grid Control to monitor database server performance.
  
  
• System Triage Part III - Finding the Top Resource Consumers
  We learned how to use 10G Grid Standalone to identify the top resource consumers for a given instance.
  
  
• System Triage IV - Access Path Identification Part I
  A high-level overview of Oracle query optimization. Also discussed the difference between estimated and runtime access paths.
  
  
• System Triage V - Access Path Identification Part II
  We reviewed the two data objects that contain access path raw data - plan_table and v$sql_plan. In addition, we discussed a few of the V$ dynamic performance views that provide information pertaining to SQL statements executing in our Oracle database environment.
  
  
• Access Path Identification - Part III
  A discussion on the tools we can use to display graphical and text versions of explain plan data including the DBMS_XPLAN package, UTLXPLS.SQL, UTLXPLP.SQL and Oracle 9I Scratchpad (graphical). We also learned how to create our own graphical displays of Oracle access paths.
  
  
• Access Path Identification - Part IV
  An investigation of everyone's favorite performance analysis tools - SQL*PLUS Autotrace and SQL Trace.
  
  
• Access Path Identification - Part V
  A discussion on a couple of 9I Oracle Enterprise Manager tools that we can use to identify Oracle access paths. We also learned how we can use 9I OEM's SQL Analyze and the Virtual Index Wizard to help us better understand access paths, monitor database performance and tune statements running in an Oracle 9I database environment.
  
  
• Access Paths VI - 10G Grid Control SQL Details Panels
  We turned our attention to the 10G Grid Control's access path display panel - the SQL Details Panel. It seems like no matter where you start your investigation in 10G Grid Control, sooner or later, you'll end up with a panel that contains a link to the SQL Details Panels. We took a look at a few of the more popular ways of finding our way to them.
  
  
• Access Paths VII - Access Path Education
  Focuses on the importance of Oracle access path education. The blog contains links to resources and numerous Metalink Notes that provide access path educational information.
  
  
• Access Path Scientific Analysis Part I
  An introduction to Oracle Access Path Scientific Analysis. We learn why it is important for us to become "database performance scientists" if we want to fully understand Oracle access paths and the affect they have on SQL statement performance. We also learned how to select a test system to use as well as document the parameters and database objects that play key roles in the optimization process.
  
  
• Access Path Scientific Analysis Part II
  We learn how to use hints to influence access paths for testing purposes. Using hints allows us to evaluate the affect that different access paths have on SQL statement performance. When we finally begin our access path scientific analysis, we will run the statement without any modification, review the access path and performance statistics, use a hint to change the access path, run the statement again and compare the before and after results. Contains a document that provides a demo on using hints to influence access paths. The demo document also contains a description of some of the basic access paths that Oracle can choose from.
  
  
• Access Path Scientific Analysis Part III
  An overview of the various types of indexes as well as indexing strategies that affect Oracle access path selection. Also includes a list of parameter that can influence the optimizer to favor index access.

Selecting SQL Statements to Analyze
We need to select a few SQL statements that we can use as input to our scientific analysis. Let's take a look at the various tools we can use to select the SQL statements we can use for testing.

• Using 9I Oracle Enterprise Manager - We can use the Session Details Panel to retrieve the SQL text of statements that are currently executing in the database. The Session Details Panel also provides the access path the statement is taking. The blog on 9I OEM will show you how to drill down to find the text of the SQL statement and the access path it is taking.
  
  
• 10G Grid Control - Grid Control' s Execution Plan Panel allows administrators to display the entire text of a SQL statement and the access path it is taking. This blog on 10G Grid Control will show you how to navigate through the various panels to access the Execution Plan Panel.
  
  
• SQL Trace - If we don't have access to the "SGTs" (Sissy GUI Tools), all is not lost. We can also use the SQL Trace utility to capture the text of the SQL statement as well as its access path. To use this method, we would contact one of our friendly application developers and ask them to run a series of batch jobs or online transactions in the environment we have selected as our test bed.
  
  We can then activate SQL Trace, contact the developer to run the selected workload, deactivate the trace and format the trace's output into an easily readable format. SQL Trace provides the benefit of allowing us to capture multiple SQL statement access paths in a single execution. The blog on Autotrace and SQL Trace provides instructions on how to activate the SQL Trace utility.

What Statements Should I Choose?
We are looking for statements that utilize the different Oracle access paths. Try and find statements that use:

• Table scan and index access paths:
  
  
  • Index only - Oracle is able to read all of the data required to satisfy the query's data needs from the index structure alone.
    
    
  • Index to table - Oracle uses a row identifier to probe the table to satisfy the data request.
    
    
  • Full table scan - Oracle reads all rows from the table. Oracle will scan the table to the last block used (as opposed to the last block that actually contains data).
    
    
• Join access paths:
  Used when the statement retrieves data based on matches between two tables (i.e. retrieve all of the employees that have the department name of "Welding"). The employee information is contained in the employee table and the department information (including the department name) is in the department table.
  
  
  • Nested loop join - Good path when the join is accessing a small subset of rows.
    
    
  • Hash join - Efficient access path for joins that access larger sets of data.
    
    
  • Sort merge join - Sorts rows to allow quicker access during the join.
    
    
  • Outer joins - An outer join returns all of the rows that satisfy the particular join condition and returns additional rows from one table that do not satisfy the join condition.

We don't have to find each and every access path I outlined above. In addition, each statement that does a join will also contain table scan and/or index access paths. Try to find statements that access a single table and statements that join two, three and four tables together. It should be relatively easy to find the nested loop and hash join access paths. Sort merge and outer joins may be a little harder to find. Try and stay away from SQL statements that use parallel processing, are 13 pages long or join 5 or more tables together. We want to start with the basics. We also don't want to use statements that have exorbitantly long execution times. We'll be executing them on a fairly regular basis during testing and waiting hours for the results will lengthen the testing process.

Next Up
We devise our scientific analysis testing process.

Almost Forgot
How are the Daylight Savings Time changes treating you? I may break in one last time to discuss their impact on our organization.

Thanks for Reading,

Chris Foot
Oracle Ace

The Problem
Before we begin, we need to understand exactly what the problem is. If you haven't read my previous blogs on this topic, I highly suggest you do so before continuing.

Metalink DST Information
In my first blog on DST, I provided you with numerous links to Metalink Notes that focused on Daylight Saving Time. In my second blog, I provided a link to Oracle's new Metalink support category on DST.

Updates
We have patched dozens of databases and operating systems since I wrote that first blog on DST. I've compiled a a few notes on what we have found:

Operating System Patches

Windows
We have patched all of our customers database servers that we are responsible for providing OS support for and have experienced no issues. But some of our customers have notified us that they have experienced problems when they patched their mail and application servers.

If one implementation experiences a problem, you have a tendency to chalk that one up to that specific environment. When multiple implementations experience problems, we decided to inform our entire customer base of the potential for issues. Especially when we know that several of the customers that reported problems have top-notch Windows administrators on staff.

All of the environments had test and production operating systems that were configured to mirror each other as closely as possible. The test patches worked as advertised but a few of their production environments failed. Either the patch wasn't applied successfully or the environment refused to work after the patch was applied. In one case, Microsoft support had the administrator apply the patch again using the exact same steps and it worked the second time. Go figure. Several of the other customers were asked by Microsoft support to make various changes to their O/S configuration and attempt the patch a second time.

A few of our customers experienced mail performance problems because the DST patch caused the mail engine to generate dozens (upon dozens) of meeting time updates. The DST patch forces the mail engine to change the meeting times for all meetings that were sheduled between March 11 (new DST date) and April 1 (old DST date).

I do apologize that I don't have specifics on the the other issues. I do know that each fix was specific to that environment. I know that this may seem not very helpful at this time but I think a general warning was warranted.

Recommendations
Just because all of your test patches worked, don't assume that the production environment will also be successful. When you patch the production environment, be prepared for issues that did not occur during the test patches. You can't blame Microsoft either; test and production environments that are configured "as closely to each other as possible" aren't mirror images of each other. Close doesn't count when it comes to operating system/hardware configurations. Visit Microsoft's website regularly to ensure that there is no additional information you need to be aware of.

UNIX, LINUX
Our O/S admins experienced a few minor issues with UNIX and Linux. Please remember - if the server has its own Java engine (and you run Java programs on the O/S), your best bet is to patch both the operating system and the Java engine. We did find that several O/S vendors were updating their DST patching instructions on a regular basis.

Recommendations
Don't forget to patch that Java engine too! Visit your O/S vendor's website regularly to ensure that there is no additional information you need to be aware of.

Oracle Database Patches
The biggest issue we had was with Oracle's updates to the DST patching documentation. Since we were required to do a LOT of patching, it was important for us to get a head start. In the middle of January, we download all of the DST documentation and studied the information. We then created our plan of attack and began implementing it.

We also checked the DST area on Metalink every few days for updates. Here's where the problem arose. Some of the database patching documentation (10G specifically) can be described as "somewhat fluid" in nature. Our DST coordinator, Brian "Captain Patch" Hays found that some of the changes forced him to assemble his DST team to review their impact.

Although the updated documentation did not force us to redo any patches, we did have to revisit several of the environments that we already patched to ensure that we didn't have to reapply them. In one case, the initial documentation did not provide a complete listing of DST datatypes that could be used in table definitions. We reviewed each of the environments looking for tables containing the additional DST datatypes and were pleased to find that none were in use. I can state that when we did apply the database patches (both for the database and the Oracle Java Engine), they worked flawlessly.

Recommendations
We intend to check for updates to all Oracle DST documentation until March 11. I would highly suggest that you also do so.


Oracle Application Server
Oracle's application server was the biggest challenge. Oracle recommended numerous patches for the infrastructure database to ensure DST compliance. Many of the patches required that the infrastructure database be at a specific release. Oracle also recommended that we apply DST patches to all of the target databases that could potentially be accessed by the application server. Once again, all of the patches worked.

Recommendations
If you haven't patched your application server yet, you had better get started! Review the Metalink DST documentation for updates.

Oracle E-Business Suite Applications
All of the patches worked flawlessly.

Recommendation
Review the Metalink DST documentation for updates.

General Recommendations
Visit Oracle's and your operating system vendor's websites regularly to ensure that you have the most up-to-date information possible. Ensure that your database and application support technicians are readily available during the week of March 11th. The challenge is that some of the problems that could occur may not become readily apparent. They could very well crop up days, weeks and possibly months later. Be ready.

Thanks for reading and good luck on DST. Let me know how it works out for you!

Thanks for Reading,

Chris Foot
Oracle Ace

Introduction to Oracle Indexes
Generally, the fastest way to access Oracle data is with an index. The Oracle database contains several different indexing types that are designed to provide complementary performance functionality.

While standard B-tree indexes are most effective for columns containing a high number of different values (high selectivity), bitmapped indexes are most appropriate for columns with a relatively limited number (low selectivity) of different values. The low selectivity statement above comes with several caveats. Instead of spending this entire blog on the bitmap index/low selectivity issue, please turn to Jonathan Lewis's DBAZine article titled "Bitmap Indexes Part 1". Oracle also provides function-based indexes to allow index access using SQL that contains column manipulations in the WHERE clause.

Administrators supporting large data stores use partitioned indexes to decompose large index structures into smaller, more manageable pieces called index partitions. Starting with the 8i release, Oracle places index data in the separate index partitions based on the index's partitioning key.

Before we begin our reviewing some of the more popular Oracle index types, let me provide you with a few thoughts on indexes in general.

How Many Indexes Can I Build?
This subject has always been a matter for great debate. The DBA must balance the performance of SELECT statements with their DML (INSERT, UPDATE and DELETE) counterparts. SELECT statements that return a limited number of rows from a table yet access non-indexed columns will suffer from poor performance. Conversely, if you have too many indexes on a particular table, DML statements may be adversely affected.

The DBA must take the business requirements, application processing workload and workload scheduling into consideration when determining how many indexes to build. If you compare the performance improvements an index makes on a SELECT statement to the negative affect it has on DML statements, you will find that the benefits of building the index usually far outweigh the performance drawbacks.

Indexes on columns in the WHERE clause of SELECT statements can reduce query times by minutes and even hours. The creation of additional indexes may add additional time to on-line transactions that execute DML statements. Additional indexes will have the greatest negative impact on DML statements that access a large number of rows. The more rows that are inserted, deleted or changed, the greater the negative impact will be. Traditionally, programs that process large volumes of rows are scheduled to execute during off-hours.

The DBA must also consider the needs of the business. What process is more important to the business unit - getting the data in or getting the data out? Who complains the most? Is it the business user that must wait minutes (or hours) for their transaction or report to retrieve data or the business user that is waiting an extra few seconds for their update transaction to complete? Is the nightly batch window tight on time?

The DBA will need to find out how much time the additional indexes add to programs that process large volumes of rows. In addition, the DBA must determine when these programs run. If they run at night or do not require high-performance, consider building the index.

If the transaction update performance requirements are excessive (dot com applications are one example), keep the number of indexes to a minimum. A good recommendation is to build a set of tables that have no indexes for lighting-fast update performance and move the data to historical tables (with proper indexing) during off-hours to improve retrieval performance.

Index Monitoring
Determining if an index will increase performance is a pretty straightforward process. The administrator is focusing their tuning efforts on a particular query and is able to gather the specific information necessary to assist in the decision making process.

Dropping unused indexes is also an important part of application tuning. We learned previously that indexes force Oracle to occur additional I/O every time a row is inserted or deleted into the table they are built upon. Every update of the table's columns incurs additional I/O to all indexes defined on those columns. Unused indexes also waste space and add unnecessary administrative complexity. Since unused indexes (excluding those used to enforce integrity constraints) do not add any benefits to the environment, why keep them?

Determining if indexes were being used in releases prior to Oracle9i was a time consuming and error-prone process. EXPLAIN plan and trace output could be used but there was no single mechanism that monitored index usage at the database level.

Starting with release 9i, the Oracle database simplifies the index usage monitoring process by providing the ALTER INDEX……… MONITOR USAGE command. The statement below turns monitoring on for the index SCOTT.EMPIDX while the second statement ends the monitoring session:

ALTER INDEX scott.empidx MONITORING USAGE;
ALTER INDEX scott.empidx NOMONITORING USAGE;

The V$OBJECT_USAGE table can then be accessed to determine if the index was used during the monitoring session. When the session is started, Oracle clears the information in V$OBJECT_USAGE for the index being monitored and enters a new start time identifying when the index monitoring session started. After the index monitoring session is concluded, the USED column in the V$OBJECT_USAGE table will contain the value 'YES' if the index was used during the monitoring session and the value 'NO' if it was not.

Parameters that Impact Index Usage
The parameters listed below influence the Oracle cost-based optimizer to favor or not favor index access. Please note that this is not an all inclusive list. It is a listing of parameters that I feel have the greatest chance to influence the optimizer to choose, or not choose, index access paths. We'll use these parameters during our scientific analysis.

• OPTIMIZER_MODE = first_rows or first_rows_nnnn - The optimizer chooses the best plan for fast delivery of the first few rows or the first nnnn rows. The first_rows_nnn replaces the first_rows parameter in later Oracle releases. The first_rows is available for backward compatibility. More often than not, that access path will include an index. This optimizer mode tends to favor nested loop joins over hash and merge scan.
  
  It is important to note that using this mode is not a switch. It won't change each and every table scan and hash join to index access and the nested loop join method. The optimizer, at times, will favor index access and nested loop joins. The reverse goes for the all_rows optimization mode below.
  
  
• OPTIMIZER_MODE = all_rows - The optimizer chooses the best plan for fast delivery of all of the rows that queries return. The optimizer may decide to choose a full table scan over index access and hash joins instead of nested loop.
  
  
• OPTIMIZER_INDEX_COST_ADJ = xxxx - This parameter lets you tune the optimizer to be more or less index "friendly." It allows the administrators to influence the optimizer to make it more or less prone to selecting an index access path over a full table scan.
  
  The default for this parameter is 100. This setting tells the optimizer to evaluate index access paths at the regular cost. Any other value makes the optimizer evaluate the access path at that percentage of the regular cost. For example, a setting of 50 makes the index access path look half as expensive as normal.
  
  
• OPTIMIZER_INDEX_CACHING = xxxx - You set this parameter to a value between 0 and 100 to indicate the percentage of the index blocks the optimizer should assume are in the cache. Setting this parameter to a higher value makes the indexes on the inner table of a nested loop joins look less expensive to the optimizer. The end result is that the optimizer may for favor a nested loop join using an index.
  

Index Affects on Access Paths
If a table only contains a few hundred rows, queries may run faster if the optimizer chooses to read all of the blocks in the table as opposed to using an index. The I/O generated traversing the index blocks to get to the table row entries would be higher than if Oracle read just the blocks allocated to the table being accessed.

What if we access a larger table using a column with poor selectivity? Selectivity describes the number of different values stored in a column. Poor selectivity means that the column has many values that are the same. If our statement contains a WHERE clause that searches for a column value that is contained in 90% of the table's rows, it is best that Oracle, once again, read each and every row in that table. Conversely if a WHERE clause searches for a column value that appears in 1% of the table rows, it would be beneficial for the optimizer to choose an index.

Here are some examples of index access paths that we will see during our testing:

• Index only - Oracle is able to read all of the data required to satisfy the query's data needs from the index structure alone.
  
  
• Index to table - Oracle searches through the index structure looking for the key value(s), then uses row identifiers to probe the table to satisfy the data request.
  
  
• Index unique scans - The SQL statement accesses an index using a column (or columns) that are defined in a unique or primary key index with an equality condition.
  
  
• Index range scans - Oracle scans a set of entries in the index to satisfy a query.
  
  
• Index skip scans - Oracle is able to break down a multi-column index and view them as smaller subindexes. This is achieved by Oracle "skipping" the leading columns in the index and using columns that appear later in the index's definition.
  
  
• Full scans - Oracle scans all of the index entries. Kind of like a tablescan on an index. Oracle drops down to the leaf blocks and traverses the leaf blocks using the leaf pointers.
  
  
• Fast full scan - Oracle uses multi-block reads to read both leaf and non-leaf blocks. Non-leaf (branch blocks) are discarded.

But indexes influence more than just index-related access paths. Indexes can also impact the type of join operations used. Here's an "over the top" example to clarify. If you are joining two tables together in a SQL statement on join columns that have good selectivity (returns relatively few rows compared to the table size), Oracle will favor index access paths and a nested loop join. If you don't have indexes on the join columns, Oracle may choose table scans using a hash join instead of the nested loop join method.

Index Types
Let's continue our discussion by reviewing some of the more popular types of indexes: B-Tree, Bitmap and Function.

B-Tree Indexes
A traditional B-Tree index stores the key values and pointers in an inverted tree structure. The key to good B-Tree index performance is to build the index on columns having a lot of different values. Oracle describes this as "good selectivity" Oracle is able to quickly bypass rows that do not meet the search criteria when searching through indexes built on columns having a high degree of selectivity.

Bitmap Indexes
As I stated previously in this blog, there is a sense of confusion about bitmap indexes and the benefits they provide to columns with low selectivity. Instead of spending this entire blog on the bitmap index/low selectivity issue, please turn to Jonathan Lewis's DBAZine article titled "Bitmap Indexes Part 1".

The optimizer can be stubborn at times. It can be particularly stubborn when you want it to choose a single bitmapped index for an access path. A single bitmap index may not be chosen at all. The optimizer will be more inclined to choose bitmapped indexes as an access path if it can use multiple bitmapped indexes simultaneously. That's where the benefits of bitmaps are realized.

Bitmap Indexes and Concurrency
Anyone accustomed to database programming understands the potential for concurrency problems. When one application program tries to update data that is in the process of being changed by another, the DBMS must sometimes forbid access until the modification is complete in order to ensure data integrity.

Each entry in a B-Tree index entry contains a single ROWID. When the index entry is locked during an update, a single row is affected. A bitmap lock affects a range of entries which could have a negative impact on other transactions attempting to update rows already locked.

Locking issues affect data manipulation operations in Oracle. As a result, bitmapped indexes are not appropriate for OLTP applications that have a high level of concurrent insert, update and delete operations. Concurrency is usually not an issue in a data warehousing environment where the data is maintained by bulk loads, inserts and updates.

Function Based Indexes
Oracle 8i solved an indexing problem that had been affecting database performance for close to a decade. Before Oracle8i, any SQL statement that contained a function or expression on the columns being searched on in the WHERE clause could not use an index.

For example, the statement:

SELECT * FROM employee_table
WHERE Upper(first_name) = 'CHRIS';

would not use an index. A full table scan would be required to retrieve the desired result set. We now know that we are able to use B-tree and bitmap indexes to speed query performance.

In Oracle8i and later releases, we are able to build both bitmap and B-tree indexes on columns containing the aforementioned functions or expressions. The following index could be used to increase performance of the query:

CREATE INDEX upper_first_name on employee_table (upper(first_name));

Function based indexes will affect the performance of DML statements that manipulate the columns contained in the function based index. The more complex of the expression used, the more time the database will require to update the index.

Wrapup
The intent of this blog was not to provide you with an all-inclusive education on Oracle indexes. There are folks that spend dozens of hours learning about access paths and the affects that indexes, hints, statistics and parameters have on Oracle optimization. We are laying the groundwork to begin our own scientific analysis of Oracle access paths. As I stated previously, we need to begin experimenting on our own to fully understand the Oracle optimization process. In my next blog, we'll use the information we learned in this series to start our experimentation.

SQL Hints
Administrators embed hints in a SQL statement to influence the optimizer to choose a particular access path.

By using hints, you are telling Oracle that your access path is better than the one the optimizer is choosing. It’s a safe assumption that most of us aren’t as smart as the optimizer. Let it make the choice, unless you are certain the optimizer is choosing the incorrect access path.

But what happens if the optimizer is making incorrect decisions? Before you begin adding hints to SQL or freezing access paths using Optimizer Plan Stability or 10G Profiles, consider taking the following steps first:

• Determine if it is actually an incorrect access path that is causing the performance problem. It may be some external influence affecting the SQL (hardware, workload, and so on).
  
  
• Identify and review the SQL taking the bad access path for proper SQL coding techniques.
  
  
• Verify that statistics have been generated on the tables and indexed columns. The Oracle-supplied utility DBMS_STATS is currently the preferred method of collecting statistics.
  
  
• Review parameters that affect SQL optimization (optimizer_mode, optimizer_index_cost_adj, optimizer_index_caching, optimizer_dynamic_sampling, optimizer_features_enable, optimizer_max_permutations).
  
  
• Investigate system statistics. Is it activated? Is it configured correctly if it is activated? Should it be activated?
  
  
• Does the application use bind variables? If so, investigate bind peeking quirks.
  
  
• Check for skewed data. Consider using histograms to compensate.
  
  
• Go to Metalink and review optimization bugs for your release. Oracle could have already identified your issue and fixed it.
  
  OK, so you have performed all of the actions cited previously and you find that the optimizer is actually making an incorrect decision. Regardless of what some industry pundits may tell you, the optimizer is NOT infallible; it can make mistakes. Oracle created hints for a reason, and wouldn’t have made them public if it didn’t think we really needed them from time to time. If you are forced to add hints to the query to improve its performance, do so intelligently and judiciously.

Using Hints to Compare Oracle Access Paths
OK, now that I have provided you with my standard warning on hints, the intent of this blog is to learn how to use hints to influence access paths for testing purposes. Using hints will allow us to evaluate the affect that different access paths have on SQL statement performance. We will run the statement without any modification, review the access path and performance statistics, use a hint to (hopefully) change the access path, run the statement again and compare the before and after results. Since hints can be embedded in virtually any SQL statement, they will provide us with an easy mechanism to learn more about access paths. We are on our way to becoming database performance scientists!

We'll begin our access path scientific analysis by using a very basic set of hints to influence the optimizer to choose a different access path. The hints I will be using in my introductory demo are:

• Hints for optimization mode - We will be asking Oracle to optimize the statement using different optimization goals. Since we are using Oracle9i for my demo, we'll be asking it to use choose, first_rows, all_rows and rule.
  
  
• Hints for access paths - Access path hints ask the optimizer to choose the access path it recommends. We'll be asking Oracle to use an index that it didn't choose in the original access path it generated. We'll also be asking the optimizer to choose a full table scan instead of using an index.
  
  
• Hints for join operations - Oracle provides several different join methods for statements that join one, or more, tables together. We'll ask the optimizer to choose nested loop, merge scan and hash joins.
  
  
• Hints for join order - Oracle only joins two tables at a time. If multiple tables are joined, join order also describes the overall order of the tables being accessed. Oracle will join two tables and create an intermediate result set which is then used as input to the next join.
  
  Join order plays a significant role in query performance. Both in the outer and inner tables selected and the overall join order. In general, you want to reduce the number of rows processed as soon as you can during the processing of a given SQL statement. The sooner you can reduce the number of rows being sent to future operations, the faster the query will usually run. We'll ask the optimizer to choose different join orders to determine the impact it has on SQL performance.

The bullets above are just a subset of all of the hints that are available. For a complete listing (and their definitions), please refer to the Oracle Database Performance and Tuning Guide for your release. You'll find the documentation on Oracle's Technet website.

Recommended Toolsets
The two tools that I most often use to review SQL performance during my own scientific analysis is SQL*PLUS Autotrace and SQL Trace. Please refer to my blog titled "Access Path Identification - Part IV" to learn more about using SQL*PLUS Autotrace and SQL Trace utilities to evaluate SQL access paths and SQL statement performance.

Demo Document
The intent of this demo is not to train you to identify which access path is most optimal for a given situation. Its intent is to help you gain experience interacting with the Oracle optimizer. Take it from your friendly ex-Oracle instructor, spending time experimenting with the optimizer and analyzing the performance statistics that different access paths generate is critical to your tuning education. There really is no substitute for time spent "in the seat" performing your own scientific analysis on query optimization.

The demo will show you how to use hints and the ALTER SESSION SQL statement to influence the optimizer to take a different access path than it normally would. You can then compare the the access paths and their associated performance statistics to obtain a better understanding of what accces path is best for your test queries.

Here's a link to the demo document. It is in Word format for ease of reading.

In my next blog, we'll discuss what SQL statements to use for your testing and what to look for when you compare the results.

Thanks for reading.

Chris Foot
Oracle Ace

I have found that most good tuners share a common set of traits. They are inquisitive by nature, understand that there is no substitute for experience and dedicate lots of time performing scientific analysis on SQL statement and database performance.

One of my favorite tuning gurus is Jonathan Lewis. Jonathan used terms like "scientific analysis" in his DBAZine podcast. It's really a very appropriate description of his activities. After reading many of his works, I would describe him as a database performance scientist. He identifies something he wants to learn more about, creates a test environment, executes a series of tests, analyzes the data and provides documented results to reienforce his theories.

That's what we all MUST DO to expand our knowledge on the Oracle optimization process. We need to become database performance scientists. We do that by creating a test environment, running and documenting performance baselines, changing the environment to influence the statement's access path and documenting the results.

We'll begin this series by learning how to select a test environment and document it. In upcoming blogs, we'll discuss the tools we will use to measure and compare our results, select a set of SQL statements to use for our testing, create our test cases, execute the tests and document the results.

Setting up a Test Environment
Running test cases to identify the affects that different database environmental settings have on a statement's access path is not as monumental as it may seem.

Most production databases have a test counterpart. Select an environment that is actively used by your application developers. Don't worry, if one of our test causes a statement to "run longer than anticipated", we can use the trusty ALTER command to kill our session. We can also run our workloads during lull times. Lastly, the majority of changes we will make to influence the statement's access path will be done in our own session. We won't be making changes that affect the entire database environment.

I prefer to use an active test environment because it allows me to easily select SQL statements to use as test cases, the data is usually more representative of what will be found in production and the developers will most likely have a firm understanding of the active workload being executed.

You need to talk to the developers who are responsible for running workloads on that test environment to ensure that the changes being made to the test data doesn't skew your results from one test execution to the next. You can't run a test, have a load insert another 100 thousand rows in the table, run another test and expect to have a good comparison. You want your test bed to be active but not so active that it complicates your testing process or causes your tests to generate incorrect results. You'll need to use common sense when selecting the test environment as well as determining the most optimal time to run your test cases.

We also want to choose a test environment that is providing adequate performance for the workloads being executed upon it. We really don't want to use an environment that isn't performing well to begin with.

There's a myriad of options available to you. If you don't want to impact any of your test environments, create a test environment of your own. Clone one of your test databases to a sand-box environment if you can.

You'll also want to make sure that statistics are up to date on the database you will be using as a test bed. If you are running 10G databases, the database will run statistics jobs for you automatically (isn't 10G great?). If you don't have statistics run automatically, it will be up to you to analyze the data objects to ensure that the statistics optimally represent what is stored in the data structures.

Documenting the Parameters That Affect the Optimization Process
The next step is to document the environment. There are a couple of dozen parameters that affect optimization and SQL statement performance. To begin, we are going to choose the basic parameters that are easy to change and have the biggest impact on optimization. These are not all of the parameters that can influence the optimization process, just the ones that are easy to change and provide the best chance of successfully achieving an access path change.

It is important that we read the documentation beforehand for these parameters for the specific Oracle release that we are using as our test environment. We know that each Oracle release may contain enhancements to these parameters that change the affect they have on the optimization process and how we alter them to different values.

We'll want to document the following parameters to begin our scientific analysis:

• cursor_sharing - For our first set of initial tests, we'll hardcode values in our selection criteria to ensure that our statements aren't affected by cursor sharing. To learn more about cursor sharing and the impact it has on statements using bind variables, please turn to my blog titled "System Triage IV - Access Path Identification Part I"
  
  
• db_file_multiblock_read_count - The number of blocks that Oracle will read in one I/O when performing a sequential scan of data (i.e. table scan). We also need to note that if we are using Oracle 10.2 as our test environment, the database itself may adjust this parameter dynamically if it identifies that the buffer cache is being flooded by blocks being retrieved using a table scan.
  
  
• optimizer_features_enable - This parameter allows you to make the optimizer behave as it would for a specific release of the Oracle database. You set the value to a release identifier (the listing of the optimizer releases that you can set this parameter to is provided in the Reference Manual) and the optimizer will act as it would if the database were at that release.
  
  
• optimizer_index_caching - Administrators will often set this parameter in conjunction with optimizer_index_cost_adj to influence the optimizer to use more indexes and Nested Loop joins. Setting this parameter makes Nested Loop joins look less expensive when compared to Hash or Sort-Merge joins.
  
  
• optimizer_index_cost_adj - Allows the administrator to make index access more, or less, costly to the optimizer. The default value of 100 tells the optimizer to evaluate the index using the regular cost. The lower you set this value, the less costly the indexes will look to the optimizer. Conversely, the higher you sent this parameter, the more expensive the indexes will look.
  
  
• optimizer_mode - Sets the approach the optimizer will use when analyzing queries. Since there have been a few changes made between Oracle 9i and Oracle10g, I'll provide information on both sets.
  
  

  Oracle 9i provides the following settings:
  

• Rule - The optimizer will use the rule based approach during analysis. The rule based optimizer does not use data statistics as input when generating the access path. Each access path is assigned a numerical ranking. The rule based optimizer chooses the access path that has the most favorable numerical ranking. The rule based optimizer has been superceded by the cost based approach. There are a few cases where I have seen the rule based optimizer choose a better access path than the cost based method - but not many.
  
  
• Choose - If optimizer_mode is set to choose, the optimizer is able to switch between rule and cost-based optimizations. When optimizer_mode is set to CHOOSE, the optimizer uses the all_rows cost-based approach for a SQL statement if there are statistics in the dictionary for at least one table accessed in the statement. If you generate statistics on one table, every query that accesses that table will use the cost-based optimizer. What happens if other tables in the query do not have statistics collected? The optimizer will make an educated guess on the statistics for those tables. The problem is that Oracle isn't always a good statistics guesser and the end-result is a "less than optimal" access path. To learn more about the affect that statistics have on cost based optimization, please refer to my blog titled "System Triage IV - Access Path Identification Part I".
  
  
• first_rows - Influences the optimizer to choose an access path that minimizes response time. Most often used for online transaction processing systems that return small result sets. The optimizer favors Nested Loop joins and index access.
  
  
• all_rows - Influences the optimizer to choose an access path that minimizes total execution time. Most often used for decision support and data warehouse environments. The optimizer tends to favor full table scans, Hash and Merge-Scan joins.
  
  

  Oracle10g settings for optimizer_mode:
  

• first_rows - The same influence on the optimizer as it did in Oracle9i.
  
  
• first_rows_n - Where "n" = 1, 10, 100, 1000. Influences the optimizer to optimize queries to provide the fastest response when returning the "n" number of rows. Acts as a throttle, which allows you to better balance the optimization process.
  
  
• all_rows - The same influence on the optimizer as it did in Oracle9i.

We will use the ALTER SESSION SQL statement to alter these parameters during our scientific analysis on the affects they have on the Oracle optimization process.

Documenting Our Test Tables
After we document some of the parameters that affect optimization, let's turn our attention to documenting the tables we will be accessing. In my next blog, I'll provide you with a few hints and tips on how to select or create SQL statements to use in your test cases but let's continue our discussion on documentation.

The following information will provide you with a good base of information on the data objects our statments will be accessing. Since we are just beginning our scientific analysis, we'll use basic storage objects (tables and b-tree indexes). We'll discuss some of the more complex objects (bitmap indexes, partitioning, etc.) in later blogs.

• Row counts for all tables that are accessed by our test queries. Can be found in the num_rows column in dba_tables if we have statistics generated for our tables.
  
  
• Number of blocks the table is using. Can be found in the blocks column in dba_tables if we have statistics generated for our tables.
  
  
• Index listing for all indexes on our tables. The query below will provide you with a listing of indexes for a given table:
  

  select b.index_name, b.column_name, a.uniqueness, b.column_position
  from sys.dba_indexes a, sys.dba_ind_columns b
  where a.index_name = b.index_name
  and a.table_owner='&table_owner' and
  a.table_name = '&table_name'
  order by b.index_name, b.column_position
  /

• Once we find all of the indexes and columns in those indexes, let's check them for both selectivity and data skew. Selectivity is the number of unique values for a given column. Skew means that some values can occur a few times in a column while other values can occur many, many times. Since data skew will affect optimization, that information will also be important to us.
  
  We can easily find the selectivity for a single or multi-column index by accessing the distinct_keys column in our dba_indexes table if we have statistics generated. For multi-column indexes, we will want to check the individual selectivity for each column in our multi-column index. We can do this with the following query:

  select count (distinct index_colname) from owner.table_name;
  
  Where index_colname is one of the columns in our multi-column index,owner is the table owner and table_name is the name of our table. We'll need to do this for all columns in our multi-column indexes.

  We'll use the following queries to identify data skew and determine if any histograms are on our tables.

  select index_colname, count(*) from owner.table_name
  group by index_colname;

  Where index_colname is the column name, owner is the table owner and table_name is the name of our table. We'll need to do this for all columns in our indexes. We'll do this for both single column and multi-column indexes.

  select * from dba_tab_histograms where owner='&owner' and table_name='&table_name';
  

  Where owner is the table owner and table_name is the name of our tables. We'll need to do this for all tables that our queries will be accessing.

Coming Up Next
We'll continue our discussion in the next blog when we review the hints we will be using to influence the optimizer's choice of access paths. In addition, we'll discuss the various tools we will use to measure and compare our results.

Thanks for Reading,

Chris Foot
Oracle Ace

Important New Information
In that last blog, I provided you with numerous links to Metalink Notes that focused on Daylight Saving Time. Since then, super-DBA Mark Shore informed me that Oracle has created a new DST support category in Metalink's Knowledge Base.

It's pretty easy to find the new DST support area. After you log on to Metalink, you'll notice that Oracle provides a list of navigation tabs at the top of each page.

Click on the second navigation tab from the left titled "Knowledge". Metalink will respond by displaying the Knowledge Browser home page. Scroll half-way down the page until you find the section titled "New in the Knowledge Base."

At the end of that paragraph, there will be a link titled "USA DST 2007 Category". If you click on that link, Metalink will respond by displaying the Daylight Savings Time (DST) USA 2007 Support Category home page.

Daylight Savings Time (DST) USA 2007 Support Category
The new support category in Metalink's Knowledge Base contains links to ALL of the notes on Daylight Saving Time. It is your one stop shop for DST information. If you scroll down to the Topic Listing Table, you'll see that the first note (Note:412789.1) provides information on Oracle DST Webcasts.

Oracle DST Webcasts
We have already attended the DST Webcasts and have found them to be very informative. Oracle provides DST Webcasts for all of their products that are affected by the 2007 DST changes:

• Webcast 1 - Oracle Database and Client. Provides information on the Oracle database, Oracle client software, Oracle Enterprise Manager and the Oracle JVM.
  
  
• Webcast 2 - Oracle E-Business Suite Applications. Provides information on all of the components affected by DST including the data server and the middle-tier services, such as Forms, Concurrent Manager, Apache, JServ and the JVM. The presenter will tell you which components are affected, which aren't, and what to do to prevent the 2007 DST changes from becoming a problem.
  
  
• Webcast 3 - Peoplesoft Application. Describes the changes to Peoplesoft that will be required to support the 2007 DST changes.

The remaining DST Webcasts combine the Application and Database DST Webcasts into a single presentation. All of the Webcasts begin with a presentation on DST and finish with a Q&A session. We found the Q&A sessions to be just as helpful as the presentations.

If you haven't attended one of the DST Webcasts, all is not lost. There are a few dates and times still available. In addition, Note:412789.1 contains information on how to replay prior DST Webcasts. Oracle also provides you with a couple of different options to choose from. You can replay a prior DST Webcast by accessing it straight from your browser or you can download it to your PC and replay it at your leisure. I highly suggest that you replay the DST Webcasts or attend one of the future sessions.

Our Status
I thought it may be beneficial to provide you with a status on our own internal 2007 DST project:

• Since we support both the database and E-Business Suite applications, we have created two teams that focus on 2007 DST changes. Each team member is required to become an expert on the impact that the changes will have on the components they are responsible for. This includes identifying what needs to be patched, learning the patching process and coordinating patching efforts with their fellow technicians and our customers.
  
  
• All of our technicians have read the Metalink Notes pertaining to DST and have attended one, or more, of the DST Webcasts.
  
  
• We executed the scripts provided in Metalink Note:412971.1 and have determined that none of the databases we are responsible for supporting use the TIMESTAMP WITH TIMEZONE datatypes.
  
  
• We executed the scripts provided in Metalink Note: 397770.1 to identify all of the databases that have the Oracle JVM installed. We will be applying the JVM patch to all Oracle 8.1.7, 9x and 10x databases that support applications using Java programs.
  
  
• We will be applying both the timezone and JVM patch to all 10G databases.
  
  
• We are patching all Oracle E-Business Suite applications.
  
  
• All Grid Control databases will have both the JVM and timezone patch applied.
  
  
• We have sent notifications to all customers to review all third-party applications, middle-tier products, operating system and external Java engines to ensure that they are 2007 DST compliant.

Your Status?
Twenty some years ago, Craig Mullins and I had the good fortune of being trained and mentored by someone who I would describe as being one of the top managers in the database profession.

We once had to revert to a previous backup in DB2 because an operator had allowed one of the current backup tapes to be overwritten. Since DB2 is intelligent enough to go back to a prior version without problem, the impact on the recovery was minimal. He called me to his office and asked "Who is ultimately responsible for the availability, reliability and recoverability of the databases here?". I stated that I was. He then calmly declared "Then you should understand that since you are ultimately responsible, you are also responsible for ensuring that there are no weak links in the chain."

I continue to live by that rule. Its blatantly obvious that we need to ensure that our databases, administration toolsets and applications are 2007 DST compliant. But we also need to play a dominant role in our organizations to ensure that all of the components (operating systems, third-party applications, middle-tier products, external Java engines, etc.) that interact with our databases are also 2007 DST ready. I am confident that my organization has done everything we can to ensure the transition is smooth.

My challenge to you is "Is your shop ready for 2007 DST?.

Thanks for Reading,

Chris Foot
Oracle Ace

To improve energy conservation, the Energy Policy Act of 2005, Pub. L. no. 109-58, 119 Stat 594 (2005) changed the start and end dates of Daylight Time. Starting in 2007, Daylight Time begins on the second Sunday in March and ends on the first Sunday in November. That means that in 2007, Daylight Time begins on March 11 and ends on November 4. In 2008, Daylight Time begins on March 9 and ends on November 2.

Impact on the Oracle Database Ecosystem
Most computer systems have been configured to accept the Daylight Time changes that have been in effect since 1986. My personal desktop computer pops up a box in April and October to let me know that it needs to change the date to adhere to the Daylight Time policy.

If I don't apply the recommended Microsoft fixes, my PC will have the incorrect time for the duration of the new DST 2007 change periods (March 11, 2007 - April 1, 2007 and between October 28, 2007 and November 04, 2007). I'm assuming that if I manually change the time and don't apply the fix, I'll wake up one morning to find that my PC operating system made another change for me using the old Daylight Saving Time dates. One way or another, my PC's date will be off.

You understand how this could have a pretty dramatic effect on virtually every database ecosystem that we support.

Warnings and Recommendations
I am by no means a 2007 DST impact expert. Personally, I find a lot of the documentation provided by the vendors on this subject to be vague and confusing. I'm providing you with links to assist you in your own education.

We are taking the safe route. When in doubt, we are asking the vendor detailed questions to clarify their statements. Our feeling is that this is what we pay support for - to get answers to questions. Many of the vendors responses will tell you to "play it safe" and patch the systems.

Operating Systems and Java Engines
If your organization does not apply the operating system patches to correct the timezone change, the timestamps those systems use will probably be incorrect by one hour for the duration of the new DST 2007 change periods (March 11, 2007 - April 1, 2007 and between October 28, 2007 and November 04, 2007). That means any dates and times that the database pulls from the operating system for SYSDATE and DATE will be incorrect. In addition, the operating system's Java engines are also affected because they use their own embedded time zone information.

Recommendation - Visit your operating system provider's website to determine what patches are required for the operating system and the Java engine.

Oracle Database Impact
Oracle is providing two patches, one for the JVM and one for the time zone files. Each patch has a different set of requirements on when to implement it.

Oracle JVM
Like its operating system counterpart, Oracle's embedded Java engine uses embedded time zone information and is impacted by the new Daylight Time start and stop dates.

Here's a snippet from one of Oracle's white papers:

JVM patch - for all Oracle versions 8.1.7, 9.x, 10.x (database only):
If you have the Oracle JVM installed you are advised to apply the JVM patch. See section "JVM Fixes" below for more details.

The problem is that a lot of shops install the Oracle JVM when they install the database but don't run Java programs in Oracle. I'll provide you with a Metalink Note that will provide you with instructions on how to determine if the Oracle JVM is installed and what to do if it is.

Time Zone Files
Depending on the release, there are Oracle datatypes that will be affected by the Daylight Time changes. Oracle states that the TIMESTAMP and DATE datatypes do not have time zone knowledge. But the 10G TIMESTAMP WITH LOCAL TIME ZONE (TSLTZ) and TIMESTAMP WITH TIME ZONE (TSTZ) datatypes and the TZ_OFFSET function take their time zone information from Oracle's time zone files. These datatypes can be used as column types or as PL/SQL types.

The time zone files are installed during the database installation. Oracle states that the version 3 and higher time zone files include the 2007 Daylight Time changes. I'll provide you with a Metalink Note that will show you how to determine what version you have.

Here's the killer, Oracle is stating that if you use the affected datatypes, you will be required to patch the database and all clients that need to access those datatypes. Once again, read the notes I am providing you and make your own decision on what to do.

Oracle Grid Control and Agents
Oracle Grid Control agents must have their times synchronized with the operating system. If not, the agents will not start. Both the O/S and agents must be patched or not-patched in order for Grid Control Agents to work. Oracle has supplied a manual work-around in case of emergency. The repository databases must also be patched. We have asked Oracle to provide us with clarification on the 9i OEM agents. We are patching all 9i OEM and 10G Grid Control repositories.

Oracle E-Business Suite Applications and Application Server
I have included an Oracle Metalink noted that discuss the impact the Daylight Saving Time changes have on Oracle applications and the Oracle Application Server.

Recommendation for all Oracle products - Visit the Metalink website, use the Metalink notes I provide as a starting point.

Third-Party Applications
There are numerous applications that will require a patch in addition to the required operating system patches.

Recommendation- Your organization will need to contact all third-party applications to determine the impact that timezone changes have on your applications.

Oracle Metalink Notes
The following Metalink Notes will provide you with a good starting point to begin your 2007 Daylight Time changes education. Good luck!

Start your education by reading these notes:
Note 359145.1 - Impact and overview of 2007 USA daylight saving changes on the Oracle database.

Note 402742.1 - USA 2007 DST Changes: Frequently Asked Questions for Oracle RDBMS. Probably the most beneficial of all the overview documents.

Note 397281.1 - USA 2007 Daylight Saving Time (DST) Compliance for Oracle Server Technologies Products.

Generic time zone information:
Note 357056.1 - Impact of changes to daylight saving time (DST) rules on the Oracle database.

Note 340512.1 - Timestamps & time zones - Frequently Asked Questions.

Note 396906.1 - Patches to Update the Time Zone Transition Rules.

Note 412160.1 - Updated Time Zones in Oracle Time Zone File patches.

Installing and using the utltzuv2.sql script:
Note 406410.1 - Dealing with existing Database time zone use when updating Oracle TZ-files.

Note 396670.1 - Usage of utltzuv2.sql before updating time zone files in Oracle 9.

Note 396671.1 - Usage of utltzuv2.sql before updating time zone files in Oracle 10.

Note 397013.1 - Advanced use of utltzuv2.sql.

Note 402614.1 - Time Zone Data in the Data Dictionary During a Time Zone File Update.

Note 399832.1 - Workarounds when utltzuv2.sql patches are not available for your patchset.

Applying the time zone file patch:
Note 359145.1 - Impact of 2007 USA daylight saving changes on the Oracle database.

Note 396387.1 - Workarounds when Database time zone patches are not available for your patchset.

Note 396426.1 - Effects on client and middle-tier of applying time zone patches on the Oracle Database.

Note 402316.1 - Database Time Zone Patching Strategy in the Enterprise.

Platform specific information for those platforms with some special patching issues:
Note 406196.1 - VMS: DST 2007: Information for OpenVMS platforms.

JVM issues:
Note 359145.1 - Impact of 2007 USA daylight saving changes on the Oracle database.

Note 397770.1 - How to check if Oracle JVM is installed in the Database.

Note:276554.1 - How to Reload the JVM in 10.1.0.X and 10.2.0.X.

Note:209870.1 - How to Reload the JVM in 9.2.0.X.

Oracle Applications:
Note 403659.1 2007 - Daylight Saving Time Changes For Oracle Applications.

Note:403311.1 -Impact of U.S. 2007 Timezone Changes on Oracle E-Business Suite, Release 11i Environments.

Grid Control Issues:
Note:401834.1 - EM Agent will not start - due to a Daylight Savings change - (Workaround). Explains why it is impacted, how to fix it and the workaround.

Thanks for Reading,

Chris Foot
Oracle Ace

Gaining a well-rounded and deep understanding of Oracle access paths and SQL performance is a wonderfully challenging task. One of the hurdles is the time required to learn how to tune SQL. DBAs are being challenged with ever-increasing workloads, shrinking DBA staffs and increasingly complex technologies to support.

In addition, the majority of shops view database administrators as much more than just "table jockeys." The DBA is often seen as the go-to technician because of their traditionally strong problem solving skills. The DBA is also viewed as the IT staff's technical generalist because of the working knowledge they have in many different facets of information technology. Those of us that have been working in this profession for any time at all understand that the term "database administration" is really a misnomer. We have to know everything from application and data design to network communications and operating systems (and everything in-between).

As a result, many of us don't have enough time to dig deep into the many different facets of the Oracle database environment. Oracle has recognized this issue and has created the various advisors in 10G to assist us in the monitoring and tuning process. I have written dozens of blogs on the toolsets provided in 10G that are designed to reduce the amount of time we spend administering, troubleshooting and tuning the Oracle Ecosystem (database, operating system, hardware server).

The Importance of Understanding Oracle Access Paths
That being said, all DBAs must have a firm understanding of Oracle access paths and SQL tuning. During my career, I've learned that there is nothing that can drag down an Oracle Ecosystem than poorly performing SQL statements.

That's when management suddenly doesn't care about how much you know about RAC, standby databases and data design. They want the database "fixed" and running smoothly again. They suddenly have a single-minded purpose. They quickly begin the chant, the chant that can only be described as the management mantra - "is it fixed yet, is it fixed yet, is it fixed yet, is it fixed yet, is it fixed yet….."

I've actually had my entire chain of command stand behind me in order of where they fit into the management food chain (team lead, project manager, section manager, division manager, VP…) while I was working on a database performance issue. I turned around and chuckled when I saw the lineup of managers. Funny, they didn't share my humor.

You need to get the performance problem fixed, you're nervous and you have a 15 page SQL statement on your screen. This is where a strong education in Oracle access paths becomes "somewhat handy."

SQL Tuning Topics You Must Understand
Here's a quick laundry list of topics you'll need to know. All of them are important. I have tried to include resources later in this blog that will provide you with information on all of them. You can use this listing to check off the topics during the education process.

• The Oracle release's impact on optimization and SQL tuning. Each new release contains features that affect access paths and SQL performance. Sometimes good and sometimes not so good.
  
  
• Oracle parameters that affect the optimizer. There are a handful or two of startup parameters that can influence the SQL optimization process. I have provided several articles below on this topic.
  
  
• Optimizer modes - Rule, choose, first_rows, all_rows. Each of the modes influence the optimizer to create access paths for the type of workload the database is responsible for supporting. For example, the first_rows optimizer mode may be OK for online transaction processing (read a record/write a record) but probably won't generate efficient access paths for a data warehouse database where millions of records are summarized.
  
  
• Oracle data statistics - How the optimizer uses them, how they affect access paths.
  
  
• Oracle system statistics - Later releases of Oracle can also incorporate the system load information during optimization.
  
  
• Basic access paths
  • Index only - Oracle is able to read all of the data required to satisfy the query's data needs from the index structure alone.
  • Index to table - Oracle uses a row identifier to probe the table to satisfy the data request. Why read all of the rows in a table if you can use an index structure to retrieve just the rows you need?
  • Full table scan - Oracle reads all rows from the table. If the statement is going to read the majority of a table's rows, why would you want it to needlessly traverse an index to get the data? You are reading extra index blocks for no reason. You also need to learn the impact that the high-water mark has on full table scans. Oracle will scan the table to the last block used (as opposed to the last block that actually contains data).
    
    
• Join access paths - Used when the statement retrieves data based on matches between two tables (i.e. retrieve all of the employees that have the department name of "Welding"). The employee information is contained in the employee table and the department information (including the department name) is in the department table.
  • Nested loop join - Good path when the join is accessing a small subset of rows.
  • Hash join - Efficient access path for joins that access larger sets of data.
  • Sort merge join - Sorts rows to allow quicker access during the join.
  • Cartesian join - The tables being joined do not have join clauses that relate the two tables together.
  • Outer joins - An outer join returns all of the rows that satisfy the particular join condition and returns additional rows from one table that do not satisfy the join condition.
    
    
• Join order - Oracle only joins two tables at a time. If multiple tables are joined, join order also describes the overall order of the tables being accessed. Oracle will join two tables and create an intermediate result set which is then used as input to the next join.
  
  Join order plays a significant role in query performance. In general, you want to reduce the number of rows processed as soon as you can during the processing of a given SQL statement. The sooner you can reduce the number of rows being sent to future operations, the faster the query will usually run.
  
  
• Subqueries - A select within a select statement. Can be one of the trickier statements to tune, especially when you have multiple subqueries embedded within each other.
  
  
• Indexes and selectivity
  • B-tree indexes are good for column(s) that have many unique values (high selectivity)
  • Bitmap indexes are used for column(s) that do not have many unique values (low selectivity)
  • How SQL statement predicates can determine if an index can be used. There are times when the way a statement is coded prevents Oracle from choosing an index as the access path. A common problem that often leads to poor performance. You do get a chance to flog the application developer responsible, though.
    
    
• Types of index access paths
  • Index unique scans - The SQL statement accesses an index using a column (or columns) that are defined in a unique or primary key index with an equality condition.
  • Index range scans - Oracle scans a set of entries in the index to satisfy a query.
  • Index skip scans - Oracle is able to break down a multi-column index and view them as smaller subindexes. This is achieved by Oracle "skipping" the leading columns in the index and using columns that appear later in the index's definition.
  • Full scans - Oracle scans all of the index entries. Kind of like a tablescan on an index.
  • Fast full scan - Oracle uses multi-block reads to retrieve the index blocks.
    
    
• Sorting - Many operations require the database to sort a result set. It could be that the query wants to return the data in a particular order (ORDER BY, GROUP BY). In addition, some joins require that the data be sorted during the operation's execution. You'll need to understand why sorts are performed and the impact they have on performance.
  
  
• Views - Views can really complicate the tuning process. You think you are accessing a few tables in a query and then find that you are actually joining views together that also contain join operations.
  
  
• Hints - My next blog will show you how to use hints to influence access paths and how hints can be used to educate yourself on the performance of a particular access path operation.
  
  
• Bind variables and bind peeking - In the first and second blogs of this series, I described the impact that bind peeking can have on SQL statement optimization. Bind peeking may lead to the predicted access path not matching the access path taken during execution.
  
  
• Query transformation - Oracle can rewrite a query during the optimization process. Learn how Oracle uses view merging, predicate pushing, OR expansion and subquery unnesting to attempt to improve execution performance. This will occur on a regular basis and you need to understand how query transformation works and how it affects SQL performance.
  
  
• Local predicates vs join predicates - A local predicate accesses a bind or hardcoded variable (i.e. emp_id = :empid, emp_id = 13344) while a join predicate is used to join two tables together (i.e. emp.dept_id = dept.dept_id)
  
  
• Predicate usage - You need to understand how predicate usage affects indexes and access path generation. I have provided information below that discusses predicate conditions (=, >, <, etc) as well as how predicates affect index utilization and access paths.
  
  
• Operation cardinality - The number of rows returned by a particular access path operation. Importance of operation cardinality is magnified as the number of tables accessed in the query increases. As stated previously, the sooner you can reduce the rows sent to future operations, the better your query will perform.
  
  
• Skewed data and histograms - What happens when you have an index built upon a column in a million row table that has twenty occurrences of the value "OUT OF STOCK" and the rest of the column values contain the value "IN STOCK". This is an extreme case, but the impact is that even though you may access the table looking for an "OUT OF STOCK" value, Oracle will most likely perform a table scan. You'll be searching close to a million values, while you need to retrieve only twenty of them. An index would be a much better access path, but Oracle sees that the column has such poor selectivity that it won't choose it. A histogram identifies skewed data and is able to provide the optimizer with the information it needs to make a more educated decision when choosing between a table scan and index access.
  
  
• Parallel processing and partitioning - A best practice for large data stores is to partition data into smaller subsets of data. Partitioning allows data to be broken down into these smaller subsets yet still be viewed as a single-entity by the application. Parallel processing breaks a single request for data into one or more processes that access the data in parallel and return the data to the calling application.
  
  
• Parsing - Learn the differences between hard parses vs soft parses. When the application sends a statement to the database for processing, one of the first steps in the execution process is called a parse. Oracle will check the statement's syntax, check security, generate the access path, etc.. Like most operations the less steps it needs to perform the better. You will need to understand the impact that bind variables have on the parsing process and how the hard parse/soft parse ratio affects query and database performance.

Access Path Education
You can start your education on the different access paths that are available to the optimizer by reading Oracle's Database Performance Tuning Guide that is provided in each Oracle release's documentation. Before you buy third-party books on any topic, I highly suggest that you read Oracle's documentation first. The importance of this suggestion bears repeating - READ ORACLE's DOCUMENTATION FIRST. Here's a link to the 10G Database Performance Tuning Guide.

You'll need to create an account, but its free to register and the process is painless. Virtually every topic that I described above is covered in the Database Performance Tuning Guide. It is very important that you read the guide that pertains to the release that you are working with. Each release contains enhancements to the optimizer as well as new features that affect the optimization process.

Oracle Classroom Education
OK, since I'm an ex-Oracle Instructor, you could have predicted that my next recommendation was to sign-up for an Oracle class on SQL performance tuning. Here's a previous blog that provides you with a few hints and tips to obtain the most from your classroom experience.

Oracle Education offers a class that focuses on SQL tuning. Here's an excerpt from the class description provided on the Oracle Education Website:

What you will learn:
--------------------------------------------------------------------------------
This course is designed to give the experienced SQL Developer or DBA a firm foundation in SQL tuning techniques. The participant learns the necessary knowledge and skills to effectively tune SQL in the Oracle Database 10g. They learn about tuning methodology as well proactive tuning and reactive tuning methods. Students are introduced to the benefits of the new automatic tuning mechanisms available in Oracle Database 10g. On completion of the course they are able to compare and contrast the steps involved to tune manually as in prior releases as well as use the automatic SQL tuning features provided in the current release. Students gain a thorough conceptual understanding of the Oracle Optimizer, and reinforce instructor-led learning with structured hands-on practices. The course uses a series of challenge-level workshops, allowing students to "play, discover, and learn" at their own level and pace. The students learn to use the Oracle diagnostic tools and facilities: Automatic SQL Tuning components, EXPLAIN, SQL Trace and TKPROF, SQL*Plus AUTOTRACE. Students also learn to influence the behavior of the Optimizer by changing the physical schema and modifying SQL statement syntax.

Metalink Notes
Oracle's premier web support service is available to all customers who have current support service contracts. Oracle MetaLink allows customers to log and track service requests. Metalink also allows users to search Oracle's support and bug databases. The website contains a patch and patchset download area, product availability, product life-cycle information and technical libraries containing whitepapers and informational documents. A few of the white papers and notes that pertain to SQL tuning, optimization and access paths are provided below (do a search using the note number on the main page in Metalink to retrieve the note):

• 199083.1 - Query Tuning Overview - Lots of good links to other articles.
• 398838.1 Frequently Asked Questions. Very good discussion on the optimizer. Discusses queries not using indexes.
• 248971.1 - Query tuning best practices. LOTS of links to other notes. Links discuss a wide range of topics. From parameters that affect optimization to system statistics impact on optimization.
• 35934.1 - Common Issues and Misconceptions about the cost based optimizer.
• 68735.1 Diagnostics for Query tuning.
• 67522.1 - Diagnosing why a query doesn't use an index. Very helpful.
• 207434.1 - Tuning Queries - Quick and Dirty Solution.
• 372431.1 - Troubleshooting a new queries.
• 154354.1 - Using the FIRST_ROWS for fast query response.
• 100229.1 - Measuring Index Selectivity.
• 41954.1 - Hash join operations - a little dated but still valid.

Third Party Books
If you want to learn how the optimizer works, I highly suggest that you read Jonathan Lewis's book titled Cost-Based Oracle Fundamentals. It is one of the most educational and informative books I have ever read on the cost-based optimizer. Here's my review of the book for DBAZine. If I could only recommend one book on the Oracle optimizer, Jonathan's book would be it. Jonathan also maintains a blog that focuses on a wide range of topics but does include a lot of discussions on Oracle tuning.

Tom Kyte's Ask Tom Website also provides a lot of information on SQL tuning, access paths and proper coding techniques. One of Tom's trademarks is to use a snippet of code to reinforce the information he is conveying. I'm a big fan of using examples and a big fan of Tom's website.

Articles and Presentations
Article by Don Burleson on cost based optimization. Discusses parameters that affect the cost based optimizer. Statistics, helpful hints and tips.

Advanced CBO Article and Presentation. Advanced discussion on how statistics influence the optimizer, why the optimizer doesn't always take the correct access path. Written By Wolfgang Breitling.

Article on the Oracle 10G Costing model changes and how they influence access paths. Listing of parameters that affect optimization. Author is Kimberly Floss.

Oracle Statistics Article - System statistics affects on the optimizer. Very good article that provides lots of details. Written by Jonathan Lewis

Article on why Oracle won't use an index (when you think it should). Excellent article written by Jonathan Lewis.

Test, Test, Test
Experience pays. You need to spend time "in the seat" learning how to tune. Read the above information and find a database that you can use as a test environment. You need to work with tables that have small numbers of rows and queries that return small result sets. You also need to run queries that access tables with high numbers of rows and return large result sets.

I'll show you how to use hints to change access paths in then next blog. Change the access path with a hint and run the query. Influence the optimizer to take a different access path and join order and keep a tuning record of the changes and runtimes.

Don't' fall into the trap of favoring one access path or join method for all situations. I once overheard a conversation between a developer that just moved from an online transaction system to the data warehouse team and the data warehouse DBA. The developer was looking at an access path and stated "I hate hash joins." The warehouse DBA stated "Not in this environment you won't". All access paths and join methods have a place in Oracle optimization. Its up to you to learn which ones apply for a given situation.

Thanks for Reading,

Chris Foot
Oracle Ace

All Roads in 10G Grid Control Lead to the SQL Details Panels
10G Grid Control's primary display tool for providing information on specific SQL statements is the SQL Details Panels. It seems like no matter where you start your investigation in 10G Grid Control, sooner or later, you'll end up with a panel that contains a link to the SQL Details Panels. Let's take a look at a few of the more popular ways of finding our way to them.

Database Performance Home Page
The Database Performance Home page provides a couple of links that lead to the SQL Details Panels. If we start from the top, we'll see the Sessions: Waiting and Working Chart in the middle of the screen. This display allows us to determine exactly what resource our disgruntled users are waiting for.

Clicking on one of the colored areas of the Sessions: Waiting and Working Chart allows us to drill down to the specific resource being waited on. Clicking on any of the colored areas will display a drill down panel that provides details on that specific resource. For example, if we click on the purple I/O color on our chart, we'll navigate to the I/O drill down page.

No matter what resource you drill down into, they all have the same look and feel. All of the resource panels display the particular resource's historical utilization and two pie charts on the bottom of the screen that allow administrators to drill down into the top sessions and SQL statements utilizing that resource. If we click on the links listed under the Top Waiting SQL: User I/O heading, 10G Grid Control will respond by displaying the SQL Details Panels. If we click on the links under the Top Waiting Sessions: User I/O heading, 10G Grid Control will display the Session Information panel. The Session Information panel contains a link to the current SQL being executed which is displayed using the SQL Details Panels.

If you are interested in learning more about the Database Performance Home Page, please turn to my blog titled "Database Tuning Using Oracle 10G Grid Control".

Top Sessions Panel
If we return to our Database Performance Home Page, we need to scroll down to see the Top Sessions and Top SQL links. When we click on the Top Sessions link, 10G Grid Control will respond by displaying the Top Session display panel. If you look at the red box, you'll see that the Top Sessions panel is actually part of the Top Consumers set of display panels.

We navigate to the Session Details Panels for each session in our report by clicking on the link under the SID column heading. The Session Details Panel provides a link to the current SQL being executed which 10G Grid control displays using the SQL Details Panels.

Top SQL Panel
Let's return to our Database Performance Home Page. The link below Top Sessions is the Top SQL. Clicking on the link displays the Top SQL panel. The Top SQL panel has to be one of my favorite display panels in 10G Grid Control because it allows us to view SQL performance historically. The Top SQL panel provides a listing of SQL Details links that provide us with access to the SQL Details Panel.

Viewing Performance Historically
10G Grid Control allows us to take the database "back in time" and review performance historically. This screenshot shows the Database Performance Home Page in historical mode. Historical mode is activated by the drop down menu (red box) that is displayed in the upper right hand corner of the page.

The historical mode display provides a slider that allows us to go back in time and view performance statistics captured during a specific time period. When I drag the slider to a time in the past, 10G Grid control displays the Database Performance Home page statistics for the chosen time period. If we click on the Top SQL link at the bottom of the page, 10G Grid Control responds by displaying the Top SQL Period display panel. This panel provides links to the SQL Details panels for the top SQL statements that were active during the chosen time period.

SQL Details Panels
Now that we have learned how to navigate to the SQL Details Panels, let's take a look at the information they provide. The first screen displayed is the Execution Plan panel.

Execution Plan
The Execution Plan panel provides us with the SQL statement's access path taken during execution. The top section will display the entire text of the SQL statement. The next section displays where the panel retrieved this information from (cursor cache), the optimizer mode, data capture time, plan hash value and optimizer mode.

The bottom section provides us with all of the basic information we need to identify the access path this query is taking. It also includes key performance indicators including the number of rows that were used in each step, the time to perform the step and CPU and I/O cost indicators.

Current Statistics
Let's take a look at the Current Statistics Panel and the wealth of tuning information it provides. The panel will tell us if this SQL statement is being influenced by a SQL Profile. We'll have an entire discussion on SQL Profiles in an upcoming blog. But for the sake of this discussion, a SQL Profile is a stored access path that we can generate using the SQL Tuning Advisor.

Time Model information provides key performance indicators including elapsed time per execution, CPU time per execution and the statement's wait ratio. The Execution Statistics includes the number of executions, parse calls, rows per fetch, rows per execution and executions per parse call.

Execution History
The Execution History panel provides us with another key set of performance indicators. It shows us the statement's performance historically. The top chart displays the number of seconds the statement took to execute. This allows us to quickly determine if a statement's spike in elapsed and CPU times can be attributed to an event that occurred during a specific time in the past.

The Executions chart shows us how many times a statement was executed historically. This allows us to determine if an overall degradation in database performance can be attributed to an increase in the number of executions of this statement. I have used this panel to find a problem in a program that was causing it to loop and execute a particular statement millions of times.

Tuning History
The Tuning History Panel displays information on past tuning exercises for this statement. If you have generated any SQL Profiles, this panel will display information on each SQL Profile generated. It will also provide information on which SQL Profile is currently being used. The panel allows users to activate and deactivate each of the SQL Profiles that were generated for this statement. As I stated, we'll focus on SQL Profiles in an upcoming blog and I'll show you how to use this panel to switch between the different SQL Profiles created for a given statement.

Wrapup
I hope you enjoyed this blog on the SQL Details panels. I think you'll agree that it provides us with a good set of information to begin our SQL performance tuning activities.

Thanks for Reading,

Chris Foot
Oracle Ace

But 9I's Oracle Enterprise Manager (OEM) is no slouch either when it comes to tuning and diagnostic tools. Since the focus of this blog has been to cover Oracle 10G, I won't be reviewing all of the 9I tools we can use to assist us in the tuning process. But this recent series of blogs focuses on the various tools that we can use to view access path information and there are tools that 9I OEM provides that are definitely worth discussing. And lets face it, there are LOTS of Oracle9I databases out there.

9I OEM allows us to log in with a repository or in stand-alone mode. Most of the tuning and diagnostic tools require the use of a repository which is a set of tables stored in a database. If you install the Oracle9I software in a windows environment, you'll find a 9I Oracle Enterprise Manager Configuration Assistant contained in your Oracle menus. When you run the configuration assistant, Oracle will prompt you for database connection information. Once the repository is configured, you will have the ability to run the advanced diagnostic tools that Oracle9I OEM provides. After the initial configuration is complete, you'll need to run the Navigator to add databases into the Oracle9I OEM infrastructure.

Sessions Panel
After the databases are added using the Navigator drop down, each time you activate 9I OEM, the first panel shown will be a database listing. You select the database you want to administer or monitor and then activate the various tools and utilities you want to use.

Let's take a look at this first screenshot. On the far left of the screen, you'll see a listing of icons we can use to activate the various toolsets provided by 9I OEM. The display screen itself is split into two subpanels. The subpanel on the left allows me to navigate to a database and then drill down to view and administer configuration settings, schema definitions, security, locking, etc..

Please note that this screen shows that I have already activated the Sessions display panel. I use this panel a lot when I am reviewing SQL performance for a 9I database. Look at the column headings at the top of the screen. It allows users to sort the sessions by the column headings. The columns provide information on program name, machine name, OS User and process ID, which allows users to easily identify specific sessions.

The columns also provide key performance indicators. We can sort sessions by CPU, memory and physical and logical I/O. These columns make it easy for us to identify the top resource consumers. We are then able to drill down into that session to determine what SQL statement the session is executing and how well it is performing.

Drilling Down to View Specific Session Information
If I double-click on a specific session, 9I OEM displays the session drill down panels. I am able to select from general information, long operations (if available), the SQL statement currently being executed (blank if the session is not currently running a statement), performance statistics and locking information. Here's a screenshot of long operations information for a session in my database. The long operations display tells me how many blocks have been accessed and how many more blocks Oracle predicts must be read before that individual operation is complete. This display shows that I have almost completed a table scan on the HR.EMPLOYEE_TEST table.

The SQL tab at the top of this panel allows me to display the current statement being executed by the session. The panel also displays the access path that Oracle is predicting the statement is taking. Remember that we learned in previous blogs that because of certain conditions (like bind peeking for one), that the access path can change during execution.

The panel allows me to choose from either a text or graphical description of the access path. In this case, I have chosen to view the graphical display of the access path. The cool feature of this panel is that the icons on the right hand side of the screen allow me to step through the access path. As we step through the access path, Oracle will display a description of the step being performed at the bottom of the panel. Not only does it allow us to more quickly identify how the query is accessing the data, but it also provides us with a quick education on the access paths themselves. If you aren't an expert in Oracle access paths, this is a quick way for you to start your education.

SQL Scratchpad
Let's close this subpanel and navigate back to the main 9I OEM screen. When I click on the top tool icon, 9I OEM will respond by displaying a horizontal menu listing of additional icons. The icon on the far right will be for the SQL Scratchpad utility. If I click on the far right icon,9I OEM responds by displaying SQL Scratchpad. Scratchpad allows me to run SQL statements, edit them and then save them to a folder. For this discussion, we are most interested in Scratchpad's graphical access path display capabilities.

If I click on the icon that looks like an open book, 9I OEM responds by displaying the predicted access path the query will use when it executes. Like its session drilldown counterpart, SQL Scratchpad allows me to view the access path in text or graphical mode and allows me to step through the individual steps. Once again, the tool provides us with an easy way to educate ourselves in Oracle access paths.

SQL Analyze
Another tool that 9I OEM provides to view access path information is SQL Analyze. SQL Analyze is only available if you have purchased the tuning pack option for 9I OEM. This screenshot shows me navigating through 9I OEM's menus to activate SQL Analyze.

When you activate SQL Analyze, the first step the utility performs is to display a listing of all of the top resource consuming SQL statements that are contained in the library cache. If the statement has been flushed from cache, you won't find it in this display.

The tool lists all of the databases on the left side of the panel that you worked with in the past. As you can see, I am a pretty active user of SQL Analyze.

I can double-click on any statement in the top SQL listing to view its access path information. I can also use the menu system at the top of the screen to create a blank worksheet for new queries that I can enter manually. I found the statement I wanted from the listing and double-clicked on it to activate the SQL tuning panel.

The panel displays the SQL statement, and like our other utilities, allows us to step through the access path the query is taking. I can click on the SQL drop down menu at the top of the screen to ask Oracle to explain the statement using the various optimizer modes that are available.

SQL Analyze Tuning Tools
SQL Analyze provide us with tools that facilitate the SQL tuning process. The tools menu at the top of the screen allows me to choose from three different wizards. We'll review the Virtual Index Wizard in just a moment. The Hint Wizard allows us to choose hints from a drop-down menu system, while the SQL Tuning Wizard activates an intelligent advisor that provides us with SQL tuning recommendations. Once you have run the SQL Tuning Wizard a few times, it's pretty easy to see that it was the precursor to some of the more advanced utilities that are now available in Oracle 10G.

Virtual Index Wizard
The Virtual Index Wizard allows me to create virtual indexes on tables that our query is accessing. The key word in that last sentence is "virtual". Oracle does not permanently create the index. The intent of the wizard is to allow users to determine the potential performance impact a new index would have on the statement being analyzed. It is then up to the user to permanently create the index.

If I click on the Virtual Index Wizard menu selection, 9I OEM responds by displaying the Virtual Index Wizard introduction panel. When I click on Next, the wizard displays a panel that allows me to select the columns for my virtual index. Clicking next again displays a panel that allows me to attempt to force Oracle to use the new index or allow it to choose the new index on its own. My personal preference is to allow Oracle to choose the index on its own without providing any additional prodding.

Clicking Next again displays the projected cost improvement the new index will have on the statement being analyzed. I used a full table scan on a rather large table as an example. Based on the display, we can be pretty sure that the new index will have a positive affect on our statement's performance. The last panel also displays a button that allows us to view the before and after access paths.

The Virtual Index Wizard does a fairly good job of estimating the impact that new indexes will have on a SQL statement. It is an excellent tool to use when you are starting your tuning education as well as evaluating access paths in Oracle 9I databases.

I hope you enjoyed this blog on Oracle 9I access path identification. I think that after reading this blog, you'll agree that Oracle9I OEM provides users with a host of utilities to facilitate the access path identification and SQL statement tuning processes.

Next Up
We'll take a look at the various tools in 10G Grid Control that we can use to display access path information.

Thanks for Reading,

Chris Foot
Oracle Ace

SQL*PLUS Autotrace and the SQL Trace command are tools that we can use to display access path information. In addition, SQL Trace also provides additional statistical information that we can use to evaluate a given statement's performance.

You can easily tell how popular these tools are by doing a Google search on them. You'll find hundreds of different articles, blogs and whitepapers that cover Autotrace and SQL Trace. Instead of me just regurgitating information that is already provided by a myriad of other sources, we'll keep this blog short. I will point you to a few of my favorite "go to" websites for more information.

SQL*PLUS AUTOTRACE
The SQL*PLUS AUTOTRACE command can be used to create a report of the execution path used by a particular SQL statement. Statement execution statistics can also be generated if desired. This is my favorite explain tool because it's a quick and easy way to display the access path the query is taking. The following parameters are used as input to Autotrace to generate different report output:

• SET AUTOTRACE OFF- No AUTOTRACE output is generated. This is the default value.
• SET AUTOTRACE ON EXPLAIN - The AUTOTRACE output displays the optimizer access path.
• SET AUTOTRACE ON STATISTICS - The AUTOTRACE output displays SQL statement execution statistics.
• SET AUTOTRACE ON - The output will display the optimizer access path and the SQL statement execution statistics.
• SET AUTOTRACE TRACEONLY - Will display the same output as SET AUTORACE ON but will suppress the display of the query output.

Take a look at this output file I generated in SQL*PLUS. It displays the output of all of the different variations of the Autotrace command. Tom Kyte's Ask Tom website provides an abundance of information on setting up and running Autotrace. Here's Tom's brief description of the performance statistics generated by Autotrace.

Bind Peeking and Autotrace
In addition, you will have to watch out for those pesky bind variables. I discussed the impact bind variables had on generating different access paths at explain time versus run time in a previous blog.

Here's Something I didn't Know
You will also need to be aware that SQL*PLUS and Autotrace will assume that all bind variables are strings - regardless of how you define them. Tom Kyte's blog (do a search on this value "to_number(:bv)" twice when the page is displayed) to see Tom's comments on this.

I agree with Tom's recommendation that if you want to attempt to get the access path displayed by Autotrace to match the access path taken during execution, your best bet is to replace the bind variables with hard-coded values. The issue is the values you choose to replace the bind variables with may not match the values actually used in production. So, once again, your access paths may differ.

Some Interesting Information from Jonathan Lewis on Autotrace
Jonathan Lewis's blog entry on Autotrace provides some interesting hints and tips on how to better format Autotrace output.

The first comment in the blog, by one of my favorite bloggers - Howard Rogers, asks Jonathan to provide evidence that Autotrace does only an explain when using the SET AUTOTRACE TRACEONLY EXPLAIN command. Please follow the link that Jonathan provides to show his explanation. You'll see that, once again, you need to be careful when using Autotrace to display access paths.

SQL TRACE
If you have been reading my previous blogs on access paths, you'll know that I stated that the data in V$SQL_PLAN and the output provided by a SQL Trace command will show the actual access path a statement used during execution.

SQL Trace is used to create a raw trace file that contains SQL performance information for all statements that were executed while the trace was active. The raw trace output can be found in the database's UDUMP directory. You use the TKPROF utility that resides in the $ORACLE_HOME/rdbms/admin directory to read the raw trace data and format it for easier readability.

There is an absolute myriad of ways to activate tracing. The range of tracing options ranges the spectrum, from tracing every statement that executes in the database to tracing individual SQL statements. The two primary methods to activate a trace are SQL statements to alter the session setting to activate tracing ("alter session set sql_trace = true;") or setting the 10046 event parameter.

Instead of me regurgitating information that is already on the web, here's a link to an extremely thorough discussion on the various ways to activate SQL tracing.

It is from another one of my favorite bloggers, Pete Finnigan. I think that after reading Pete's document, you'll agree that he has done a great job at showing us how to activate tracing.

We learned in previous blogs how to identify poorly performing queries using the various tools available in 10G Grid Control. You can also activate a trace from the Grid Control Panels. In my blog titled "System Triage Part III - Finding the Top Resource Consumers", I show you how to drill down from the Top Activity Page to the Session Details page. If you notice, the Session Details page has a link to activate a SQL trace.

I'm also a big fan of Mark Rittman. Mark shows us a couple of additional tricks on activating a SQL trace and how to read its output.

We now know that we have numerous alternatives to choose from to activate tracing. Here are the ones I most commonly use:

• If I can quickly identify the statement, or statements, causing the problem (discussed in previous blogs in this series), I will cut the statement out and attempt to run it in the production environment. This is easily done if it is a SELECT statement. I'll try to mimic the execution environment as closely as I can in production. Here's an example of a PL/SQL block I have often used to trace a SQL statement that I think may be affected by bind peeking. I'll set my session to trace using the "alter session set sql_trace=true" command, run the query, format the dump file and review the output.
  
  
• If I identify the program or user having the problem, I'll use one of the methods described by Mark Rittman and Pete Finnigan to trace another user's session.
  
  
• If I can narrow it down to a specific user or program, I'll activate a logon trigger.

  Login trigger for schema login. Replace SCHEMA_OWNER with user name.
  
  CREATE OR REPLACE TRIGGER SYS.LOGINTRIGGER AFTER
  LOGON ON SCHEMA_OWNER.SCHEMA BEGIN
  execute immediate 'alter session set sql_trace = true';
  END;
  
  Login trigger to trace program. Replace "name of program".
  
  CREATE OR REPLACE TRIGGER SYS.TRC_PKREVCSORT AFTER
  LOGON ON DATABASE DECLARE
  progname v$session.program%type;
  BEGIN
  select max(program) into progname from v$session where audsid = userenv('sessionid');
  if (progname = 'name of program') then
  execute immediate 'alter session set sql_trace = true';
  end if;
  END;
  

These two logon triggers were provided to me by Ron Berner. Ron states "you have to match the program name exactly. Selecting from v$session and pulling the whole value from the column PROGRAM. If the program name is "/opt/oracle/programs/ron.exe" you have to include the full path in the name."

Tom Kyte's Ask Tom website has a whole host of different variations of the above two triggers. You can pop over to Tom's website to learn what the different options are and the benefits they provide when activating traces using logon triggers.

Let's take a quick look at a trace file I generated from the very simple SQL statement I have been using as an example. The intent of this discussion is to use the formatted trace output file to compare the access path that Oracle predicts the statement will take and compare it to the actual access path taken during statement execution. Before we do, here are the commands I executed in SQL*PLUS to generate the trace file:

• alter session set timed_statistics = true
• alter session set max_dump_file_size = unlimited
• alter session set tracefile_identifier = 'test_event_tracing'
• alter session set events '10046 trace name context forever, level 12'
  Could have used the "alter session set sql_trace=true" command.

I then executed this single statement:

select a.employee_id, a.last_name, b.department_id,
b.department_name,c.street_address, c.postal_code,
c.city, c.state_province
from hr.employees a,
hr.departments b,
hr.locations c
where a.department_id=b.department_id
and b.location_id=c.location_id
and a.employee_id = 174
order by a.last_name
/

The "set tracefile_identifier" allows us to easily find the raw trace file in our database's UDUMP directory. Once we find the file, we can run the TKPROF command to format the output. Executing the TKPROF command without any options lists the different options that are available. Here's the command I issued to format the trace file. The "explain=" option asks TKPROF to display the predictive access path while "SYS=no" prevents TKPROF from displaying statements executed by SYS.

If we take a look at the output, you'll see two access paths displayed. The access path under the title "Execution Plan" is the one the optimizer thinks the statement will take, while the access path displayed under the title "Row Source" Operation" shows the access path the statement actually used during execution.

You'll notice that in my example, the predicted and actual access paths are the same. Here's a trace output file that shows the access path changing during run time.

The intent of this series of blogs is to show you how to display access paths and to determine which access path the statement is taking during execution. Along the way, we learned how bind variables and SQL*PLUS/Autotrace can cause the access path generated by the explain to not match the access path taken during execution.

What's Next
We'll take a look at 9I Scratchpad and 10G Utilities that display access path information.

Here's the statement that we will be using in our example:

SQL>
SQL>
SQL> explain plan into system.plan_table for
2 select a.employee_id, a.last_name, b.department_id, b.department_name,
3 c.street_address, c.postal_code, c.city, c.state_province
4 from hr.employees a, hr.departments b, hr.locations c
5 where a.department_id=b.department_id
6 and b.location_id=c.location_id
7 and a.employee_id = 174
8 order by a.last_name;

Explained.

When we run the statement, Oracle will insert rows into our plan table that describe the statement's access path. I'm not using bind variables for simplicity sake, not because I'm afraid of an access path change during execution.

Let's use Oracle's stored procedure to retrieve the contents from our plan table and format the output:

08:54:53 orcl9i> select * from table(dbms_xplan.display);

Here is the output from the above statement's execution.

DBMS_XPLAN Package
The DBMS_XPLAN package is the "go to" procedure for displaying the contents of plan tables as well as:

• Access paths from statements (cursors) that are loaded in the cursor cache.
• Access paths from statements stored in the Automatic Workload Repository.

DBMS_XPLAN.DISPLAY
The statement "SELECT * FROM table(dbms_xplan.display);" asks Oracle to display the current contents of our default plan table. The DBMS_XPLAN.DISPLAY function accepts "TABLE_NAME", "STATEMENT_ID" and "FORMAT" as input.

• TABLE_NAME allows you to specify a non default name for the plan table you are accessing.
  
  
• STATEMENT_ID allows you to select a specific access path from a plan table that contains access paths from different statements. Please read my two earlier blogs on access paths to learn how to set the STATEMENT_ID.
  
  
• FORMAT allows a user to choose from four different output formats:
  
  • BASIC - Displays the operation ID, object name and operation option. Provides limited information for a quick review
  • TYPICAL - Default option that displays partition pruning, parallel execution information and predicates. This is the one I use the most
  • ALL - All information from TYPICAL and also includes projection information
  • SERIAL - Does not display any information on parallel execution
    

Here's a few quick examples:

SELECT * FROM table(DBMS_XPLAN.DISPLAY('save_plan_table'));
Retrieves the contents from a non-default plan table.

SELECT * FROM table(DBMS_XPLAN.DISPLAY('plan_table', 'stmt_1'));
Retrieves a statement from our default plan table that has a statement ID of "stmt_1".

DBMS_XPLAN.DISPLAY_CURSOR
Not only can we use DBMS_XPLAN to retrieve the contents of our plan table, we can also use the DISPLAY_CURSOR function to display the access path that a statement used during execution. We are able to provide the program with a few input variables that will allow us to search the cursor cache for individual SQL statements:

• SQL_ID - The statement's SQL identifier. This value can be found by accessing either V$SQL or V$SQLAREA. You can search VSQL/V$SQLAREA's SQL_TEXT column for the text of the SQL statement you are looking for and use the statement's SQL_ID as input to DBMS.XPLAN.DISPLAY_CURSOR.
  
  I've provided a few examples of how to search these two tables for a statement's text in an earlier blog. You'll need to add the SQL_ID column in the SELECT clause of your searches. The example below shows the addition of the SQL_ID column to the statement I provided as an example in my previous blog:

  SQL> select sql_id, address, executions, buffer_gets, sql_text from v$sqlarea
  where sql_text like '%select a.employee_id, a.last_name%' order by buffer_gets desc;

• CHILD_NUMBER - Child number of the child cursor. If you don't supply this value, you will retrieve all child cursors from the SQL_ID you are using as input.
  
  
• FORMAT - Allows a user to choose different display outputs (discussed earlier in this blog).

Here's an example of using DBMS_XPLAN.DISPLAY_CURSOR to retrieve access path information for all child cursors that have a statement ID of 'xuponh8k9hjyk'.

SELECT * FROM table(DBMS_XPLAN.DISPLAY_CURSOR('xuponh8k9hjyk'));

The PL/SQL Packages and Types References manual provides in-depth instructions on how to use DBMS_XPLAN to retrieve and format access path information from plan tables, AWR and the cursor area. I highly suggest that you go to Oracle's Technet Website and study the information on DBMS_XPLAN. Tom Kyte's website also provides an abundance of information on this utility, how to use it and customize its output. Just hop on over and do a search on DBMS_XPLAN.

The UTLXPLS.SQL and The UTLXPLP.SQL
Oracle also provides two scripts that format plan table contents. The UTLXPLS.SQL script is used to format output for statements that perform serial processing, while UTLXPLP.SQL is used to format output for statements that perform processing in parallel.

Other Scripts
There are literally thousands of customized scripts on the web that you can use to format access path raw data. Find one that formats the output in a style that you prefer and go for it!

Graphically Displaying Access Paths
We'll use the same query we used previously for this next discussion:

SQL> explain plan into system.plan_table for
2 select a.employee_id, a.last_name, b.department_id, b.department_name,
3 c.street_address, c.postal_code, c.city, c.state_province
4 from hr.employees a, hr.departments b, hr.locations c
5 where a.department_id=b.department_id
6 and b.location_id=c.location_id
7 and a.employee_id = 174
8 order by a.last_name;

Let's review the information displayed by DBMS_XPLAN and use it to graphically display the query's access path. We'll also take a look at the output from a query I personally use when creating graphical access path displays to help. Before we begin, it will be beneficial to show you what we are trying to achieve. Let's take a look at the graphical output from 9I Oracle Enterprise Manager's Scratchpad tool. One of the benefits that Scratchpad provides is that with one quick mouse click, we can generate a graphical display of a statement's access path. We'll be reviewing this tool in an upcoming blog.

Graphically displaying a statement's access path will help us to easily understand how the access path works. The graphical display tells us that:

• We are accessing the HR.EMPLOYEES table using a unique scan on the HR.EMP_EMP_ID_PK index. The key words "(UNIQUE SCAN") in the graphical display tells us that we are searching the index for a unique value. When we find the value we are looking for (in this case an EMPLOYEE_ID that has a value of 174), we are using the row's ROWID to access the HR.EMPLOYEES table. DBAs often use the term "local predicate" to designate predicates that use a value to access a specific table.
  
  
• The HR.EMPLOYEES table is the outer table in our nested loop join. We are using the results that satisfy the local predicate (a.employee_id = 174) to access the HR.DEPARTMENTS table. We are looking for rows in the HR.DEPARTMENTS table that have DEPARTMENT_IDs that match the results returned from our local predicate on HR.EMPLOYEES. The predicate "where a.department_id=b.department_id" joins the two tables together and is classified as a "join predicate".
  
  
• We are using another unique scan on the HR.DEPT_ID_PK index, retrieving the row's ROWID from that table and using it to access the HR.DEPARTMENTS table. The HR.DEPARTMENTS table is the inner table of our nested loop join.
  
  
• Rows from HR.EMPLOYEES AND HR.DEPARTMENTS that have matching DEPARTMENT_ID values become the intermediate result set. Rows that don't match are filtered. The result set from this join is used as the outer data set in another nested loop join. The join predicate is "and b.location_id = c.location_id".
  
  
• Like our previous two table accesses, we are using a unique index called HR.LOC_ID_PK to access the HR.LOCATIONS table. HR. LOCATIONS is the inner table in this nested loop join.
  
  
• Data from the intermediate result set and the HR.LOCATIONS table that have matching LOCATION_IDs (and b.location_id=c.location_id) is passed to the next step.

Our Own Graphical Display
Take a look at this graphic. The top query is one that I often use to help me to create a graphical display of a statement's access path. The graphic below is the graphical display I created from its results. The query at the bottom is the output from DBMS_XPLAN. Let's take a look at the information provided from my query and learn how we can use it to create a graphical display of a given query's access path:

• ID - Numeric identifier that tells us the order of the step in relation to other steps. It is important to note that this is not the order in which Oracle performs the steps during the statement's execution. Generally, Oracle will perform the steps from left to right and bottom up during execution.
  
  
• PARENT_ID - Numeric identifier of the step's parent step. Used to determine where to place the step in our graphical display.
  
  
• POSITION - Numeric identifier that provides the position of the step in relation to other child steps that have a common parent step. Like PARENT_ID, it is used to determine where to place the step in our graphical display.

The graphical display helps me to more easily understand a query's access path. You can compare the output from my query to the results returned from DBMS_XPLAN and other Oracle utilities. Once you become more experienced, you will be able to create a graphical display from virtually any output format that Oracle, or you, generate.

Next Up
We'll continue our review of the various tools that we can use to generate and format access path information.

Thanks for Reading,

Chris Foot
Oracle Ace

The EXPLAIN PLAN statement will inserts rows into the plan_table that describes a given statement's execution plans. Oracle's standard recommendation is to drop and recreate all plan tables after an Oracle upgrade to ensure that you use the latest and greatest version of the object. It is a good practice to follow Oracle's advice just in case the definition of the plan_table changes from one release to another.

Manually Running the Explain
Although the intent of this series of blogs is to review the tools that Oracle provides to automatically display access path output, it is important to begin with the basics. The only tool you need to manually run an explain plan is SQL*PLUS.

Take a look at the examples below. I'm adding the explain plan clause to the beginning of the statement to notify Oracle to update the plan_table with rows describing the statement's access path:

• EXPLAIN PLAN FOR select * from scott.emp;
  
  This statement will explain the statement "select * from scott.emp" and place the results in the plan_table owned by the account executing the statement.
  
  
• EXPLAIN PLAN INTO schema.plan_table FOR select * from scott.emp;
  
  This statement will explain the statement "select * from scott.emp" and place the results in the plan_table owned by account referenced in schema.plan_table. The account executing the statement must have the proper privileges on the plan_table to execute this statement successfully.
  
  
• EXPLAIN PLAN INTO schema.plan_table SET STATEMENT_ID = 'test1' FOR select * from scott.emp;
  
  This statement will explain the statement "select * from scott.emp" and place the results in the plan_table owned by account referenced in schema.plan_table. The statement ID allows users to retrieve the statement's access path information from a plan_table that contains rows from many different explains.
  

It's important to note that the contents of the plan_table are a prediction of the access path that the statment will take. V$SQL_PLAN, discussed below, contains information on the access paths the statements used during execution. For more information, please refer to my previous blog titled "System Triage IV - Access Path Identification Part I"

When you explain in statements in SQL*PLUS, you also need to be aware of how the tool interprets the bind variables. Tom Kyte has a discussion on bind variable interpretation on his Ask Tom website.

You need to be careful when you are copying and pasting SQL statements that you are attempting to explain. I have seen experienced DBAs forget to copy the keywords "explain plan for" when they copy the statement, dump the statement into SQL*PLUS and run the statement instead of explaining it. It's an easy mistake to make when you are rushing to solve a performance problem. If it's an update, use the ROLLBACK statement to remove the unwanted data.

V$SQL, V$SQLAREA, V$SQLTEXT and V$SQLTEXT_WITH_NEWLINES
Before we begin our discussion on v$sql_plan, let's take a high level look at some of the views we will use to gather information on SQL statements contained in the library cache. We'll need this information before we access v$sql_plan to determine what access path our queries are taking.

• V$SQL - contains information on statements in the library cache. It will contain a row per SQL statement, which allows you to review information for an individual query. This becomes beneficial when you have tables that have the same names but have different owners. If two users are both executing "SELECT last_name, first_name, emp_id FROM employees" and both users own a table called employees, you'll find both statements in v$sql.
  
  V$SQL's sql_text column provides the first 1,000 characters of the text of the SQL statement and an address column that we can use to as an identifier when we query the other V$ views. V$SQL's sql_fulltext is a CLOB column that contains the entire SQL statement.
  
  V$SQL also provides an abundant amount of information on resource utilization including sorts, memory utilization, disk reads, buffer gets, rows processed.
  
  
• V$SQLAREA - Is an aggregate of the aforementioned v$sql view and contains much of the same information as in v$sql. Because it is an aggregate you'll have to remember that if two users are both executing "SELECT last_name, first_name, emp_id FROM employees" and both users own a table called employees, you'll find only one statement in v$sqlarea.
  
  
• V$SQLTEXT - Although both v$sql and v$sqlarea do provide CLOB columns that provide this information, this view provides an easy way to retrieve the entire text of the SQL statement. If the text of the SQL statement looks like it has been truncated in v$sql/v$sqlarea's sql_text columns, use the value in the address column from v$sql/v$sqlarea to query v$sqltext to get the entire statement. If the text of the SQL statement contains any newlines or control characters, they will be replaced with whitespace.
  
  
• V$SQLTEXT_WITH_NEWLINES - Contains the text of the SQL statement without the newlines or control characters being replaced by whitespace.

It is important to note that information for specific SQL statements contained in the V$ performance views provided above, as well as v$sql_plan, could be flushed from the system based on workload and memory allocations. Although you will always find data for currently executing queries, you may, or may not, find information pertaining to statements that were executed some time in the past. If a SQL statement has been aged out of the library cache, you won't be able to find information on it.

Going in-depth into each of these dynamic performance views is far beyond the scope of this particular blog. Consult the Oracle Reference Guide for your specific release for more information on this VERY helpful set of views.

V$SQL_PLAN
Oracle provides the v$sql_plan view to hold information on the access paths SQL statements used during execution. If you want to quickly find out exactly what access path was taken during runtime, this is the view to go to for that information.

If you do a describe on a plan_table and compare it to v$sql_plan, you'll notice that although most of the column definitions are the same, v$sql_plan, has additional columns to help you match the access path information to the SQL statement that generated it.

Let's take a look at the statements below:

The first step is to execute the statement we'll be using in the example. That way we'll be able to query a few of the dynamic performance views to retrieve some identification and performance information.

SQL>
SQL> r
1 select a.employee_id, a.last_name, b.department_id, b.department_name,c.street_address, c.postal_code, c.city, c.state_province
2 from hr.employees a,
3 hr.departments b,
4 hr.locations c
5 where a.department_id=b.department_id
6 and b.location_id=c.location_id
7 and a.employee_id = 174
8* order by a.last_name

EMPLOYEE_ID LAST_NAME DEPARTMENT_ID DEPARTMENT_NAME STREET_ADDRESS POSTAL_CODE CITY
----------- ------------------------- ------------- ------------------------------ ---------------------------------------- ------------ ------------------------------
STATE_PROVINCE
-------------------------
174 Abel 80 Sales Magdalen Centre, The Oxford Science Park OX9 9ZB Oxford
Oxford

Let's see if we can find the statement in the library cache by accessing the v$sqlarea. Since I know there is just one query in the library cache, I'm not interested in finding each specific version of it.

In this case, I'm interested in the address value to identify the statement and the number of buffer gets to get a feel for the statement's workload. I'll query the v$sqlarea view to retrieve the address for the statement, the number of executions and the buffer gets it consumed during those executions.

SQL>
SQL> select address, executions, buffer_gets, sql_text from v$sqlarea
2 where sql_text like '%select a.employee_id, a.last_name%' order by buffer_gets desc;

ADDRESS EXECUTIONS BUFFER_GETS
-------- ---------- ---------SQL_TEXT --------------------------------------------------------------------------------------------------------------------------------------------------------------------
697F521C 1 349
select a.employee_id, a.last_name, b.department_id, b.department_name,c.street_address, c.postal_code, c.city, c.state_province from hr.employees a, hr.departments b, hr.locations c where a.department_id=b.department_id and b.location_id=c.location_id and a.employee_id = 174 order by a.last_name

A couple of quick points to remember. Here is the WHERE clause for the above statement:

sql_text like '%select a.employee_id, a.last_name%'

We know that v$sql and v$sqlarea only contain the first 1,000 characters of the text of the SQL statement. If your search value falls after those 1,000 characters, you'll have to find the address from v$sqltext and use that value to search v$sql, v$sqlarea and v$sql_plan. In addition, the way the statements are stored can also have an affect on whether you will be able to successfully find the statement you are looking for.

If there is a break between "employee_id" and "a.last_name", you may not be able to find it using this search. That's why I attempt to search for the first few characters for a given SQL statement. The important point is to find the smallest chunk of the statement that allows you to differentiate it from other statements and use that value in your searches. There is a whole host of different options when searching the SQL_TEXT column. Tom Kyte's blog contains a lot of good suggestions.

Although the text of this SQL statement can be found entirely in the v$sqlarea performance view, as I stated, there are times when the statement will be too large to be entirely contained in the sql_text column. If the text of the SQL statement looks like it has been truncated, use the statement's address to query v$sqltext to retrieve the entire text of the statement.

SQL> r
1* select sql_text from v$sqltext where address = '697F521C' order by piece

SQL>
-------------------------------------------------------------
select a.employee_id, a.last_name, b.department_id, b.department_name
,c.street_address, c.postal_code, c.city, c.state_province from hr.employees a, hr.departments b, hr.locations c where.department_id=b.department_id
and b.location_id=c.location_id and a.employee_id = 174 order by a.last_name

OK, let's use the same value for the address column to query the v$sql_plan view to take a look at the access path it used during execution:

SQL> r
1 select operation, options, object_name, id, parent_id, position
2* from v$sql_plan where address = '697F521C'

Here's the output for the statement above.

You'll notice that the SQL statement I personally use to retrieve the statement's access path looks different to most others you will find. This is how it was done in "the old days" and I still feel comfortable using it from time-to-time. If I'm working on simple queries (no partitioned tables accessed, etc.), I will use this one because it allows me to quickly review the access path for the query.

It is important to note that this statement does not provide cardinality information. Cardinality is the number of rows that each, individual operation will access. Like the access paths themselves, there is a predicted cardinality and the actual cardinality. Bad things can happen if the predicted and actual differ. We'll discuss the importance of cardinality in a later blog. Right now, we are just concerned about displaying basic access path information. We'll take a look at some of the queries that Oracle provides to format access path output in my next blog.

Now that we have identified that our database is the cause of our application's poor performance, let's continue our system triage analysis by trying to identify the top resource consumers. During our investigation, we'll also find out what resource the top resources are consuming and the SQL they are executing.

For sake of clarity, we will make the assumption that the database ecosystem (database, operating system and hardware platform) has been performing fairly well historically. In addition, this blog also builds upon my previous two blogs. In those blogs, we created a not so mythical performance problem that began with a business user notifying us that they are experiencing a performance problem. During our analysis, we have ruled out the network and application tiers and narrowed the problem down to a database issue.

Finding the Top Resource Consumers
We begin our investigation by logging into 10G Grid DBConsole and displaying the Database Home Page. The first indicator that we are having a problem is the Host CPU graphic displayed in the middle of the screen. The DBConsole has placed a small red 'X' next to the title 'Host CPU' to notify us that CPU is an issue. If we scroll down to the lower half of the screen, we notice that ADDM has already identified the problem and is reporting its recommendations. I've discussed drilling down into ADDM recommendations in a previous blog. We can certainly use the ADDM output to help us solve our problem but it is important for us to discuss all of the tools that 10Grid DBConsole provides to us.

Let's continue our investigation by using the navigation tab on the top left hand side on the Database Home Page to navigate to the Database Performance Home Page. This panel certainly makes it easy for us to determine what resources are being utilized. We immediately see that we have a high CPU utilization by the large green area displayed at the top of the screen.

We are able to use DBConsole's drill down capabilities to drill down to the specific SQL and Sessions that are consuming that resource. It is important to note the the DBConsole provides drill down capabilities for all major resources displayed on this page. We activate the drill down for CPU utilization by clicking on the green area displayed on the screen. DBConsole responds by displaying the CPU drilldown panel. DBConsole provides a slider that allows us to go back in time and review the utilization of this resource historically. If we click on the SQL statement drill down on the left side of the screen, DBConsole responds by displaying the SQL Details Acitivity Page.

Once again, a large green area shows that this statement is consuming a LOT of CPU resources. You will begin to notice that almost every page DBConsole displays during our ananlysis provides a button to run the SQL Tuning Advisor. For more information on the SQL Tuning Advisor, please refer to my blogs titled The SQL Tuning Advisor and The 10G Automatic Tuning Optimizer. These blogs provide a wealth of information on the SQL Tuning Advisor.

I have personally seen the tool executed numerous times and can attest that the recommendations it makes are worthwhile. Please remember this point - the SQL Tuning Advisor often recommends that you implement a Profile that changes the SQL statement's access path. This tool is currently not a substitute for having a firm understanding of SQL access paths. The advisor will allow you to view the new access path. Review it, determine if you want to activate it and then monitor the new access path for the statement very closely. Don't blindly implement the profile without monitoring the performance of the new access path.

We can use the navigation tab on this page to navigate to the SQL Details Statistics Page. The SQL Details Statistics Page provides us with a wealth of information on this particular SQL statement. Note the large green area in the Activity By Waits pie chart. The page's Activity By Time section also denotes both high elapsed and high CPU times. In addition, the page provides us with other pertinent information including the number of times this statement was executed, buffer gets, disk reads, etc.. Once again we see a navigation button for the SQL Tuning Advisor.

If we click on the Plan navigation tab, DBConsole responds by displaying information on the SQL statement's access path. It's blatantly obvious in this case that we have a cartesian join occurring. We also see that DBConsole is displaying a Schedule SQL Tuning Advisor button.

The Tuning Information Panel provides us with historical information on past tuning changes. When we finally take Oracle's advice and run the SQL Tuning Advisor, it will display the activated profile on this page. If you have implemented several different profiles in the past, DBConsole will provide a listing of them on the Tuning Information Panel. You then have the option of deactivating one profile and reactivating a past profile if you desire. Here's a screenshot of a Tuning Information panel showing multiple executions and profile recommendations generated by the SQL Tuning Advisor. As I stated previously, the intent of this blog is not to instruct you how to use the advisor. I've covered that in a previous blog.

Let's continue our discussion by navigating back to the Database Performance Home Page. If we scroll to the bottom of the page, we can see a column heading titled "Additional Monitoring Links". If we click on the Top Activity link, DBConsole responds by displaying the Top Activity Page. This is the panel that I personally use the most when I am trying to find top resource consumers in DBConsole. The Top Activity Page displays both Top SQL and Top Sessions. If we click on the SQL ID link provided in each line of the Top SQL section of this panel, DBConsole will display the SQL Details Activity page we reviewed earlier in this blog. We are also able to drill down into more specific session information by using the drill down links provided in each line of the Top Session Panel. As I stated, when I use DBConsole to determine what resources are being consumed, I will navigate to the Database Performance Home Page and look at the graph at the top of the page. Although I may drill down into the resource, I most often find myself dropping down to the Top Activity link to display the Top Activity Page. It allows me to quickly identify both the top SQL and sessions.

There is a link displayed in the middle right hand side of the Top Activity Page titled 'Run ASH Report'. Since the Automatic Workload Repository, by default, takes snapshots every 60 minutes, performance information could be up to 60 minutes old. As a result, snapshots do not contain enough information to allow administrators to perform analysis on the active workload currently being performed in the database system. Oracle10g contains a new internal utility, called Active Session History (ASH), to provide administrators with access to current performance information. ASH samples data from the V$SESSION dynamic performance table every second and stores the information in V$ACTIVE_SESSON_HISTORY. The information contains the events for which current sessions are waiting. The information pertains to active sessions only; information from inactive sessions is not recorded. The view contains one row per active session for each one-second sample. Administrators are able to access V$ACTIVE_SESSON_HISTORY as they would any other V$ dynamic performance table. DBConsole also provides this handy report to provide you with information derived from statistical data stored in V$ACTIVE_SESSON_HISTORY. The ASH report provides a wealth of top resource consuming components.

The Top Activity Page also allows me to drill down into individual session information. Each Session ID displayed in the Top Sessions chart on the Top Activity page is a link to a set of session information panels. If we click on the Session ID link, DBConsole will resond by displaying the Session Activity page. The Session Activity Page provides us with information on the resources being used and navigation buttons that allow us to kill the session and enable or disable SQL Tracing. You'll notice that these buttons are displayed on all of the panels dealing with session information.

If we click on the General Tab, DBConsole will display the General Information Page. The General Information page provides us with a wealth of pertinent information on this specific session. Depending on the application's architecture, the Client information could help us identify and contact the specific user experiencing the problem. In addition, the panel provides information on the application and server. The most important piece of information displayed on this page is under the heading titled "Wait". It tells us what resource is currently being waited for, and in this case, the file and object. Great diagnostic information when you are trying to debug a performance problem. The Session Statistics Page is a dump of all the raw statistics for this session. The Open Curors Page displays information on the SQL statement or statements being executed by this specific session. It provides the text of the SQL statement and drill downs that allow you to access the SQL Details drilldown screens discussed previously in this blog. The Blocking Tree Page helps you determine if this session is being locked out by another process or locking out other processes. For more information on locking and blocking, please refer to my blog titled "10G Grid Control Hang Analysis Feature." Lastly, the Wait Event History Page provides drilldown capabilities into the specific waits this session is experiencing.

Let's take a look at another set of panels that provides information on top resource consumption. We will navigate back to the Database Performance Home Page and click on the Top Consumers link to display the Top Consumers Home Page. Although I don't use this page frequently when performing database triage, it does provide some very useful information. My personal favorite is the Top Modules by Service chart on the upper right hand side of the panel. It organizes the resource consumers by the type of service. This information is very beneficial when you have numerous sessions consuming a high level of resources because it allows you to determine if the problem is localized to a specific area of the application.

Currently, it is my SQL*PLUS service that is dominating our database resources. I have seen personally viewed report writer programs, application server programs, TOAD, SQL*PLUS, etc. all displayed on this chart. The information allowed me to quickly identify if the problem sessions were all coming from that one specific component of the application environment. For example, I once viewed TOAD being displayed as the top resource consumer during one of my investigations. I drilled down into the session information, found the accounts that were being used, stopped by the development area and found that a group of developers just had TOAD installed and were testing it by accessing production data. The panel provides drilldowns for Top Services, Top Clients, Top Clients and Top Sessions. The Top Sessions Panel allows me to once again drill down into individual session information. The panel also provides a link that takes you to the SQL being executed by the top sessions.

As we have found, the 10G DBConsole provides many different features that allow us to identify the top resource consumers in the database it is monitoring. With time and experience, you'll be able to quickly determine what sessions, SQL and components are the top resource consumers. In addition, you will also be able to determine what resource the top consumers are utilizing and the SQL they are executing.

Thanks for Reading,

Chris Foot
Oracle Ace

Introduction
If you haven't read my previous blog titled Using Deductive Reasoning and Communication Skills to Identify and Solve Performance Problems, I would highly suggest that you do so before continuing. The blog will provide you with recommendations that can be used to determine what architectural component is causing the problem. This blog assumes that you have determined that the performance issue is somewhere in the database ecosystem (database, operating system, hardware).

Determining the Scope of the Problem
We'll begin with the premise that the entire application is running slow. If the problem is localized to a specific transaction, you won't need to perform all of the steps I am providing. The next blog will be more pertinent for transaction specific performance problems. But you will be able to use a subset of the investigative activities in this blog because both sets of problem determination activities are very close regardless if it is a specific transaction or an application-wide performance problem. The remainder of this blog will focus on Host Performance Analysis. We'll continue the discussion on System Triage in subsequent blogs when we use the 10G Grid Control R2 toolset to indentify the specific transactions and SQL statements that are causing the performance problem.

Host Performance Analysis
We need to determine the health of the server that the database is running on. We are able to use 10G Grid Control's host performance analysis capabilities in conjunction with O/S commands to determine the current system load. In this blog we'll be using UNIX as an example. Although the commands may be a little different in LINUX, the output we will be evaluating will be very close. I'll cover Windows system performance in an upcoming blog.

Let's begin by activating 10G Grid Control R2 and navigating to the Host's Home page. We accomplish that by selecting the 'Targets' tab at the top of 10G Grid Control R2's Home Page. 10G Grid Control R2 displays the Hosts Home Page.

10G Grid Control R2 will provide you with a listing of all hosts that is monitoring. If you would like to learn how new database ecosystems are added into Grid Control, please refer to one of my previous blogs. Although the blog doesn't provide detailed installation guidelines, it does provide a high level overview of the Grid Control Management System.

Your next step will be to select the host that runs the database having the performance problem. Once the host's home page is displayed, you will want to click on the performance tab at the top of the screen to display the Host Performance Home Page.

You will want to review the utilization of the three primary resources: CPU, Memory and Disk. Take a look at the CPU chart, you'll notice that our server was experiencing a high level of CPU usage a short time ago. Here's another performance home page showing some definite CPU utilization problems.

Each of the primary resources displayed on the page provides drill down capabilities that allow you to further investigate the problem. The drilldown panels allow you to review past performance of CPU, memory and disk.

For more information on Using 10G Grid Control to evaluate Host Performance, please refer to the following blogs: Using 10G OEM Grid Control's Host Performance Monitoring and Tuning Features and Host Performance Monitoring Using 10G Enterprise Manager Grid Control. The information they provide will be pertinent to both 10G Grid Control R1 and R2 users.

Operating System Monitoring Tools
There is a host of UNIX performance monitoring tools that display performance information. Two of my favorites are NMON and TOP. Here's an NMON display verifying that we are indeed experiencing CPU problems.

NMON
Take a look at the CPU utilization display at the top of the NMON page. The letter 'U' designates the CPU being consumed by User processes, while the letter 'S" designates the CPU consumed by the system. If you see a high number of S characters on the display and very little U characters, it is time to contact your friendly system administrators and ask them why their system is consuming such a high level of CPU. If you see a lot of U characters and Oracle processes are the top resource consumers, your database is probably the culprit.

For more information on NMON (including how to show the top processes), please refer to the IBM NMON home page. IBM's NMON is a free tool that is available for download and is available for most UNIX and LINUX systems. I personally prefer it to over the Top command.

Top
Let's take a look at a Top screenshot. The important indicators are CPU States, Memory Utilization and Top processes. Like NMON, Top also provides settings to provide you with disk performance information. On both NMON and Top, you can use the Process IDs to confirm the information displayed on the 10G Grid Control R2 Host Performance Home Page.

Jonathan Lewis provides some helpful hints on the Top command in Appendix B of his book titled Practical Oracle 8i - Building Efficient Databases. Here's an addendum to the appendix that describes the output of the Top command that you may find useful. Although the book is on Oracle8i, the Top information is still pertinent.

Here's some additional information on the Top command that will help you understand Top. It is a little hard to read but the information it provides will be very useful. Lastly, if your system administrators have configured the MAN command, you can use the operating system manuals to retrieve informtion about most of the system commands we are discussing. If they don't have MAN configured, the first question you need to ask them is "why not?".

VMSTAT
The VMSTAT command is also available on many flavors of UNIX and Linux. VMSTAT provides you much of the same information that TOP and NMON do, but it also provides you with disk performance and system queueing information. Instead of me regurgitating information on VMSTAT, here is an excellent description of the VMSTAT utility. The author also shows you how to interpret VMSTAT output. Although the author's discussion pertains to LINUX, like other commands, the majority of information he provides will also pertain to the various UNIX flavors.

Two of the display columns I review frequently when I am performing host performance analysis are the numeric values listed under the "R" and "B" columns in the output. The numbers under the "R" column in VMSTAT designate how many processes in the system are queued up and waiting to run. The higher the number, the bigger the problem. The numbers under the "B" column designate how many processes are blocked and can't do anything.

IOSTAT
LINUX and the many of the UNIX variations provide the IOSTAT command to allow users to analyze disk performance. IOSTAT displays kernel I/O statistics on terminal, disk and CPU operations. By default, IOSTAT displays one line of statistics averaged over the machine's run time. The use of -c presents successive lines averaged over the wait period. Here's an excellent article that shows you how to use IOSTAT and interpret its output.

SAR
SAR is another command that can be used to evaluate host performance. One of the benefits that SAR provides is that it allows you to run it via CRON on a regular basis and spool the performance data to a SAR output file. Users are then able to use the SAR command to retrieve historical performance statistics. Once again, here's an article on how to use SAR to create historical performance reports. The article also provides information on how to intrepret its output.

Using 10G Grid Control in Conjunction with UNIX Monitoring Commands
10G Grid Control's Host Performance Home Page, provides you with a quick snapshot of the host's key performance indicators (CPU, Memory and Disk). Oracle's intent was to provide you with this information so that you would not be forced to log on to the operating system to evaluate CPU, memory and disk performance indicators. 90 percent of the time, I will use the key performance indicators provided by Grid Control. They provide me with just the right amount of diagnostic information I need to continue my investigation.

There are times when the system is so locked up that 10G Grid Control will "act up" when you attempt to access the host system's performance panels. That's a good indication that you are definitely experiencing some server resource problems.

In that case, your only choice is to revert back to the tried-and-true O/S commands that provide server performance information. I will also use O/S commands when I want to retrieve more detailed information than 10G Grid Control is able to provide.

Not all of the performance monitoring commands I have provided will be available on all flavors of UNIX and LINUX. Even if they are available, some of the commands must be specifically installed. You need to determine what tools are available, have your O/S admins install the ones that are and use the information in the links I have provided to understand them.

In my next blog, we'll continue our performance analysis using 10G Grid Control's database performance analysis features.

Thanks for Reading,

Chris Foot
Oracle Ace

Well, we all know those days are looong gone. We can also agree that application complexity will continue to grow and continue to complicate the problem determination process. From Service Oriented Architectures and BPEL to advanced disk drive hardware configurations (and everything in-between), our environments are growing more challenging by the moment.

Performance Problem Analysis - System Triage
One of the common mistakes that is often made during a tuning exercise is not gathering enough high level information during the initial stages of the problem determination process. I often have the same issue. I get an inkling of what I think the problem is and jump right in. As I stated in my introduction, the key to success is to begin the analysis at a high level and then continue by narrowing the scope of information collection until the problem is identified.

Once the problem is identified, we tune and tweak, record the same statistics we used to identify the problem and compare the before and after information. This iteration of recording/changing/recording comparing is done again (and again) until the problem is corrected.

Asking the Right Questions to Help Identify the Problem
OK, so you have been called about a performance problem. What do you do? Suggestion - blame the network! OK, I'm kidding. What you need to do is contact the parties affected by the problem and perform a little detective work.

Before we continue, let's take a quick look at the application architecture diagram for the sample application that we'll be using in this discussion. It is a simple multi-tier application that uses a lightweight application on the PC, a workload balancer (CoyotePoint, HydraWeb, etc.), one or more web/application servers and finally, an Oracle database back end. Denver and Atlanta are remote sites while the application hardware, additional business users and you are located in Dallas. Like most applications, each site runs a common set of programs with a few programs that are site specific. We also can assume that most load balancers are configured to transfer connections to the least utilized web/application server.

It's 2:30 PM on Friday. You are counting down to the weekend. You're thinking to yourself "this week wasn't so bad" when the phone rings. It's Mike the applications manager telling you that users are complaining about poor database performance and you need to fix it. The first question that comes to your mind is "Why does everyone always assume that it is a database problem?"

The DBA Detective
You decide to begin your detective work by going straight to the end-user (or users) experiencing the problem.

You ask Mike the manager for the end-user's contact information. He gives you the number for the business team lead in Denver. You make the call and begin your interrogation. Your first question is as follows:

How many users are affected by the problem?

If it is all users at a specific site:
You continue your investigation by calling the other remote site to determine if they are also affected by the problem.

If the other remote site is not experiencing the problem, continue you're your own investigation but you should also place an emergency call to your friendly network specialists and ask them for help. Explain to them that you have one remote site that is running fine, while the other site is affected by poor application performance.

If the other remote site is also experiencing the problem, call one of your favorite users in Dallas. Since the hardware for the application is also in Dallas, this is an easy way to determine if the problem is localized to just the remote sites. Ask the user in Dallas if they can run the application. If the Dallas users are running fine, the performance of the remote network is now a suspect. Once again, continue your own investigation but also ask assistance from network support.

If all sites are affected:
The list of possible suspects now grows. It could be any piece of the puzzle - network, load balancing, application server, database, hardware server. One of the tricks I have used to assist in my analysis of performance sensitive applications is to talk with the application developers and ask them to send me a half dozen to a dozen performance sensitive SQL statements from their application that don't perform data updates.

I will try and mimic the application as closely as possible. If the application is using stored procedures, it's easy, I'll create my own stored procedure. If the application uses bind variables, I'll code my program to use bind variables. Here's an example (courtesy of ace DBA Ron Berner) of how to use inline input to SQL bind variables that will be executed in SQL*PLUS. Please note that executing a SQL statement in SQL*PLUS requires that you define both input and output variables.

I'll run the statement when performance is good with timing on in SQL*PLUS to get a baseline. When I'm troubleshooting a performance problem, I'll execute the SQL statements and compare their performance against my baseline. I like to run the statements twice, once from my PC and another execution on the database server itself. If it runs fine on the database server yet runs poorly from my PC, the network is once again one of my prime suspects.

I'll l continue my own quick database ecosystem (database, operating system, hardware server) triage before contacting other units. I'll be looking at CPU, memory and disk utilization and identifying processes that are consuming large amounts of resources. I'll log on to the database itself and look at top SQL, session waits, locking and system event statistics for more information. If the database is 10G, I'll use an entirely different set of tools than I would with 9I. I'll cover the different toolsets in upcoming blogs. I want to continue to focus on high-level investigation activities in this blog.

If the statements run close to the baseline from both my PC and on the database server and my initial review of the database ecosystem looks like it is working as it should, I don't immediately rule it out as the cause. But it will affect the way I continue my analysis. Before I begin my own more detailed database ecosystem performance analysis, I will contact the technicians responsible for the application and network tiers. All groups involved can work in unison to identify the problem.

I'll ask the folks responsible for the application tier to log on to their web/application servers and review the same CPU, memory and disk utilization statistics that I would on a database server. The application tier folks can also review connection input/output queues. I will also ask them to compare performance statistics amongst all hardware components of the web/application tier. If one web/application server is at 100% utilization while the others are at 20%, the workload balancer now becomes one of the suspects.

I will also ask the network and server specialists to review network performance of all hops from the database server to the remote and local sites. As I stated previously, I don't arbitrarily rule out the database ecosystem if the SQL statement runs fine on the server, but I know that other components of the application also need to be investigated.

If it is a select group of users having a problem:
If Mike the Manager states that only a subset of users is experiencing a problem, you will need to identify the specific transactions they are executing. It's time to talk one-on-one with a business user that is having the problem. If only one user is experiencing the problem, ask them if they can log on to the application using a fellow user's computer and execute the same transaction. If performance is acceptable, ask the user to run other transactions. If they continue to run with acceptable performance, ask them to log on to their original PC and run an additional workload. If performance is once again poor, their PC and/or network connection could be the culprits.

If it is a group of users, or the user continues to experience poor performance on all PCs they try, it's time to get the developers involved and ask a few more questions:

• Is it the entire application running slow or just specific screens/transactions?
• How much slower is it?
• Is the problem intermittent or continuous?
• Has this problem occurred before?
• If so, is it getting worse?
• Does the problem occur at the same time every day?

After the line of questioning is complete, you will need to determine exactly what transaction is causing the problem. It's a relatively simple process if all users come in with unique accounts and/or program identifiers. Log on to Grid Control or 9I OEM, ask the user to log on and look at the sessions or top sessions screens. Both 9i and OEM provide components that allow you to drill down and identify what SQL statements a specific transaction is executing. In addition, both toolsets provide screens that show database locking information.

An application that uses a common ID and program complicates the problem identification process. Having an application developer on hand that knows the SQL being executed by the various transactions can be invaluable. Oracle 9I and Oracle10G both provide components that allow you to monitor the current SQL being executed. More on that later...

As I stated previously, I don't' want to get too in-depth in this blog. In upcoming blogs, we'll take a look at what tools you can use to identify poor performance when you find out that it is your database causing the problem.

The intent of this blog was not to persuade you to follow the steps it provides verbatim, it was to provide you with a few high-level recommendations to help you quickly determine what component is causing the problem. Look for more detailed information in upcoming blogs.

Thanks for Reading,

Chris Foot
Oracle Ace

In a recent interview with Bill Gates and Warren Buffet at the University of Nebraska, Buffet was asked what one piece of advice he would give to others wanting to be successful. He stated that students "should polish their public speaking skills."

As I wrote in a previous blog, take a look at your last performance appraisal forms, I'm betting that a lot of the criteria you are being judged upon depends upon communications. The key words and phrases to look for are "ability to work in a team environment", "keep supervisors informed", "maintain good communication with the user community", "ensure the content of the communication is at the appropriate level for the intended audience", "provide system and user documentation for projects and system enhancements." I pulled all of the aforementioned phrases verbatim from one of my own recent performance appraisals. I reviewed all of the criteria that I was being evaluated upon and found that almost ninety percent of the items depended upon verbal or written communications.

So, let me get off my soapbox and get to the topic at hand. How exactly do you improve upon your verbal and written communications? If you don't have good communication skills, all is not lost. Like anything else, these skills can be learned.

Improving Verbal Communications
We have all heard that speaking in front of an audience is the number one fear for most folks, surpassing both financial and health problems. Let me give you a couple of hints and tips that have helped me. Although, I still don't view myself as an accomplished speaker, I'm a LOT better than I used to be.

Toastmasters International - I can not recommend this organization highly enough. Toastmasters International is a non-profit organization that focuses on improving communication and leadership skills. Toastmasters is also a great place to network and advance your career (it makes for an impressive bullet on your resume).

Toastmasters International consists of hundreds of Toastmasters Clubs that are governed by a Board of Directors that is elected by Toastmaster members. The Toastmaster website provides a search function that allows visitors to find nearby clubs. If you want to learn how to speak effectively in public, do yourself a favor and search for the clubs in your area. I think that you will be surprised at how many clubs there are.

Everyone that attends a Toastmasters event is there for one reason - to improve their public speaking and leadership skills. You are not only taught the skills you need but you are also provided with the opportunity to showcase your new found talents by giving various speeches to fellow members. The key benefit of speaking at Toastmasters is that you are giving speeches to others in a supportive environment. Everyone there wants to improve their communication and leadership skills. A great place to become an accomplished speaker.

Formal Classroom Education
It would be a rare event indeed if you found an institution of higher learning that didn't provide classes on public speaking. If you are attending school to obtain that next degree, you need to make sure one, or more, of those classes pertains to verbal communications. The more the merrier. Many colleges offer adult education curriculum at night. Review the night class curriculum for local colleges. I would be surprised if you didn't find one or two classes on public speaking.

Books
This list will be short. I am an avid fan of anything written by Dale Carnegie. My all time favorite book from this author is titled The Quick and Easy Way to Effective Speaking. Amazon carries this book in stock. The benefit it provides far outweighs its current price tag of $7.99. If I could only have one book on public speaking, it would be this one.

Personal Recommendations
Here's a quick list of recommendations that have helped me improve my own public speaking skills:

• When you attend presentations from others, pay attention to both the material they are presenting and how the present it. Two of the most engaging speakers I have ever had the pleasure of meeting are Craig Mullins and Richard Niemiec. I actually attended one of Niemiec's presentations a second time. The first time I attended was to learn the material he was presenting and the second time I focused on the techniques he used to engage the audience.
  
  I once saw a representative from Microsoft give a presentation on their new operating system to a group of 700 people. By the end of his speech, he had the entire audience "high fiving" each other on a regular basis. I thought to myself, "he generated this much enthusiasm over an operating system?". I stayed right where I was and watched him give the presentation to a second group of 600 with the same result. The second time I paid close attention to his presentation style and stage mannerisms.I'm not saying that you should attempt to clone your presentation style from a particular speaker, but you can improve your own communication skills by learning from accomplished speakers.
  
  
• The more you speak in front of an audience, the better you will become. Absolutely, positively no doubt about it that "practice makes perfect" in this case. Start your public speaking career by starting with a small audience. Maybe you know a particular database feature or a tip or trick that you think your fellow technicians may benefit from. Invite them to a meeting, create a presentation and present it! Work your way up to larger audiences. Join local users groups and ask if they would allow you to give a presentation. Every time there is a call for presentations from IOUG or Open World, I submit several presentations. I may not get picked all of the time but I still try.
  
  
• I have found that the more prepared I am, the less I am affected by anxiety before and during a speech. I will present a speech to myself, my wife, my dogs numerous times before I give the speech in public. The more times I do it, the more confident I am.
  
  
• I always visit the room I am going to speak in. It makes me more comfortable when the time comes to stand up in front of the audience. I also stand right in the middle of the aisle and introduce myself to as many participants as I can. It helps me to feel more comfortable when I step up on stage.

Improving Written Communications
Have you ever read a document or e-mail that was so poorly written that you had a hard time understanding what the writer was trying to convey? What was your opinion of the author? Not good, I'll bet. Whether we like it or not, we will be continuously judged upon our written communications througout our careers. From entry-level to CEO, you will be judged on the words you write.

You'll find that these recommendations closely parallel my public speaking recommendations. They focus on formal education, gaining experience and best practices.

Formal Classroom Education
Universities, colleges and high schools all offer adult night classes. Like public speaking courses, you will be able to find a local educational institution that offers classes on effective writing skills. I still have a couple of my college textbooks on my desk that I use as references.

Books and Websites
You need the tools of the trade to write effectively. As I stated previously, I still use a few of my aging college textbooks as references. I also have Anne Stillman's book appropriately titled Grammatically Correct: The Writer's Essential Guide to Punctuation, Spelling, Style, Usage and Grammar. This is the book I refer to about 90% of the time when I am attempting to improve my grammar, spelling and punctuation.

I also visit one website regularly when I write. If you hop on to your browser of choice and navigate to www.thesaurus.com or www.dictionary.com, you will go to the same website. This very helpful website allows me to quickly find definitions, synonyms and antonyms. If I want to obtain information specifically on grammatical rules and concepts, I'll visit the Guide to Grammar and Writing website. The numerous dropdown lists allow me to quickly navigate to the topic I am looking for.

You can also take advantage of Microsoft's Spelling and Grammar Checker tools in Word. You will need to be careful with some of the recommendations. Use Anne Stillman's book or the Guide to Grammar and Writing website as a reference if you suspect that the Microsoft tool's recommendation is incorrect.

Obtaining Assistance from Accomplished Writers
When I first started working in a corporate environment, my writing skills were terrible. But I had the good fortune of having a manager that understood the importance of both verbal and written communications. I would write a memo, she would correct it with her red pen and send it back to me for a rewrite. Many of them had a "Nice Try!" and a smiley face on top. Even though all of the rewrites (and smiley faces) were somewhat exasperating, her persistence forced me to become a better writer.

I also had the good fortune of being able to use a very accomplished writer, Craig Mullins, as an editor during the initial stages of my writing career. His first piece of advice stays with me to this day. I told him that I often get writer's block when I first start. He stated "Hey, who says you have to start at the beginning of the article? After you do the initial article layout, start anywhere you want to. If your initial design and layout are good, it will be easy to tie it together."

It's relatively easy to find fellow technicians that excel at written communications. Think about all of the e-mails and documents that come across your desk on a daily basis. You can read the content and also review the text for ease-of-reading and grammatical style. Find the folks that are good at written communications and ask for their input. I think you'll find that most of your requests will be warmly received.

Practice Makes Perfect
Like public speaking, the more you write, the better you'll get. That's one of the benefits of writing this blog. It keeps my skills sharp. When I first started this blog, I would often agonize over every written word. Although I still get stuck from time to time, the words now flow a little more easily for me.

You also need to become involved with as many company newsletters and related communications as you can. When I asked to join a newsletter, I always started with "I'm not the greatest writer, but I'm trying to learn." I also asked my peers that worked on the newsletter to critique my work. The more I was critiqued and the more I plugged away at writing, the better I became.

Wrapup
IT shops are no longer evaluating technicians purely on their technical skills. It is the total package of skills that you bring to the table that you are being evaluated upon. We all know the importance that our technical skill sets have upon our success in this field. But you also need to be well rounded in all of the skill sets your managers are looking for.

Thanks for Reading,

Chris Foot
Oracle Ace

I was also employed as an Oracle Certified Instructor for a number of years. The greatest accomplishment in my career came was when I was awarded the OAEC Quality Achievement Award. These awards are given to the top instructors in the OAEC (Oracle Authorized Education Center) program. That recognition was quickly followed by Oracle allowing me to certify a few of my fellow instructors. Certifying my fellow instructors was a responsibility I took very seriously. One of the folks I certified continues to have a very prosperous career in Oracle Education. It should be interesting if I ever attend one of his classes in the future. "The student becomes the teacher" syndrome would be an interesting dynamic. I'll try not to critique him in the middle of his class…

Oracle Classroom Education
When is the best time to take the classes? This may sound trite, but it is best to follow Oracle's recommendations on the sequence of classes. Take the intro classes before taking the more advanced classes. If you have the luxury (meaning you aren't the only DBA in your shop), gain some day-to-day experience before taking the more advanced classes (SQL or database tuning, backup and recovery, etc.). You shouldn't be asking questions like "What is an init.ora parameter file, anyway?" in a tuning or backup and recovery class. Instructors don't have the time and your fellow students won't have the patience to bring you up to speed before continuing on to more advanced topics.

Preparation
Being prepared is the single most important factor to obtain the most benefit from any formal training class. I can't stress this highly enough - PREPARATION IS THE KEY TO SUCCESS.

It's a pretty simple process to prepare for class. Oracle Education's Website will provide you with all of the information you need to prepare for your upcoming class.

Step one is to read the information provided by the Oracle Corporation that pertains to the class you are attending. Sounds simple enough, right? That's because it is. Oracle Education's website provides a detailed description of each class it offers. It will tell you EXACTLY what to prepare for. Go to the Course Curriculum page on Oracle Education's website, find the course you are attending, navigate to the course description page and then click on "Course Objectives" and "Course Topics". You will find a detailed listing of all of the topics the course will cover.

If you are a new Oracle DBA and will be attending your first database administration class, read the Oracle Concepts Manual, Administrator's Guide and review the Reference and SQL Reference books. Follow this same advice for whatever class you are taking. If it's a new features class, read the new features guide for that release. Tuning and performance - read the Performance Tuning Guide. Same thing for backup and recovery, standby, RAC, etc.. Do yourself a favor and perform some self-education first. You may not know the mechanics, but you'll be ahead of the game if you know the lingo and the concepts used.

Showing Enthusiasm and Willingness to Learn
If an instructor sees that you are making an extra effort to overcome your lack of day-to-day experience by coming in early, staying late and being prepared, they will be more prone to help you.

It's basic human dynamics. If the instructor walks by your work area and sees you continuously reading "NFL Today" when you should be doing labs, they are going to spend less time with you and more time on students who are exhibiting a willingness and enthusiasm to learn.

Instructors are weird; we like to see people excited about what we are teaching. Seeing someone enthused about learning makes us want to make sure they get the most out of class.

Class Participation
Class participation means more than just answering questions that are asked. You also need to ask questions. If you still don't understand, ask again. If you are having a hard time understanding a particular concept, ask the instructor during a break to explain the information to you again.

Trust me, your instructor will be glad to help. Instructors don't choose training as a profession because they like living out of a suitcase; education is a passion for them. I knew it was time for me to retire when I woke up one morning in a hotel room, looked out the window and couldn't remember what city I was in. After I cleared my head (and looked at the local phone book), I found out where I was and started contemplating that it may be time for a career change. Coming in my front door that weekend and having my German Shepherd attack me because he forgot who I was pretty much confirmed my decision.

If you are experienced in Oracle and you have a lab partner that isn't as well versed in Oracle as you are, take the time and help them. Don't take control of the keyboard and blast through the labs. Be patient with them, show them the little tricks of the trade you picked up and help them along. You'll probably have their undying gratitude and you'll be contributing to the health and welfare of our profession.

If you are an Oracle expert taking one of the more advanced classes, don't let your ego get in the way of you obtaining the utmost benefit from the class you are taking - ask questions and get involved!

Over the years, I found that the discussions that were held during class breaks were often just as illuminating as the instructor led training. One DBA would ask a question about a problem that occurred or the challenges they were facing. Their fellow DBAs would immediately provide recommendations to help them. Don't be afraid to ask your fellow DBAs and instructor for advice. If you have a problem, an Oracle class is the perfect forum to obtain the information you need to get it resolved.

Familiarize Yourself with the Next Day's Material
I know this can be hard to do. Especially if you are attending class in a place you've never been to before. It's a tough decision between sightseeing in San Francisco or planting your nose in an Oracle Admin I class guide. You'll need to make some compromises here. Go out for a few hours, come in early and prepare for the next day's class.

The Importance of Becoming Certified
Becoming certified is an essential ingredient for all Oracle technicians who want to prosper in their chosen profession. Certifications are especially important to those seeking employment in a field that often has many candidates competing for a single position. An Oracle certification shows the potential employer that the candidate has made the commitment to "learn their trade" and has the background and education to quickly become a productive member of the their staff.

The Oracle Certification also gives a distinct advantage to DBAs looking to advance in their careers. Being an Oracle Certified Professional raises their visibility in the organization, which is a requirement for career advancement in most shops.

Preparing for the Oracle Certified Professional Exams
The best time to take the exam is a week or two after taking the Oracle class that the exam pertains to. Passing the OCP test is much easier when the information is fresh. The class workbook should be used as the primary study guide. I have passed every exam I have taken by studying only the information contained in the class workbooks.

The Oracle Education website allows students to purchase practice exam tests. Free sample questions are also available. Practice tests will provide you with a firm understanding of the areas you are strong in as well as the areas where you need to shore up your knowledge. Oracle University has chosen two partners (Self Test Software and Transcender) to provide authorized Oracle Certified Practice Exams.

I am fully aware that there are many competing websites that offer Oracle Certification practice tests. If you want to prepare for an Oracle Exam by purchasing a practice test, why not purchase the test from the vendors that Oracle has chosen to partner with?

Self Test Software and Transcender practice tests provide a thorough coverage of the Oracle certification requirements and use the same test question technology as the real exams including simulations, scenarios, hot spots and case studies. This makes the choice clear - go with the Oracle partner practice exams.

The Day of Your Test
Oracle partners with Prometric Testing Centers to provide testing centers throughout the United States. The Prometric Testing Center website provides a test center locator to help you find testing centers in your area.

The following hints and tips will prepare you for the day you take your certification tests:

• You must have two forms of identification, both containing your signature. One must be a government issued photo identification.
• Try to show up early (at least 15 minutes) before your scheduled exam. If you show up late, the testing center coordinator has the option of canceling your exam and asking you to reschedule your test.
• You cannot bring any notes or scratch paper to the testing center. Paper will be provided by the testing center and will be destroyed when you leave.
• Testing center personnel will provide you with a brief overview of the testing process. The computer will have a demo that will show you how to answer and review test questions.
• Don't leave any questions unanswered. All test questions left unanswered will be marked as incorrect.
• Your exam score is provided to you immediately and the exam results are forwarded to Oracle Certification Program management. Make sure you keep a copy of your test results for your records.
• If you fail a test, you must wait at least 14 days before retaking it (except for exam #1Z0-007 Introduction to Oracle9i: SQL).
• Don't let your ego prevent you from studying. I have seen numerous experienced DBAs receive "less than stellar" test scores because they thought they knew the material they were being tested on.

Wrapup
I hope you enjoyed this blog on Oracle Education.

Thanks for Reading,

Chris Foot
Oracle Ace

Initial Analysis
Clarifying the business needs takes place during this phase. Individual, departmental and corporate requirements are evaluated. The focus of the analysis is on solving the business or technical problem as opposed to buying a product. End users may have conflicting requirements. Needs clarification helps the users in synthesizing a common statement of the overall requirements. Analysis to determine if existing products can solve the problem is performed at this time. The make vs buy analysis is also completed during the initial analysis phase.

The following questions need to be addressed during the initial analysis:

• Can the solution be provided with existing product sets?
• Should the product be purchased or built in-house?
• Are other business or technology units able to utilize the product?
• Does the product provide any other additional benefits?
• What is the impact of not solving the business or technology problem?

Determine Impact to the Information Technology Support Infrastructure
The product's affect on the support infrastructure is determined during this phase. Although the major focus should be on solving the problem identified in the initial analysis, it is proper to allow external considerations to affect the final outcome. A product is useless if it cannot be successfully implemented and integrated into the existing information processing environment. The product's impact on individuals and groups as well as its impact on existing products and technologies is determined at this time.

Collect the following information during this phase of the analysis project:

• The use of new products often increases development time and training costs. Can the additional time and higher application development costs be justified by an increase in functionality or competitive advantage?
• If the product requires support from the development area, can the application personnel effectively administer the selected product?
• Risk vs Benefit. Is the additional functionality/competitive advantage worth any additional risk the product brings to the business or IT units? This is often described as the product's comfort ratio.
• Identify changes to the organizational infrastructure required to support the product (staff additions, training, etc.)
  • Identify changes to other technologies and products.
  • Identify additional products required to support the selected product.
  • Identify changes to existing technologies and products the selected product will require.
  • Identify changes to policies and procedures the selected product will require.

Analysis Evaluation
The analysis evaluation determines the resources required to evaluate, select, implement and support the selected product. Some problems may not be cost-effectively solved by existing technologies. Evaluation personnel must estimate the costs required to evaluate the competing products. In other words, don't spend $100,000 to evaluate a $1,000 product. As a rule of thumb, the evaluation, selection and procurement costs should range from three and ten percent of the project budget. The time required to perform the evaluation is also determined during this phase.

Perform the following activities during this phase:

• Determine and estimate the resources required to further evaluate the business needs.
• Determine and estimate the resources required to perform the product selection.
• Estimate general implementation and customization costs.
• Estimate on-going support costs, including staff requirements and training.
• Determine timelines and deliverables for the evaluation, selection and implementation processes.
• Determine date when the recommendation will no longer be valid (recommendation life-cycle).

Obtain Business Unit and IT Management Commitment
One of the major activities of any evaluation process is acquiring management commitment for the evaluation, selection and implementation stages. Representatives from user and technical management need to be a part of the evaluation process to ensure their continued support. Management always asks the same questions. As a result, it is relatively easy to prepare for them. A satisfactory answer to "What business problem are you trying to solve?" is usually followed by "How much will it cost to purchase, implement and support?", and "What is the risk if we don't solve the problem?". Management wouldn't be doing their job if they didn't ask these questions. To be successful in this phase, you need to be prepared to answer them.

Obtain business unit and IT management commitment on the evaluation process, selection process and possible implementation by providing the following information:

• Reasons why you are performing the evaluation (What business problem are you trying to solve?)
• How the product solves the business problem.
• What you intend to test.
• Time and cost estimates for product selection, product purchase, implementation and on-going support (Is the solution worth the cost?)

Create Evaluation Team
The evaluation team is created in this phase. For products that do not have a wide impact and are relatively inexpensive, the evaluation team will consist of a few select individuals. But for products that have a wide impact or are expensive, it is prudent to involve a larger group, some with a technical background, others with a user perspective. Wide representation and diverse viewpoints obtained early in the decision process produce better decisions and fewer surprises later. It is imperative to include all those affected: IT personnel who will be supporting the product, IT groups affected by its usage and end-users. Team leaders are also identified during this phase.

The evaluation team consists of representatives from:

• Infrastructure support (personnel who are experienced in related technologies or who would be affected by the product implementation).
• Application developers who will be supporting the application.
• Business unit champions and end-users.

Locate Potential Vendors
Identifying the vendor offerings that will be evaluated takes place in this phase. The problem is usually too much information as opposed to too little. The best way to create a potential vendor list is to take a structured view of the marketplace and investigate all avenues to give your evaluation team the best possible chance of finding the alternatives.

A few sources of locating potential vendors follow:

• Internal personnel who are experienced in related technologies (evaluation team).
• Industry research groups- Gartner, Forrester, Burton, IDC.
• Internal research - trade journals and trade shows.
• Vendors that have existing relationships with your company (i.e. IBM, MSOFT, Oracle, etc.)
• Internet search engines.

Initial Elimination
This phase will eliminate products that should clearly not be considered as viable alternatives. The overall intent is to reduce the number of candidates being evaluated. With each additional candidate, the cost of the evaluation increases. The key to a successful elimination process is to create an in-depth set of evaluation criteria that will be used to evaluate the potential candidates. Using weighted measures helps to identify the measurement criteria that are most important to solving the business problem. If the product passes this phase, it is passed to the vendor evaluation phase for a more thorough examination.

Perform the following activities during this stage:

• Create the vendor selection process (methods used to perform the evaluation) and evaluation metrics (evaluation criteria used to compare the products). Document both the evaluation methods and metrics.

  The evaluation metrics will include:

  • Vendor Assessment (financial stability, size, alliances, etc.). The best product doesn't always win. The more expensive the products are, the more time you need to spend reviewing the vendors making them.
  • Functional requirements - evaluate product based on its ability to solve the business problem.
  • Technical requirements - evaluate product based on its technical capabilities.
  • Create a general requirements document containing a compliance matrix that uses weighted measures to show how each vendor meets the requirements.
    
    
• Create and distribute a Request for Information (RFI) document using the evaluation metrics created above as the basis for the document.
• Evaluate the vendor responses to the Request for Information request.
• Reformulate the requirements based on vendor input.
• Create a vendor short list to keep evaluation costs to a minimum.

Vendor Evaluation
An in-depth evaluation of each of the remaining products takes place in this phase. The processes and methods created in this step are more detailed than those created in the initial elimination phase. Vendor responses to the Request for Information may also require alterations to both evaluation processes and evaluation metrics. Remember that an individual product's feature set is usually a compromise of function, speed and cost when compared to its competitors. You need to determine which factors are most important to you and weight them more heavily during your evaluation. A Request For Proposal (RFP) helps to formalize the evaluation process. The Request For Proposal details the scope of the sales process and defines the procedures and rules to be followed by the vendors. Vendors are more enthusiastic about committing resources to the sales process when it is clear to them how they are to respond and how their responses will be evaluated.

Perform the following steps in the Vendor Evaluation phase:

• Create a final vendor selection process (methods used to perform the evaluation) and weighted evaluation metrics (evaluation criteria used to compare the products).
• The evaluation processes and metrics created in this phase are more in-depth than those created during the Initial Analysis phase.
• Vendor responses to the Request for Information may require alterations to both evaluation processes and evaluation metrics.
• Functional requirements - evaluate the product based on its ability to solve the business problem.
• Technical requirements - evaluate the product based on its technical capabilities.
• Create Request for Proposal (RFP)
  • The Request for Proposal is more formal than the Request for Information.
  • The document details the scope of the sales process (meetings, presentations, product demos, trial periods, timelines and deliverables).
  • It also defines the procedures and rules to be followed by vendors during the evaluation process. Ground rules are written and shared with the qualified vendors.
• Notify the vendors of how they will be evaluated (overview of compliance matrix).
• Vendor meetings and presentations are held during this phase.
• Create a questionnaire and contact the vendor supplied customer references.
• The evaluation matrix is used to measure product conformance to the business unit IT unit requirements.

Communicate Results
The results of the evaluation process are communicated during this phase. There are many communication avenues to choose from. A formal evaluation document is created and distributed. Presentations detailing the evaluation process are also given. Before you make your recommendation, be prepared to justify it. The major questions to ask yourself are "Did you choose the best product?", "Did you keep the level of risk at an acceptable level?" and "Did you achieve these objectives at a reasonable price?"

The documentation provided to business and IT management includes:

• An executive summary.
• A detailed description of the vendor chosen.
• A financial analysis of the chosen vendor.
• The reasoning behind the vendor's selection.
• What other products were evaluated and the reasons why they weren't selected.
• An overview of the evaluation process.
• The metrics used as the basis for evaluation.
• An overview of the responses to the Request for Information and the Request for Proposal.
• The benchmark results of all competing products, if any.

Wrapup
I hope you enjoyed reading this blog on third-party product evaluations. As I stated in the introduction, the intent of this blog was to not influence you to follow the steps provided verbatim. This contents of this blog should be considered as a set of general recommendations that can be used to increase the quality of the third-party product evaluation process. Happy evaluating!

Thanks for Reading,

Chris Foot
Oracle Ace

Communication Skills
The importance of improving your communication skills cannot be understated. I don't care how strong of a technician you are, if you can't communicate effectively with your peers, you won't be able to succeed in this profession. In the old days, you might have been able to get by with just your technical skills. That is definitely not the case in today's business world. This will be a reprise of a previous blog which I feel bears repeating in this series.

Knowing More than Just the Database
One of the benefits of being a database administrator is that you are exposed to a wide range of technologies. From application design to the operating system that the database runs on, you'll need to "know more than just the database" to be successful in this profession.

In many shops, the DBA is known as the "GOTO" technician when problems occur. DBAs are given this role (whether they like it or not) because they have a reputation as having strong problem solving skills and are known to possess an in-depth understanding of the entire application architecture. As a result, the DBA is often viewed as the IT staff's technical generalist and is often called upon to provide advice on many different facets of information technology. I'll provide a few helpful hints on what technologies you'll need to learn and what the best sources are to help you quickly educate yourself.

Obtaining the Most Benefit From Oracle Classroom Education
I've been teaching Oracle, on and off, for most of my career. I have taught literally hundreds (and hundreds) of students during that time. The greatest accomplishment in my career came was when I was awarded the OAEC Quality Achievement Award. These awards are given to the top 4 instructors in the OAEC program. Like most instructors, I took my teaching responsibilities seriously. It was my job to ensure that all students taking my classes were prepared for the challenges that would face them. But it takes more than just a good instructor to ensure the transfer of technical information from teacher to student successfully occurs. The recipients (students) must be ready, willing and well prepared to receive this information. It truly is a "two-way street". Now that I am officially retired as an Oracle instructor, I thought I would provide you with a few unofficial hints, tips and tricks that will help you to obtain the most benefit from Oracle Classroom Education.

Evaluating Third-Party Products
As we learned in previous blogs, database administrators are much more than just "table jockeys." DBAs often become involved in the evaluation of third-party business applications, application development tools and database administration and monitoring products. Over the years, I have developed a Product Evaluation Methodology that you may find helpful.A methodology can be loosely defined as a body of practices, procedures and rules used to accomplish a given task. The task in this case is to evaluate and select information technology products.

You are Moving into Management - Now what?
I've stated more than once that being a good technician doesn't mean that you'll make a good manager. But most of you can learn. If I can do it, you certainly can. I'll provide you with a primer on what to do when your manager calls you in their office and offers you that promotion "you've always dreamed of." I have had the good fortune of being supervised by folks that I would rate as being among the best managers in the IT industry. In the last twenty years, I've worked for many different businesses and I've seen my share of both good and bad management styles. I'll show you what to do and not to do to make your migration from DBA to manager a success.

The Non-Technical Art of Being a Successful DBA - Wrapup
I'll spend the last blog of this series summarizing what we have learned. This blog will also provide me with one last chance to reinforce my opinion that it takes much more than technical expertise to become a successful database administrator.

Thanks for Reading,

Chris Foot
Oracle Ace

The Perils of our Profession
One of the common challenges that all DBAs face, no matter what vendor's database they work on, is the absolute attention to detail our profession demands. Switch a couple of characters in a script, forget to set your SID, set the wrong flag at the wrong time and the end result usually isn't very pretty. Many commands we issue on a regular basis are destructive by their very nature. This is the reason why I have a great respect for all technicians who have selected database administration as their chosen profession.

I know they have all experienced that uncontrolled "eye-twitching" at 2 AM when they are ready to hit the final enter key to execute the command. You know what command I'm talking about too. It's that one command that you really, really, really hope is going to come back with a successful return code and ultimately end with a database that is finally usable. Whether it's a recovery, a file fix or corrupt data is immaterial, it's the wait that we are talking about.

There is no longer wait in our profession than waiting for the message below after a database recovery:

SQL> Database opened.

Time always seems to stand still. The longer the recovery, the messier the recovery, the more critical the database - the longer you wait. You stare at the screen hoping beyond hope that the above message will appear. It's the ritual cross your fingers, spin around three times, face towards Oracle headquarters and pray to everything that is Larry Ellison wait. I've actually caught myself mumbling "come on, please open, please open…" I don't care how sure you are of your capabilities - you know the anticipation I'm talking about.

You then either breathe a sigh of relief or you are in absolute disgust when you see an Oracle error message appear. How about the old "File 1 needs more recovery to be consistent" or the "File 2 not restored from a sufficiently old backup"? Those messages are enough to make anyone cringe. I'm an ex-Oracle instructor, I've seen those messages A LOT in class. I still cringe.

At a previous job, I once had to run through 36 hours of tapes to restore a multi-terabyte warehouse. A disaster occurred that required us to do a recovery. THAT was the longest wait for a database open message I ever experienced. One of my fellow DBAs asked if I needed a brown paper bag to breathe into.

Or it's the command that drops the schema in the test environment that will allow you to do a refresh from production. It's that test database that runs on the same box as production. The environment that makes you do a "SELECT name FROM V$DATABASE" command in SQL*PLUS about 15 times in a row before you finally execute the "DROP USER CASCADE" statement.

Not only must we try to prevent our own mistakes, we must safeguard our environments against the mistakes of others. Operating system administrators, disk storage technicians and application developers are just like us. We are all part of the human community that makes mistakes from time to time.

If you never make mistakes, send me a resume. I'm always looking for a "Patron Saint of Oracle" here at Remote DBA Experts. It will also save us on travel costs because I'm sure you'll be able to spread your wings and fly here on your own.

But as my old boss Dan Pizzica used to tell me (when I was a VERY junior DBA) "It really doesn't make a difference who broke the database. You are the technician who is ultimately responsible for fixing it. The buck stops with you. If you can't protect your environments, you aren't doing your job." We all know he's absolutely correct.

Then there's the software glitches. The problems that pop up out of the blue and make you go:

"WHAT THE ? - how did THAT happen? I've done this 317 times in a row and it worked every time."

For you math majors, here's my calculation for this:

I don't care what software you are using, you will run into the "only occurs on this release, on this version of the operating system, using this particular feature on the third Tuesday of the sixth month when it's cloudy outside" BUG. Be sure to expect management to stop by and ask "well why didn't you test this on the third Tuesday of the sixth month when it was cloudy outside?"

The more complex the database ecosystem, the more paranoid I become. Which is why I'm not a follower of "the database is getting so easy - we won't need DBAs" mantra that mindless industry pundits profess on a seemingly endless basis.

So now we know that our jobs are somewhat unforgiving and we do make a mistake from time to time. What can we do to reduce the chance of an error occurring?

The Second Set of Eyes
As I have stated in previous blogs, I have over 20 years of experience using Oracle and have done my fair share of database backups and recoveries. During my career as an Oracle instructor, I have assisted in hundreds of database recoveries in Oracle's classroom environments. If possible, I still have others review my recovery strategy and recovery steps before I begin the recovery process. I used backup and recovery just as an example. Whatever the process is you are are performing, a second opinion may prevent you from making a mistake. A review from a fellow DBA has saved me more than once. I may be described as having an ego (I have no idea where they get that opinion) but it doesn't prevent me from asking for help from others.

We were recently correcting a very poor backup script created by a customer's previous database support vendor. The customer described this particular environment as "if it goes down we lose our ability to make money" application. After our massive set of changes were complete, two of us went line-by-line verifying each line of the backup script. At the end of each script we asked each other "are you OK with this?". Only then did we move on to the next one. I don't care how much time you have "in the seat" using Oracle, you need to put your ego aside at times and have someone check your work on critical activities.

Concentration
I used to work for a shop that subscribed to "the everybody in one big room" philosophy. I guess it was supposed to allow everyone to work together as a team and become as "one with each other". It may have achieved that purpose but it sure didn't allow you to concentrate on your work very well. You could hear so many different conversations they had to pump white noise in. The constant 'whhhsssssshhhssshhh" noise made me feel like I was a crewmember of the Starship Enterprise. I always wondered when Captain Picard would walk through.

Like all DBA units, our particular area was often populated with various developers and O/S technicians. Many different conversations were occurring, some that could be described as somewhat animated. The environment did not allow you to concentrate on the task at hand. We often had to go into small conference rooms to work on critical tasks.

The point I'm trying to make is that no matter what type of environment you work in; if you can concentrate OK, but if you are like me and you can't, find a spot where you can. Block off some time, send questions to other DBAs and concentrate on the task at hand. Don't attempt to answer questions and code a complex script at the same time. May seem obvious, but throughout my career I have personally watched numerous DBAs attempt to multitask when they are working on a critical process. It's a recipe for a problem. Once you are done, follow rule number one and have someone review your work.

What Database Are You Working IN?
Working in the wrong database is a common problem for database experts as well as their less experienced counterparts. How many times have YOU found yourself running statements in the wrong environment? Feel free to include me in that not so select group. The operating system command SET can be used in Windows systems to display environment variables. The ENV command can be used to display the environment variables in UNIX. Many seasoned database administrators change their UNIX shell prompt in their.profile to display the current Oracle SID. Displaying the current Oracle SID in the shell's prompt provides a continuous reminder to the DBA of the database they are working in.

The entire paragraph above was from a recent blog I wrote on the new capabilities of 10G SQL*PLUS. I highly recommend you read this blog and use the 10G Version of SQL*PLUS to take advantage of this safeguard.

Setting your SID Automatically During Logon
Here's an easy recommendation - don't do it. Many of the servers we work on contain several databases that use different database software releases. It is our standard to not set environments in logon scripts. We create shell scripts that have the database name in their title to set our SID, ORACLE_HOME, etc..

It is a little harder in Windows environments. During installation, Oracle often uses the first database created on the server as the default SID. You can override this configuration, but you'll need to read the Oracle installation guides to find out how. The manuals will show you how to override the default database on Windows operating systems.

Saving Time VS Creating a Problem
I once watched a fellow DBA perform a rather complex set of administrative tasks to solve a problem. He was rapidly flipping back and forth between at least 15 active screens, copying and pasting and editing and copying and pasting and editing… I describe this particular activity as "Multiple Screen Syndrome". He also had several other screens open that were connected to other databases. He was multi-tasking to its highest degree.

How about the rm -r /u0*/ora*/prod*/*/*.* command in UNIX? It's the command that drops multiple databases in multiple directories. All in one painful swoop. How many times have you heard of a mistake caused by commands like this causing mass mayhem? When you make a mistake like this, you become immortalized in conversations for years to come. Get a few technicians together after work and ultimately the conversation will include "remember when Bob so-and-so ran that big rm -r command by mistake and wiped out the entire O/S on our production web server? You can't tell me you haven't heard stories like this.

My opinion as a database manager is that I would rather you take your time than showcase your multi-tasking and time saving skills. The more complex and critical the activity, the more basic you should become in your plan of attack.

Safety First Mindset
I once saw a DBA log in to a database using a particular schema account. He then logged into a different database using a DBA account and dropped the schema with the same name he was logged into on the first database. I asked him why he logged into the first database using the schema account he just dropped in the second. He stated "Oracle won't let you drop a schema that is connected. No matter what happens after this, I'm positive that I won't drop the user in this database by mistake." I like that Safety First mindset in a DBA.

You need to think Safety First when you are performing any particular complex or critical activity. Take the time and put one or two safeguards in place like the DBA did when he dropped the user.

Other DBAs may call you paranoid, I'll call you an experienced DBA that would rather be safe than sorry.

Wrapup
The intent of this blog was to not provide you with a laundry list of recommendations. It was intended to help jumpstart your creative juices to think about different methods to protect yourself against problems. If you have any helpful hints, please feel free to respond to this blog with your Safety First Tips and Tricks.

Thanks for Reading,

Chris Foot
Oracle Ace

You begin creating your own rules of thumb by reading opinions from folks that you can trust. The industry experts that are both learned in their chosen profession and aren't afraid to provide you with their opinion. If you are interested, I wrote a blog listing my favorite authors and bloggers.

In addition to learning from others, you need to create your own test environments and experiment. One of Oracle's great benefits is that it runs virtually anywhere. If you aren't sure how something works and the manuals aren't explicit enough, build a test case and execute it. I experiment with my Oracle9i and Oracle10G environments on a daily basis to teach others and continue my own database education. If someone asks me a question that I can't answer, we'll log on to my test environment and try it.

How do you make your own rules of thumb? Follow Jonathan Lewis's advice and test, experiment and learn! Afraid that performance will suffer if you place 4 columns in an index to obtain index-only access? Build the index in test, have the developers run transactions that access the new index, monitor on-line transaction performance and find out. Then document the results.

Try different column combinations, multiple index usage, etc. When prototyping complex joins between several tables, build a set of indexes that favors one access path and test the queries in question. Drop the first set of indexes and create indexes that favor another access path. Build the different permutations of indexes that influence the optimizer to join a different set of tables first or allow different join types to be performed. Test the different access paths and record the performance statistics.

The above recommendation may seem to be a time consuming process, but until you learn access paths this is the best approach to take. I have spent hundreds of hours working with access paths and if I could impart one bit of wisdom, it would be the following one: No matter how much time you spend reading other's opinions on how the database works, you need to spend time "in the seat" experimenting and learning on your own.

The same advice goes for learning OEM, backups and recoveries, advanced database features and virtually every other facet of the database. You need to do as all the other experts do - test, experiment and learn.

Repeatable Processes
Repetition, even though it can be boring, is the foundation for a high quality support environment. If the scripts and administrative process worked correctly the first time, chances are they will continue to work correctly in the future.

Automating and documenting complex administrative processes such as production to decision support database refreshes and application upgrade activities will allow future iterations of these activities to be executed more quickly and with less errors. If you have been reading my blogs, you'll understand the importance I place on documentation.

As I have stated in the past, have you ever tried to refresh an ERP application test environment from production when that test environment didn't have enough space to hold all of production's data? 4,000 steps later and you begin to second-guess your choice of professions. The more complex the process is, the greater the need for detailed documentation.

The moral of this story is: If you don't want to be the only one that can perform that 900 step ERP Application production to test refresh, script it and then document it.

DBA Report Cards and the 360-Degree Review Process
Effective measurements are required to judge the success of any activity. The quality of support the DBA team provides should be reviewed on a regular basis. A DBA report card allows business and application development units to provide feedback on DBA support activities.

The report card will allow them to measure how well they feel you are meeting your Service Level Agreement objectives. Here are a few sample questions to start you on your way:

• How would you rate the turnaround times for DBA unit work requests?
• How would you evaluate the DBA unit's responsiveness to questions?
• How would you evaluate the DBA unit's responsiveness to requests for assistance?
• Please rank the quality of communications the DBA unit provides.
• Please rank the overall quality of work the DBA unit provides.

All questions are ranked from 1 to 5 with 5 being the highest. You can also allow the respondents to rank the importance of each question from 1 to 5 with 5 being the highest.

A short survey should be attached to the DBA report card to gather additional information that can be used to improve the quality of support the DBA unit provides to their customers:

• What are your top three technical challenges that you face?
• What are the top three non-technical challenges?
• Please list your current priorities. Rank them in order of importance.
• List the most important services the DBA unit provides. Rank them in order of importance.
• What support services does the DBA unit do a good job of providing?
• What support services should the DBA unit improve upon?
• What additional services would you like the DBA unit to provide?

Meetings can be held with the respondents to discuss their reviews. DBA team members participating in the reviews must be prepared to respond to criticism in a professional manner. But just as its title describes, the 360-degree review process also allows support units to provide feedback on their customer's support requests and work activities. The 360-degree review process provides important feedback to both support units and their customers. Once again, your customers may not know that some of their expectations are unachievable until you tell them the reasons why.

Corrective Action Reports
Oracle is a challenging database to administer. As much as we would like to prevent mistakes, we do make them. That's one of the benefits of being human; we learn from our mistakes. During my 20 years in this field, I've seen all kinds of mistakes made - from little ones to catastrophic. I've also made my fair share of them.

The worst one I ever experienced is when we brought a technician in from a third-party disk vendor to format what we thought was to be a single disk in a huge disk array. We just purchased the arrays and were in the process of assuming support responsibilities from the vendor. The vendor assisted us in the initial setup and data migration from our old storage devices to their new whiz-bang storage system.

The array stored data from dozens of Oracle databases. After 20 minutes of showing our folks how to set the necessary switches and enter the appropriate values at the prompts, the technician hit the enter key with much flair and bravado.

I assumed we would see the one activity light activate for the disk being formatted. Instead, I saw a whole wall of lights begin flashing and twinkling. I thought to myself "Hmmm, this can't be good…". I glanced at the technician and the look of horror on his face confirmed my suspicions. He then reinforced my conclusions when he stammered, "I think I just formatted the entire array." I looked at the DBA next to me and said "Looks like we are in for a looong night."

I continue to be amazed at the rapid improvements in hardware redundancy and reliability. Hardware platforms now have the ability to recognize individual component failures, bypass them, produce diagnostic information and "phone home" to report the problem to the manufacturer. But hardware components will fail. Just as the sun comes up every morning, hardware components will fail, operating systems will get tied up in knots and database bugs will manifest themselves. Such is life for a computer technician.

The key is to document everything that happened during the time period when the problem occurred. Documenting provides you with the information you need to prevent the problem from occurring again and your customers with information on exactly what happened. Its my experience that the more informed a customer is about the problem and the steps your unit is taking to prevent its reoccurrence, the better they feel.

The corrective action report should contain:

• A detailed description of the error that caused the problem.
• The customer impact the problem caused.
• A timeline of the activities that were performed.
• The steps that were performed to correct the problem.
• Mitigating factors that contributed to or exacerbated the problem's impact.
• The steps that will be taken to prevent the problem from occurring again. Include who is responsible for the activity and a date the activity will be completed.

Thanks for Reading,

Chris Foot
Oracle Ace

Each one of our DBAs can go to any platform and quickly navigate to the script directories, find the script they are looking for and execute it. I feel that this is so important to the quality of support we provide that I have selected a few DBAs to be members of a team that are responsible for creating and auditing naming conventions here at CTi. These folks, led by Ace DBA Jeff Kondas, review environments to ensure that our naming conventions and best practices are being followed. They find few deviations because this best practice has become integrated into our organization's psyche. Very good news.

I'll continue our discussion on naming conventions by providing you with a few recommendations. As always, these recommendations are not intended to be all-inclusive. They are a few examples to help start you in the right direction.

Follow OFA Naming Conventions
OFA stands for Optimal Flexible Architecture. The OFA standard is a set of naming conventions and configuration guidelines that:

• Distributes I/O among different disk drives.
• Facilitates ease of administration by creating naming conventions for mountpoints, directories, file suffixes, database binary directories and database output (i.e, background dump, core dump).
• Improves the DBA's ability to manage and administer database growth.

These standard naming conventions and placement guidelines are intended to improve database performance by spreading I/O, protect against drive failures and also allow administrators to more easily assume the responsibility of supporting new database environments. In addition, because OFA standards are well documented by the Oracle Corporation, newly hired DBA and consultants are able to more quickly assume administration responsibilities.

To learn more about OFA, please visit Oracle's Technet website. Go to the top right hand corner of the page and type "OFA" in the search box. You will find all of the documentation you need!

Create and Standardize Monitoring and Administration Scripts
All DBAs have their own set of favorite Oracle administration and monitoring scripts. They find them on the web, get them from third-party books and trade them like bubble-gum cards. They then tune, tweak and tailor them until they fit their own unique style of administration.

It is highly recommended that database administration units create a set of scripts for daily monitoring, hot and cold backups, exports and common administrative activities. This library of scripts can then be installed on each server administered by the team. Personalizing the scripts should be highly discouraged. Any modifications to the library can be reviewed during team meetings. Assign a team member the responsibility of being the script library owner. The script library owner will be responsible for making all script modifications and installing the new scripts on the supported servers. Here at CTi, we have a virtual team that is assigned this responsibility.

All scripts and directories should have names that allow them to be easily recognized. It irks me to no end when I log in to a server, navigate to a script directory and find files named SQL1.SQL, SQL2.SQL, FIXIT.SQL, TUNEIT.SQL… Fix what? Tune what? What the? Personally, I would prefer to have a SQL file name that is long, but descriptive, as opposed to a name that doesn't provide me with any clue on its contents.

It is also important to follow a strict naming convention for your output files. Whether the output is from a SQL, DDL or operating system command, the output file should be placed in the appropriate directory and have a descriptive name. The file name should also contain the date the output was created.

For example:

create_mcfinemp1_table_012606.out would be the output from the DDL statement execution that created the MCFINEMP1 table on January 26, 2006.

Because many of our customers have a preference where we store our scripts and internal monitors, we have created our own directory naming conventions:

Object Naming Conventions
This is the area where having a Data Administrator available is certainly beneficial. These are the folks that are trained in naming convention best practices. If you don't have the good fortune of having a DA available, it will be up to you and your application developer counterparts to create a set of naming convention guidelines. It is important to create a naming convention that contains abbreviation guidelines that are used to shorten object names.

Here's an example of an index naming convention I have used in the past:

INDEX
Definition: An index is an object that ensures efficient access to the data by using an ordered set of pointers to the table's data rows.

Naming Conventions:

Example: MCFINEMP_IDX_01 would be the first index on the MCFINEMP table.

There are dozens of guidelines available on the web. In addition, many of the books on Data Administration provide suggested naming conventions. We also have our own personal preferences. For example, many DBAs like to include column name abbreviations in their index names.

What the standard naming conventions are is not as important as creating them and adhering to them religiously. That is the key to success.

These recommendations are not intended to coerce readers into using the provided naming convention examples verbatim but to emphasize the importance of a creating a set of standardized naming conventions and then strictly adhering to them.

Thanks for Reading,

Chris Foot
Oracle Ace

Craig and I learned that a key ingredient of a trouble-free mainframe environment is ensuring that there are no "surprises" when a change is implemented in production. Changes that can affect the day-to-day business operations of an entire organization. Throughout my career, I have applied these "mainframe style" best practices to all other database ecosystems that I was responsible for supporting.

It works. The first company I applied these best practices to was selected by Oracle as a "Showcase Environment". This was back in the old days, when Scott's tiger was just a cub. Oracle identified shops that it thought had rock-solid support environments and asked them to host visitors from other organizations that wanted to create their own high-quality support infrastructures.

I thought I would provide you with a few helpful hints and tips on change management best practices.

Database Design Reviews
One of my earlier blogs provides information on database design reviews. Database design review meetings foster effective communications between the DBA unit, system support personnel and application developers throughout the entire application design and implementation process. When Oracle design issues are addressed early in the development lifecycle, problems are minimized and the migration from test to production is more easily accomplished.

If you haven't read the design review blog, you should. I'm intentionally not covering the importance of rigorous testing of any change before it is implemented in production because it is covered in-depth in the design review blog. From simple changes to new application implementations, there is simply no reason not to perform the test, review, change, test, review, change iteration lifecycle. Although the blog covers new application implementations, the blog will show you how important I think it is to follow a rigorous test plan.

Proceduralize the Change Request Process
Database administrators usually support different business units with each unit having their own set of unique procedural requirements. Formalizing and documenting the change request process minimizes the potential for miscommunication between the business units, application development areas and the database administration unit.

The notification lead-time required for the database administration team to perform a requested change should be documented and distributed to business and application development units. This will prevent your team for getting a request to migrate a database from test to production in the morning with a required date for that afternoon. Of course, we all know that never happens.

If your organization doesn't have a formal change request process in place, (and many shops don't) create your own! There are dozens of change management and source code versioning tools available on the market today. The prices can range from thousands to tens of thousands of dollars.

Although I highly recommend these types of products, I wouldn't let the lack of having one prevent me from formalizing the change management process. You do the best with what you have.

OK, so you don't have the good fortune of having a formal change management process in place. What do you do? You can begin the formalization of the change request process by:

• Creating standardized change request documents
• Establishing change management meetings
• Creating Service Level Agreements (SLAs), which include change request turnaround times.

Standardized Change Request Documents
Standardized request documents help to increase the quality of the change request process. The forms are sent to the database administration unit by the application owner of the data to be processed. Any requests not sent or signed off by the application owner should be rejected. Keep copies of all completed work requests for auditing purposes. Application owners can be virtually any member of the organization that is identified as having the authority to sign off on change requests. The most common persons are application development team leaders, section heads, department heads, etc..

Each request form contains the following common information:

• Form identifier - naming convention that allows the form to be easily identified
• Application name
• Database name
• Name and contact information of the person requesting the change
• Request date
• Required date
• Application owner signoff
• Data security signoff (if required by shop standards)
• A free form request area for nonstandard requests
• An area that the DBA will fill out when executing the change that contains the following information:
  • DBA executing change
  • DBA contact information
  • Date and time change was processed
  • Verification procedures followed

Here are a few examples of specific forms that will help formalize the change request process:

Oracle Authorization Request Form
This form is used for requesting authorization changes to the Oracle database environment.

The Oracle Authorization Request Form contains the following information pertinent to Oracle authorization requests:

• Grantee listing for security grant or revoke
• Type of security granted or revoked

Oracle Test Environment Change Request Form
The Oracle Test Environment Change Request Form will be used for requesting physical changes to Oracle test databases. It is used for requesting both new objects (tables, indexes, views, stored programs, etc.) and object alterations (adding new columns, changing datatypes, adding columns to indexes, etc.). In addition, the form notifies the DBA team to perform ad-hoc activities (test database refreshes, exports, etc..) in test environments.

The Oracle Test Environment Change Request Form contains the following information pertinent to Oracle change requests:

• Schema owner of object to be changed or created
• Object name to be changed or created
• Object type (i.e. table, index, view) of the object to be changed or created
• Detailed description of change requested
• Data administration sign off for new data objects
• Free from area for ad-hoc activities

Oracle Production Environment Change Request Form
This form will be used for requesting the migration of Oracle objects (databases, tablespaces, tables, indexes, etc.) from test to production and the alteration of existing production objects. For new database implementations, please see my blog titled "Database Design Reviews". In addition, the form notifies the DBA team to perform ad-hoc activities in production environments.

Each Production Environment Change Request Form must have an associated Oracle Test Environment Change Request Form counterpart. If the change wasn't made in test, you don't implement it in production. To facilitate this process, the identifier for the test change request form that was used to notify the DBA team to create or alter the test objects should be provided on the production change request form.

The Production Environment Change Request Form contains the following information pertinent to production environments:

• Oracle Test Database Change Request Form. Allows DBA to determine if the change was implemented in test. If no change request is found, the DBA needs to determine the reason why
• Object name of the object to be migrated or changed
• Object type for the object to be migrated or changed
• What time these activities should be performed
• The name of the database the object will be migrated from (source database)
• The name of the database the object creation/alteration will be implemented in (target database)
• A freeform area for special processing required during the migration or alteration process
• Free from area for ad-hoc activities

Change Management Meetings
If you read my earlier blog on database design review meetings, you know I'm a proponent of constant communication between all units that are involved in the change management process. How often should you hold these change management meetings? As often as you implement objects in production. If your organization makes changes to production environments on a daily basis, the meetings should be held daily. This is not as big of an imposition on your time as you may think. We provide remote database services for several very large organizations that have these change management meetings on a daily basis. The process takes about 15 to 20 minutes. Not a lot of time spent to ensure that everyone knows what is happening.

To shorten the amount of time these meetings consume and to make them as productive as possible, the following discussion items should be a standard part of the meeting's agenda:

• Application name being changed
• Date and time change will be implemented
• Change description
• Potential business impact if the changes don't go as expected (include both units affected and how they will be affected)
• Backoff procedures
• Requestor
• Tested by

Service Level Agreements
Identifying support needs and expectations is required to provide high quality support. You probably won't be meeting all of your customer's expectations if you don't know what any of them are. As stated previously, each application has it own unique set of support requirements and expectations. Service Level Agreements (SLA) help to solidify support requirements and dispel any inflated expectations a business or application development unit may have. They probably won't be aware of your current workload and resulting work request lead times until you tell them. The DBA team lead should meet with each unit supported to establish a set of measurable Service Level Agreements that include work request lead times, support activities required and application performance and availability objectives.

Wrapup
This is by no means an all inclusive list of activities you need to perform to standardize the change request process. It is intended to give you a head start in the right direction.

Thanks for Reading,

Chris Foot
Oracle Ace

That always seemed to ensure that my students paid attention during the remainder of the class. The backup and recovery class was one intense week of instruction that consisted of me pounding information and best practices into my students' collective heads. I would often stop after an important topic and bellow "DO YOU UNDERSTAND?!?!!" as loud as I could. By the end of the week, the class would immediately yell back "YES!!!" as loud as they could.

What can I tell you, that style worked for me. Trust me when I say that when my students left that class, they could all backup and recover an Oracle database. The backup and recovery classes were the ones responsible for my courses being labeled as "Foot Camp" by the Oracle student population. That was OK by me.

My first "mentor", if you could call him that, was an ex-Marine Corps drill instructor that went into IT after he retired. When I started my career, he was the first senior DBA I worked for. Every so often, he would walk up to the back of my chair as I was facing my terminal, lean in real close to my ear and say "You know what Foot, the next time I see you make a mistake, I'm not even going to tell you. I'm just gonna wait 5 minutes then come back and kick your *&^%$ up around your shoulder blades." Not motivational for sure, but I made few mistakes. Maybe some of that rubbed off...

Oracle Backup and Recovery
Recovering an Oracle database is a wonderfully complex task. Data files, log files, control files, full backups, hot backups, RMAN and point-in-time recoveries all combine to make many administrators lie awake nights thinking about whether their databases can be easily recovered (or not).

The next few sections will provide some useful information on the Oracle backup and recovery process. My intent is to not cover any technical topics in-depth. You can get that information from a myriad of sources. My focus will be on the non-technical tips and tricks that will help you improve your recovery skills and ensure trouble free recoveries.

It's the Little Things That Bite You
Most botched recoveries can be attributed to human error. Make sure all tapes have proper retention periods, verify that all backups are executing correctly and run test recoveries on a regular basis. Don't let missing tapes or backups cause you to lose data. You don't want to hear UNIX support say "the retention on that tape was supposed to be how long?" in the middle of a recovery. COMMUNICATE with others that are responsible for all other pieces of the recovery "pie" (system admins, operators) on a regular basis to ensure you have everything you need to recover a crashed database. Pick a database, identify the backup output files and verify that they are available when you need them. Remember, YOU are the technician that is ultimately responsible for ensuring that your organization's databases can be recovered. Not O/S support, operations, application developers....

Document Your Recovery Environment!
OK, I'm yelling already. I already covered the importance of good documentation in a previous blog. You know by now that I work for a remote database services provider. It is absolutely imperative for us to know EVERYTHING about our customer's existing backup and recovery strategies. Part of our assimilation process is to document our customer's environment. Here's a quick list of some of the questions we ask. The document we actually use is a standardized Word template that uses drop downs, text boxes and help buttons but this shortened text document should provide you with a starting point to help you build your own backup and recovery documentation library.

Keep Your Skills Sharp
Don't let your recovery skills get rusty. The more test recoveries you do the easier the production recoveries become. Create one database that you and your fellow administrators can trash on a regular basis. Take turns and make a game out of it. DBAs can be pretty creative when causing problems for others when it's all in fun.

It can actually become quite an interesting game competing for bragging rights over who has the current title of "the most devious database destroyer." During one of my test recoveries, I couldn't even bring up the monitor. I waltzed down to the server room and saw an open drive bay on our test server with a bundle of unconnected wires sticking out. There was a single note attached below the opening telling me to look for the next note. 15 notes later and I found the drive. Dumped it in, fired it up and found that the database was deleted. I recovered it from a tape backup. THAT was devious.

If you are a senior-level DBA, make sure you keep the junior folks on their toes. I have never personally seen the database make a mistake during the recovery process. That leaves incomplete backups and DBA error as the most likely causes of "good recoveries gone bad."

At RemoteDBAExperts, we have dozens of customers that we have to support. We test recoveries and failovers on a regular basis. Not four hours ago, we had three of our folks perform a cold database backup and recovery on a Linux platform. Ensuring our recovery skills are sharp is that important to us. I still do test recoveries. Its important to me to ensure that I am ready to go when the time comes. If it were up to me, I would have our receptionist test her recovery skills too.

RELAX and Plan Your Attack
When you are notified of a database failure, take a deep breath and relax. Don't immediately begin to paste the database back together without a plan. Create a recovery plan, put it on paper, have others review it if you can, and then execute it. You shouldn't be trying to determine what the next step is in the middle of the recovery process. I will plan my attack on paper for all recoveries, no matter how simple they are.

Don't Be Afraid to Ask Others
I have over 20 years of experience using Oracle and have done my fair share of database backups and recoveries. During my career as an Oracle instructor, I have assisted in hundreds (and hundreds) of database recoveries in Oracle's classroom environments. If possible, I still have others review my recovery strategy and recovery steps before I begin the recovery process. A second opinion may prevent you from making a mistake or overlooking a key part of the recovery process.

Don't be afraid to ask others and don't be afraid of calling Oracle support if you have to. That's what they get paid by your company to do - support you. Don't make a database unrecoverable by "guessing." When I first took over as the Database Group Manager for a large financial organization many years ago, I viewed the execution of over 70 different commands in an alert log after a botched recovery performed by a junior DBA. An ego that was too big to allow that person to ask questions created a database that was almost unrecoverable.

The Importance of Formal Education
Read the Oracle Backup and Recovery Guides before reading third-party books. The manuals will provide you with a firm foundation of knowledge on backup and recovery strategies and procedures. Then move on to third-party books (like this one) for helpful hints and tips that may assist you in the recovery process.

Take the Oracle classes! Oracle's instructors understand the importance of backups and recoveries. You will receive days of instruction and hours of hands-on labs. You'll learn everything from simple O/S cold backups to RMAN incomplete recoveries using backup control files.

Oh, and now that I'm retired from teaching, you won't have to worry about me yelling at you.

Thanks for Reading,

Chris Foot
Oracle Ace

It is the DBA’s responsibility to ensure that the overall design and implementation of the new application is proceeding according to expectations. Although application developers may be experts in SQL, procedural languages (Java, PL/SQL, C variations, etc.), they still turn to the DBA for help with standards, procedures, performance design, error recovery and complex SQL.

A continuous and effective dialogue between the DBA unit, system support personnel and application developers is of utmost importance throughout the entire application design and implementation process. One method to foster effective communications is for the DBA unit to create a series of new application design review meetings. These meetings can be organized in a manner that promotes efficient application usage of the Oracle database environment. The design review meetings can be scheduled during logical break points between the different stages of the application development process.

The database administration team should invite representatives from the development and business units to assist in the creation (and enforcement) of the standardized review process. Application development teams will be able to help tailor the design review process to satisfy their specific design and implementation needs. Business users who will be impacted by the production turnover will provide valuable input on implementing new applications for their specific business areas. Customized checklists are created to ensure that all application and business specific issues are addressed during each meeting of the design review process.

It is recommended that the database administration team then communicate to all application development and business areas that any internal applications created without following the standardized review process will not be migrated to production. Having the application teams and business units participate during the creation of the standardized review process allows the DBA team to enforce the policy without being viewed as dictatorial or making rules in a vacuum. In addition, every effort should be made to “sell” the standardized design review process in all communications to application development teams and business units.

The following list of meetings can be used as a starting point in the creation of a structured application design review process:

The kickoff meeting is held with datacenter operations support managers, operating system administrators, application team leaders, system analysts, end-user management and data administrators to discuss the application’s general purpose and function. This information will allow support technicians to determine the impact the new application will have on existing systems and allow them to begin planning the application design implementation process. The following information should be covered in this meeting:

• General description of purpose and function
• Application scope (enterprise wide application that affects all business units, intra-departmental that affects several business units or departmental)
• General application size (estimated number of programs, data objects)
• Criticalness of the application (mission critical, application critical, non-critical)
• Application availability requirements and downtime windows
• Application type (decision support, business intelligence, data warehouse, data mart, online transaction processing)
• Architecture design (web server, application server, N-tier, distributed database environment)
• Advanced features required (replication, advanced queuing)
• Data feeds required from other architectures
• Load processing
• Batch processing
• Online transaction processing
• Development tools used to build front-end application screens
• Third-party tools used to provide Ad-Hoc query access
• Procedural language used to build business logic (Java, PL/SQL)
• Application development methodology to be used (JAD, RAD)
• Number of concurrent users
• Disk storage and data growth estimates
• Highly available architecture discussion (RAC, Oracle Fail Safe, Data Guard, hardware vendor clustering and failover)
• Performance expectations
• Criteria used to judge performance
• Security and auditing requirements
• Hardware platform, O/S preferences/selection and sizing discussion
• Hardware platform, O/S installation, operation and administration
• Division of duties between the DBA, application development and business units

This meeting is convened as soon as the logical data model is complete. The major emphasis of this meeting is to determine if the logical data model is complete and correct. The application’s process model (if one is available) can also be verified at this time. Volume statistics, object growth rates, purge criteria, referential integrity needs and application-naming conventions are also discussed. Knowing your data before hand is essential to designing processes to manipulate that data. The following topics are covered in this meeting:

• Determine if the data model is fully documented (entities, attributes, relationships)
• Attributes have correct datatype, length, NULL status, default values
• General discussion of business rules that are to be enforced by database level constraints
• Not null constraints
• Check constraints
• Unique constraints
• Primary key constraints
• Foreign key constraints
• Business rules to be enforced by triggers and procedures
• Business rules to be enforced by application code
• Logical/process model comparison
• Volume statistics
• Growth rates and purge criteria

This meeting is held with the application development units before any physical DDL is generated by the DBA. Proper transaction design and efficient SQL coding results in less performance-oriented database alterations being made during the latter stages of the design and implementation process. Determining predicate usage will allow the DBA to create additional database objects (indexes, materialized views, index organized tables) to increase SQL, and subsequently, application performance. The following information is discussed:

• Normalization vs denormalization
• Table access requirements and predicate usage
• Database objects used to increase SQL performance including:
• B-Tree Indexes
• Bitmap Indexes
• Bitmap join indexes
• Function-Based indexes
• Materialized views
• Index organized tables
• External tables
• Data partitioning algorithms
• Range partitioning
• Hash partitioning
• List partitioning
• Composite partitioning
• Process parallelism (parallel query, parallel DML, parallel load)
• Transaction modeling
• Oracle SQL performance features
• Full table scan vs index access
• Hash joins
• Star joins
• Index skip scans
• Index fast full scans
• Cursor sharing (SQL statement reuse and soft parse vs hard parse)
• Bind variables
• Reinforcement of effective SQL coding techniques

This meeting is held as soon as the application developers are ready to begin the actual coding process. The ultimate goal of this meeting is for application developers to have a firm understanding of what is required to create and maintain a successful Oracle test environment. Discussions on Oracle utilities, SQL statements, procedural code, Oracle security, batch script testing, benchmarking, testing and monitoring are of prime importance at this time. The DBA must make it clear to all involved that the DBA unit is always available to answer questions and troubleshoot poorly performing SQL throughout the entire application development life cycle. It is imperative that the DBA make every effort to find and correct problems during this stage of the development process. Maintaining a proactive stance instead of a reactive one will always prove to be beneficial when it comes to correcting design problems. There are always enough potential problems lurking in the system for the DBA to solve without adding any additional ones through faulty application design. Topics to be discussed in this meeting include:

• Division of duties between DBA, systems, application development and business units
• Test hardware platform, O/S installation, configuration and administrationD
• Data feeds required from other architectures
• Data load and update processing and scheduling
• SQL Loader control cards
• Test system availability requirements
• Test system backup and recovery requirements
• Oracle security authorization
• Tools to aid in application development
• Oracle Enterprise Manager toolset
• In-house monitoring tools
• Third-party vendor products
• Benchmarking and performance measurements

Because this meeting and the previously discussed Setting Up a Successful Test System are closely related, they may be discussed together to save time and provide greater clarity. The DBA and developers need to have an understanding of what information is required to effectively monitor the performance of the Oracle application throughout the entire testing process. The DBA can offer to show developers how to use the various toolsets provided by Oracle (Oracle Enterprise Manager, SQL Trace, explain plan, BSTAT/ESTAT, autotrace, Oracle traces, V$ tables and Statspack) so that developers can play an active role in performance measurement. Suggested discussion topics are as follows:

• Names of all SQL Loader control cards and batch scripts
• Performance measurement goals and expectations
• Determine the test load volume (data volume, concurrent users) required to accurately predict production performance
• Comparison of test system load to estimated production load
• Oracle performance measurement tools. Suggested tools include:
• Explain
• SQL*Plus autotrace
• Oracle SQL trace
• Statspack
• Oracle Enterprise Manager with Tuning Pack option
•  V$ performance tables
• Index utilization monitoring via ALTER INDEX MONITORING statement
• Third party performance monitoring tools
• LAN performance monitoring tools
• Operating system performance monitoring tools

Information collected from the various performance monitoring tools is discussed at this time. One meeting may be sufficient, but large application development efforts usually require several discussions. If the DBA has maintained good communications throughout the initial stages of application design, there should be few surprises when these meetings are held. SQL tuning and tweaking recommendations are covered in this meeting. Depending on the length of the development process, follow-up meetings can be held to ensure that the application is performing as expected. Some suggested topics include:

• Load testing assessment: Is the load being placed on the test system large enough to accurately predict production performance?
• Review performance statistics collected during the testing process
• Assess SQL coding techniques by reviewing explain plan output for performance critical transactions
• Index usage
• Local/join predicates
• Join methods
• Subselects
• View materialization
• Determine if additional Oracle objects need to be created to enhance SQL performance
• B-Tree Indexes
• Bitmap Indexes
• Bitmap join indexes
• Function-Based indexes
• Materialized views
• Index organized tables
• External tables

This meeting is held to determine if there are any last-minute questions and to make certain that everyone knows what is expected of them during the final turnover process. All units that have participated in the application design or application design review process are invited to attend. To ensure that all steps necessary for a successful migration to production are executed, the use of a standardized Oracle migration checklist is highly recommended. This document will allow the DBAs and developers to concern themselves with topics that are related to this specific turnover rather than spending time on the more mundane turnover tasks that are just as important, but easily forgotten. Having a complete, well thought-out migration checklist produces a less stressful and less error prone production migration process. Topics include:

• Division of duties between DBA, systems, application development and business units
• Production hardware platform, O/S installation, configuration and operation
• Data feeds required from other architectures
• Data load and update processing and scheduling
• SQL Loader control cards
• Backup and recovery
• Oracle security authorization
• DBA forms and turnover procedures
• Contact information and escalation procedures
• Post-production monitoring tools to be used

This final set of meetings is held after the production turnover is complete. Application logic errors and performance problem resolution are the prime topics of discussion. A comparison of the actual performance to the anticipated performance of the application is also discussed. The review and correction process, by its nature, is iterative. The problems are identified, a plan of attack to solve them is agreed upon and additional meetings are scheduled to review the results. As additional problems are identified, they are added to the list of problems to be solved. This iterative process continues until all major performance issues and application logic errors are addressed. The post production turnover meetings should include discussions on:

• Review performance statistics collected (Oracle Enterprise Manager, SQL Trace, explain plan, BSTAT/ESTAT, autotrace, Oracle traces, V$ tables and Statspack)
• Assess SQL coding techniques by reviewing explain plan output for transactions experiencing performance problems
• Index usage
• Local/join predicates
• Join methods
• Subselects
• View materialization
• Determine if additional Oracle objects need to be created to enhance SQL performance
• B-Tree Indexes
• Bitmap Indexes
• Bitmap join indexes
• Function-Based indexes
• Materialized views
• Index organized tables
• External tables

These recommendations are not intended to coerce readers into using the Oracle application design review meeting examples verbatim but to emphasize the importance of a structured approach to the design review process. The seemingly endless combinations of application architectures and software products used for application development may require the creation of a fully customized design review process for each application development project. The overall goal of design review meetings is to ensure the involvement of technical support units during the application’s design and implementation. When Oracle design issues are addressed early in the development lifecycle, problems are minimized and the migration from test to production is more easily accomplished.

Thanks for reading.

Chris Foot
Oracle Ace

Data administrators view data from the business perspective and must have an understanding of the business to be truly effective. Their expertise allows them to define, describe, organize and categorize data. DAs build models of the data that are a based upon data entity relationships and the business rules that govern them. Data administrators provide the framework that allows disparate business units within an organization to share timely and accurate data. The end result is: higher quality data = better business decisions. Enough said.

Because folks in this profession must understand the business itself to perform their duties, I have never met a DA that didn't know more about the overall business than any one else in the organization. That includes the business personnel that were responsible for providing that service. You know what surprised me the most after watching a DA interview business unit personnel about data meaning and business rules and processes? How much more that businessperson learned about the service they were providing to their internal and/or external customers.

As a result, if I were a CIO and I wanted to learn more about a particular business function my organization provides, I would include the Data Administrator in the list of people I needed to talk to. No doubt about it.

I have had the good fortune of working with many excellent Data Administrators during my career. I made every effort to learn as much as I could from them. Rebecca Duffy and Jim McQuade come to mind when I think about top-notch Data Administrators that have made an impact on me. I have also had the good fortune of working with Craig Mullins who I feel is one of the few database experts that firmly understands the Data Administration function.

The greatest positive impact I think these folks had upon me was that they were the conduits that allowed me to understand and appreciate the benefits that high quality data driven design provides. They taught me the importance of their chosen profession.

Like all DBAs, I know that the key to creating a high quality database environment is to build it upon a firm foundation. A foundation that consists of a well-designed hardware platform and an operating system and database that were tailored and configured for high performance, high availability and protection against unauthorized access.

I came to understand that high quality data driven design, led by a competent Knowledge Management Worker (Data Administrator to simplify) was the foundation for an application that was easily administered, easily changed, less error prone, needed less rework to add new functionality, and the list goes on and on and on….

Because Data Administration maps the business rules into constraints that can be enforced by the database and turns raw data into information stored in tables, the Data Administrator acts as the intermediary between the business and Database Administration units.

You can plop me down into any database and within 15 minutes I will be able to tell you if the application objects and business rules were designed by someone who understood Data Administration best practices. Since I work for a Remote Database Services provider, I am provided with that opportunity on a daily basis. When I assess a new customer's database, one of the items on my assessment document is adherence to Data Administration best practices. When I see poor naming conventions, the same piece of data named 10 different ways and stored using every different data type Oracle provides, I immediately begin to become highly concerned about the quality of the rest of the database ecosystem (hardware platform, O/S, database, application). I also know that we are in for a lot of complex data changes in the future.

All this being said, the data administration role, although important, is often undefined in many IT organizations. The smaller the shop, the more prevalent this becomes. Those responsibilities, by default, are usually awarded to the shop's database administration unit.

If you are employed at an organization that does not have a full-time Data Administration unit, the first question you need to ask is why not? Here's my two cents. Any organization that is large enough to employ a staff of DBAs, operating system administrators, network engineers and application developers is being shortsighted if it doesn't have a full service Data Administration Unit.

We previously learned that the services a DA provides ranges the spectrum. As a DBA/DA, the scope of services you can provide will be severely limited by time constraints and knowledge. Most of your time will be spent understanding the Data Modeling process as opposed to defining enterprise-wide information frameworks. You must have the basic understanding of how data is modeled and the logical to physical transformation process. Your responsibilities will require that you work closely with the business folks and application developers. This must be a team effort.

There are dozens of data modeling tools to help you perform the data modeling process. Personally, I am a neophyte Erwin user. Here's a web page with a more comprehensive listing.

I fully understand that small shops can't afford to employ a full time Data Administrator. So what happens if you are the "chosen one" to perform these services for your organization?

Here's my quick-start listing:

• Realize that the data administration function you are providing is just as important as any database administration service you can perform. You need to spend as much time, if not more, learning this trade. Even if it is only a part-time responsibility, it is a highly important part.
• Don't allow yourself to become overwhelmed. As with anything, start by learning the basics. I'll provide some resources later in this blog to jumpstart you.
• Understand that this topic is complex. I have run across few Data Administrators that didn't have a passion for their job and weren't highly intelligent. They will also speak a different language than you do. My first meeting with Jim McQuade started at the same intellectual level and ended by him slowly saying "DO…. YOU….. UNDERSTAND…?" after each sentence. I shook my head in several different directions and said "NOOOO." But I kept trying and kept learning from him.
• You need to learn the basics. Start by beginning to understand the Data Modeling process: normalization, naming conventions, entity relationships, business rules. The book I often refer to is "Data Modeling Essentials 2nd Edition: A Comprehensive Guide to Data Analysis, Design, and Innovation" by Graeme C.Simsion. A introductory book I think highly of is "Data Modeling Made Simple: A Practical Guide for Business & Information Technology Professionals" by Steve Hoberman.
• Sign up for DAMA and attend their conferences. As I stated previously, start with the basic presentations.
• Spend dedicated time on the The Data Administrator Newsletter website. This is the FIRST site I would go to. I can't stress this recommendation highly enough. The folks that contribute to this site are top notch.
• While you are on that site, go to the Book Center page and read some of their reviews.
• Read Jim McQuade's excellent article titled "Loving to Hate the Data Administrator" on DBAzine.
• Many of my personal Data Administration books are outdated, so I have asked Jim McQuade to respond to this blog with his personal book recommendations. I hope that once word gets about about this blog, other DAs will chime in too.
  

The key to your success is to start with the basics and understand that there is much to learn before you can perform the DBA/basic DA functions.

Thanks for Reading,

Chris Foot
Oracle Ace

But a robust detailed documentation library creates an environment that is less complex, less error-prone, reduces the amount of time DBAs spend learning new database environments and reduces the overall time spent on day-to-day support activities. DBAs are able to spend more time administering the environment than finding the objects they are trying to support and the processes and programs used to administer them.

The nature of my business as a remote services provider demands excellent documentation. The majority of environments we administer weren't designed by my organization. The only way that we can ensure high quality and high-speed administration of these environments is to document them thoroughly. We document everything, from initial connectivity and customer contact sheets to detailed information on batch job streams and individual program execution. If we need to be aware of it, we have it documented.

Documentation is also the foundation of many of the other disciplines I will be discussing in future blogs. Let's continue our discussion with a few helpful hints to get you started.

Understanding the Important Role Good Documentation Plays
We all generally understand the benefits that documentation provides. I think that all readers will see the importance I personally place on documentation in upcoming blogs. Virtually every topic that I will discuss, from repeatable processes to paranoid administration best practices, I will be mentioning what to document and the important role that documentation plays.

Let me reaffirm my opinion in this one sentence:

Documentation is one of the foundations for virtually everything I will be discussing in this series on The Non Technical Art of Being a Successful DBA. Enough said.

Creating an Organizational Environment That Fosters Good Documentation
I've been a database group lead off and on for 15 years now. It is my responsibility as manager to create an environment that fosters the production of robust and high quality documentation. Let me describe some of the challenges that I have faced in the past and how I have overcome them.

Since I view high quality documentation to be my responsibility as a database unit manager, I ensure that it becomes part of every DBA's performance appraisal criteria, including my own. If it isn't on my, and my unit's, performance appraisal forms, I will ask to have it added or make my own personal addendum and notify both the DBA team and management that I have done so. I will dedicate an entire blog on tailoring performance appraisals to what DBAs do for a living in this series.

I will add time for documentation when I estimate the amount of time it will take me to perform an administrative task during project planning meetings. I don't settle for "we can do that after the project is complete" as an answer from an application project manager.

If I get overruled and lose one battle, I still make sure I document my activities after the project's critical path is complete. And I don't let a single setback prevent me from telling everyone in the next planning meeting that I need so many hours to document what I have done.

If you continuously sell the importance of documentation sooner or later you will begin to wear your opponent's down. Although I prefer to call it "being relentless", I'm sure that many of the application development managers (and my own managers) viewed it as "being a …." (insert your favorite description here).

Every document I have created that provides a list of activities I , or my unit, needs to perform during a project has documentation included. It helps to integrate it into the fabric and culture of my organization's environment.

Making Documentation Second Nature
You also need to ensure that generating documentation becomes a natural part of your daily activities. You must continuously remind yourself that documentation is a primary and integral part of providing high quality support services to your customers.

You must also remind yourself that it makes your job easier and benefits your fellow DBAs. You know what has really irritated me the most during my career? When a fellow DBA needs to be out of the office for a time and asks me to "help them out" by performing a complex, application specific administrative activity and then tries to verbally tell me how to perform the 300 steps it takes to do it.

Ever try to refresh an ERP application test environment from production when that test environment doesn't have enough space to hold all of production's data? 4,000 steps later and you begin to second-guess your choice of professions. That was the exact request from one of my fellow DBAs a few years ago. Not only did he get me to do the refresh but I also had to document the process for him along the way. Some call that being a good coworker; I would view that as having a big sucker taped to my forehead. Live and learn, I guess.

The moral of this story is: If you don't want to be the only one that can perform that 900 step ERP Application production to test refresh, document it! If you don't want to be called by the on-call DBA because he doesn't know exactly where to add a datafile in an emergency situation (like an application developer forgetting to tell youu that they were loading 10 million additional rows into that 100 row table), document it! The more you document, the easier your life as a DBA becomes.

I've never had a great memory. It makes documentation easy for me. I also like to write and that helps. But I will admit that there are times that I would rather perform virtually any other activity than document.

But it has become easier because I continuously reaffirm to myself the importance of documentation. The more you reinforce that to yourself, the more second nature (and easier) it becomes.

Making Documentation Easy
I'm a huge fan of documentation templates. Here at Remote DBA Experts we have, or are in the process of creating, templates for everything we document. We have templates for documenting connections to our customer's environments, their backup and recovery environments, their application specific processes to name a few. If it needs to be documented on a regular basis, we have a template for it. We also have generic templates for documenting environments and activities that don't fit into other templates.

Microsoft Word document templates provide many features that streamline the documentation process and help to improve the quality of the content they store. I try to take advantage of as many features as I can. I use drop down selection menus, check boxes and radio push buttons to improve the speed and quality of the documentation process. I also take advantage of the help pop-up feature that Microsoft Word provides to create a detailed description of what information is to be entered into that field, check box or radio button.

I created my first template close to 15 years ago. It wasn't created to document a process, but to document a change request. I'll be covering change management in an upcoming series in this blog. Was the document crude? You bet. But it did the job. It has gone through numerous iterations and migrations from one Word release to the next. I continue to use it at organizations that don't have a formalized and documented change management process. And you can't believe how many I have come across that don't.

There are dozens of software companies that offer content management solutions. Database vendors have also recognized this as a lucrative market. All of the major database vendors now offer advanced content management software, each one trying to outdo the other in the number of bells and whistles that their product's offer.

Do a quick search on Google for documentation content management software and you will find out just how many competing products there are.

Content management products offer check-in/check-out features, document versioning, web portal access and advanced workflow capabilities to name just a few of the features desgined to improve content management. The competition in the content management market space is fierce to say the least. Content management vendors know that continuously adding new bells and whistles to their products is not just important for increasing market share; it is critical for their survival. Product costs can range from thousands to tens of thousands of dollars (or more).

If you have the funds and your management understands the benefits that a full-blown content management package provides, by all means begin a content management product analysis. But if you don't have the funds, create a shared drive on your network and declare it to be the "DBA Documentation Portal."

What to Document
By all means this is not an all-inclusive list of what can be documented. Consider it as a starter kit to help you begin your quest for "documentis nirvanas". Is some of this overkill for your particular environment? Maybe, but just consider this a general, high-level list.

Database Environment Documentation

• Naming conventions
• Servers (server names, operating system release, hardware vendor)
• Databases (vendor, database version, features enabled)
• Directories (everything from Oracle software installations that don't follow Oracle standards, application code storage to personal script locations). If you, or a fellow DBA needs to find it, document where it is!

Application Specific Documentation

• Application type (i.e. data warehouse, online transaction processing, decision support, third-party application name and functionality it provides).
• Business unit requirements and related information for supported databases
• Uptime requirements (i.e. 24 X 7, 8 X 5)
• Database downtime windows
• Critical job processes
• Business unit and application developer contact lists
• Turnover windows for database changes
• Problem notification and escalation procedures
• Security Sensitivity. How sensitive is the data?

Process Documentation

• Repeatable administrative processes (covered in an upcoming blog)
• Backups - Probably the most critical set of documentation you will ever create. Document how it is backed up, what scripts back it up, where the back up is going to, tape retentions and backup message directories. If it is involved with a backup, DOCUMENT IT. Review the document with other units that are involved in the backup and recovery process. It is your responsibility to ensure that you don't hear an operator say "What retention period? Nobody told me we were to have a retention on these tapes." when you are in a recovery situation. Remember that Oracle states that human error, including miscommunications, is responsible for over 90% of failed recoveries. If you want to reduce recovery failures DOCUMENT THE PROCESS AND REVIEW IT.
• Oracle utilities. Document Oracle Import/Export input and output directories and where the files are that execute the utilities. The same with Oracle's load utility. Document the execution scripts and input and output directories. A short note on why you are running it is also required
• Anything else you run on a regular basis to support a specific application
• Change Management. I'll be spending an entire blog, or two, on this
• A daily monitoring activity checklist to ensure that no activity is missed. We have daily, weekly and monthly activities that are to be performed for each of our customers
• Complex administrative activities performed regularly
• Test and reporting database refreshes
• Data reorganizations
• Disaster recovery tests. The processes required to perform the recovery AND the criteria that will be used to evaluate whether it was successful or not

Object Documentation

• DBA specific stored PL/SQL programs
• Day-to-day support scripts (where they are and what they do)
• Monitoring scripts (where they are and what they do)
• Scripts used to perform database administrative changes. I have specific directories that provide output from critical database changes that I have performed and other directories containing the SQL used to make that change
• Operating system scripts. Document what the script does in the beginning of each of your scripts. Ever try to determine what a 400 line script does that was created by someone that knows much more about UNIX scripting than you do? We have all been in that position at one time or another during our career. I'm personally still not good at it. Which is why I never was a great developer (all that IF-THEN-ELSE-OTHERWISE-GOTO-EXIT stuff bored me to tears). Make it easy on your coworkers to understand what the script does by putting comments at the top of the script as well as in the body. Also keep a running history of script changes. What they were and the time they were made

Database Administration Unit Organizational Documentation

• Contact Information
• DBA responsibilities. Which databases and tasks they are responsible for
• DBA unavailablity. Allows application developers to plan for a DBA not being available

It is a good practice to distribute this information to all business units supported by the database administration unit

I hope you enjoyed this blog on documentation and the important role it plays in the Non Technical Art of Being a Successful DBA.

Thanks for Reading,

Chris Foot
Oracle Ace

There are numerous jobs where you have to be on top of your game at all times. The medical profession and police work come to mind when I think about folks that have to be "mistake free" to keep us all safe and healthy. I often remind myself that the stress I let my job generate for me is nothing compared to what these aforementioned folks must experience. Knowing that there is an 18 year old somewhere fighting and trying to stay alive in a foreign land helps me to keep my own career's stress in perspective.

As Oracle database administration goes, being an Oracle Instructor is about as unforgiving as it gets. You get dropped into a classroom that has twenty PCs running Oracle and twenty students depending upon you to teach them. My first attempts at teaching a dozen years or so ago could be described as "less than stellar." I had a review from my first class state "this was the most boring person I have ever met." I felt bad about that until my senior instructor showed me one of his first reviews. It had in big red letters at the top of the page "I HATED YOU." I felt better after reading that.

I must admit I was a little timid when I started, but those students that attended later classes know that I decided to let my own bombastic personality show through. Let's say that none of my subsequent classes were ever boring after that. I'm sure that previous statement will generate a comment or two from my fellow blogger and lifetime buddy Craig Mullins and possibly a few from my students.

After that less than glorious start, I worked hard to become a better instructor, eventually winning one of Oracle's top instructor awards with a 95% student approval rating. How I ever ended up in a teaching profession where the students graded the teacher, I'll never know. Just as a joke I started one class by stating "Oracle has instituted a new procedure that will allow me to grade your participation and learning skills and send them to your employers." The look on everyone's faces was priceless. I still view being asked by Oracle to certify fellow instructors as one the greatest achievements of my career. I took that job very seriously.

I can tell you one thing about Oracle DBA students, they are a rough bunch. I started a DBA I class by blowing up a database create and couldn't get it to work correctly after a second attempt This after doing it successfully hundreds of times. A corrupt drive on my PC was the culprit. After turning around and seeing the "who is this idiot" look on my student's faces, I suddenly felt like a wounded fish in the middle of a pack of hungry piranha.

Oracle instructors describe that problem with a single phrase "losing creditability with the class." You did that and you were lunchmeat. But I never let that potential loss of creditability prevent me from saying "I don't know, I'll get back to you." Or I would look up the information for the student on the spot. That provided two benefits: answering the question and showing students the importance of knowing where to look for the answers.

Assisting a class of twenty students through backup and recovery labs certainly helped my debugging skills. If you ever want to find out how bullet proof an Oracle database can be, watch 20 installations being abused by students in a backup and recovery class. I always started my backup and recovery class with this statement "The fastest way to lose your job in this profession is to lose data for your company. You can be a Tom Kyte and a Jonathan Lewis X 2, but if you can't recover a database, you aren't of any use to your employer." That always seemed to ensure that they paid attention during the remainder of the class. That was one class where I absolutely pounded information and best practices into their collective heads. We were all worn out by Friday.

I enjoyed teaching and became friends with each and every group of students I taught. I always felt sad when we parted and always worried that I missed that one tidbit of information that could help them.

On-site consulting is also a tough profession. The customer brings you into an environment that you know nothing (or little) about and expects you to be productive and solve their problems - immediately. That is as it should be. They pay a premium and expect good service. It is the consultant's fault if that doesn't happen. I liked that job because it was very challenging and forced me to continuously refine my skill sets and talents to keep my customers happy.

But I have had my fair share of sweaty palms at many a client site. I didn't have the luxury of being an Oracle consultant. They always seemed to have an endless stream of people to draw from for support. I usually ended up in a wiring closet with a manual in one hand and a keyboard in the other. I actually spent three weeks in a wiring closet at one client's site.

Remote database administration also requires excellent technical and non-technical skills. You must continuously strive to keep your customers happy, all of which hold you to a high standard. Describing competition in the remote DBA industry as fierce is like describing the Titanic as "having a small leak." To reinforce this point - right now I'm concerned that others will view this as me using this blog as a sales pitch, which often leads to a competitive firestorm. It is not intended that way. I admire everyone that works in the remote DBA profession. It is a very challenging job. I'm just describing the work that we do and what is expected of all remote services providers.

You become successful as a remote services company by providing a superior level of support to your customers. As a technician, it's a great job! You can be installing a RAC/LINUX environment in the morning and be debugging an Oracle Applications concurrent manager problem in the afternoon. From warehouses to HUGE web databases having thousands of concurrent users and from manufacturing to financials, you work on it all.

The Non-Technical Art of Being a Good DBA
I started off this blog describing some of my jobs not to impress you with my background but to hopefully help you understand how important it is to me to provide you with this new series of non-technical recommendations. These jobs quickly laid bare every hole in both my technical and non-technical approach to database administration. They not only laid them bare but also amplified their negative results. I found that becoming lax in the non-technical areas of database administration and not following my own best practices could create just as much havoc as a technical problem. As I have stated numerous times, some of us are brilliant and some of us are lucky. The rest of us have to be thorough.

To ensure that I did not allow a failure in non-technical areas to complicate my life as a DBA, I needed to learn how to document, communicate and follow my own best practices. I learned the tricks of the trade to reduce the amount of time it took me to administer my environments and ensure I didn't get paged at night. I found that I could trust some author's information and not others. I learned that by keeping my recovery skills sharp, I reduced my level of anxiety when I got one of those two in the morning "I think we just formatted one of your Oracle disks by mistake" calls (actually the caller stated they accidentally formatted the entire array consisting of dozens of disks by mistake). The list goes on and on…

To continue, here's a laundry list of what will be covered in upcoming blogs:

• Why good documentation is so essential and documentation best practices
• Naming conventions
• The benefits of Data Administration or "Loving to Hate the Data Administrator". A reprise from Jim McQuade's excellent article here on DBAZine.
• The key to successful application implementation - Design Review Meetings
• Improving the speed and quality of database administration by keeping your environments organized and uncluttered
• Creating and maintaining tips, tricks and problem resolution libraries
• Script foraging
• Reducing stress and preventing failed recoveries by keeping recovery skills sharp
• The benefits of repeatable processes. If it worked that way once, it'll probably work that way again
• Proceduralizing the change management and production turnover process
• Time Management and the art of project and process time estimation
• Maintaining good relationships with customers, end-users and fellow technical personnel
• The Foot Rule of Thumb
• Knowing where to look when you don't know the answer (reprise from earlier blog)
• Using Service Level Agreements to ensure your internal and external customers know what to expect from you
• Monitoring by day so you don't get called at night
• Monitoring space growth and performance statistics historically
• Paranoid administration best practices. Many of which are stolen from probably the most paranoid (and error free) DBA I have ever met. Jeff "I never met a problem prevention method I didn't like" Kondas.
• Preventing errors from reoccurring by creating and following Corrective Action Procedures

Some of the topics will require an entire blog to cover while other blogs may group many topics together. The intent of this series of upcoming blogs is to help beginners create an approach to improving in non-technical areas and act as a refresher to those of us that have been in this profession for a while. Some of these recommendations come from problems that have occurred as a result of me not placing the same emphasis on non-technical areas as I did the technical ones. Many of the recommendations come from my experiences working as a mainframe DBA that, as we all know, is a very structured environment that demands rigorous adherence to best practices. The remainder comes from my experiences working as an educator, consultant and services provider.

Thanks for Reading,

Chris Foot
Oracle Ace

Readers of this particular blog entry have the added benefit of an Oracle expert's opinion that I think highly of. Howard J. Rogers, author of the Dizwell Informatics Site, has made some additional points that are important to consider when contemplating the use of Bigfile tablespaces. I have added them to this blog after the fact. It's important for me to provide as much pertinent information on the topic being discussed in all of my blogs. You can find much of his helpful information in this blog entry.

Oracle manuals state that a single Bigfile tablespace file can be up to 128 terabytes for a 32K block tablespace and 32 terabytes for an 8K block tablespace. Now THAT is a big data or temp file!

A key point to remember is that a Bigfile tablespace can only contain one file. A traditional Smallfile tablespace can contain up to 1,022 files. With Smallfile tablespaces, administrators have the option of expanding an existing tablespace file or adding a new file to accommodate application data growth.

Since Bigfile tablespaces can not contain multiple files, administrators must ensure that there is enough free disk available to allow the single file to grow to its capacity when mature. As a result, Oracle recommends that Bigfile tablespaces be used with Automatic Storage Management (ASM) or logical volume managers that support RAID and dynamically extensible logical volumes.

Rogers adds that ASM is not only recommended, but a requirement because of inode issues: "If you create a truly large bigfile tablespace on a traditional file system, you will suffer from horrendous inode locking issues. That is why ASM is not really an optional extra with these things."

For those of you that would like to know what inodes are, please refer to this Wikipedia definition.

Rogers also adds that if you use a single Bigfile temporary tablespace, you could be negatively impacting database performance for sort operations. He states "the number of tempfiles is one factor that affects the ability of the optimiser to parallelise a sort."

Rogers' blog entry provides this additional information to reinforce his point: "But parallel operations in 9i derive a lot of their performance benefits from the fact that their slaves can sort to different tempfiles within the one temporary tablespace, thereby distributing I/O (potentially across different hardware devices) and not blocking each other at the O/S-inode level. If, therefore, in 10g we are going to prevent you from having multiple tempfiles, because you've chosen to implement bigfiles, then your parallel query performance is going to be stuffed to buggery, putting it bluntly. Hence the temporary tablespace group: it restores the 'multiple tempfile' capability of parallel query slaves by allowing them to write to multiple different temporary tablespaces. But if you have a single temporary tablespace comprised of multiple tempfiles (because you chose to go smallfile), then you already have precisely that capability, and tablespace groups provide you with nothing extra, except a management headache arising from the additional level of redundant complexity you've added to your database."

So before you even consider creating one of these things, make sure that your environment is able to support its expansion. If not, you could have a tablespace that is not able to accommodate the normal growth of the application data it contains. In addition, heed Rogers' recommendation to use ASM for large Bigfile tablespaces.

An interesting benefit of using a Bigfile tablespace is the ability to run file alteration operations without specifying a file name. The attached file shows a Bigfile tablespace creation and alteration.

You'll notice that when I increased the size of the file from 500 MEGs to 1 GIGs and from 1 GIGs to 2 GIGs that I did not use the data file specification. I only used the tablespace name to alter the size of the datafile. Since the Bigfile tablespace can only contain one datafile, the file specification is not required.

Bigfile data tablespaces must be created as locally managed with automatic segement space management. These are the default specifications and don't have to be specified in the CREATE BIGFILE TABLESPACE DDL statement. Oracle will return an error if either EXTENT MANAGEMENT DICTIONARY or SEGMENT SPACE MANAGEMENT MANUAL is specified. But administrators can create locally managed undo and temporary Bigfile tablespaces, both of which use manual segment management.

The DDL statement below creates a Bigfile tablespace called bigtbs that is 80 GIGs in size:

CREATE BIGFILE TABLESPACE bigtbs
DATAFILE '/u01/oradata/orcl10g/bigtbs01.dbf' SIZE 80G

Administrators are able to specify the SIZE specification in kilobytes (K), megabytes (M), gigabytes (G), or terabytes (T).

Oracle 10G allows administrators to set the default tablespace size type during database creation. Administrators use the "SET DEFAULT BIGFILE/SMALLFILE TABLESPACE" specification in the database create statement. SMALLFILE is the default specification. All tablespaces that don't have the BIGFILE or SMALLFILE option specified during creation will default to the database level specification. The database level specification can be overriden by specifying either BIGFILE or SMALLFILE during tablespace creation.

DBA_TABLESPACES, USER_TABLESPACES AND V$TABLESPACE contain a new column called BIGFILE that allows administrators to determine if the tablespace is a Smallfile or Bigfile tablespace.

The last thing I would like you to consider when you are deciding whether you should be using Bigfile tablespaces is the time required to recover it. The larger the file size, the longer it will take to recover. The smaller the size, the faster the recover and the fewer the number of tapes you have to use during that recovery.

The less tapes I have to rely upon during a recovery, the better. I personally have a tape phobia. I have had my fair share of recoveries go haywire because of bad blocks on a tape (must be my magnetic personality). I especially liked the advice our O/S admin gave my team during our last recovery that was affected by tape I/O problems "tell the operator to yank it out of the tape reader, shake it around a bit and put it back in. He'll know what to do." It worked. Its hard to believe we rely upon these things for data recovery. Disk backups here I come.

Rogers also adds that Oracle's recovery utilities provide additional features to reduce the time to recover "Sure, backing up and restoring a 128TB datafile is not something to look forward to, but they invented block level media recovery in 9i for a reason (and bigfiles are it). Similarly, they've invented the ability to use incremental backups to 'refresh' rarely-taken image copies in 10g for a reason, and 128TB datafiles are similarly it. By all means, 'handle with care' -but when you do, use the entire 10g technology stack to address some of the issues they bring."

Summary
Bigfile tablespaces will have a future in many Oracle environments but they must only be implemented with much forethought. If any of you have a spooled output showing a terabyte tablespace create, feel free to send it to me! Additional comments are always welcome!

Thanks for Reading,

Chris Foot (with much help from Howard J. Rogers)
Oracle Ace

I have a LOT of books on Oracle. I am an avid Oracle book collector. I buy them, publishers send me them to me to review and I receive them as presents from family members. I have been working with Oracle for over 18 years now. I have been using Oracle since Version 5 and I have kept every Oracle book, manual and instructor's guide I crossed paths with during that time. Remember the old paper back manuals that Oracle sent to you with the software? I've got LOTS of them! I was certified by Oracle to teach virtually the entire DBA admin track and I have kept every version of every instructor's guide I used (Version 7 to Version 10). As a result, my library consists of dozens and dozens (and dozens) of books on Oracle.

There is so much information available on Oracle that it tends to become overwhelming. How do you find that one facet of information, that one explanation you are looking for when you are confronted with seemingly endless sources of information? Here's a hint, GO TO THE MANUALS FIRST.

Call me old school, but I still read the entire set of documentation for every new Oracle release. The order of the first few books is always the same: "New Features", "Concepts", "Administrators Guide", "Reference", "SQL Reference", "Performance Tuning Guide" and "Data Warehousing Guide". The remaining books are read in no particular order. It's pretty much what I feel like reading at the time. When I was teaching, I always reinforced to my students that they needed to reference the Oracle manuals before moving on to other forms of documentation.

A very experienced co-worker of mine was at a customer site installing an Oracle9i database on LINUX. He was reading the installation manual when the customer demanded to know why he was reading the manual when he was supposed to be "the high-priced expert." He quickly replied "I'm reading the manual because I am an expert." As your experience grows, you'll find that you'll become just like my co-worker, an avid user of the reference guides and not afraid to admit it.

Reference Manuals
It is good practice to keep a set of reference manuals for each major release of the database you are administering. Oracle does have a tendency to change default values for object specifications. In addition, each new release contains new parameters that affect the database's configuration. When you receive the latest and greatest version of Oracle's database (one of the benefits of purchasing support), turn straight to the "OracleX New Features" section to find out what impact the new release will have on your daily administrative activities. You'll also find many new features that haven't been covered by Oracle's new release whitepapers and marketing propaganda.

Oracle Internal Resources
The Oracle websites contain a wealth of information on the Oracle product sets. The following Oracle websites are favorites of mine:

• metalink.oracle.com - Oracle's premier web support service is available to all customers who have current support service contracts. Oracle MetaLink allows customers to log and track service requests. Metalink also allows users to search Oracle's support and bug databases. When you experience an Oracle problem, look up the return code (if one is provided) in the Oracle reference manuals. If you are unable to solve the problem, search Metalink using the return code or error message as the search criteria. The website also contains a patch and patchset download area, product availability and life-cycle information and technical libraries containing whitepapers and informational documents.
  
  I've been able to solve 80% of my problems using the search feature. Here's a couple of tips: make sure you specify "ALL SOURCES" when searching for problems, use the Advanced feature to narrow your search to specific products and remember that Oracle now adds a listing of products and asks you if you want to limit your search to them at the top of each search result set.
  
  While you are on Metalink, click on the "Knowledge" and "Forum" tabs. You will find that they provide a wealth of information. You will need to drill down into the different sections under the "Knowledge" tab to retrieve all of the gems of information available.
  
  
• technet.oracle.com - Technet's software download area allows visitors to download virtually any product Oracle markets. This website stores technical reference manuals for Oracle database releases 7 through 10, Oracle RDB, Oracle Gateways, Oracle Applications, Oracle Application server - the works.
  
  The site contains links to many Oracle blogs (including this one) and has forums, sample code and tutorials. A quick and easy way to get access to the information you need. Visitors are also able to view Oracle discussions, download product whitepapers and obtain information on Oracle education. Can you tell that I love this site? YOU BET!
  
  
• partner.oracle.com - If you are an Oracle partner, (and there are a lot of us), then this is the website for you. Oracle's partner website contains information on partner initiatives and provides customized portlets categorized into partner activity and job role.
  
  
• education.oracle.com - Oracle University's web site contains information on Oracle education including course descriptions, class schedules, self-study courses and certification requirements.
  
  When you go to education.oracle.com, look on the left side of the screen. You will see a link to the Knowledge Center. The Knowledge Center is Oracle's online education portal. You can take dozens of online courses from Oracle's experts. Want to learn from the people that wrote the code? This is the place that will allow that to happen.
  
  There is a link on the right side of the Knowledge Center that says "Take a Test Drive". Do yourself a favor - TAKE THE TEST DRIVE. You will have to pay for a full-blown subscription, but the benefits the Knowledge Center provides is well worth the cost. Take the test drive and then weasel the money out of your company for a full subscription. Tell them you'll work late, wash the boss's car. It's well worth it.
  
  
• www.oracle.com - Oracle's home page on the web. Contains white papers on Oracle features, product descriptions, etc.. Good place to get a high-level overview of the Oracle database's bells and whistles.

External Resources
Non-Oracle websites are also excellent sources of information. The Internet has an abundance of web sites containing hundreds of scripts, tips, tricks and techniques. I often find myself heading straight to Google when I can't quickly find the information I need on Metalink. Some of my favorites are:

• www.dbazine.com - How can you not love this website? The contributing authors list reads like a "who's who" of the database industry. Topics range from entry-level discussions to information that even the most experienced database user would find enlightening. Experts like Mullins, Inmon, Ensor, Celko, Nanda and Lewis provide readers with articles that are topical and interesting. Great articles and a pleasing, easy-to-navigate website makes DBAZine the place to go for database information. Am I biased? You bet. But I would still go to this site whether I wrote for it or not.
  
  
• www.orafaq.com - Orafaq discussion forums are excellent sources of information. Post a question to hundreds of experienced Oracle DBAs and you'll find out just how helpful Orafaq can be. Orafaq provides an intelligent search engine that visitors can use to search the discussion forums for topics of interest. The website also provides hints, tips, script and whitepapers.
  
  
• www.oracle.com/oramag - Oracle Corporation's own technical magazine. Oracle Magazine provides readers with product announcements, customer testimonials, technical information and upcoming events. Oracle magazine is available in hardcopy and on the web.
  
  
• asktom.oracle.com - Kyte maintains an Oracle Q&A website aptly tiled "Ask Tom". If you scan the Ask Tom website, you'll quickly find that his primary method of simplifying complex technical concepts is to use a snippet of code as an example. Based on the positive responses he receives, the code examples work. They also work for me too. A great website.
  

Blogs and Wikis
Blogs and Wikis are a fairly recent phenomenon when compared to other information sources. What I like about blogs is that their free-form style makes them interesting to visit. Some blogs contain small snippets of information, while others are more like articles. I've always been grammatically long-winded by nature, so my blog is often longer than most.

Wikis are collections of information that can be added and edited by the internet community. They are similar to a blog in structure but can contain the works from many different contributors. Personally, I find blogs to be more to my liking. I pick a blogger who I find to be interesting and noted for technical correctness and stick with them. I know that Wikis are supposed to be "self policing" but I just seem to gravitate to blogs.

Blogs that I like are:

• Mark Rittman's Blog - Warehousing and general Oracle information. His writings are so easy to read, that no matter what he writes on, it is a fun and interesting read.
  
  
• Pete Finningan's Blog - Oracle Security blog and general Oracle information. More than just a blog, the website has whitepapers, discussions, scripts, Oracle news....
  
  
• Oracle WTF - Not real fond of the name but very fond of the content. General Oracle information. Great content.
  
  
• FeuerThoughts - Oracle PL/SQL language. I'm absolutely terrible at coding anything (all that IF-THEN-ELSE-OTHERWISE-GOTO-EXIT stuff drains me). I go to his site to attempt to improve my PL/SQL skill sets. If his teachings can't help, no one's will. Excellent writer and creator of some pretty neat tools. Make sure you traverse the link to his Oracle PL/SQL Programming site. Tons of great information.
  
  
• Dizwell Informatics - Probably one of my favorite Oracle blogs. Excellent writer, always entertaining, never know what you'll find when you visit. Intelligent and humorous which is a good combination for a technical expert. Unafraid to express his opinions.
  

Book Recommendations
Third-party books are another excellent source of information. The big advantage third-party books have over the technical reference manuals is that technical reference manuals must provide all of the information on the entire Oracle environment while third-party books are able to focus on just what the author felt was important. If you choose a good author, you'll learn quickly.

Instead of listing books, I'll provide you with my favorite authors. You'll have the best chance of buying a great book if you select one from the following authors: Thomas Kyte, Jonathan Lewis (his book titled "Cost Based Oracle Fundamentals" is one of my favorites), Arup Nanda, Craig Mullins, Michael Abbey, Michael Correy, George Koch, and Kevin Loney.

Oracle Press Books provides a lot of great Oracle titles and Appress Publishing also comes to mind as having a great set of Oracle literary offerings.

Wrapup
I hope you enjoyed this brief discussion on where to find Oracle information. Don't attempt to remember everything about Oracle just try to remember the best place to look when you need information. Its out there!

Thanks for Reading,

Chris Foot
Oracle Ace

GLOGIN.SQL and LOGIN.SQL
Administrators are able to use two operating system files, GLOGIN.SQL and LOGIN.SQL to customize their SQL*PLUS environment (including the prompt). When SQL*PLUS starts, it will look for GLOGIN.SQL in $ORACLE_HOME/SQLPLUS/admin and will execute its contents as soon as the connection to the database is complete.

Once the GLOGIN.SQL file has been executed, the LOGIN.SQL file will be executed. Oracle will look for the LOGIN.SQL file in the current working directory (where you started SQL*Plus) and the operating system environment variable SQLPATH. Remember that LOGIN.SQL can overwrite the actions of GLOGIN.SQL. The exact names of these files may be different on some operating systems. Check the Oracle installation and administration guides provided for your operating system for the exact names.

Here's an example of my GLOGIN.SQL file on my PC that displays the time and instance name in my SQL*PLUS prompt. The file also contains a few formatting commands to format SQL*PLUS output. In this example, I'm using a SELECT statement to populate a variable that is displayed in the prompt. I'll show you how to use a SQL*PLUS pre-defined variable to do the same thing later in this blog.

COLUMN file_name FORMAT a44
COLUMN tablespace_name FORMAT a20
COLUMN owner FORMAT a15
COLUMN segment_name FORMAT a20
set lines 132
set pages 100
set termout off
col dbname new_value prompt_dbname
select instance_name dbname from v$instance;
set sqlprompt "&&prompt_dbname> "
set termout on
set time on

The Problem in Releases Prior to 10G
When I first tested this feature many years ago, I thought to myself "hey, this is a pretty cool feature." A fellow administrator walked by and saw that I was changing my prompt. He told me "you'll have to watch that because if you connect to another database, you won't execute the GLOGIN.SQL file again. That means you'll still have the original database name displayed in the prompt but you will be connected to the new database. There's a few other tricks you can do but just putting the display commands in GLOGIN won't work and its actually kind of dangerous."

10G SQL*PLUS comes to the rescue. GLOGIN.SQL and LOGIN.SQL are now executed after each successful connection to a target database. This is in addition to the files being run during the initial execution of the SQL*PLUS tool. Each time you connect to a different database in 10G SQL*PLUS, the prompt will change accordingly. The two demos below will show you the differences between 10G and 9I SQL*PLUS. Both 9I and 10G installations have the script above embedded in the GLOGIN.SQL file.

10G SQL*PLUS DEMO

SQL*Plus: Release 10.2.0.1.0 - Production on Thu Jun 22 07:14:10 2006

Copyright (c) 1982, 2005, Oracle. All rights reserved.

Connected to:
Oracle9i Enterprise Edition Release 9.2.0.6.0 - Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.6.0 - Production

07:14:11 orcl9i> * Prompt shows database I connected
07:14:17 orcl9i> * to when I activated SQL*PLUS
07:14:18 orcl9i>
07:14:18 orcl9i>
07:14:18 orcl9i>
07:14:18 orcl9i> connect system/maxwell@orcl
Connected.
07:14:26 orcl>
07:14:28 orcl> * Prompt changes when I connect
07:14:28 orcl> * to a different database
07:14:28 orcl>

9I SQL*PLUS DEMO

SQL*Plus: Release 9.2.0.6.0 - Production on Thu Jun 22 07:16:01 2006

Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.

Connected to:
Oracle9i Enterprise Edition Release 9.2.0.6.0 - Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.6.0 - Production

07:16:02 orcl9i> * Prompt shows database I connected
07:16:03 orcl9i> * to when I activated SQL*PLUS
07:16:03 orcl9i>
07:16:04 orcl9i>
07:16:04 orcl9i>
07:16:04 orcl9i>
07:16:04 orcl9i> connect system/maxwell@orcl.com
Connected.
07:16:14 orcl9i>
07:16:15 orcl9i> * Notice that the prompt doesn't change
07:16:16 orcl9i>
07:16:16 orcl9i> select name from v$database;

NAME
---------
ORCL

07:16:23 orcl9i>

Pre-Defined Variables
Pre-defined SQL*PLUS variables can also be used to customize a user's prompt. Currently, SQL*PLUS provides the following pre-defined variables (variables that are new in 10G are highlighted in bold):

• _CONNECT_IDENTIFIER - Connection identifier used to establish the database connection
  
  
• _DATE - Current date, or a user defined fixed string
  
  
• _EDITOR - Specifies the editor that will be used when the EDIT command is executed
  
  
• _O_VERSION - Oracle database version
  
  
• _O_RELEASE - Full release number of the Oracle Database
  
  
• _PRIVILEGE - The privilege level the current user has assigned to them
  
  
• _SQLPLUS_RELEASE - Full release number of the SQL*PLUS program being used
  
  
• _USER - Account name used to make the connection

For example, the command below could be embedded in GLOGIN.SQL or LOGIN.SQL to display the account used to connect to the database and the connection string:

SET SQLPROMPT "_USER'@'_CONNECT_IDENTIFIER > "

The demo below shows the results:

>
>
>
>SET SQLPROMPT "_USER'@'_CONNECT_IDENTIFIER > "
SYSTEM@orcl >
SYSTEM@orcl >
SYSTEM@orcl >

I don't care how experienced of a DBA you are, you are always in danger of working in the wrong database. It takes but a momentary distraction to lay the groundwork for a catastrophic error. Anything I can do to reduce my anxiety when I execute that 'DROP USER…. CASCADE" command, makes me a more efficient and more productive administrator. Taking five minutes to customize my prompt is well worth the effort - when you compare it to the disastrous results that can occur as the result of administering the wrong database. Now I can finally quit executing "SELECT NAME FROM V$DATABASE" to tell me what database I'm in.

Enhancements to the SPOOL Command
The SPOOL command tells SQL*PLUS to send all output to the specified flat file. Think how many times you have used the SPOOL command to save your output for later review. Well, 10G improves the usablity of the SPOOL command by adding the following syntax:

• APPEND - Appends output data to an existing file. The command will create a new file if the specified file is not found
  
  
• CREATE - Creates a new output file and will return an error if the file already exists
  
  
• REPLACE - This is the default option. REPLACE will replace an existing file or create it if it is not found

DBMS_OUTPUT Line Length and Output Limit Increases
Oracle provides the DBMS_OUTPUT package to display debugging information from PL/SQL code. The package was not designed to display reports or format output to standard output. But there are times when developers need to pump out large amounts of diagnostic information to help them debug their programs.

But developers using 10G R1 and earlier versions were frequently constrained by DBMS_OUTPUT's 255 byte single line limit and total session limitation of 1 million bytes. As a result, when using SQL*PLUS to debug their PL/SQL code, developers often received the following error messages:

ORA-20000: ORU-10028: line length overflow, limit of 255 bytes per line
Single line limit of 255 bytes exceeded.

ORA-20000: ORU-10027: buffer overflow, limit of 1000000 bytes
Maximum amount of 1 million bytes per session exceeded.

10G R2 increases the single length line limit to 32,767 bytes and the overall session limit is removed altogether.

Using PUPBLD.SQL
PUPBLD.SQL can be used for more than stopping those irritating "PRODUCT PROFILE TABLE NOT LOADED" messages from occurring when you log on to SQL*PLUS. PUPBLD.SQL creates a product profile table that is read by SQL*PLUS when users log on to the product.

Front-end applications can be designed to prevent users from updating data incorrectly. Other applications don't require database accounts because the user logs on the application and not the database. But there are still numerous applications that require database defined accounts and passwords. What happens if the user logs on to the database using their database account and password using SQL*PLUS? The application code is not available to prevent them from making incorrect or unwarranted changes to the database data.

Administrators can prevent this from happening by inserting rows into the product profile table to disable a SQL*PLUS user's ability to execute:

• SQL*PLUS commands - COPY, EDIT , EXECUTE, EXIT, GET ,HOST (or your operating system's alias for HOST, such as $on VMS, and ! on UNIX),QUIT, PASSWORD, RUN, SAVE, SET, SPOOL, START
  
  
• SQL statements ALTER, ANALYZE, AUDIT, CONNECT, CREATE, DELETE, DROP, GRANT, INSERT, LOCK, NOAUDIT, RENAME, REVOKE, SELECT, SET ROLE, SET TRANSACTION, TRUNCATE, UPDATE
  
  
• PL/SQL commands - BEGIN, DECLARE

SQL*PLUS reads restrictions from PRODUCT_USER_PROFILE when a user logs on using SQL*PLUS and maintains those restrictions for the duration of the session. To disable a SQL or SQL*PLUS command for a given user, insert a row containing the user's username in the USERID column, the command name in the ATTRIBUTE column, and DISABLED in the CHAR_VALUE column.

Conclusion
I hope you enjoyed this blog on SQL*PLUS.

Thanks for reading.

Chris Foot
Oracle Ace

It is my opinion that database administrators will always need to just as technically proficient as they had to be in the past. I will agree that database environments are becoming easier to administer. Oracle Grid Control allows us to administer, monitor, tune and troubleshoot an Oracle database without having to go as deep technically as we had to in the past. Although we won't have to know as much about the internals as we used to, our skill sets will become much more broad in scope.

The database engine is taking on a much more strategic role in most organizations. It provides the mechanism to store physical data along with business rules and executable business logic. The entire application environment (data storage, business rule enforcement, application program storage, communication, system monitoring) is now controlled by the database. Over time, the database engine will store more information related to the understanding of the business, the meaning of the data stored (metadata) and the mechanisms to control and track versions of the database, access programs and related software. As the database's area of influence expands, so does the DBA's. Can any one of us predict what IBM, Oracle and Microsoft have up their sleeves in their next "latest and greatest" release? Not me. But THAT is what makes this job exciting. Our area of technical influence will be expanding, not contracting.

The Importance of Soft Skills
Now that I have expressed my opinion on the expanding role of the DBA, let''s talks about the importance of soft skills. Before we begin, I define soft skills as the ability to communicate in both written and verbal forms and the ability to interact with fellow employees in a positive manner. In the past, a technicians' lack of interpersonal skills was often overlooked. The more technically proficient the technician was, the more leeway they were given.

Let me provide you with a couple of quick examples. When I first started in this profession, I attended a meeting at a large financial institution that included some pretty high-level representatives from both the business and technical areas. One of the technicians that attended the meeting was a mainframe operating system support technician.

The guy was well known to be very good at as his job and as nasty a person as you would want to meet. The meeting started and it quickly became apparent that most of the issues being discussed would be about business processes. The O/S expert slammed his pen and pad down on the table and declared "It looks like this is going to be a waste of my time. I have work to do. Call another meeting with me when you need technical advice." He then promptly walked out. Being a junior level programmer, I was in awe. I asked my boss after the meeting who he was. He stated "one of our mainframe gurus, don't act like he does until you get as good as he is."

A dozen years later, I saw a technician raise his hand at an enterprise-wide IT meeting when the CIO asked (rhetorically I think), who was the most important person in the organization. I think the CIO thought the answer would be "the CEO." The techie who raised his hand said "I am." I think every manager attending that meeting shrunk down in his or her seat. When the CIO asked why, he stated, "when my computers go down, all business stops." Just a few months ago, I had a 20-minute conversation with a UNIX admin who never bothered to turn around from her screen to look at me.

Although my examples may be over the top, they show you the mindset that often plagues our profession. I have been guilty of falling into that trap myself. In the past, I could be described as being "somewhat temperamental". I can also say that once you get a reputation for being "somewhat temperamental", it becomes hard to shake.

As the years have gone by, I have migrated from DBA to DBA Unit Manager and now DBA Operations Manager. I have seen too many excellent technicians end up with a mediocre career because they achieved a reputation for "not playing well with others." I talk from experience. If you want to excel as a technician, you will need to be technical, but you will also need to work well with others.

Verbal and Written Communication Skills
I think people read this blog because they take pride in their work and want to become better at their chosen profession. So here's my second piece of non-technical advice. The importance of improving your communication skills can not be understated. I don't care how strong of a technician you are, if you can't communicate effectively with your peers, you won't be able to succeed in this profession. In the old days, you might have been able to get by with just your technical skills. That is definitely not the case in today's business world.

Take a look at your last performance appraisal forms, I'm betting that most of the criteria you are being judged upon depends upon communications. The key words and phrases to look for are "ability to work in a team environment", "keep supervisors informed", "maintain good communication with the user community", "ensure the content of the communication is at the appropriate level for the intended audience", "provide system and user documentation for projects and system enhancements." I pulled all of the aforementioned phrases verbatim from one of my own recent performance appraisals. I reviewed all of the criteria that I was being evaluated upon and found that almost ninety percent of the items depended upon verbal or written communications.

If you don' have good communication skills, all is not lost. Like anything else, these skills can be learned. I still consider myself to be only a fair writer. I am in awe of people like Craig Mullins (a fellow blogger here on DBAZine) who can just sit down and let the words flow. I often find myself agonizing over every word and sentence.

When I first started working in a corporate environment, my writing skills were terrible. My original career was not database administration, it was construction. A job that didn't require you to excel at written and verbal communications. One on the job accident, 9 operations and 11 months of vocational rehabilitation training later and I had a new career as a COBOL programmer. I went from working with a construction crew to working with computer programmers. My first employer was a very large and somewhat stuffy financial institution. When I was employed there, men couldn't leave their floor without wearing their suit coat.

I quickly learned that getting caught wiping your computer screen off with your tie didn't really show your managers that you were good at thinking "outside of the box." That was about the only thing I thought that piece of knotted cloth around my neck was good for. For the first six months, I refused to tie them. Being the non-conformist that I was, I just loosened them up, slipped them off and hung them up.

Saying that my communication skills were rough around the edges when I started my carrer would be an understatement. But I had the good fortune of having a manager that understood the importance of both verbal and written communications. I would write a memo, she would correct it with her red pen and send it back to me for a rewrite. Many of them had a "Nice Try!" and a smiley face on top. After becoming exasperated because of the numerous rewrites (and seeing all of those smiley faces), I thought I had better improve my writing skills. I read books, practiced writing and became involved with as many company newsletters and related communications as I could. When I asked to join a newsletter, I always started with "I'm not the greatest writer, but I'm trying to learn." I also asked my peers that worked on the newsletter to critique my work. The more I was critiqued, the better I became.

Same way with public speaking. My first speech could be described as being "somewhat less than stellar." Craig Mullins would gently prod me from time to time until he finally convinced me that speaking was something I should be doing from time to time. Craig promised to sit in the back of the room for my first speech and give me hand signals if I was speaking too fast, too slow, too loud or too soft. Halfway through the speech, my knees were knocking and his hands were in constant motion.

I found that like anything else, experience helps. But I will say that my speaking career was not without excitement. I learned that you really shouldn't drink a carbonated beverage wearing a tie mike that is attached to a set of 6 12-foot speakers. When I was done chugging the pop before the speech, I looked around and saw everyone laughing at the noises I had just made. I then found out that a tie mike doesn't help when you try to stifle a burp.

I also learned that some podiums are on wheels and those wheels aren't always locked. I started my first sentence, leaned against the podium and it began to move. I tripped a little trying to stop the podium from moving and ended up heading for the end of the stage at a very rapid rate. It was a raised stage too, about six feet higher than the first row of seats. As I quickly approached the end of that raised stage, I noticed that the people in the first row were making motions just like the extras did in the old Godzilla movies- right before they got stomped on. Lucky for me one of my work buddies in the first row had the good sense to jump to his feet and stop the podium (and me) from killing a few members of my audience. I got over those little snafus and kept plugging away. With each subsequent speech, I started to improve.

The point I am trying to make is that you can improve upon your communication skills. IT shops are no longer evaluating technicians purely on their technical skills. I have seen the soft skill evaluation pendulum swing a little more each year.. It is the total package of skills that you bring to the table that you are being evaluated upon. We all know the importance that our technical skill sets have upon our success in this field. But you also need to be well rounded in all of the skill sets your managers are looking for.

Thanks for reading,
Chris Foot

Data Pump Import
Let's begin our discussion with a quick review of a sample Data Pump Import command:

impdp foot/foot directory=data_pump_dir1 dumpfile=expfull.dmp logfile=impfull.log full=y

"impdp" tells the operating system to run the Data Pump Import utility. The parameters that follow are used to control the job's execution. In our example, the account and password are both "foot", the file that will contain the input data is "expfull.dmp", the logfile is"impfull.log" , and a full database import will be performed

We learned previously that you must precreate a directory using a SQL DDL statement on the database being accessed. If you don't pre-create the directory, Oracle provides a default directory called DATA_PUMP_DIR. That means that users are unable to fully qualify the output file and log file as they were able to do in the non Data Pump versions of Export and Import. You specify a fully qualified directory on the server when you create the new database directory object and then you must grant access to that directory. My last blog contains more information on the creation and administration of Oracle directories.

Data Pump provides the following import modes:

• Full - Activated by specifying "FULL=Y". Notifies Import to import the entire content of the input file. The amount of data actually imported is dependent upon the data contained in the input file. If the input file contains all of the objects and data from a full database export, all of those objects will be imported. If the input file contains a schema export, all of the objects in that schema will be imported.
  
  
• Schema Import - Activated by specifying "SCHEMAS=schema, schema, schema….). Imports the entire schema and all dependent objects (tables, indexes, grants, synonyms, PL/SQL programs, views). The problem with the older version of Import was that because it didn't recreate the user (just the user's objects), the administrator was forced to save the user creation and grant statements to recreate the user with the same privileges it had before the drop was performed. A Data Pump schema import will recreate the user and execute all of the associated grants, user password history, etc..
  
  
• Table Import - Activated by specifying "TABLES=[schemas].tablename, [schemas].tablename,…. Imports a table and all dependent objects.
  
  
• Tablespace Import - Activated by specifying "TABLESPACES=tablespacename, tablespacename, tablespacename… All of the objects contained in the specified tablespace are imported. If a table is imported, its dependent objects are also imported regardless of the tablespace they were stored in.
  
  
• Transportable Tablespace Import - Activated by specifying "TRANSPORT TABLESPACES=tablespacename… Here's how this feature works. The administrator first runs Data Pump Export with the Transportable Tablespace feature activated. Data Pump exports only the metadata for the objects contained in the specified tablespaces. The objects in the tablespace set being exported can not have any references to objects contained in tablespaces outside of the tablespace set. In addition there can't be any objects in the tablespace set that are referenced by objects outside of the tablespace set being exported. Having a index in a tablespace that is not contained in the same tablespace set that contains the table it is built upon would be an example of a dependency reference problem. Administrators use the TRANSPORT_FULL_CHECK parameter to perform the dependency verification.
  
  The administrator then uses an operating system command to copy the tablespace datafile(s) to the target database. The transportable tablespace import feature is then used to attach the datafile (and its contents) to the database. Transferring the datafile using operating system commands is much faster than exporting and importing the data using the traditional Data Pump Export and Import operations. Administrators are only limited by hardware and network performance.

It is important to note that you are not forced into using the same import mode as you did for the export. You can use the schema and table option using an export file that was created by a full database export. You can also import one table from a schema level export (and so on…). Lastly, Data Pump Import can only read data that was created by a Data Pump Export.

Let's continue by reviewing a few of the 10G Data Pump Import parameters that you may find interesting. Many of these parameters will be the same as for Data Pump Export.

Data Pump Import Parameters

• JOB_NAME=jobname_string - You are able to assign a job name of up to 30 bytes to a Data Pump operation. This allows the job to be more easily identified when querying the DBA_DATAPUMP_JOBS and DBA_DATAPUMP_SESSIONS data dictionary views that provide information on Data Pump operations. If you do not specify a job name, Oracle will create a default job name for you. The form will be SYS_IMPORT_mode_nn, where mode is the type of export (FULL, TABLE, etc.) and "nn" is a incrementing number starting at 01. The job name is used in the ATTACH command (shown below) to attach, and reattach, to running jobs.
  
  
• ATTACH [=[schema_name.]job_name] - Used to attach a client session to an existing Data Pump operation. An example of the ATTACH command is provided below:
  
  > impdp foot/foot ATTACH=foot.importfulljob
  
  
• DIRECTORY=directory_object - All of the dumpfile set I/O is generated by Data Pump processes that run within the constructs of the database engine. As a result, the O/S account performing the I/O is Oracle, which usually has a higher level of privileges than most other accounts. Oracle attempts to prevent unwarranted operations from occurring by using Oracle directory objects that require read and write privileges granted by the DBA. Users running Data Pump operations are only able to read and write files in the directories that they have been granted access to.
  
  Before a user is able to run any Data Pump operation, the administrator must pre-create a directory and grant privileges to the user on that directory. This means that users are unable to fully qualify the output file and log file as they were able to do in the non Data Pump versions of Export and Import. If you don't pre-create the directory, Oracle provides a default directory called DATA_PUMP_DIR.
  
  
• DUMPFILE=[directory_object:]file_name [, ...] - Contains the metadata and/or data output from a Data Pump Export operation that Data Pump Import is using as input. The directory object specification is not required to be specified if a directory is specified in the DIRECTORY parameter. If one is specified in the DIRECTORY parameter and the DUMPFILE= parameter, the DUMPFILE=directory: specification will take precedence.
  
  
• ESTIMATE={BLOCKS | STATISTICS} - Used to estimate the amount of data a Data Pump Import network operation will generate. This differs from the estimate option that is used with Data Pump Export. Using the ESTIMATE parameter with Data Pump Export will ask Data Pump to estimate how large the output dumpfile will be. I'll provide more information on Data Pump network import operations later in this blog.
  
  
• Fine Grained Selectivity Parameters (Exclude, Include, Query, Flashback, Content) - Allows the administrator to include or exclude objects. For more information on these parameters, please turn to my previous blog titled "10G Data Pump Part 1".
  
  
• HELP = {y | n} - Displays a listing of all the parameters that can be specified for the Data Pump operation. For example:
  
  > impdp help=y
  
  Would provide a listing of parameters for a Data Pump Import operation.
  
  
• LOGFILE=[directory_object:]file_name - Contains work-in-progress, work completed and errors generated during a Data Pump Operation. For a real time update on a Data Pump operation's activities, you are able to attach to a running Data Pump operation using the ATTACH command and then execute the STATUS command. Like its dumpfile counterpart, The directory object specification is not required to be specified if a directory is specified in the DIRECTORY parameter. If one is specified in the DIRECTORY parameter and the DUMPFILE= parameter, the DUMPFILE=directory: specification will take precedence.
  
  
• NETWORK_LINK=database_link_name - OK, I must admit that this is one of my favorite new features provided by Data Pump Import. The network transfer option allows administrators to attach to any source database that can be accessed from the network, transfer the data and write it to the target database. No dumpfile required! Administrators use the NETWORK_LINK parameter to specify a pre-created database link that points to the source database.
  
  This parameter must be specified when using the FLASHBACK_SCN, FLASHBACK_TIME, ESTIMATE, or TRANSPORT_TABLESPACES parameters.
  
  
• PARALLEL=integer - Multiple worker processes are able to perform inter-table and inter-partition work in parallel. This allows administrators to load and unload database tables using parallel processing. Data Pump Import takes advantage of parallelism to build indexes and load package bodies.
  
  The most beneficial aspect of Data Pump's parallel processing feature is that it is dynamic. Administrators are able to attach (connect) to a Data Pump job that is currently executing and dynamically alter the number of parallel processes it uses.
  
  Oracle documentation states that the parallel option may cause file contention if too many parallel processes are used to read the same input dumpfile. As a result, you'll need to be careful when specifying the level of parallelism.
  
  
• REUSE_DATAFILES={y | n} - Tells Data Pump Import to reinitialize existing tablespace datafiles. Specifying "N" tells Data Pump Import to generate an error message from the failing create tablespace execution. The Import will then continue executing. Specifying "Y" tells Import to issue a warning and reinitialize the tablespace datafile(s) which results in a total loss of existing data in the target tablespace.
  
  
• STATUS=[integer] - Specifies the number of seconds between each status information display.
  
  
• TABLE_EXISTS_ACTION={SKIP | APPEND | TRUNCATE | REPLACE} - Provides Data Pump Import with instructions to perform when it finds a table in the target database with the same name as a table contained in the input file.

  TABLE_EXISTS_ACTION=SKIP - Existing tables are bypassed if they already exist. No action is taken.

  TABLE_EXISTS_ACTION=APPEND -Data Pump Import will append rows to the existing table.

  TABLE_EXISTS_ACTION=TRUNCATE - Data Pump Import will truncate the existing table and load rows from the source.

  TABLE_EXISTS_ACTION=REPLACE - Existing tables will be dropped, recreated and then loaded from the source.
  
  

• TRANSFORM = transform_name:value[:object_type] - Allows the administrator to alter object creation DDL for specified objects . For more information on these parameters, please turn to my previous blog titled "10G Data Pump Part 1".

Commands Available in Interactive Mode
We learned previously that administrators are able to attach and detach from actively running Data Pump operations. Administrators use the ATTACH and EXIT_CLIENT commands to connect and disconnect from an active Data Pump job. This output file provides a demonstration of detaching and attaching from an active Data Pump operation. You'll notice that I dynamically change the parallel setting after I reattach.

Let's take a look at some of the other parameters for Data Pump Import that you can dynamically change during its execution:

• CONTINUE_CLIENT - Switches from interactive mode to status (logging) mode. An example of this command follows:
  
  impdp> CONTINUE_CLIENT
  
  
• EXIT_CLIENT - Detaches the client from an active Data Pump operation. The job continues to run. The administrator is then able to use the ATTACH command to reattach to the job at any time. The attach can be performed if the job is executing or in a stopped state. An error message will be returned if an attempt is made to attach to a completed Data Pump Operation.
  
  
• HELP - Provides help on interactive mode Data Pump commands.
  
  
• KILL_JOB - Kills the active Data Pump operation and detaches all client sessions. This command differs from the STOP_JOB command which is discussed below. The KILL_JOB will remove DUMPFILEs that were created which prevents the user from taking advantage of Data Pump's restart capabilities. The log files will remain.
  
  
• PARALLEL=integer - Change the number of parallel processes for a Data Pump operation.
  
  
• START_JOB / STOP _JOB - Restarts and stops the Data Pump operation the client is currently attached to. Data Pump jobs update a master table with all pertinent information pertaining to the job being executed. The master table knows about the objects being exported or imported and their location in the dump file set.
  
  As a result, a Data Pump job can be restarted in the event of a planned or unplanned stoppage. The Data Pump job accesses the master table to determine where to start and picks up right where it left off. Administrators are able to use the STOP_JOB AND START_JOB commands on the client to temporarily stop jobs and restart them when desired.
  
  Oracle provides the SKIP_CURRENT option of the START_JOB parameter to allow administrators to restart a Data Pump Import operation that failed to restart because of a failing DDL statement. SKIP_CURRENT tells Data Pump Import to skip the object having the problem and restart processing on the next object.
  
  
• STATUS=[integer] - Specifies the number of seconds between each status information display.

WRAPUP
I think after reading these last four blogs, we should all have a pretty good understanding of 10G Data Pump. We can finally retire Data Pump's ancestors Export and Import. As we now know, they just can't compete with the features and functionality provided by Data Pump Export and Import.

Thanks for reading.

Chris Foot
Oracle Ace

• Database Backup - Export can be used to take a time consistent snapshot of the data in an Oracle database. The data contained in the flat files can be reloaded back into the database at any time. A common question I received from students during my classes was if redo logs could be used to roll the database forward to the current point-in-time after using Import to reload the data. Sorry, Export and Import work within the database data files. The synchronization information required by Oracle during a database restore and recovery relies upon the data contained in the control files and data file headers. Since Export/Import does not change any of the synchronization data in the file headers, the utility can not be used to perform traditional Oracle recoveries executed using the RECOVER statement.
  
  
• Data Archival - I have seen several shops that used the transportable tablespace feature to archive data to offline storage. If the data was required for reporting purposes, the tablespace was reattached to a reporting database for quick access.
  
  
• Database Upgrades - Export/Import is one of the upgrade options that is almost always specified in the Oracle migration manuals. It is often the only method if the difference between the old source release and the new target release is substantial. Since it requires that a new database be created using the new release, backoff is certainly simplified.
  
  
• Data Transfer - Administrators have been using FROM USER/TO USER to transfer data from one schema to another since SCOTT/TIGER was a cub.
  

Let's begin our discussion with a quick review of a sample Data Pump Export command that I provided in a previous blog:

expdp foot/foot directory=export_dir dumpfile=expfull.dmp logfile=expfull.log full=y

"expdp" tells the operating system to run the Data Pump Export utility. The parameters that follow are used to control the job's execution. In our example, the account and password are both "foot", the file that will contain the output data is "expfull.dmp", the logfile is"expfull.log" , and a full database export will be performed.

We learned previously that you must precreate a directory using a SQL DDL statement on the database being accessed. If you don't pre-create the directory, Oracle provides a default directory called DATA_PUMP_DIR. That means that users are unable to fully qualify the output file and log file as they were able to do in the non Data Pump versions of Export and Import. You specify a fully qualified directory on the server when you create the new database directory object and then you must grant access to that directory. You will need to do this for both Data Pump Export and Import operations.

Creating the directory:

CREATE DIRECTORY data_pump_dir1 as '/u01/oradata/prod1/datapump';

We can then view all of the directories by executing the select statement below:

SQL> r
1* select directory_name, directory_path from dba_directories

DIRECTORY_NAME DIRECTORY_PATH
--------------- -------------------------------------------------------------
SUBDIR /app/oracle/product/10.2/db/demo/schema/order_entry//2002/Sep
XMLDIR /app/oracle/product/10.2/db/demo/schema/order_entry/
MEDIA_DIR /app/oracle/product/10.2/db/demo/schema/product_media/
LOG_FILE_DIR /app/oracle/product/10.2/db/demo/schema/log/
WORK_DIR /ade/aime_10.2_lnx_push/oracle/work
DATA_FILE_DIR /app/oracle/product/10.2/db/demo/schema/sales_history/
DATA_PUMP_DIR /app/oracle/product/10.2/db/admin/ctitst/dpdump/
ADMIN_DIR /ade/aime_10.2_lnx_push/oracle/md/admin
DATA_PUMP_DIR1 /u01/oradata/prod1/datapump

Before we are able to begin using the directory for Data Pump Operations, we'll need to grant access to it:

GRANT READ, WRITE ON DIRECTORY data_pump_dir1 TO foot;

There is one important point to remember. Even though you have granted the database user the authority to read and write to that directory, the user does not have the authority to access those files outside of the Oracle database. If the only task the user needs to do is execute the Data Pump Export and Import utilities, the database privileges will suffice. But if the user needs to view or manipulate the output and log files after they are created, they will need to have an operating system account created on the database server with the proper operating system privileges granted to it.

In addition, you will have to ensure that the Oracle server itself has the appropriate read/write operating system privileges on the directory structure used in the CREATE DIRECTORY statement. If the Oracle engine can't access the directory, the Data Pump Export and Import won't be able to access it.

Data Pump provides the following export modes:

• Full Export - Activated by specifying "FULL=Y". Notifies Export to export the entire database. Everything you need to recreate all of the application data objects will be included in the output file including tablespaces, tables, indexes, views, synonyms, PL/SQL programs, users, grants - the works.
  
  
• Schema Export - Activated by specifying "SCHEMAS=schema, schema, schema….). Exports the entire schema and all dependent objects (tables, indexes, grants, synonyms, PL/SQL programs, views). The problem with the older version of Import was that because it didn't recreate the user (just the user's objects), the administrator was forced to save the user creation and grant statements to recreate the user with the same privileges it had before the drop was performed. A Data Pump schema import will recreate the user and execute all of the associated grants, user password history, etc..
  
  
• Table Export - Activated by specifying "TABLES=[schemas].tablename, [schemas].tablename,…. Exports a table and all dependent objects.
  
  
• Tablespace Export - Activated by specifying "TABLESPACES=tablespacename, tablespacename, tablespacename… All of the objects contained in the specified tablespace are exported. If a table is exported, its dependent objects are also exported regardless of the tablespace they are stored in.
  
  
• Transportable Tablespace Export - Activated by specifying "TRANSPORT TABLESPACES=tablespacename… Only the metadata is exported for objects contained in the specified tablespaces. The administrator then uses an operating system command to copy the tablespace datafile(s) to the target database. The transportable tablespace Import feature is then used to attach the datafile (and its contents) to the database.

I provided a listing of parameters that I thought readers might find interesting in a previous blog titled "10G Data Pump Part 1". It may be beneficial for those of you that haven't read that blog to do so before continuing.

Here are a few other parameters I thought you may find to be beneficial during your day-to-day usage of Data Pump Export:

• JOB_NAME=jobname_string - You are able to assign a job name to a Data Pump operation. This allows the job to be more easily identified when querying the DBA_DATAPUMP_JOBS and DBA_DATAPUMP_SESSIONS data dictionary views. If you do not specify a job name, Oracle will create a default job name for you. The form will be SYS_EXPORT_mode_nn, where mode is the type of export (FULL, TABLE, etc.) and "nn" is a incrementing number starting at 01. The job name is used in the ATTACH command below to attach, and reattach, to running jobs.
  
  
• ATTACH [=[schema_name.]job_name] - Used to attach a client session to an existing Data Pump operation. An example of the ATTACH command is provided below:
  
  > expdp foot/foot ATTACH=foot.export_full
  
  
• DIRECTORY=directory_object - All of the dumpfile set I/O is generated by Data Pump processes that run within the constructs of the database engine. As a result, the O/S account performing the I/O is Oracle, which usually has a higher level of privileges than most other accounts. Oracle attempts to prevent unwarranted operations from occurring by using Oracle directory objects that require read and write privileges granted by the DBA. Users running Data Pump operations are only able to read and write files in the directories that they have been granted access to.
  
  Before a user is able to run a Data Pump operation, the administrator must pre-create a directory and grant privileges to the user on that directory. This means that users are unable to fully qualify the output file and log file as they were able to do in the non Data Pump versions of Export and Import. If you don't pre-create the directory, Oracle provides a default directory called DATA_PUMP_DIR.
  
  
• DUMPFILE=[directory_object:]file_name [, ...] - Contains the metadata and/or data output from a Data Pump Export operation. The directory object specification is not required to be specified if a directory is specified in the DIRECTORY parameter. If one is specified in the DIRECTORY parameter and the DUMPFILE= parameter, the DUMPFILE=directory: specification will take precedence.
  
  
• ESTIMATE={BLOCKS | STATISTICS} - Used in conjunction with the ESTIMATE_ONLY parameter to estimate the amount of space a Data Pump Export dumpfile will consume. For more information on these parameters, please turn to my previous blog titled "10G Data Pump Part 1".
  
  
• Fine Grained Selectivity Parameters (Exclude, Include, Query, Flashback, Content) - Allows the administrator to include or exclude objects. For more information on these parameters, please turn to my previous blog titled "10G Data Pump Part 1".
  
  
• HELP = {y | n} - Displays a listing of all the parameters that can be specified for the Data Pump operation. For example:
  
  > expdp help=y
  
  Would provide a listing of parameters for a Data Pump Export operation.
  
  
• LOGFILE=[directory_object:]file_name - Contains work-in-progress, work completed and errors generated during a Data Pump Operation. For a real time update on a Data Pump operation's activities, you are able to attach to a running Data Pump operation using the ATTACH command and then execute the STATUS command. Like its dumpfile counterpart, The directory object specification is not required to be specified if a directory is specified in the DIRECTORY parameter. If one is specified in the DIRECTORY parameter and the LOGFILE= parameter, the LOGFILE=directory: specification will take precedence.
  
  
• PARALLEL=integer - Multiple worker processes are able to perform inter-table and inter-partition work in parallel. This allows administrators to load and unload database tables using parallel processing.
  
  The most beneficial aspect of Data Pump's parallel processing feature is that it is dynamic. Administrators are able to attach (connect) to a Data Pump job that is currently executing and dynamically alter the number of parallel processes it uses.
  
  
• SAMPLE=[[schema_name.]table_name:]sample_percent - Allows an administrator to sample a percentage of the object specified. If you don't specify a table name, the percentage will be applied to the entire Data Pump operation.
  
  
• STATUS=[integer] - Specifies the number of seconds between each status information display.

Commands Available in Interactive Mode
We learned previously that administrators are able to attach and detach from actively running Data Pump operations. Administrators use the ATTACH and EXIT_CLIENT commands to connect and disconnect from an active Data Pump job. This output file provides a demonstration of detaching and attaching from an active Data Pump operation. You'll notice that I dynamically change the parallel setting after I reattach.

Let's take a look at some of the other parameters you can dynamically change:

• ADD_FILE=[directory_object]file_name [,...] - This feature allows the administrator to add additional files to a Data Pump Export operation. The ability to add data files "on the fly" would come in very handy when the first dumpfile's target drive begins to fill up. The administrator would be able to add a new file on a different drive.
  
  
• CONTINUE_CLIENT - Switches from interactive mode to status mode. An example of this command follows:
  
  export> CONTINUE_CLIENT
  
  
• EXIT_CLIENT - Detaches the client from an active Data Pump operation.
  
  
• HELP - Provides help on interactive mode Data Pump commands.
  
  
• KILL_JOB - Kills the active Data Pump operation and detaches all client sessions. This command differs from the STOP_JOB command which is discussed below. The KILL_JOB will remove DUMPFILEs that were created which prevents the user from taking advantage of Data Pump's restart capabilities. The log files will remain.
  
  
• PARALLEL=integer - Change the number of parallel processes for a Data Pump operation.
  
  
• START_JOB / STOP _JOB - Restarts and stops the Data Pump operation the client is currently attached to. Data Pump jobs update a master table with all pertinent information pertaining to the job being executed. The master table knows about the objects being exported or imported and their location in the dump file set.
  
  As a result, a Data Pump job can be restarted in the event of a planned or unplanned stoppage. The Data Pump job accesses the master table to determine where to start and picks up right where it left off. Administrators are able to use the STOP_JOB AND START_JOB commands on the client to temporarily stop jobs and restart them when desired.
  
  
• STATUS=[integer] - Specifies the number of seconds between each status information display.

WRAPUP
I think after reading these three blogs on 10G Data Pump, we should have a pretty good understanding of Data Pump Export. In my last blog of this series, we'll turn our attention to DATA Pump Import.

Data Pump Components
Let's take a look at the first diagram. Data Pump provides four client utilities that execute the procedures provided in the DBMS_DATA_PUMP PL/SQL package. The client utilities accept input in the form of parameters which are passed to the DBMS_DATA_PUMP PL/SQL package to enable the exporting and importing of metadata and data into and out of an Oracle database.

A brief overview of the four client utilities is provided below:

• EXPDP - Client interface to the Data Pump Export utility. The interface closely resembles its predecessor, Oracle Export.
• IMPDP - Client interface to the Data Pump Import utility. The interface closely resembles its predecessor, Oracle Import.
• OEM Interface - Provides access to the DBMS_DATA_PUMP PL/SQL package from the DB and Grid Control web interfaces.
• Custom Interface - Allows for the movement of complex data mining models.

Because Data Pump is made available through the DBMS_DATA_PUMP PL/SQL package API, users are able to call Data Pump natively from application programs.

Data Access Methods
Data Pump provides two access methods to load and unload data:

• Direct Path - Direct Path unload has been available since Oracle 7.3 and its direct load counterpart has been around since Oracle 8i. Oracle will attempt to use the Direct Path first but there are a few situations that can force Data Pump to switch to the External Tables access method. Oracle documentation provides a listing of all the reasons why Data Pump was unable to use the Direct Path data access method. Since Data Pump will automatically switch from the Direct Path access method to External Tables, it is transparent to the client.
  
  
• External Tables - The capability of reading data from external sources has been available since Oracle9i. Oracle10G also support writing data to external destinations. Data Pump provides the ORACLE_DATAPUMP access driver that allows it to read and write to external files. Data Pump is required to use this access method if parallel processing is activated to increase performance.

The user is not required to determine which data access method best fits each job's requirements. The Data Pump utility itself will automatically choose the most appropriate data access method for each table being accessed. No user intervention required! The output created by both the Direct Path and External Table data access methods has the same external format. This allows Data Pump to choose the best access method to load data, regardless of which data access method created the output.

Metadata API
Data Pump uses the DBMS_METADATA package to load and unload metadata. The Metadata API stores the metadata as XML documents. This differs from the original Export utility which stored the metadata as DDL. The Data Pump Import utility takes advantage of the flexibility provided by the XML format by allowing users to change object ownership, storage characteristics and tablespace destinations.

Data Pump Data Flow
In order for us to truly understand how data pump works; we need to learn how the processes and memory structures interact with each other. The second diagram closely resembles the one that Oracle uses in many of its presentations on the Data Pump Export data flow. Let's break down the data flow diagram into its individual components:

Shadow Processes
This should be an easy one for us to understand. The shadow process is the foreground process that is created when a client logs on to an Oracle database. The shadow process creates most of the other components found on this diagram. After receiving the Data Pump initiation request, the shadow process creates the master table, advanced queue structures for the command and control and status queues and the master control process.

After the Data Pump job is executing, the shadow process is delegated the responsibility of services the GET_STATUS requests from the client. If the client detaches, the process is no longer needed and is removed. We learned previously that detaching a client from Data Pump does not cause the Data Pump job to stop executing. Users are able to reattach to a Data Pump operation as often as required during the job's life cycle. Each time the reattach is performed, Oracle will start a new shadow process on the client's behalf. In addition, multiple clients can be attached to the same Data Pump job concurrently, with each client having its own shadow process.

Status Queue
Oracle takes advantage of its Advanced Queuing (AQ) feature to provide communication paths between the different Data Pump processes. The Data Pump master control process writes work progress and error messages to the status queue. Shadow processes subscribe to this queue to retrieve the status information.

Master Control Process
The master control process is the "traffic cop" of the Data Pump environment. It controls the execution of Data Pump jobs by delegating work requests to worker processes based on the particular execution phase (metadata capture, data unloading, data loading, etc.) the job is currently performing. The master control proces updates the master table with high-level information (job's description, state, restart and dumpfile info.) required to restart stopped Data Pump jobs. The process is also responsible for communicating to the clients through the status queue and performing logging operations.

Master Table
The master table contains information about the details of the current Data Pump operation being performed. You can equate it to a running log book that provides information on each object being exported or imported and their locations in the dumpfile set. In addition, the master table also records the parameters supplied by the calling program, worker status information and output file information.

The table provides all of the information required to restart a Data Pump job that has stopped because of a planned, or unplanned, failure. The table is owned by the account running the Data Pump job and is written to the dump file at the end of each Data Pump Export operation. Data Pump Import read's the master table and loads it into the user's schema during the initial phase of execution. The table is then used to keep track of the Data Pump Import operation (job status, object being imported, etc.).

Command and Control Queue
The command and control queue provides an avenue of communications between the master control process and worker processes. All of the work requests created by the master process and the associated responses are passed through the command and control queue.

Worker Process
During the initial phases of a Data Pump job, the master control process creates worker process to perform the actual data and metadata load and unload operations. The worker processes are responsible for updating the master table with information on the status (pending, completed, failed) of each object being processed. This information is used to provide the detailed information required to restart stopped Data Pump jobs.

Parallel Query Process
For those of you that have used Oracle's Parallel Query Option (PQO), these processes should be familiar to you. Data Pump's Parallel Query Processes are standard parallel query slave processes that are controlled by the worker process which assumes the role of the parallel query coordinator.

Dump Datafile
Data Pump dump files are created by EXPDP and used as input by IMPDP. Since their format is incompatible with their predecessors, Export and Import, the utilities are not interchangeable. Import can only read a file created by Export and IMPDP can only read a file created by EXPDP.

The dumpfile contains data and/or metadata and the master table which is written to the dumpfile at the end of each EXPDP operation and read at the beginning of each IMPDP operation.

All of the dumpfile set I/O is generated by Data Pump processes that run within the constructs of the database engine. As a result, the O/S account performing the I/O is Oracle, which usually has a higher level of privileges than most other accounts. Oracle attempts to prevent unwarranted operations from occurring by using Oracle directory objects that require read and write privileges granted by the DBA. Users running Data Pump operations are only able to read and write files in the directories that they have been granted access to.

Before a user is able to run a Data Pump operation, the administrator must pre-create a directory and grant privileges to the user on that directory. This means that users are unable to fully qualify the output file and log file as they were able to do in the non Data Pump versions of Export and Import. If you don't pre-create the directory, Oracle provides a default directory called DATA_PUMP_DIR.

You specify a fully qualified directory on the server when you create the new database directory object and then grant access on that directory to the user running the data pump operation:

CREATE DIRECTORY data_pump_dir1 as '/u03/oradata/datapump';

GRANT READ, WRITE ON DIRECTORY data_pump_dir1 TO foot;

The user running the Data Pump Operation must specify the directory in one of two ways:

By specifying the DIRECTORY parameter on the command line:

EXPDP foot/foot DIRECTORY=data_pump_dir1 dumpfile=users.dmp schemas=eul, kap, mal logfile=users.log

Or by specifying it in the dumpfile and logfile parameters:

EXPDP foot/foot dumpfile= data_pump_dir1:users.dmp schemas=eul, kap, mal logfile=data_pump_directory:users.log

Notice that in the above command I have chosen to write the logfile to a different directory than the Data Pump output file. If you specify the DIRECTORY parameter, the directory specifications embedded in the dumpfile and logfile parameters are optional. Directory specifications embedded in the file parameters override the directory specified in the DIRECTORY parameter.

Up Next
In the next blog, we'll learn how to run the Data Pump Export and Import Utilities.

The intent of this blog is to generate interest in the Data Pump utilities. In my next blog, I'll continue our discussion on Data Pump by providing you with a general overview of its architecture and finish the series by showing you how to use Data Pump. I'll provide a few examples and screen shots to enhance the learning process along the way.

Before we begin let's look at a very quick example of a Data Pump Export command line:

expdp foot/foot directory=export_dir dumpfile=expfull.dmp logfile=expfull.log full=y

"expdp" tells the operating system to run the Data Pump Export utility. The parameters that follow are used to control the job's execution. In our example, the account and password are both "foot", the file that will contain the output data is "expfull.dmp", the logfile is"expfull.log" , and a full database export will be performed.

Most of the parameters are simple, but a further explanation of the DIRECTORY parameter is in order. Because a Data Pump job is executed in the database, you must create a directory using a SQL statement on the database being accessed. If you don't pre-create the directory, Oracle provides a default directory called DATA_PUMP_DIR. That means that users are unable to fully qualify the output file and log file as they were able to do in the non Data Pump versions of Export and Import. You specify a fully qualified directory on the server when you create the new database directory object and then you must grant access to that directory. I'll show you examples of this in next week's blog.

The Oracle manual provides an entire listing of parameters that can be used as input to the Data Pump Export and Import utilities. I can choose to export a single table, an entire schema, parallelize the job's execution to name just a few of the options available. I can exclude objects, include only a set of objects and transfer data across the network directly from one database to another. Let's get started by looking at just a few of the benefits Data Pump provides.

Performance Benefits
Oracle documentation states that Data Pump's performance on data retrieval is 60% faster than Export and 15 to 20 times faster on data input than Import. I have run some simple tests to roughly compare the performance of the older versions of Export and Import to their Data Pump counterparts (also called Export and Import) and have found the claims to be pretty close. But like all claims "your performance may vary - as mine did." I will say that Data Pump is very fast compared to its ancestors. To keep the discussion from getting confusing, I'll refer to the 10G Versions of Export and Import as Data Pump Export and Data Pump Import.

Oracle documentation states that the following factors contribute to the performance improvements:

• Parallelism - Multiple worker processes are able to perform inter-table and inter-partition work in parallel. This allows administrators to load and unload database tables using parallel processing. Data Pump Import takes advantage of parallelism to build indexes and load package bodies.
  
  The most beneficial aspect of Data Pump's parallel processing feature is that it is dynamic. Administrators are able to attach (connect) to a Data Pump job that is currently executing and dynamically alter the number of parallel processes it uses. I'll show you how to dynamically activate parallelism in the demo at the end of this blog.
  
  
• DBAs that ran Export and Import remotely (not logged on to the target server) were required to transfer data to their local machines. Data pump Export reads and writes files directly to the server and does not require any data movement over the network to the client machine. Data Pump jobs are scheduled and processed within the database engine as opposed to being a separate client-side process.
  
  
• The format of the Data Pump Export output files is very similar to the format of the data stored in the Oracle tablespace datafiles. Sounds pretty logical to me. If you want to transfer data from a table to a flat file and back into another table, it makes sense to keep the format of the output datafile as close to the format of a database datafile as possible.
  

Attaching and Detaching
Because Data Pump runs in the database, you can detach the client from a Data Pump job and reattach it at any time to monitor and/or change execution parameters. The reattach can be performed as often as required during the job's lifecycle. If there are numerous jobs running, you attach to a specific job by specifying the job's name in the ATTACH command.

This feature allows administrators to start a Data Pump job, detach from it and attach to it from time to time to monitor its execution. Detaching from an active Data Pump job will be a little strange at first because you have to hit the CONTROL- C combination that we all use to try and break out of something that is stuck. First time I did it, I thought "I'll break out of this and the job will stop." Wrong, as you will see in the demo, you can attach and detach as often as you would like.

Job Status Information
Oracle provides the STATUS command to allow administrators to view the status of the job they are attached to. It provides virtually all of the information needed to monitor the job's execution. In addition you can use the STATUS="seconds" command to obtain a status at timed intervals.

You can also use DBA_DATAPUMP_JOBS to view general information on all active Data Pump jobs and DBA_DATAPUMP_SESSIONS to view information on user sessions that are attached to active Data Pump jobs. In addition, administrators are able to use the tried-and-true V$SESSION_LONGOPS to review how much work the Data Pump job has completed and how much more work it needs to perform.

Space Estimation
Administrators are able to estimate the size of a Data Pump's Export output file by using the ESTIMATE and ESTIMATE_ONLY parameters. I have heard comments from some administrators that the estimates can be time consuming. Although, I haven't performed this command on very large sets of data, it seemed to work pretty well for me. The ESTIMATE parameter has two options:

• ESTIMATE=STATISTICS - Estimates the size of the output file by using the data dictionary statistics associated with the objects being exported. The statistics will need to be current for this option to be accurate.
  
  
• ESTIMATE=BLOCKS - Estimates the size of the output file by multiplying the number of blocks each object being exported consumes times the database block size.

You would place the parameter above on the command line. But if you only want to estimate the size of the output file, you'll need to remember to use the "ESTIMATE_ONLY=y" parameter in conjunction with the ESTIMATE parameter. If you don't you will estimate the size of the output file and perform the export.

Interruptible and Restartable
Data Pump jobs update a master table with all pertinent information pertaining to the job being executed. The master table knows about the objects being exported or imported and their location in the dump file set. As a result, a Data Pump job can be restarted in the event of a planned or unplanned stoppage. The Data Pump job accesses the master table to determine where to start and picks up right where it left off. Administrators are able to use the STOP_JOB AND START_JOB commands on the client to temporarily stop jobs and restart them when desired.

Database to Database Transfers
Data Pump Import allows users to transfer data directly from a source database to a target database without using disk as a temporary staging (and stopping) point. You use the NETWORK_LINK command to specify a pre-created database link that points to the source database. The server connects to the source database specified in the database link, retrieves the data and writes it directly to the target database. No dump files required!

Recreates Schema Users and Grants in Schema Mode
As stated previously, users have the option of choosing to export an entire schema. Let's say you wanted to refresh an entire schema in a target database from an export file you have available. The easy way to create all of the tables from scratch is to drop the user with the cascade option, recreate the user, grant the user the privileges they used to have and use Import utility to recreate the schema objects. The problem with the older version of Import was that because it didn't recreate the user (just the user's objects), the administrator was forced to save creation and grant statements to recreate the user with the same privileges it had before the drop was performed. A Data Pump schema import will recreate the user and execute all of the associated grants, user password history, etc..

Fine-Grained Selectivity
Being able to narrow the scope of the work performed is critical to most of the administrative functions we perform. The tighter we can focus on the set of objects we are administering, the more efficient and safer our administrative activities will be. Data Pump provides several parameters that help administrators narrow the scope of objects being processed:

• Exclude - Notifies Data Pump to exclude the specified object type from the export or import operation. Multiple excludes can be used together to exclude more than one object type from the operation. For example, "EXCLUDE=function" would exclude all of the functions from a Data Pump operation. SQL-like syntax can be used to provide even greater selectivity.
  
  
• INCLUDE - Notifies Data Pump to include only those object types specified. All other object types will not be included in the export or import operation. Like its EXCLUDE counterpart, a SQL-like syntax can be used to provide greater selectivity.
  
  
• CONTENT - Notifies data pump to include only the metadata (CONTENT=METADATA_ONLY), only the data (CONTENT=DATA_ONLY) or both metadata and data (CONTENT=ALL).
  
  
• QUERY - Provides users with the ability to use SQL statements to limit the rows processed by the Data Pump operation. A limited version of this feature was available in the older releases of Export and Import. The QUERY feature has been improved by allowing the statement to be qualified with a table name to apply the statement to a specific table. In addition the QUERY option can now be used in both Data Pump Export and Import operations.
  
  
• FLASHBACK - Allows Data Pump Export and Import to restrict data to a specific time in the past. Being able to go "back in time" and retrieve data seems to be an extremely valuable Data Pump feature to me. One example would be to view before images of data that was changed incorrectly. If an application program runs amok and changes data erroneously, the Data Pump utility could be used to go back in time and populate data in a different schema to allow the development team to identify the differences at their leisure.
  

Transformations
Older versions of Export and Import did not provide users with the ability to change an object's storage characteristics. For example, there wasn't a real simple way to export a set of object from one tablespace and import them into another tablespace. There were a few tricks that could be performed to make this happen but they certainly weren't as easy or as straightforward as Data Pump's transformation features.

Data Pump allows users to easily change the target tablespace that the objects will be created in (REMAP_TABLESPACE), migrate objects from one schema to another (REMPAP_SCHEMA) and move databases across platforms that have different file system symantics (REMAP_DATAFILE).

The TRANSFORM parameter can also be used to alter object creation DDL for specific objects. For example, this synax:

TRANSFORM=SEGMENT_ATTRIBUTES:n:table

Could be used to change this object creation statement:

CREATE TABLE foot.dept
(emp_id NUMBER(9),
first_name VARCHAR2(40),
last_name VARCHAR2(50))
PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255 NOCOMPRESS LOGGING
STORAGE(INITIAL 10240 NEXT 10240 MINEXTENTS 1 MAXEXTENTS 500
PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1 BUFFER_POOL DEFAULT)
TABLESPACE hr_data1 ;

To this one when running a Data Pump Import operation:

CREATE TABLE foot.dept
( emp_id NUMBER(9),
first_name VARCHAR2(40),
last_name VARCHAR2(50));

The transformation feature provides numerous variations to allow administrators to tailor each object's storage characteristics to meet their specific needs.

Generating DDL Files
Data Pump's Import utility can be used to create an output file containing DDL that is ready for execution. The DDL statements are not executed during the Data Pump operation so the target system will not be changed.

Target Table Exists Operations
DBAs have been using Export and Import to populate existing tables with data since the Export and Import utilities have been available. I was using Oracle when the original Export and Import utilities first became available. I used to be proud of this, now it makes me feel like I'm ready to retire. I have used Export and Import dozens of times to load existing data tables. These new existing object features are certainly helpful.

Oracle has improved upon this functionality by providing administrators with four different options when loading preexisting tables:

• TABLE_EXISTS_ACTION=SKIP - Existing tables are bypassed if they already exist. No action is taken.
  
  
• TABLE_EXISTS_ACTION=APPEND -Data Pump Import will append rows to the existing table.
  
  
• TABLE_EXISTS_ACTION=TRUNCATE - Data Pump Import will truncate the existing table and load rows from the source.
  
  
• TABLE_EXISTS_ACTION=REPLACE - Existing tables will be dropped, recreated and then loaded from the source.
  

Demo!
Let's take a quick look at the attached file that shows a few of the new features that Data Pump provides. I've added some comments along the way.

Wrapup
As I stated previously, the intent of this blog was not to be a primer on Data Pump administration and execution. Rather, it was intended to provide you with a laundry list of interesting and beneficial features to pique your curiosity. Hopefully this curiosity will spur your to learn more about these utilities (and read my upcoming blogs!).

The two types of Oracle checkpoints are full and incremental. Full checkpoints will occur when the administrator manually issues a checkpoint command or the database is shutdown using the normal, transactional or immediate options. A full checkpoint ensures that all of the dirty blocks contained in the buffer cache are written to the data files. The database is thought of as being in a synchronized state at this time. Although a normal, or incremental, checkpoint does write blocks that have been dirty the longest time to disk, it does not ensure that all of the dirty blocks contained in the cache are written to the data files. Oracle technicians coined the phrase "lazy checkpointer" to describe incremental checkpoints.

One (out of several) reasons Oracle can fire an incremental checkpoint is when it switches from one log group to another. When Oracle fills a log group, a checkpoint is fired and Oracle then begins writing to the next redo log group. This continues in a circular fashion; when all log groups are filled, Oracle reuses the first group.

A common rule of thumb is to adjust the redo log file's size so that Oracle performs a log switch every 15 to 30 minutes. Log switches that occur more frequently may have a negative impact on performance. Log switches that occur several times a minute have a definite impact on database performance.

Using the Redo Log File Size Advisor in Oracle 10G
If you have the FAST_START_MTTR target parameter set for your database, you will be able to use 10G Grid Controls' redo log file advisor to help you determine the initial size of your redo log files.

We start by activating 10G Grid Control or 10G Database Control and navigating to the Administration Home page. Since I'm using an early version of Database Control in this exercise, those of you using a later version of Grid or Database Control may see something a little different. When I click on the Redo Log Group navigation link, Database Control responds by displaying the Redo Log Groups Main Administration page. Notice that I have selected "Sizing Advice" from the drop down menu.

When I do that, you'll notice that Database Control responds by displaying an error message telling me that setting FAST_START_MTTR target is required to be set before the tool will offer advice on the proper sizing of the online redo logs. If you would like to learn more about FAST_START_MTTR_TARGET, please refer to my blog titled "A Few More of Oracle 10G's Automatic Features."

This next screenshot shows sizing advice from 10G Grid Control for a database that does have FAST_START_MTTR_TARGET set. 10G Grid Control is advising me that the optimal size for the online redo log group files is 104 megabytes.

You don't have to use the 10G Database/Grid Control SGTs (Sissy GUI Tools) to provide this advice. If you have FAST_START_MTTR target set, the sizing advice will be contained in the OPTIMAL_LOGFILE_SIZE column in the dynamic performance table V$INSTANCE_RECOVERY.

If you look at the Oracle documentation, every section that discusses the Redo Log File Size Advisor provides the same advice "It is recommended that the user configure all online redo logs to be at least this value." I think the key words are "at least this value." Personally, I use the recommended value as a starting point and then double-check the advisor's advice by performing a manual sizing review of the redo log files.

Manually Determining the Optimal Redo Log File Size
Checking messages in the alert log is another easy way to determine how fast Oracle is filling and switching logs. Here's an example of a production database's alert log that shows redo log switches occurring too frequently. Notice that the timestamps show that the logs are switching every minute or so.

If the following messages are found, you can be sure that performance is definitely being affected:

Thread 1 advanced to log sequence 248
Current log# 2 seq# 248 mem# 0: /orant/oradata/logs/redolog2a.log
Thread 1 cannot allocate new log, sequence 249
Checkpoint not complete

The "checkpoint not complete" messages are generated because the logs are switching so fast that the checkpoint associated with the log switch isn't complete. During that time, Oracle's LGWR process has filled up the other redo log groups and is now waiting for the first checkpoint to successfully execute. It can't overwrite the contents of the redo log until the information covered by that log is flushed to the data files. Oracle will stop processing until the checkpoint completes successfully.

Performance can be dramatically improved by increasing the log sizes so that logs switches occur at the recommended interval of 15 to 30 minutes. Identify the current size of the redo log members from V$LOG, record the number of log switches per hour and increase the size of the log to allow Oracle to switch at the recommended rate of one switch per 15 to 30 minutes.

For example, if the database log size is 1 megabyte and you are switching logs every 1 minute, you will need to increase the log size to 30 megabytes in size to allow it to switch every 30 minutes.

The problem is that many applications have workloads that vary dramatically throughout each 24-hour time-period. In addition, application-processing workloads may vary according to the days of the calendar month. Month-end and mid-month processing may increase the number of changes occurring in the database. This higher level of activity may cause the redo logs to be filled much more quickly than during off-peak times. The DBA must make sure that the on-line redo logs don't switch too often during periods of high activity and switch often enough during times of low processing workloads. In addition, administrators must also be concerned about how the size of the online redo logs affect instance recovery. The less checkpoints taken, the longer instance recovery will take.

The LOG_CHECKPOINT_TIMEOUT, LOG_CHECKPOINT_INTERVAL and FAST_START_MTTR_TARGET parameters can help to decrease the time of instance recoveries. For more information, please refer to my blog titled "A Few More of Oracle 10G's Automatic Features."

Data Loss
It is important to note that these parameters will not reduce the amount of data lost in certain database recovery scenarios. For example, let's say we lose a single database datafile. We restore the datafile from the backup and begin the recovery process. Since we will be recovering the datafile up to the current point-in-time, we'll need all of our archived and online redo logs that were active since the backup was taken. We'll use all of the information contained in the archived and online logs to "redo" all of the changes that took place. The only data that will be lost will be from transactions that were in-flight (active) when the failure occurred.

If a lost or damaged archived or online redo log is used as input to our database recovery, the amount of data lost will be directly proportional to the amount of time that redo log or logs were active. We will be forced to restore all of the database datafiles from the backup and recover the database up to the bad or missing archived or online redo log. All data after that point will not be recovered.

In case of a bad archive log, the farther back in time the bad archive log corresponds to, the more data that will be lost. Since we now know that the size of the redo logs determine how long the log will be active, the larger the log, the more data loss there will be if we use a bad or missing log as input to our recovery.

That's why administrators multiplex both their online and archived redo logs. They ensure that their online redo log groups have multiple members on different disk drives and send their archives off to more than one destination.

I have seen administrators size their online redo logs to accommodate their highest periods of activity and create PL/SQL programs to ensure that the logs switch every 15 to 30 minutes during times when activity is low. The intelligence in the programs vary from just waking up and doing a log switch to checking to see when the last log switch occurred before executing the log switch command.

I have also seen the initialization parameter ARCHIVE_LAG_TARGET be used to force log switches at regular intervals. Although the parameter is intended to control the amount of data that is lost in a standby database environment during a catastrophic failure scenario, some savvy administrators set their online redo logs to accommodate peak activity and set ARCHIVE_LAG_TARGET to ensure the logs switch at regular intervals during lower activity time periods.

Whatever method you use, one of the keys to good database performance is to ensure that you aren't switching logs at too fast of a rate. Use the sizing advice provided above to help you on you way.

Thanks for reading (and happy sizing),

Chris Foot

10G Grid Control automates the entire process. Administrators are able to quickly and easily capture or generate workloads, use them as input to the SQL Access Advisor and view the recommendations, along with their associated benefit ranking. The output also contains links to the DDL statements that administrators can use to implement the recommendations.

Like most of the advisors in Oracle, you can also activate the SQL Access Advisor by executing the PL/SQL package DBMS_ADVISOR. The Oracle10G Database Performance Tuning Guide dedicates an entire chapter to the SQL Access Advisor. After comparing the steps necessary to run the advisor using the command line interface to the quick and easy GUI interface provided by 10G Grid Control, I think I'll make every effort to use the 10G Grid Control tool to activate the SQL Access Advisor. Stating that the command line interface is more labor intensive than its Grid Control counterpart is like stating that the Titanic "sprung a small leak."

But the command line interface does provide users with the ability to tune a single SQL statement. The DBMS_ADVISOR.QUICK_TUNE procedure allows administrators to quickly and easily create a task for a workload that consists of one statement. The SQL Access Advisor will perform the same analysis and generate the same recommendations as it does with larger workloads. The Quicktune utility accepts a user-defined task name that identifies the task and a SQL statement to be analyzed. Here's an example of how to run Quicktune on a single SQL statement:

EXECUTE DBMS_ADVISOR.QUICK_TUNE(DBMS_ADVISOR.SQLACCESS_ADVISOR, 'my_first_quicktune', 'SELECT * from scott.dept where dept_id=140');

In addition, the command line method allows you to import Oracle Database 9i Summary Advisor workloads using the IMPORT_SQLWLD_SUMADB procedure.

Advice from your friendly ex-Oracle Instructor
Please remember that the quality of the output produced by the SQL Access Advisor will depend upon the quality of input. Make sure that the tables and indexes being analyzed have up-to-date statistics. You do this by generating statistics using the Oracle supplied package DBMS_STATS.

In addition, we know that an index has the possibility of influencing all SQL statements that access the table that the index is built upon. An index that provides fast access for one query may negatively impact other statements. Analyzing larger workloads will produce index and materialized view recommendations that take all of the queries contained in the representative workload into consideration. The larger the workload you use as input to the SQL Access Advisor, the more high-quality the recommendations will be.

It is important to note that the SQL Access advisor does generate some overhead during analysis. You need to balance the size of the workload with the impact the analysis process will have on database performance. My personal preference is to schedule my larger workload analysis during off-peak hours.

SQL Access Advisor Input
Before we continue, we need to understand what a task and a workload is. They are used together to generate SQL Access Advisor recommendations and can be created using either Grid Control or the command line interface.

Tasks
A task is an object that contains the setup and execution options of a particular SQL Access Advisor job. It allows administrators to tailor an evaluation to meet their specific requirements and can be used over and over again. A task can be created using Grid Control or by command line by executing the DBMS_ADVISOR.CREATE_TASK procedure. Command line users would then use the DBMS_ADVISOR.SET_TASK_PARAMETER to set attributes that control the workload collection, tuning analysis and final reporting.

These attributes provide a myriad of customization options to the administrator. You are able to establish default owner names, tablespaces and naming conventions for object creation recommendations.

The attributes also allow administrators to use workload filters to restrict the tuning analysis to a set of SQL statements that meet a given criteria. Administrators are able to filter on the module name, SQL statement syntax, table and schema names or the account executing the statement. In addition, administrators have the option of limiting the analysis to the top "n" resource consumers based on buffer gets, disk reads, executions, etc.. There are multiple filter options available that can be combined to narrow the scope of the analysis and recommendations. For a complete list of task attribute settings, please refer to the DBMS_ADVISOR package description contained in the 10G Oracle Database PL/SQL Packages and Types Reference Documentation Manual.

Workloads
Workloads consist of SQL statements and statistics and attributes that describe those statements. Workloads are classified as either full (representative of the entire application) or partial (representative of a portion of the application's entire workload). One important difference between full and partial workloads is the SQL Access Advisor will not generate drop recommendations for partial workloads.

If you read my previous blog on the SQL Access Advisor, you'll know that the advisor is able to accept several different types of workloads as input. The input workload can consist of a single SQL statement, SQL statements currently in the SQL Cache, a user defined set of SQL statements contained in a workload table or a Grid Control generated SQL Tuning Set. The SQL Access Advisor is also able to generate a hypothetical workload for a specified schema.

Templates
Once you fine tune your task and workload, you can create a template to simplify future operations. Both tasks and workloads can be used to create templates. The template can be created using either Grid Control or command line.

Analysis
The SQL Access Advisor performs an in-depth analysis on each statement contained in the workload that is not eliminated by the workload filtering options. The utility reviews every clause contained in the statement for efficiency. If it identifies that a statement accesses a table using a particular column, it will review the statistics on the accessed table to determine if a new index is warranted. Conversely, if it finds an index that is not used, it may generate a recommendation to drop the index in question. The SQL Access Advisor will also review the index types. If it finds that an index should be a B-Tree, as opposed to a bitmap, it will make a recommendation to change the type of the index.

The SQL Access Advisor will also evaluate the workload for queries that would benefit from materialized views. In releases previous to Oracle8i, administrators used summary tables to increase SQL performance for data warehousing applications. A summary table stores the pre-computed results of data calculations, such as sums, counts, averages and data table joins. It is more efficient to access a pre-computed result instead of computing the desired result from detail data every time a query is executed. In Oracle8i and later releases, administrators are able to use materialized views to provide the same benefits as summary tables.

SQL Access Advisor Output
The output the advisor produces depends upon the extent of analysis performed. Administrators are able to choose from:

• Comprehensive - The SQL Access advisor reviews materialized views, materialized view logs and indexes. The advisor will assume that the SQL being analyzed is representative of the application's general workload that is being processed during the course of normal day-to-day operations.
• Limited - The SQL Access Advisor will assume that the workload consists of poorly performing SQL statements. The advisor will provide advice for improving the performance of that portion of the total workload.

The types of recommendations provided depend upon the extent of analysis performed:

For those running the SQL Access Advisor via command line, Oracle provides a series of data dictionary views that allow administrators to view the output generated from the SQL Tuning Advisor. The view that contains the actual recommendations is DBA_ADVISOR_RECOMMENDATIONS. Oracle also provides the GET_TASK_SCRIPT procedure that produces the output in a easy to read, graphical display. The display contains links to individual SQL statements which allows administrators to quickly see which SQL statements benefit from a recommendation.

Grid Control users are able to view the Summary Output page to view recommendations that are sorted by the benefit they provide. Clicking on the number on the far left side of the tabular output drills down into the specific recommendation.

Grid control's recommendation details page displays links for the specific objects that will be created if the recommendation is implemented. The bottom of the page displays the SQL statements that will be improved by the recommendation. Notice that the output DDL uses the table schema as default if none is specified. The page also displays the space that will be consumed if the object is created. Finally, users can hit the OK button to begin creating the recommended objects or save the DDL for execution at a later time. When you click on the materialized view link, 10G Enterprise Manager displays the DDL for the recommended object.

Summary
I think you'll find that the SQL Access Advisor provides administrators with a powerful performance analysis tool. The intent of this blog was to pique your curiosity on the tool and the benefits it provides. The Oracle documentation contains a wealth of knowledge to continue your education.

Oracle has integrated a new feature into the optimizer that allows it to provide automatic SQL tuning. The optimizer has been enhanced to perform in-depth tuning of SQL statements that are fed to it by the SQL Tuning Advisor. For more information on how to run the SQL Tuning Advisor, please refer to my blog titled "The SQL Tuning Advisor."

In that blog, I stated that it is the optimizer's job is to create the most high performance access path in the shortest time possible. It's easy to understand how these two objectives can be viewed as "mutually exclusive." In a perfect world, you would give the optimizer as much time as it needs to create the most optimal access path. In the real world, you need to get the statements executing as quickly as possible. That includes optimization.

Since the SQL Tuning Advisor does not have time constraints, the tool can take as much time as necessary to optimize the statement. This allows the optimizer to generate high-quality recommendations. The premise is the more time it spends performing the analysis, the higher the quality of recommendations it will provide. The SQL Tuning Advisor Home Page provides a drop down menu that allows the administrator to "throttle" the amount of time the utility spends optimizing the statement.

In tuning mode, the optimizer is able to perform a higher level of analysis than in normal mode where it is required to generate access paths as quickly as possible. The output the optimizer produces in normal mode is an access path. The output the optimizer produces in tuning mode is a series of recommendations. Those recommendations include information justifying why it generated those recommendations. This prevents the advisor from becoming a "black hole" that pumps out advice without supporting information. The recommendations may include collecting statistics on objects, new index creation, restructuring the SQL statement or the creation of a SQL Profile to create a more optimal access path. When the optimizer is called by the SQL Tuning Advisor, Oracle refers to the optimizer as the Automatic Tuning Optimizer or ATO.

The Four Types of Analysis Performed by the ATO
When the SQL Tuning Advisor activates the ATO, the optimizer performs four types of in-depth analysis:

Statistics Analysis
Craig Mullins, well known DB2 Guru, provides an excellent description of a database optimizer in The DB2 Developer's Guide "the optimizer is equivalent to an expert system. An expert system is a standard set of rules when combined with situational data can return an expert opinion."

The cost based optimizer uses statistics generated by the DBMS_STATS procedure as the situational data when creating its expert opinion on which access path to the data is most optimal. These statistics are stored in the data dictionary and describe the data object's space characteristics, data uniqueness and data distribution.

The cost-based optimizer is only as good as the statistics it uses as input. Statistics collections should be run on a regular basis to ensure that the statistics are current (representative of the data being accessed). The optimizer is then able to create a highly accurate access path that is based on the least cost. If statistics are not available, the optimizer uses an algorithm to calculate the statistics, which often leads to "less than optimal" access paths.

The ATO checks all of the objects accessed by the SQL statement evaluated for missing or stale statistics. The ATO will make recommendations to generate statistics for objects that have statistics that are missing or stale. In addition, the ATO will collect auxiliary information to compensate for substandard statistics.

SQL Profile Generation Analysis
Before we begin discussing how the ATO creates a SQL Profile, a quick comparison between Stored Outlines and SQL Profiles is in order. A common misconception is that a SQL Profile is like a Stored Outline.

A Stored Outline is a set of optimizer hints that are attached to a SQL statement that freezes its access path. When the Stored Outline is enabled, Oracle uses the stored hints to generate the access path. Administrators are able to manually change the access path of a given statement by using the Outline Management Editor in Oracle Enterprise Manager.

Another method to change the access paths is to manually update the hints that are attached to the SQL statement. Since the hints are stored in a table, administrators are able to use SQL DML statements to alter them. I just find the SGT (Sissy GUI Tool) method to be so much easier that I use Oracle Enterprise Manager religiously to change a Stored Outline's access path. For more information on Stored Outlines, please refer to my blog appropriately titled "Optimizer Plan Stability in Oracle9i". Although the title says Oracle9i, the information is still pertinent in 10G.

A SQL Profile is a collection of information gathered by the ATO that is specific to a particular SQL statement. The information is stored in the data dictionary and is used by the optimizer to make more informed, and more intelligent, access path decisions for that statement.

As stated previously, we expect the optimizer to generate access paths as quickly as possible. As a result, the optimizer often makes estimates on cardinality and selectivity to generate the access path. During the SQL Profile analysis phase, the ATO will verify predicate selectivity, cardinality and optimizer settings. The ATO accomplishes this by:

• Dynamically sampling the data being accessed and applying the predicates used in the SQL to the sampled data. If the ATO determines that the difference between its original estimates and the statistics generated by the dynamic sampling is large enough to warrant a correction, it will change its original estimates to more accurately reflect the data being accessed.
  
  
• Decomposing the SQL statement into fragments and executing the fragments to gather access path and performance related information.
  
  
• Evaluating the SQL statement's past execution history. One of the outcomes of this evaluation could be a change in the optimizer mode (first rows vs all rows) for the statement being analyzed

If you take a look at this screenshot from my blog on the SQL Tuning Advisor, you will see the ATO recommending that I accept a SQL Profile.If I accept the tool's recommendation, the optimizer will use the information contained in SQL Profile in addition to the normal database statistics to generate an optimal execution plan.

Unlike its Stored Outline counterpart, a SQL Profile does not freeze a statement's access path. As the data in the objects being accessed changes as a result of normal database operations, the information contained in the SQL Profile may become outdated. This could result in less than optimal access paths being generated. This problem can be corrected by running the SQL Tuning Analyzer again to regenerate fresh information or removing the SQL Profile altogether.

Access Path Analysis
When the SQL Tuning advisor activates the ATO to perform detailed SQL analysis, it also provides advice on indexes. When the SQL Tuning Advisor generates an index recommendation, the output usually includes a recommendation to run the SQL Access Advisor.

We know that an index has the possibility of influencing all SQL statements that access the table that the index is built upon. An index that provides fast access for one query may negatively impact other statements. The scope of analysis the ATO performs does not include a global review of how the new index will affect the entire SQL workload. As a result, the SQL Tuning Analyzer will often recommend that the SQL Access Advisor be run on a representative SQL workload along with the query that is being analyzed. The result is a much more global set of index recommendations that takes all of the queries contained in the representative workload into consideration.

SQL Structure Analysis
The ATO also examines the structure of the statement being analyzed to determine if it can be coded differently to obtain a more optimal access path. During analysis, the ATO compares the structure of the SQL statement against a set of coding best practices. The ATO does not rewrite the query, but will provide a recommendation for an alternative SQL statement structure that it thinks would generate a more optimal access path. The ATO attempts to identify design mistakes such as Cartesian products, predicates that don't match the datatype of indexed columns, UNION vs UNION All etc.

Summary
The SQL Tuning Advisor, SQL Access Advisor and the ATO combine to make a powerful tuning tool that database administrators can use to ensure optimal SQL execution performance. The future of Oracle database tuning will be administrators interpreting and implementing the recommendations generated by the advisors and ADDM. I absolutely and firmly believe that the advisors and ADDM will become more intelligent in their recommendations. As their intelligence increases, the usage of statistics dumps to attempt to improve database performance will decrease.

Thanks for reading,
Chris Foot

10G Automatic Statistics Generation - Discusses how the 10G database automatically generates performance statistics and stores them in a repository for analysis.

Automatic Tuning Recommendations - I create a not-so mythical performance problem and use the output from ADDM and the advisors to provide the diagnostic information and recommendations I need to solve the issue.

Automatic Storage Management - An overview on Oracle's disk management system.

Automatic Shared Memory Management - A detailed discussion on 10G's Automatic Shared Memory management feature and memory advisors.

10G OEM Grid Control's Automatic Alert Notification System - Information on how to create automatic notifications that allow database administrators to identify (and correct) performance and availability issues before they negatively impact the database environment.

The Changing Role of the DBA - My opinion on the notion that all of these automated features will be reducing the need for experienced DBAs blogs.

Automatic Retention Tuning
Most of us thought that automatic undo would be the solution to all of our before image management problems. Certainly, before image administration has been simplified by reducing the number of tuning knobs to one (UNDO_RETENTION). The UNDO_RETENTION parameter specifies the length of time that undo data for committed transactions will be kept available to provide read consistency. But sizing the UNDO_TABLESPACE to hold the amount of before images generated during the time period specified in the UNDO_RETENTION parameter can still be a challenge.

The goal we are trying to accomplish is to set the UNDO_RETENTION parameter and size the undo tablespace to ensure that we don't run out of space in the undo tablespace and our queries don't receive the dreaded ORA-01555 "Snapshot too old" error messages.

ORA-01555 errors occur when a transaction is unable to successfully read a before image of data to maintain read consistency. Read consistency ensures that all data returned by a query comes from the same point-in-time (query start time). If a block of data changes after the query has started, Oracle will read the before, or undo, image of the data to maintain a read consistent view.

To guarantee read consistency, Oracle returns the ORA-01555 error if it cannot retrieve the before image of the changed data block. This occurs because the undo block has been flushed out of the system to allow more current before images to be stored. The less undo space you have and the lower your UNDO_RETENTION period is, the more often this problem will occur.

The trick is to balance the size of the undo tablespace with the amount of undo retention you need to satisfy your long-running queries. But this is easier said than done. Database applications, by their nature, are very dynamic. It is hard to predict how much undo will be generated for a particular time period. Sizing the undo tablespace and setting the UNDO_RETENTION time period is certainly simpler for small databases that don't have a lot of activity. But the problem becomes much more complex on systems that have a high level of update activity and long-running queries.

Oracle10G automatically tunes the amount of undo data kept available. It is important to note that the automatic tuning process will take place even if the UNDO_RETENTION parameter is set. The database will use the value contained in the UNDO_RETENTION parameter as input to the tuning process. Oracle10G records the times of the longest-running queries and the amount of undo generated to automatically tune the UNDO_RETENTION parameter.

The value of the UNDO_RETENTION parameter also depends on the undo datafile's AUTOEXTEND setting:

• Undo datafile is autoextensible - The undo retention period is set to a time that is slightly longer than the longest-running query. The UNDO_RETENTION parameter must be set when using the autoextend option.
  
  
• Undo datafile is fixed - The undo retention period is set to obtain the maximum amount of undo retention allowed by the amount of free space available in the undo tablespace datafile. Oracle will ignore the UNDO_RETENTION initialization parameter setting unless retention guarantee is enabled. If retention guarantee is enabled, Oracle will not let the undo retention time period fall below the value specified in the UNDO_RETENTION parameter. If undo space becomes constrained, Oracle will let the transactions fail before it will flush before images out that are within the undo retention time period.

Checkpoints, Performance and Instance Recovery
Oracle's checkpoint mechanism ensures that data contained in the buffer cache is safely written to disk on a regular basis. During a checkpoint, the DBWR background process flushes dirty blocks contained in the buffer cache to the database datafiles and the CKPT background process updates the datafile headers with the checkpoint information.

The point-in-time identified by the last successful flush of the buffer cache is called the Checkpoint Position. Oracle knows that all of the data past the point in time identified by the Checkpoint Position does not need to be replayed during instance recovery. That data has already been flushed to the datafiles. If the database crashes and an instance recovery is required, the instance recovery process will begin at the Checkpoint Position.

If you would like to learn more about how instance recovery is performed, please refer to an article I wrote on the database synchronization process. Scroll through the article until you see the title 'Oracle Internals'.

There are a handful of parameters that manually control when a checkpoint is performed in Oracle10G:

• LOG_CHECKPOINT_TIMEOUT - Specified in seconds. This parameter ensures that the Checkpoint Position does not lag beyond the last write to the redo log (sometimes called the tail) by the value specified. For example, a LOG_CHECKPOINT_TIMEOUT value of 300 will tell Oracle to keep the Checkpoint Position no more than 300 seconds behind the last write to the redo log. Setting this parameter to 0 disables time-based checkpoints.
  
  
• LOG_CHECKPOINT_INTERVAL - Specified in system blocks, not Oracle blocks. It ensures that the Checkpoint Position does not lag beyond the last write to the redo log by the specified value. For example, a LOG_CHECKPOINT_VALUE of 500 will tell Oracle to keep the Checkpoint Position no more than 500 blocks behind the last write to the redo log. Setting this parameter to 0 disables block-based checkpoints.
  
  
• FAST_START_MTTR_TARGET - Specifies the amount of time, in seconds, that Oracle will take to perform instance recovery. The database will adjust the frequency of checkpoints to ensure that instance recovery will take no longer than the value specified. This parameter is the more optimal method of defining instance recovery time because it sets the actual amount of time the instance recovery will take. Oracle currently recommends that you use this parameter, as opposed to the previous two parameters to define the amount of time instance recovery takes.
  

A checkpoint is a relatively expensive process. Checkpoints that are taken too frequently have a tendency to "monopolize finite database resources" (one of my favorite terms). The key to optimal database performance is to balance fast instance recovery with optimal database performance.

10G Automatic Checkpoint Tuning
If you do not set FAST_START_MTTR_TARGET, or set it to a very large value, Oracle10g will provide automatic checkpoint tuning. The database will write out dirty blocks from the cache as fast as possible without negatively impacting database performance. The DBA is no longer required to set any of the aforementioned checkpoint parameters.

Automatic DB_FILE_MULTIBLOCK_READ_COUNT Tuning
The Oracle database improves the performance of tablescans by increasing the number of blocks read in a single database I/O operation. If your SQL statement is going to read all of the rows in a table, it makes sense to return as many blocks as you can in a single read. In releases prior to Oracle10G R2, administrators used the DB_FILE_MULTIBLOCK_READ_COUNT initialization parameter to tell Oracle how many block to retrieve in the single I/O operation.

But setting the DB_FILE_MULTIBLOCK_READ_COUNT parameter too high can affect access path selection. Full table scans use multi-block reads, so the cost of a full table scan depends on the number of multi-block reads required to read the entire table. The more blocks retrieved in a single multi-block I/O execution, the more favorable a tablescan looks to the optimizer.

In releases prior to Oracle10G R2, the permitted values for DB_FILE_MULTIBLOCK_READ_COUNT were platform-dependent. The most common settings ranged from 4 to 64 blocks per single multi-block I/O execution.

In Oracle 10G R2, Oracle defaults the DB_FILE_MULTIBLOCK_READ_COUNT to the maximum number of blocks that can be effectively read. Although this value is also platform-dependent, Oracle documentation states that it is 1 MB for most platforms. This 1 MB size allows much more data to be read in a single operation in 10GR2 than previous releases. In addition, this larger value does not make tablescans look more favorable to the optimizer. As a result, it is now recommended to not set the DB_FILE_MULTI_BLOCK_READ_COUNT parameter and let the database determine the number of blocks read in multi-block I/O operations.

Summary
As you can see, Oracle10G automates many of the features we used to have to wrestle with on a daily basis. The automated features reduce the amount of time we spend performing mundane administrative actvities and allows us to spend more time on higher ROI activities.

Thanks for reading,
Chris Foot

But data generated by sort operations is by far the most common topic of discussion. Administrators new to Oracle quickly learn that efficient management of sort data is critical to achieving and maintaining good query performance.

Sort data is generated explicitly when the user executes a statement that contains an ORDER BY or a GROUP by and implicitly generated by the database for joins, bitmap merges, index creation operations, etc..

Sorting in Memory
The key to fast sort operations is to sort as much data as possible in memory. As I used to tell my students in my Oracle Admin I classes, "memory fast (both hands raised high)- disk slow (both hands pointing to the floor)." Silly, but it worked. Depending on the release, Oracle provides a different set of parameters to allocate memory for sort operations.

In releases previous to Oracle9i, DBAs were required to manually set the SORT_AREA_SIZE, HASH_AREA_SIZE, BITMAMP_MERGE_SIZE initialization parameters to control the amount of memory allocated to sort operations. The SORT_AREA_SIZE parameter allocated memory specifically for sort operations generated by user SQL statements.

It is important to note that SORT_AREA_SIZE defined the amount of memory allocated to sort operations for each individual user. It was not shared amongst all users. User processes requiring sort work areas did not immediately consume all of the memory allocated to them by the SORT_AREA_SIZE parameter. They allocated chunks of memory until they reached the total amount of memory allocated to them by SORT_AREA_SIZE. If the user's sort work areas consumed all of the memory allocated to them by SORT_AREA_SIZE, they began writing sort data to the temporary tablespace that was assigned to them.

We know that each platform has a finite amount of system resources (disk, memory and CPU) available. The problem with defining memory areas at a per user level was that the total amount memory allocated to sort operations at any one time was very dynamic by nature. The more users that required sort work areas, the more total memory Oracle consumed. That's the one thing I like about Oracle, it'll consume every system resource it can if you let it. That's what makes tuning Oracle exciting, I think.

Back to the discussion… A large SORT_AREA_SIZE parameter setting that worked wonderfully when only a few users were sorting often impacted the database server's performance when dozens of users were sorting. As a result, DBAs were often very conservative when setting the SORT_AREA_SIZE parameter. Better to let users sort to disk than lock up an entire box.

In Oracle9i, Oracle introduced two new parameters called PGA_AGGREGATE_TARGET and WORKAREA_SIZE_POLICY to automate and improve the process of allocating memory for sort work operations.

PGA_AGGREGATE_TARGET defines a global pool of memory that all users performing sort operations share. Both PGA_AGGREGATE_TARGET and WORK_AREA_SIZE_PARAMETER must both be set to activate automatic management of sort work areas.

WORK_AREA_SIZE_POLICY can be set to 'MANUAL' or 'AUTO'. Setting it to 'MANUAL' disables the automatic adjustment of sort areas while 'AUTO' enables the automatic adjustment of sort work areas in memory. The DBA then must set the various parameters that allocate sort areas manually.

When the WORK_AREA_SIZE_PARAMETER is set to 'AUTO', Oracle will automatically allocate and adjust memory used for sorting to ensure that each user's sort operations are fully optimized without exceeding the value defined by PGA_AGGREGATE_TARGET.

The two key words in that last sentence were "automatically adjust". Oracle will adjust the amount of memory allocated to each, individual user based on the amount of total memory available in the global pool defined by PGA_AGGREGATE_TARGET. The more memory that is available, the more memory the individual user gets.

Oracle's Metalink website has numerous articles on the monitoring and administration of both manual and automatic sort work areas. If you are interested in furthering your education on how to ensure that your database environment is configured optimally for sorting, start with the Oracle documentation for your release and then move on to Metalink.

Sorting to Disk
When your user populations exhaust the amount of memory available for sorting, Oracle will begin to write sort data to disk. Oracle introduced temporary tablespaces in 7.3, but true temporary tablespaces (created using the CREATE TABLESPACE….TEMPFILE command) were introduced in Oracle8i.

The TEMPORARY keyword in the CREATE TABLESPACE command tells Oracle to create a tablespace that manages segments differently than those stored in permanent tablespaces. When you create an object in a permanent tablespace, it allocates a segment and extents within that segment. A temporary tablespace contains a single segment that all users share. Although multiple transactions share the same sort segment in a temporary tablespace, they do not share extents. The first sort operation allocates the sort segment. Future sort operations do not incur the additional overhead required by segment allocation, which improves performance.

The temporary tablespace should be created as locally managed and with a uniform size extent allocation. You'll see an example of the syntax in the demo contained at the end of this blog.

The first time I created a true temporary tablespace that used the TEMPFILE parameter, I immediately went to V$DATAFILE and DBA_DATA_FILES to view the information the data dictionary provided on them. I said to myself, "Hey, this database has a bug already - where's the information on my datafile?" I should have read the new features manual first, I think. It took me a while to figure out that tempfile information is recorded in v$TEMPFILE and DBA_TEMP_FILES. If you are looking for information on tempfiles, those are the views that you'll need to access.

If you are looking for information on sort operations, Oracle provides the V$SORT_SEGMENT and V$TEMPSEG_USAGE (Oracle 9.2 and above) views to provide that information. You'll need to remember that if your sorts are contained entirely in memory, you won't see any entries in these views. The sorts must overflow to disk. You can also run this statement to see how much data is being sorted in memory vs disk:

SELECT NAME, VALUE
FROM V$SYSSTAT
WHERE NAME IN ('sorts (memory)', 'sorts (disk)');

You assign temporary tablespaces to accounts during their creation. You can also use the ALTER USER DDL statement to change an account's temporary tablespace. One of the problems in releases prior to Oracle9i was that if administrators did not explicitly assign a temporary tablespace during account creation, Oracle assigned the System tablespace as the user's temporary tablespace. This oversight allowed user processes to create sort work areas in the System tablespace, which impacted database performance and had a tendency to fragment the tablespace.

In Oracle9i, administrators could create a default temporary tablespace at the database level. Accounts that were not explicitly assigned a temporary tablespace during creation were assigned the default temporary tablespace defined at the database level.

Temporary Tablespace Groups
Oracle10G allows administrators to create temporary tablespace groups, which can consist of one, or more, temporary tablespaces. There is no upper limit on the number of temporary tablespaces that can be a member of a tablespace group.

Tablespace groups allow administrators to provide a large pool of disk storage to users that sort large volumes of data. With data warehouses continuing to increase in size, it is easily understandable why this amount of disk space is often required.

It is interesting to note that there is no CREATE TEMPORARY TABLESPACE GROUP DDL statement. The group is implicitly created when the first tablespace is assigned to it. Administrators have the option of assigning a temporary tablespace to a group during tablespace creation or by altering the tablespace at a later time. The group name cannot match an existing group or tablespace name.

Oracle provides the DBA_TABLESPACE_GROUPS view to provide information on tablespace groups. In addition, temporary tablespace groups can also be assigned as the database's default temporary tablespace.

A temporary tablespace can be a member of a group, which is assigned to users and also be assigned as a single temporary tablespace to other users.

The benefits that tablespace groups provide are:

• A single user that is assigned to a tablespace group is able to use multiple tablespaces in different sessions.
• Child sessions in a single parallel operation are able to use multiple tablespaces.
• Enables multiple tablespaces to be defined as the default temporary tablespace at the database level. Oracle will assign users to the tablespaces in the group in a round-robin fashion.

DEMO
Let's finish the blog with a brief demo.

I hope you enjoyed this blog on temporary tablespaces and temporary tablespace groups. Thanks for reading,

Chris Foot

One of the common mistakes that is often made during a tuning exercise is not gathering enough high level information during the initial stages of the problem determination process. As I stated in my introduction, the key to success is to begin the analysis at a high level and then narrowing the scope of information collection until the problem is identified.

Once the problem is identified, we tune and tweak, record the same statistics we used to identify the problem and compare the before and after information. This iteration of recording/changing/recording/comparing is done again (and again) until the problem is corrected.

Almost every component in Oracle that has an affect on database performance is measured by the database system. One of the most common measurements is the amount of time the user processes and system spend performing a particular operation.

V$SESS_TIME_MODEL and V$SYS_TIME_MODEL Statistics Views
V$SESS_TIME_MODEL and V$SYS_TIME_MODEL views provide information on exactly that - where the user processes are spending their time. All of the timings are recorded in microseconds. One of the key indicators recorded in the time model views is DB Time, which is the total amount of time spent performing database operations.

The time model views differ from each other in that the V$SESS_TIME_MODEL view stores timing information for individual sessions while the V$SYS_TIME_MODEL view provides information at instance level. As a result, you won't find the column for SESSION_ID in the V$SYS_TIME_MODEL view. In addition, V$SYS_TIME_MODEL records information historically from instance startup, so don't be concerned if you add up all of the time spent by the current indiviudal sessions and it doesn't match the DBTIME value in V$SYS_TIME_MODEL view. One last thing, use the timings as a relative reference, they may not add up exactly because of the way they are recorded by Oracle.

Let's take a look at the contents of these views:

V$SESS_TIME_MODEL and V$SYS_TIME_MODEL Contents

• SID (NUMBER) - Session Identififer. This is the same value that you will find in all views that record information about individual sessions. This column is useful if you want to join V$SESS_TIME_MODEL to V$SESSION to retrieve additional information on the session being evaluated. As stated previously, since V$SYS_TIME_MODEL records information at the instance level, you won't find this column in the view.
  
  
• STAT_ID (NUMBER) Statistic identifier for the time statistic.
  
  
• STAT_NAME (VARCHAR2 64) Name of the statistic being recorded. A listing of all of the statistics is provided below.
  
  
• VALUE (NUMBER) - Amount of time, in microseconds, the session has spent performing the operation identified in the STAT_NAME column.

Values for STAT_NAME:

Knowing the hierarchy of the the contents is important. I have included a hierarchy chart that will tell you what timings consist of sub-timings.

• DB Time - Amount of time spent performing operations in the database. The V$SYS_TIME_MODEL view does not record and add any time spent by the database background processes. As a result, the V$SYS_TIME_MODEL view records the total user workload, while the V$SESS_TIME_MODEL view records information about the workload produced by the individual session identified in the SID column. You compare this value against all of the other values contained in this table to determine where the bulk of the time is being spent.
  
  
• DB CPU - Amount of CPU time spent performing operations in the database. Like DB TIME, DB CPU only records user workload. If you see a relatively high value in this time look for complex calculations and poor SQL plans that perform a high level of buffer gets.
  
  
• Connection Management Call Elapsed Time - This value represents the amount of time processes spent performing CONNECT and DISCONNECT calls. This time should be much lower than most of the other values contained in this table. If it is high in V$SYS_TIME_MODEL, review program code to ensure that the application isn't attempting to make a connection and disconnection for each interaction with the database. This is a very common problem for applications built using a middle tier application server.
  
  
• Sequence Load Elapsed Time - Amount of time spent obtaining the next sequence number from Oracle's Data Dictionary. If the sequence is cached, the time is not recorded. A sequence is a user created object that generates numbers according to a specific pattern. It is most often used to generate unique identifiers to identify a specific object.
  
  
• SQL Execute Elapsed Time - This is a very important measurement. It records the amount of time that SQL statements are executing. This will also record the amount of time SELECT statements spend fetching the query results. Typically, user processes spend the bulk of their time accessing data. If it is extremely high, look for poorly performing SQL statements. If the user process is having performance problems and this indicator does not make up the majority of overall DB Time, check for poorly written programs, parsing problems, poor connection management, network issues.
  
  
• Repeated Bind Elapsed Time - Elapsed time spent on re-binding.
  
  
• Parse Elapsed Time - This records the amount of time the process spent hard parsing and soft parsing SQL statements. Oracle parses each statement before it is executed. The parse process includes syntax checking (making sure you spelled "WHERE" right), ensuring the objects being accessed are actually in the database, security checking, execution plan creation and loading the parsed representations into the shared pool.
  
  
• Hard Parse Elapsed Time - The amount of time spent hard parsing a SQL statement before execution. Before a statement enters the parse phase, Oracle matches the statement being executed to statements that are already parsed and stored in the shared pool. When Oracle finds a matching statement in the shared pool, it will do a soft parse on the SQL statement. If Oracle does not find any matching SQL in the shared pool, it will perform a hard parse, which requires that more steps be performed than its soft parse counterpart. As with almost everything else, fewer steps = faster performance.
  
  The key to obtaining a high number of soft parses is to match as many incoming statements to statements already stored in the shared pool as possible. This is trickier than it sounds and requires vigilant and rigorous SQL coding standards. When Oracle looks for a matching SQL statement in the shared pool, the statements must match exactly.
  
  If you see a high percentage of hard parses being performed, the most common culprit is SQL statements that don't use bind variables. A bind variable holds values that are passed from application programs. It allows the programs to change the search values in the WHERE clauses without having to perform a hard parse. Application programs that don't take advantage of bind variables continues to be a common problem.
  
  
• Hard Parse (Sharing Critieria) Elapsed Time - This value represents the amount of elapsed time the database was forced to hard parse a SQL statement because it was unable to find an existing cursor in the SQL Cache. It is a subset of the Hard Parse Elapsed time. If this value is high, look for programs that don't use bind variables.
  
  
• Hard Pase (Bind Mismatch) Elapsed Time - Signifies the amount of elapsed time spent performing hard parses because the bind variable's type or bind size did not match existing cursors in the cache. Oracle documentation also states that bind type mismatches often causes indexes not to be used. If this value is high, look for poor programing standards.
  
  
• Failed Parse Elapsed Time - Records the time spent attempting, and ultimately failing, to parse a statement before execution. Check the program code to ensure that the statements are syntactially correct and review Metalink notes for issues.
  
  
• Failed Parse (Out of Shared Memory) - Pretty self explanatory. The amount of elapsed time recorded when a parse failed because of a lack of adequate resources allocated to the shared pool. If this time is high, you need to add more memory to the shared pool, identify if bind variables are being used, etc.
  
  
• PL/SQL Execution Elapsed Time - Amount of elapsed time spent running the PL/SQL interpreter. It does not include the time executing and parsing SQL statements or the amount of time the process spent recursively executing the Java VM.
  
  
• PL/SQL Compilation Elapsed Time - Elapsed time spent running the PL/SQL compiler. It is the compiler's job to transform PL/SQL source code into machine-readable code (m-code).
  
  
• Inbound PL/SQL RPC Elapsed Time - Records the elapsed time PL/SQL remote procedure calls spent executing including executing SQL and JAVA.
  
  
• Java Execution Elapsed Time - Amount of time spent running the JAVA machine. This time does not include the time spent executing and parsing SQL statements or recursively executing PL/SQL.

Quick Demo
If you look at this attached text file, you'll see a series of statements that I executed against the time model views to analyze a performance problem.

I hope you enjoyed reading this article on Oracle's time model views.
]]> http://www.dbazine.com/blogs/blog-cf/chrisfoot/blogentry.2006-03-29.0050872616 Readers of this blog know how important I think 10G Grid Control is to the future of Oracle database administration. DBAs must learn to leverage the intelligence built into the Grid Control product set to reduce the amount of time they spend administering, monitoring and troubleshooting Oracle databases. <BR><BR>That being said, there are times when you don't have access to a full blown, enterprise-wide Grid Control environment. Not to worry, Oracle also provides the 10G Database Control administration toolset. 10G Database Control can be used to effectively administer a single Oracle database. Although 10G Database Control might not have the enterprise-wide scope that Grid Control does, it does provide a very robust set of features and functions that are guaranteed to make database administrators more productive. This blog will provide you with a brief overview of the 10G Database Control product set. We'll also compare and contrast Database Control to its big brother, Grid Control. Mon, 03 Apr 2006 05:18:59 -0500 Oracle Grid Control
Those of you that have been reading this blog are aware that Grid Control is architected as a multi-tier architecture consisting of the HTML console, a management service with an integrated information repository and management agents running on all monitored targets.

The management service receives monitoring data from the various agents and loads it into the management repository. The management console retrieves data from the management repository, organizes it and then displays it as information to the administrator via the HTML console interface.

The agents are programs that run continuously on all servers that are controlled by the Enterprise Manager architecture. Examples of the more popular targets found on the servers are databases, application servers and listeners.

The number of components that Grid Control manages and monitors continues to grow as the product matures. 10G Grid Control is now able to monitor and/or administer listeners, databases, RAC environments, Application Servers, Collaboration Suite implementations, and hardware servers. In addition, Oracle also provides plug-ins that allow Grid Control to monitor non-Oracle databases, non-Oracle middleware, network devices and storage systems. As Grid Control matures, its area of influence can only continue to expand.

Oracle Database Control
The environment that Database Control manages is much more limited in scope than its more robust counterpart, Grid Control. But it does provide a full set of administration and monitoring tools that allow administrators to quickly and effectively monitor and administer a single Oracle database environment (host server, listener and database). When I began to review Database Control for the first time, I was surprised at how robust the tool actually was.

Database control's architecture can be loosely compared to Grid Control's. It consists of the following components:

• A web-based console that provides a GUI interface.
• A local version of the management service that works with the local database.
• A local version of the management repository. The repository is installed in the local database. Unlike Grid Control's repository, which can be used to manage many databases, the repository for Database Control is used to contain configuration and historical information for one, single database environment.
• A local version of the management agent that sends monitoring data to the local management service.

Notice that the key word is local? Database Control provides DBAs with a web based toolset that allows them to administer and monitor a single database environment.

Oracle Database Control Installation
If you intend to register the database with Grid Control, installing Database Control is really not necessary. But if you do not have a Grid Control environment available, then I highly recommend that you install and use Database Control. Database Control can be installed during the installation of the database binaries or can be done separately if so desired. One of the installation panels will ask you if you prefer to administer the database with Grid control or Database Control. If the Grid Control agent is not installed on the server, the Grid Control option will be unavailable and you will be forced to use Database Control for database management.

After the installation is complete, a Database Control log in panel will pop up in a browser window that allows users to immediately log in to the local database being managed. Unlike Grid Control's installation, which requires user input during the installation, installing Database Control requires virtually no input from the end-user. It installs everything (web interface, management repository, management service and management agent) that is needed on the database server.

Automatic installation of the local agent also differs from Grid Control which requires that separate agent installations be performed on all managed targets. If you use DBCA to create your database, one of the panels that will appear during the configuration asks you if you want to use Grid Control or Database Control to administer the new database.

10g Database Control Demo
Let's begin our review of Database Control by activating the web interface. Activating Database Control will differ according to the operating system you are running it on. For the purposes of this demo, I installed a database and Database Control on a Windows environment. The icon for the Database Control toolset was found in the standard Oracle10G program grouping that can be found when you click on START - ALL PROGRAMS on the desktop.

A standard browser window will pop up that prompts you for your account and password. You will need to enter a valid database account and password to continue. Once you enter the account and password, Database Control will respond by displaying the Database Control Administration Home page. You'll note that this page looks pretty close to 10G Grid Control's Database Home page.

The Database Control Home page begins with a brief overview of system performance. If you click on the host name, in this case CFOOTE, 10G will respond by displaying the Host Environment Home page. The Host Environment Home page allows administrators to view performance and configuration information for the host server.

At the bottom of the Database Control Home page, you see the same alert and job activity information that is displayed on Grid Control's Database Home page. Below the alert and job information, 10G Database Control provides links to various utilities and informational panels. These are the same utilities that are provided by Grid Control. You can establish metrics for the database, run advisors, view ADDM data, etc.. You can also establish notification policies for alerts and metric violations, blackouts, etc.

If we go to the top of the Database Control Home page and click on the performance tab, Database Control responds by displaying the Database Performance Home Page. If you would like to learn what features this panel provides, please refer to my blog titled Database Performance Monitoring using 10G Enterprise Manager. 10G Database Control and 10G Grid Control are almost identical in navigation and information that is displayed.

It is important to note that you have a robust AWR repository available to you. The AWR repository is the center of the Common Manageability Infrastructure and provides services to collect, store, maintain, process and provide access to statistics for self-tuning and problem detection. In addition, Database Control also provides Oracle10g's Automatic Database Diagnostic Monitor (ADDM) which automatically identifies performance bottlenecks and makes recommendations to resolve them. Lastly, you have access to all of the intelligent advisors (SQL Tuning Advisor, SQL Access Advisor, Segment Advisor, etc.) that will allow you to quickly identify and resolve database performance problems.

If we click on the Administration tab that is displayed at the top of the Database Control Home page, Database Control responds by displaying the Database Administration Home page. The Database Administration Home page provides links to virtually any administrative task you can perform in an Oracle database.

The last home page is the Database Maintenance Home Page. Once again, Database Control provides a tab that links to the Database Maintenance Home Page at the top of each Database Control panel. Like its big brother, Grid Control, there is a complete set of maintenance activities available to you.

Summary
If you don't have access to a full blown Grid Control support infrastructure, using 10G Database Control is certainly an excellent alternative. Although it doesn't allow you to view your entire environment from a single web interface, it will provide you with all of the functionality you need to monitor, administer and troubleshoot your 10G database environment.

If you are using Database Control, you'll find that most of the Grid Control topics I have been blogging about will apply to Database Control.

Thanks for reading.
Chris Foot

RAC stands for Real Application Clusters. The optimal RAC environment uses an interconnected set of servers with each server containing one to four CPUs. The servers are connected to a shared disk system using network-attached storage (NAS) or a storage area network (SAN) technologies as the connectivity mechanism. High-speed network connections between the hardware servers themselves and from the hardware servers to the shared disk system allow end-users and administrators to view the environment as single application architecture.

Real Application Cluster enhancements in Oracle10G allow hardware servers to be seamlessly added to an application. Oracle10G enhances Oracle Grid Control functionality to manage RAC environments with a single interface. Oracle's Grid Control is a web-enabled toolset that allows administrators to group hardware platforms, databases, and application server installations and manage them as a single entity. Administrators are able to call other utilities (Data Pump, Transportable Tablespace, Oracle's new job scheduler) from within Grid Control to modify, monitor and tune databases contained in the RAC environment.

Grid Control also simplifies RAC administration by automating the installation, configuration and cloning of Application Server 10G and Database 10G implementations across multiple nodes. Grid Control monitoring views the entire grid as a single unit and provides drill down capabilities to identify problems with individual components.

Question 1
How long have you been working with RAC?

I have been working with RAC environments for about two years now. My last employer was building a very large OLTP application on 9iRAC running on Sun Solaris servers. But, we were only using part of the functionality that RAC provides. We set up multiple instances and used our application server tier to do all of the load balancing and network hardware for failover.

At Remote DBA Experts, building and administering RAC environments is one of the most popular services that we offer. I currently provide primary remote database administration services for several clients that have RAC implementations. These implementations include Oracle 9i, 10.1 and 10.2 releases running on Windows and LINUX. The job is challenging and requires a lot of knowledge in many different areas of technology.

Question 2
RAC seems to be receiving a lot of good trade press lately. What is your personal opinion of RAC?

It really does provide a lot of benefits. The primary ones are availability and scalability. I'd say RAC, and the Cache Fusion technology that improves its performance, has been a huge success for Oracle. This is especially true with the 10.x releases. With RAC technology and commodity server hardware both rapidly evolving and maturing, RAC is becoming not only a viable alternative for smaller enterprises but also a preferred method of achieving their availability requirements.

Question 3
What is Cache Fusion?
Cache Fusion is essentially a memory-to-memory transfer of data between the nodes in the RAC environment. Before Cache Fusion, a node was required to write some of the data to disk before it could be transferred to the next node in the cluster. Cache Fusion does a straight memory-to-memory transfer. In addition, each node's SGA has a map of what data is contained in the other node's data caches.

The performance improvement is phenomenal. Oracle leverages the vendor's high speed interconnects between the nodes to achieve the cache-to-cache data transfers. Before Cache Fusion, when you added a node to the cluster to increase performance of the application, it didn't always provide you with the performance improvement that you hoped for. With Cache Fusion, you can easily cost justify the addition of another node into a RAC cluster to increase the performance of the application running on it. Oracle sales pitches describe it as 'near linear horizontal scalability'.

Question 4
What are the three greatest benefits that RAC provides?

The three main benefits are availability, scalability, and the ability to use low cost commodity hardware. As I stated earlier, RAC is quickly finding its way into enterprises of all sizes to meet their availability and scalability requirements. RAC provides fault tolerance at a level that was out of reach for many organizations just a few years ago.

Prior to RAC, many organizations would run standbys to minimize outages in case of a hardware or operating system failure. The problem with standby database technology is that you will most likely lose transactional data during a failover. In addition, the failover itself can be time-consuming. RAC overcomes both of those problems.

In a RAC environment, if a node in the cluster fails, the application continues to run on the surviving nodes contained in the cluster. If your application is configured correctly, most users won't even know that the node they were running on became unavailable.

The scalability of a RAC environment goes hand in hand with the ability to utilize low cost commodity hardware. The x86/x86_64 platforms have achieved phenomenal price/performance ratios over the last couple of years. The problem was that they couldn't scale like their big iron proprietary UNIX server competitors.

RAC allows an application to scale vertically, by adding CPU, disk and memory resources to an individual server. But RAC also provides horizontal scalability, which is achieved by adding new nodes into the cluster. RAC also allows an organization to bring these resources online as they are needed. This can save a small or midsize organization a lot of money in the early stages of a project.

The key to achieving this level of scalability is Oracle's Cache Fusion technology, which we just talked about Now that Oracle is able to leverage the hardware vendor's high speed interconnect technology, the cache-to-cache data transfer provides near linear horizontal scalability. If your application needs more horsepower, you add another low cost node into the RAC environment.

Question 5
What areas of RAC do you think need to be improved?

Although Oracle has made great progress in both of these areas over the last few years, I'd say installations and tools. I'd like to see an installation, including Oracle Clusterware, ASM, and the database be performed in an hour or two. I would also like to see the installer be more intelligent.

Oracle is doing a very good job on RAC administration tools. They are maturing pretty quickly. But like all technicians, I am impatient. I want the most robust administration tools I can as quickly as possible. Oracle is certainly making headway.

The new 10.2 Grid Control tool has a robust reporting mechanism, which I think you covered in one of your blogs. Grid Control now provides administrators with a good view into ASM. In addition, the Oracle Clusterware, OCFS2.0, has just been included in the latest LINUX kernel.

I'd like to see Grid Control provide a more robust cluster-wide performance dashboard and an interface for creating and managing services. I think that would help a lot with getting to a true Grid computing environment.

Question 6
What is Clusterware?

One of RAC's requirements is that clustering software (sometimes called clusterware) be used to connect the hardware platforms together. This underlying clustering software was purchased either from the hardware vendor or a third-party clustering software provider. RAC is installed on top of the cluster environment and works in conjunction with the underlying clustering software to allow the application programs to view the multiple instances as a single entity.

One of the problems using previous releases of RAC was identifying exactly whose software it was causing the problem in the first place. Was it the RAC software or was it the clustering software provided by the hardware or third-party vendor? Oracle10G solves this problem by providing its own clustering software called Portable Clusterware. Portable Clusterware can now be used in place of the hardware or third-party vendor's clustering software.

Question 7
How complex is RAC to administer?

RAC certainly bring its own set of challenges with it. You not only have multiple instances to administer but you may also have to administer multiple services. In addition, the DBA is increasingly being called upon to perform operating system and storage administration tasks for RAC environments.

The DBA needs to know more than just the Oracle database to be effective. I have the good fortune of being trained and previously employed as a LINUX and UNIX operating system administrator. That experience provides me with both hardware and operating system administration expertise. I found that knowledge to be invaluable. Operating system, disk storage and database administrators must work together to achieve a successful RAC implementation.

Oracle has been providing new and better tool sets for these tasks, from OCFS and ASM to Grid Control. It is very important for an organization considering a RAC implementation to perform an in-depth analysis to understand and plan for this added complexity early in the design and implementation process.

Question 8
Is Grid Control required for RAC administration?

Technically no, but it's difficult to be effective without it. It's hard enough to troubleshoot enterprise applications let alone RAC implementations that are far more complex. If you have an application that uses multiple instances, each running on a separate node, you can easily spend as much time jumping from one node to another as you do viewing and analyzing diagnostic and performance information.

The Grid Control software provides an overview of the complete cluster. In addition it provides administrators with the ability to drill down into each of the nodes and database instances in the cluster If Grid Control did nothing more than aggregate the information from multiple instances, it would be worth installing. It really provides a lot more features and functionality than reporting. Grid Control facilitates and simplifies RAC administration. You'll be wasting a lot of precious time if you use the command line interface.

Question 9
What is the most challenging aspect of RAC?

There are really two areas that make RAC a challenging environment to administer. First, there are a lot more moving parts. You need to understand the shared storage environment and the cluster layer to be able to troubleshoot them. You need to know where the cluster logs are, how to execute FSCK on an OCFS drive, etc..

Second, you need to realize that you have multiple instances to administer. In order to create an environment that provides the highest availability possible, you will have to implement Transparent Application Failover (TAF).

TAF automatically transfers application user connections to surviving nodes when their primary node, the node they were connected to, fails. Depending on how you configure TAF, users won't even know that their connection and work have been transferred to another server.

Question 10
How much operating system expertise is needed?

RAC is fundamentally just a clustering solution, albeit a highly specialized and tuned one. Even if you're lucky enough to have operating system and/or SAN administrators to help you, you'll still be required to have a basic understanding of all of the components. You must be able to understand what your operating system and disk storage administrators are talking about and what questions you need to ask during the installation and configuration process.

Question 11
What operating system do you prefer for RAC?

The most popular environment of our current clients is most definitely LINUX. We have several RHEL3 clients and have recently been adding RHEL4 to the mix. I definitely prefer LINUX and UNIX over Windows, and am happy with RHEL3-4, but I'm looking forward to getting a test RAC environment set up on Solaris 10 now that Oracle has indicated that Sun, once again, is one of their preferred hardware vendors.

Question 12
Can RAC be installed by an experienced DBA with no previous RAC experience?

Sure, but a traditional DBA will face a steep learning curve. It's important that the DBA have a fundamental understanding of all of the different components that comprise a clustered environment. Your success will also depend on the support staff you have available to you as well. If you're lucky enough to have a SAN and operating system administrator that understands clustering, it's a lot easier. If not, be prepared to do a lot of research up front to ensure that you understand all of the technologies involved and how they interface with each other.

It is certainly a lot easier if you have someone to help jumpstart your knowledge in this environment. That is why our RAC implementation and administration offerings are so popular. Our customers realize that RAC requires a unique knowledge set that combines database, disk storage and operating system administration expertise. We have people on staff that have extensive experience in all three of those disciplines. It makes the installation and administration of RAC environments easy for us. In addition, this expertise helps us when we are required to perform tuning and problem determination analysis, which is more complex in RAC environments.

Question 13
If you could provide advice to someone just getting started in RAC administration, what would it be?

The recommendations at the top of my list are to use the tools Oracle provides, use the Internet to learn as much as you can about RAC administration and don't be afraid to learn how the operating and disk storage systems work. If you don't have the experience or expertise, call in someone to help that does.

The Grid Control product is the best tool available for viewing your systems at the cluster level. You must install the product and learn how to use it. There are a few administration tasks that are better suited to using a command line interface but the Grid Control tool should be your first choice when administering and troubleshooting a RAC environment.

The Internet, both official Oracle sites and sites like this one, contain a wealth of information on RAC installation, configuration, and administration. There are detailed guides to help you with most of the tasks you will be required to perform. Be sure to read as many Oracle Best Practices documents as you can. These will help you to solve problems in the planning stages of your project. The earlier you find potential issues during the project the better off you will be. You certainly don't want to find out that you missed a critical configuration step after you implement a RAC environment in production. I will state that RAC can be unforgiving at times.

Finally, depending on the size and staffing of your organization, you must involve personnel that have operating system and disk storage administration expertise. The cluster and shared storage are the foundation of RAC, you need to understand and be comfortable with both of them.

Question 14
What do you think the future will be for RAC?

RAC has gained a solid foothold and will be around for many years to come. Obviously the individual pieces, both the cluster services and shared storage systems, will get more mature and become more efficient. I also look for a more intelligent Grid Control product that will have the ability to not only monitor resources but also intelligently reassign them within a large enterprise GRID containing many RAC databases.

RAC is the foundation for Grid, which Oracle views as the future of enterprise computing. Instead of connecting multiple nodes together to support one, individual application, you connect multiple nodes together to support many different applications. Whether Grid becomes a popular implementation is immaterial, as RAC continues to mature it can only gain in popularity.

What We Do
I am finally beginning to get settled in here at Contemporary Technologies' RemoteDBA Experts. CTi RemoteDBA Experts provides remote database administration services for Oracle, Microsoft and IBM databases to customers across the globe. We provide both complete and supplemental database outsourcing services.

One of our more popular offerings is database project work. This set of services includes upgrading databases and applications (database vendor and third-party), installing and configuring highly available environments for all three major database products (RAC, Failsafe, Data Guard, SQL Server Clustering, etc.), database tuning and architecture design. We also provide remote operating system support for Windows, Linux and all flavors of UNIX.

Broadening the Scope of this Blog
When you work for one company, the technologies that you work with are limited to what is implemented there. You can certainly create your own test environments to learn new technologies and database features but you can't really become an expert until you administer that environment on a daily basis.

Working for a remote database services provides me with access to many different database features and technologies. Our customer's environments range the spectrum, from data warehouses to OLTP environments that have thousands of active connections. The range of technologies and third-party applications utilized is also broad. Name a database feature, third-party product, operating system or technology and I would be surprised if we didn't support it.

Leveraging In-House Experts
Customers who "give" their environments to you to administer remotely are doing so because they feel your organization has the expertise to provide a superior level of support. That takes a lot of trust on the customer's part. When you promote yourselves as "the experts", you better have the talent to back it up. I'm pleased to say that my fellow technicians at CTi RemoteDBA Experts have that high level of expertise required to keep our customers happy.

My new company has recognized that no single technician can become an expert in everything, so they are have created silos of expertise. All technicians here have strong expertise in day-to-day administration, but we also have identified certain database features that require a high level of expertise to administer successfully.

We have identified these silos of expertise and assign administrators to them. We have personnel that focus on security, regulatory compliance (SARBOX), highly available architectures (RAC, Failsafe, Data Guard), Oracle applications support, advanced administration tactics, monitoring, database/operation system interoperability, etc.. When a person comes on board at RemoteDBA Experts, we determine what areas of database support they are most interested in. We understand that technicians will be the best at doing things they like. We then assign them to that silo.

Since I am certainly not an expert in all of facets of technology, I intend to leverage my fellow technician's expertise to improve the quality of this blog.

A Preview of What's Coming in the Near Future

10G RAC
One area that we specialize in is highly available architectures. We support numerous RAC environments for our customers. My next blog will be an introduction to one of our resident RAC experts, Scott Rupnik. Before we delve into the technical details of installing and administering RAC environments, I thought it would be interesting to interview Scott to obtain a high level perspective of RAC, the benefits it provides and how to install and administer it effectively.

Data Guard
Data Guard is another feature that is often used to improve the availability of Oracle database environments. Oracle's Data Guard is becoming a popular solution to the problem of providing highly available architectures at a reasonably low cost. Oracle Data Guard is a failover environment that uses a single system to run the user applications until a failure occurs. Then the backup system is engaged and takes over for the primary system. The primary system can then be repaired or replaced. I'm pretty well versed in setting up and administering Data Guard but we have technicians here that have much more expertise than I do.

Oracle Applications Support
One of our more popular offerings is Oracle Applications Support. This includes Oracle applications as well as Peoplesoft. Joseph Greene, Bob Kline and Sandi Starr are three of our senior-level experts. I'll start the series of blogs on supporting Oracle Applications with another high level interview and then hopefully "turn the keys" over to them for a couple of blogs

Converting Oracle Applications from UNIX to LINUX
One thing that surprised me when I came here was how popular this conversion is becoming. Once I began to talk to the folks here, I understood why. The costs of supporting an Oracle application on LINUX are so much lower than supporting the same application on UNIX that companies are able to justify the conversion costs. This is a very popular service for us. I intend to write a blog that provides the details on the lower support costs and the tricks of the trade we use to make the transition as seamless as possible.

Oracle on Windows, LINUX, UNIX
One of the great benefits of Oracle is that it runs on many different operating systems. If you have worked with Oracle for any length of time you know that the standard interfaces to the environment are not dependent on the operating system. You execute the same SQL statements to administer a database on LINUX as you do UNIX and Windows. But you must also have a firm understanding of the operating system the database runs on to administer it effectively. DBAs don't have to be full-blown operating system administrators but they must know how the operating system works. In addition, DBAs should be able to write operating system scripts and understand the operating system's command line interface.

Several of our administrators here are not only database administrators but ex-operating system support technicians. We have folks that administer LINUX, most flavors of UNIX and all versions of Windows. I'll provide a few blogs on database/operating system tips and tricks for some of the more popular operating systems we support.

Regulatory Compliance and Security
My last task at Giant Eagle was to work with the SARBOX auditors. Although Giant Eagle isn't a publicly traded company, they felt it was important to comply with SARBOX regulations. One of my fellow employees, Sandi Starr has extensive experience in SARBOX and helps our customers put the proper practices and procedures in place to ensure that they comply with the numerous regulations SARBOX requires.

Summary
It looks like I'll have a lot to blog about in the future. I've covered only a few of the technologies and database features that are now available to me. I intend to mix the content up to keep the blog fresh. I don't intend to stop blogging about Oracle 10G Grid Control. Although my next blog will be on RAC, the one after that will be on 10G R2's DBConsole. I'm using it quite heavily here and I'm impressed with its capabilities.

Thanks for reading.

I've been employed as the database unit manager and database architect at Giant Eagle for close to two years now. A great job for a great company. Although Giant Eagle has always had a good reputation locally, my fellow consultants teased me about going to a supermarket. When I came to Giant Eagle, I was amazed when I realized how high-tech the company was and how efficient the technicians were at deploying and utilizing new technologies.

I certainly enjoyed my career at Giant Eagle and made many great friends during my stay there. But I always missed the entrepreneurial attitude and freedom that a smaller company provides. Working for a small company requires that one become involved in a wide variety of activities. As a result, the job roles are less rigidly defined, which offers greater flexibility to those that work there.

In addition, I have always enjoyed consulting. I like being technically challenged on a daily basis. Consulting certainly offers that. You can be working on a RAC implementation one day and tuning SQL the next. Your customers hold you to a higher standard than they do their own internal employees. They are paying you for you expertise, so they expect you to be an expert. I also like that type of challenge. You are judged solely on your talents (both technical and soft skills).

After much deliberation, I've decided to return to the consulting field and my old company, Contemporary Technologies Inc. (CTi). CTi is a remote database services provider based in Pittsburgh, Pennsylvania.

CTi specializes in providing remote database administration services for DB2, Oracle, Microsoft SQL Server and Oracle Rdb. CTi is a well-rounded organization with multiple lines of business that will allow me to become involved in many different technologies and Oracle features. The company's services range the spectrum, from architecture design to database and application upgrades.

Having a senior-level position there will allow me to assist them in many different areas. I'm hoping to become involved in everything from day-to-day support and improving day-to-day operations to sales and marketing. The broader the scope, the more challenging the position will be. Exactly what I am looking for at this point in my career.

Since I've worked with CTI's technicians before; I know that the team is highly skilled. They will help me learn new skills that I will be able pass on to readers of this blog. I have long-established relationships with their DBAs, having worked with them at other companies in Pittsburgh. They are a group of no-nonsense, motivated IT professionals who want to do what is best for their customers. When they assume responsibility for a customer's database, they do just that. It becomes their database. You don't see that kind of attitude in many remote services outfits.

CTi is well aware of my penchant for writing. I now have the advantage of not only transferring my own knowledge to readers of this blog, but the knowledge of more than a dozen senior-level database experts. I will also have access to a much wider range of technologies and Oracle features. I intend to leverage both my fellow technician's expertise and my ability to work with new technologies and Oracle features to broaden the scope of this blog.

The result will be a higher quality blog that covers a wider range of topics.

Thanks for reading.

Chris

UPDATE foot.emp SET col1='123' where col1='abc'

We know that Oracle locks at the row level. The first session that executed the update statement will hold the row level lock until I execute a COMMIT or a ROLLBACK. The second session I created is unable to obtain the row level lock until the first session commits or rolls back its unit of work. We have successfully created our first deadlock.

Problem Analysis
Let's begin our problem determination by navigating to 10G R2's Hang Analysis screen. We accomplish this by selecting the 'Hang Analysis' link displayed on the bottom of 10G R2 Grid Control's Performance Administration Home page. 10G EM R2 responds by displaying the Hang Analysis Home page.

The Hang Analysis home page displays a graphical representation of the blocking and blocked session. The blocking session is shown at the top of the diagram, while the blocked session is shown at the bottom. This simple graphic representation may not seem very worthwhile to you now, but the benefits become readily apparent when you the contention problem becomes more complex. The more complex the locking problem, the greater benefit the Hang Analysis diagram provides. I'll show you some complex blocking diagrams later in this blog.

We can see on the diagram that the blocking session has an ID of 128. When we click on the block, 10G R2 Grid Control responds by displaying the Session Summary Page. The Session Summary Page provides basic information pertaining to the session. The panel also provides a 'View Session Details' link on the center of the panel.

Clicking on the link displays the Session Details page. This page contains a wealth of information on the session. Let's review some of the most important information provided on this page:

Server

• Status of the server process (in our case inactive).
• Serial number of the server process.
• USERID of the server process. I know I'm using SYS, but it is my own little toy database.
• OS Process ID.
• Logon time and duration.
• Connection Type (our connection is dedicated).

Contention

Since session 128 is the blocking process, there is no contention information to be displayed. We'll take a look at the blocked session contention information later in this blog.

Client Information

• OS user name - Name of the account that is running the Oracle client software (in our case SQLPLUS).
• Host - Name of the machine that is running the client software.

Wait Event

• Current Wait Event - What resource the session is currently waiting for. In our case, we are waiting from a response from the client. It is expecting me to either execute another SQL Statement or perform a COMMIT or ROLLBACK. Until I perform the COMMIT or ROLLBACK, session 128 will continue to block session 118.
• Wait Duration - How long the session has been waiting.

Application

• Current SQL - This column is blank because the SQL statement I executed has completed processing.
• Current SQL Command - This column is also blank because of the reason stated above.
• Previous SQL - If I click on the link, Oracle will display the SQL Details page for this statement. The page displays the SQL statement, performance statistics, access path information and also provides a link to run the SQL Tuning advisor.
• Program - The application program that executed the SQL statement (in our case SQLPLUS)

The Session Details page also provides a series of tabs across the top. Since we already found most of the information we need to determine what work session 128 is performing, let's click on the Blocking Tree tab at the top of the page. 10G R2 Grid Control responds by displaying the Blocking Tree panel. This panel provides information on the problem session 128 is creating. The panel displays session 128 at the top of the hierarchy stating that it is the blocking session and displays session 118 below it telling us that it is being blocked by session 128. The output panel provides links to view session 118's SQL statement and also displays wait information for both sessions.

Let's take a look at the wait information for session 118. It shows that it is experiencing row lock contention. Lastly, the Blocking Tree panel contains a 'Kill Session' navigation button that will allow us to kill one, or both, sessions.

We'll conclude our investigation of session 128 by reviewing wait history information. We accomplish this by clicking on the 'Wait Event History' tab at the top of the Session Details page. 10G R2 Grid Control responds by displaying the Wait Event History page. The Wait Event History page displays a running history of all the events that session 128 waited for during its execution life cycle.

Continuing our Analysis
Let's continue our problem determination by reviewing session 118, which we now know is being blocked by session 128. I start the analysis by navigating back to the Hang Analysis Home page and clicking on the green block titled '118' at the bottom of the panel. 10G R2 responds as it did previously by displaying session 118's Session Summary page. Nothing out of the ordinary here, so let's continue. I'll use the 'View Session Details' navigation link to ask 10G R2 Grid Control to display session 118's Session Detail page.

Session 118's Session Detail page contains much of the same information as session 128's did except for a few VERY important details. We see that session 118's Contention column shows that it is being blocked by session 128 and it displays the file that contains the row that both sessions are attempting to update.

The Wait column for session 118 also shows the resource that it is waiting for. In this case, it shows that it is experiencing row lock contention on a row contained in the FOOT. EMP table.

If I click on 'Current SQL' link on the top right hand side of the page, 10G R2 Grid Control will respond by displaying the SQL Details page for this statement. The page displays the SQL statement, performance statistics, access path information and also provides a link to run the SQL Tuning advisor.

We'll conclude our investigation of session 118 by reviewing wait history information. We accomplish this by clicking on the 'Wait Event History' tab at the top of the Session Detail page. 10G R2 Grid Control responds by displaying the Wait Event History page. The Wait Event History page displays a running history of all the events that session 118 waited for during its execution life cycle. Notice that the information displayed for session 118 is quite different than for session 128. The information shows that session 118 is experiencing row lock contention.

Solving the Problem
We can easily solve this problem by executing a COMMIT or ROLLBACK command in session 128 or killing the process which will automatically roll back the work it has performed. Problem Solved!

More Complex Problems
We have already reviewed a simple graphical lock diagram that showed one session blocking another. Let's try something more complex and see how 10G R2 Grid Control responds.

I created a third session and executed the same SQL that the previous two sessions did. Here is the new graphical lock diagram for the three sessions. Notice that we now have two blocked sessions.

Let's finish this exercise by increasing the complexity once again. I created another table called FOOT. DEPT that contains one row. I created a fourth session and executed the following two statements:

UPDATE foot.dept SET col1='123' where col1='abc'
Notice that this is updating my new table.

UPDATE foot.emp SET col1='123' where col1='abc'
Updating the same row that session 128 is holding the row lock on.

I created one more session and executed this statement:

UPDATE foot.dept SET col1='123' where col1='abc'
Notice that this is also updating my new table.

Let's take a look at 10G R2 Grid Control's blocking diagram. Our first session (session 128) is blocking our fourth session (session 149). But session 149 is also holding a lock on our new FOOT.DEPT table, which our fifth session (session 104) is waiting for.

We have created a situation where the fourth session is being blocked and is also blocking other sessions.

Summary
I think you'll agree that 10G R2 Grid Control's Hang Analysis feature will provide great benefits when debugging contention problems.

Thanks for reading!

Chris

Maintenance Windows
There are two critical components in 10G R2 Grid Control that affect statistics gathering. First, we need to make sure that Oracle's maintenance windows are configured optimally. In my last blog, we learned how to set 10G R2 Grid Control's maintenance windows to ensure that Oracle's routine maintenance tasks don't compete with application programs for finite system resources. You don't want to be forced to tell an application manager that his nightly batch job stream is running long and will be competing with his application's on-line transactions because your statistics gathering job made them run longer. Trust me, that happens.

Manual and Automatic Statistics Gathering Options
Creating an optimal statistics gathering environment also requires us to review the default statistics gathering options. These options will be used for Oracle's automatic statistics gathering jobs. In addition, these options will be used if you do not specifically set them in manually scheduled statistics gathering jobs. We'll learn how to schedule statistics gathering jobs manually in my next blog.

Navigating to the Options Page
Let's begin by activating 10G R2 Grid Control and navigating to the Manage Optimizer Statistics Home page. We accomplish that by selecting the 'Manage Optimizer Statistics' link displayed on the middle, left hand side of 10G R2 Grid Control's Database Administration Home page. 10G EM R2 responds by displaying the Manage Optimizer Statistics Home page.

On the right hand side of the panel, there will be a column name in blue titled 'Related Links'. Under that panel you'll see a navigation link titled 'Statistics Options'. When we click on the link, 10G R2 Grid Control responds by displaying the Gather Optimizer Statistics Default Options page.

Some General Advice
Before we begin reviewing the options on this page, let me provide you with some general advice. A lot of the options provide an Oracle recommended setting called "Auto". I would suggest that you leave the options set to "Auto" and only change them when you feel that a manual setting will produce a more desirable result. Remember that Oracle is building more and more intelligence into the 10G Grid Control tool set. The product is really becoming an expert system. But the default settings are intended to create the most desirable statistics gathering environment for the general database population. It could be that these generic settings may need to be adjusted to fit your environment's specific needs.

You monitor the efficiency of the automatic statistics jobs by looking at the statistics displayed on the Manage Optimizer Statistics Home page. The information on this panel tells us:

• If the database's automatic statistics gathering job is enabled
• When the statistics gathering job is scheduled to run next
• The total number of previous runs
• The date of the last run
• If the last run was successful
• Elapsed time
• The number of objects it analyzed

If you look closely at the number displayed in the Objects Analyzed display line (in this case 344), you'll notice that it is also a link. Clicking on the link will tell Oracle to display the Objects Analyzed report, which lists all of the objects that have been analyzed in the last run. The report also provides the reason why each object was analyzed (stale or missing statistics). You can use this report as well as the information provided by the LAST_ANALYZED column in DBA_TABLES and DBA_INDEXES to determine if Oracle's automatic statistics gathering job is analyzing your tables as often as you would like.

If the automatic statistics gathering job is running longer than you would like, or not gathering statistics as often as you would like, it may be beneficial to set some of the options manually.

Option Review
Let's begin our review at the top left hand side of the page. As I stated previously, I will provide you with general advice for each option. As we know, there are a myriad of configuration combinations available. If the database has been upgraded from a previous release, a common practice in the field is to set the options to match the manual statistics gathering jobs that were run previously. Administrators feel safe knowing that the automatic statistics gathering job's configuration closely resembles the old manual statistics gathering jobs that worked for them in the past.

Retention Period
The first setting we see is for the Retention Period. Retention period determines how many day's worth of optimizer statistics will be retained in the repository. In an upcoming blog, I'll show you how to restore a set of saved statistics. If your SQL statement's performance degrades due to access paths changes, you can restore the statistics to a day when you know the statements were running normally to determine if statistics changes were the cause.

Estimate Percentage
Seasoned DBAs know that statistics gathering jobs run on large tables can take a loooonnng time. It is a very common practice to reduce the amount of time the jobs take by reducing the amount of data they analyze. It is important to note, that in most cases, the higher the percentage of the object you analyze, the more intelligent the optimizer's decisions will be.

You can choose Auto if you would like Oracle to choose for you. Many DBAs choose 20% as the percentage of estimation. If the database consists of many small tables, you can set the percentage higher to ensure that the optimizer knows as much about the data as possible.

Recommendation - Choose Auto and monitor SQL statement performance. If the optimizer begins to make poor access path decisions, set the percentage manually and monitor again.

Granularity
The granularity determines what type of partitioned objects will be analyzed. The options are Auto, Global, Global and Partition, All, Partition and Sub-Partition. Do you want to analyze just the partitions and sub-partitions or do you want to analyze just global objects, like indexes, that access more than one partition?

Recommendation - Choose auto. Oracle will determine which partitioned objects have had enough data changes made to them to warrant their analysis.

Cascade
Cascade determines if just the table will be analyzed or the table and all of its indexes.

Recommendation - Choose Auto or True. In most cases, Oracle will choose to analyze the indexes along with the table when Auto is chosen. If you want to make sure that all of the table's indexes are analyzed, choose TRUE.

Histograms
This is the only option that I am a little hesitant to make a recommendation on. Histograms is a complex topic that I can't cover in a few paragraphs. Oracle's default setting is to build histograms. Histograms improve SQL statement plans that access skewed data. Before you begin deviating from Oracle's recommendation, you need to fully understand what histograms are. I recommend that you start your education by reading the Oracle documentation. If highly recommend that you also buy Jonathan Lewis's book titled Cost Based Oracle Fundamentals. I have learned more about histograms from that one book than I have from all other sources combined.

Recommendation - If you are experienced in Histograms and know you have skewed data, build the histograms. If you don't have skewed data, don't build them. If you don't have enough experience or education, build the Histograms, read the Oracle manuals, buy Jonathan's book and make an educated decision at a later time.

Degree of Parallelism
Determines how many parallel processes that will be generated. Oracle's statistics gathering jobs can create multiple child processes that analyze the data in parallel. The child processes access the data and report back to the parent process when complete. The more parallel processes that are generated, the faster the statistics job will run and the more system resources it will consume.

The administrator must balance statistics gathering job performance with the amount of resources the job consumes. DBAs use the parallel setting in conjunction with the Estimate Percentage option to maximize statistics gathering job performance.

The options are Auto (Oracle chooses), Default (based on initialization parameters), Table Default (parallel option set at the object level), Degree (number of parallel processes set manually).

Recommendation - Choose Auto once again. Many DBAs feel more comfortable setting the value manually (Degree option) because that is what it was set to before. We have manually set our Degree to 12 on most of our data warehouse databases. We have historical performance statistics that tell us that generating that number of parallel processes to gather statistics provides an optimal balance between the statistics gathering job's performance and the percentage of resources it consumes on the host platform.

Cursor Invalidation
A cursor is invalidated when one of the objects it depends on is changed. This causes the cursor to be reparsed. New statistics may change access paths and, as a result, the statements accessing those newly analyzed objects will need to be reparsed.

Recommendation - Choose Auto or Immediate. I don't have enough information yet to determine the quality of Oracle's Auto option. Most of our statistics gathering jobs run during the application's nightly maintenance windows, so the performance degradation that occurs as a result of cursor invalidation does not have a significant impact. We have chosen Immediate for most of these environments.

Target Object Class
The cost-based optimizer is only as good as the statistics it uses as input. Statistics collections should be run on a regular basis to ensure that the statistics are current (representative of the data being accessed). In Oracle9i, the GATHER AUTO option of the DBMS_STATS procedure could be used to help determine if statistics generation was required. If more than 10% of the rows changed in the table since the last analyze was performed, the DBMS_STATS procedure (with the GATHER_AUTO option activated) analyzed the table.

If you choose Auto, 10G R2 Grid Control uses the same methodology. It will identify the percentage of data that is changed for each object under analysis and determine if new statistics are warranted. Choosing All will gather statistics for all objects in the database and choosing Oracle will gather statistics for system component schemas only.

Recommendation - Choose Auto or All in most cases. If the application has numerous tables (our Peoplesoft application has tens of thousands of database objects), use the auto option to ensure that you only gather statistics on objects that have changed. Other applications that have a smaller number of objects or have the majority of objects changed by application programs and are performance sensitive, choose All.

Summary
I hope you enjoyed this blog. Please remember that my personal recommendations are just that - my personal opinion.

Thanks for reading.

• If the database's automatic statistics gathering job is enabled
• When the statistics gathering job is scheduled to run next
• The total number of previous runs
• The date of the last run
• If the last run was successful
• Elapsed time
• The number of objects it analyzed

If you look closely at the number displayed in the Objects Analyzed display line (in this case 344), you'll notice that it is also a link. Clicking on the link will tell Oracle to display the Objects Analyzed report which lists all of the objects that have been analyzed in the last run. The report also provides the reason why each object was analyzed (stale or missing statistics). You can use this report as well as the information provided by the LAST_ANALYZED column in DBA_TABLES and DBA_INDEXES to determine if Oracle's automatic statistics gathering job is analyzing your tables as often as you would like. We'll discuss this topic in-depth in the next blog.

The Optimizer Statistics Home page provides links that allow administrators to:

• Adjust automatic maintenance window start and duration times
• Edit the parameter defaults for optimizer statistics tasks. The defaults affect both automatic and manually scheduled statistics gathering jobs
• Schedule manual statistics gathering jobs
• Restore statistics to a previous time in the past. VERY helpful if you have access paths that have changed
• Lock and unlock optimizer statistics. To learn more about why you would want to lock statistics for a set of database objects, please refer to my blog titled Oracle 10G R2 New Features Part 3
• Delete optimizer statistics
• Review statistics gathering jobs already scheduled

Application Job Scheduling and Maintenance Windows
One of the challenges that DBAs face is that every application we work with is different. For many of us (include me in this group), the opportunity to work with many different applications is one of the reasons we became database administrators.

Each application has its own unique processing flows. As a result, you often need to alter a database's default maintenance window to ensure that Oracle's routine maintenance tasks don't compete with application programs for finite system resources. In other words, you don't want Oracle running the automatic Segment Advisor at the same time your heavy batch processing programs are running. The only way to prevent these conflicts from occurring is to adjust the times on the weekday and weekend maintenance windows and throttle the amount of resources they use.

We know that 10G will schedule Segment Advisor and statistics gathering jobs to run during the maintenance windows. I'm currently reviewing the Oracle documentation to determine what other routine maintenance tasks are run during the maintenance windows. Its important for us to know all of the routine maintenance tasks Oracle runs during those times.

Managing Maintenance Windows and Windows Groups
Oracle provides the 'Maintenance_Window_Group' navigation link on the Manage Optimizer Statistics Home page to allow administrators to configure the settings for the GATHER_STATS_JOB that is automatically run on objects with outdated or missing statistics.

When I click on the 'Maintenance_Window_Group' navigation link, 10G Grid Control responds by displaying the Edit Window Group: Maintenance_Window_Group panel. This panel allows administrators to:

• Enable and disable the Maintenance Window Group
• Edit individual Maintenance Windows
• Add Windows to the Maintenance Window Group
• Remove Windows from the Maintenance Window Group
• Apply this Maintenance Window Group to multiple databases

It's pretty easy to remove a Maintenance Window from Maintenance Window Group. But in order to add a new Window, you have to create it first. For example, our application may have different batch processing job streams for each day of the week. In that case, it may be beneficial for us to create a different Maintenance Window for each day. You also may want to have a separate Maintenance Window for Saturday.

The majority of applications I have worked with in the past usually had a pretty standard processing schedule for each day of the workweek. But as we discussed previously, each application is different and you may want to tailor the Maintenance_Window_Group to ensure that the database's routine maintenance tasks do not conflict with application programs.

In order to create a Maintenance Window, I need to go back to of 10G Grid Control R2's Database Administration Home page. On the upper, right hand side of the page, you'll see a column heading titled 'Oracle Scheduler'. In that column, there are two navigation links titled 'Windows' and 'Window Groups'. The links point to panels that allow us to administer Windows and Window Groups.

Before we continue, some clarification is in order. There are two ways to navigate to the Windows and Windows Group administration panels that will allow us to administer our Maintenance Windows and Maintenance Window Group. We can click on the the 'Maintenance_Window_Group' navigation link on the Manage Optimizer Statistics Home page as we learned how to do previously in this blog (about 3 paragraphs up) or I can go back to the 10G Grid Control R2's Database Administration Home page and click on the 'Windows' and 'Windows Groups' links.

But there is only one way to navigate to the panel that allows me to create a new Window that I can add to a Window Group. I have to go back to the 10G Grid Control R2's Database Administration Home page and select the 'Windows' navigation link. When I do this, 10G Grid Control R2 responds by displaying the Scheduler Windows page. This panel allows me to edit, alter and delete Windows. I'll create a new Window by selecting the 'Weeknight Window' radio button and then selecting the 'Create Like" navigation button. 10G Grid Control R2 responds by displaying the Create Window administration panel. I chose the following options for the new Window:

• The name of our Window will be Monday_Window
• The new Window will use the INTERNAL_PLAN resource plan. Resource plans allow administrators to throttle the amount of CPU resources used. To learn more about resource plans, please refer to Kimberly Floss's article titled Database Resource Manager. I didn't need to choose a resource plan to control CPU resources. We'll learn in an upcoming blog how we can control the level of CPU utilization our statistics gathering jobs consume by adjusting the number of parallel processes they create.
• I chose the default low priority. A high priority Window takes precedence over a low priority Window if the times of the two Windows overlap. If our new Window overlaps another Window that has a high priority, the high priority Window's configuration will take precedence.
• I configured the Monday_Window to be a weekly Window that repeats every week. The day for this Window to be active is every Monday starting at 12:00 midnight. The Window's duration is 8 hours.

When I select the 'OK' navigation button to continue, Oracle takes me back to the calling panel. The Scheduler Windows page shows the new window I have just created. As we learned previously, I can edit our Maintenance Window Group by navigating either to 10G Grid Control R2's Database Administration Home page and clicking on the 'Window Groups' navigation link or by navigating back to our Manage Optimizer Statistics Home page and selecting the 'Maintenance_Window_Group' navigation link.

Regardless of which way we choose, 10G Grid Control R2 will display the Scheduler Windows Group panel. I only have one Windows Group to choose from. It's the Maintenance_Window_Group that Oracle automatically created for us during database creation. I selected 'Edit' from the drop down menu to edit the Maintenance Window Group. 10G Grid Control R2 responds by displaying the Edit Window Group: Edit Maintenance Window Group panel. Since I want to add our newly created Monday_Window to the Maintenance_Window_Group, I continue by clicking on the 'Add/Remove' navigation button.

10G Grid Control R2 displays a listing of Windows that can be added to the Maintenance_Window_Group. Notice that I have deactivated the Weeknight Window and selected the Monday_Window. I can complete the process by clicking on the 'Select' navigation button.

If I want, I can add the remaining days by creating a specific Window for each day. Or I can choose to deactivate Monday from the Weeknight Window. In that case, all other weekdays will use the Weeknight Window's configuration. I'll have to remember to add the Weeknight Window back to the Maintenance_Window_Group if I choose to use that configuration.

Summary
It is important to note that this was just a quick example of how to configure Maintenance Windows. The intent of this blog was to show you that you have a myriad of configuration options available to you. You can choose to use the default Maintenance Windows and Maintenance Windows Group or you can create a totally unique set of Maintenance Windows for your database. As I stated, the options are endless. In my next blog, I'll show you how to configure the default options for the gathering statistics jobs.

So little is known about the inner-workings of the cost based optimizer that is often called the Oracle database's "little black box." The optimizer's job is to analyze the statement being executed and determine the most efficient access path to the data for that statement. Craig Mullins provides an excellent description of a database optimizer in The DB2 Developer's Guide "the optimizer is equivalent to an expert system. An expert system is a standard set of rules when combined with situational data can return an expert opinion."

The cost based optimizer uses statistics generated by the DBMS_STATS procedure as the situational data when creating its expert opinion on which access path to the data is most optimal. These statistics are stored in the data dictionary and describe the data object's space characteristics, data uniqueness and data distribution.

The cost-based optimizer is only as good as the statistics it uses as input. Statistics collections should be run on a regular basis to ensure that the statistics are current (representative of the data being accessed). The optimizer is then able to create a highly accurate access path that is based on the least cost. If statistics are not available, the optimizer uses an algorithm to calculate the statistics, which often leads to "less than optimal" access paths. In other words, Oracle guesses and it is usually not a very good guesser.

You can start your education on the different access paths that are available to the optimizer by reading Oracle's Database Performance Tuning Guide that is provided in each Oracle release's documentation. Before you buy third-party books on any topic, I highly suggest that you read Oracle's documentation first.

CPU and I/O Performance Statistics
Before we begin our discussion on object statistics, let's spend a few minutes discussing the system performance statistics that are used as input to the optimizer. Don't confuse system performance statistics with database performance statistics that are accessed using the V$ tables. We are discussing the system's CPU and I/O performance and utilization.

In Oracle 10G, the optimizer takes CPU and I/O costs into consideration during query optimization. If the optimizer understands what the performance characteristics are for I/O and CPU, it will be more effective at query optimization. Current system statistics allow the optimizer to make more intelligent decisions during the optimization process, which ultimately results in higher quality access paths being generated.

Unlike object statistics, administrators must continue to gather system statistics manually in Oracle 10G. This is done by executing the DBMS_STATS.GATHER_SYSTEM_STATS procedure. The key to gathering good system statistics is to run the procedure when a normal workload is being executed. Running it when the workload is light, or too heavy, can result in poor access paths being generated.

If you would like to learn more about how system statistics affect the Oracle optimizer, please refer to Jonathan Lewis's excellent article titled Understanding System Statistics on Oracle Technet. Please pay special attention to Lewis's information pertaining to each release of the database.

Object Statistics
As we learned previously, object statistics describe the contents of the data and how it is stored in the physcal database objects. Before we continue, it may be beneficial for me to provide you with a few examples of how these statistics could affect query performance.

If a table only contains a few hundred rows, queries may run faster if the optimizer chooses to read all of the blocks in the table as opposed to using an index. The I/O generated traversing the index blocks to get to the table row entries would be higher than if Oracle read just the blocks allocated to the table being accessed.

What if we access a table using a column with poor cardinality? Cardinality describes the uniqueness of a particular value stored in a column. Poor cardinality means that the column has many values that are the same. If our statement contains a WHERE clause that searches for a column value that is contained in 90% of the table's rows, it is best that Oracle, once again, read each and every row in that table.

Conversely if a WHERE clause searches for a column value that appears in 1% of the table rows, it would be beneficial for the optimizer to choose an index. The optimizer also uses object statistics to determine which join method should be used and the order in which the joined tables are accessed.

Here are just a few of the object statistcs that the optimizer uses as input during access path generation:

• Number of rows in the tables being accessed
• Number of blocks allocated to the table
• Average table row length
• Number of distinct values for a column
• Number of nulls in a column
• Number of leaf blocks allocated to indexes
• Number of leveles in the B-Tree
• Clustering factor - how the order of the index matches the order of the table rows

As I stated previously, the intent of this blog is to not be an all-inclusive primer on the optimizer, but we should now understand why current object statistics are so important to the generation of efficient query access paths.

Gathering Object Statistics
Oracle provides the DBMS_STATS package to allow administrators to gather, modify, view, export, import, and delete statistics. The DBMS_STATS package gathers statistics on columns, tables, indexes and partitions and stores the results in Oracle's data dictionary. The optimizer reads the object statistics stored in the data dictionary durinq query optimization.

After DBMS_STATS is executed, current statements in the shared pool that access the newly analyzed tables are invalidated. This forces the statement to be re-parsed which will allow the optimizer to generate access paths based on the new statistics.

There are many different options available in DBMS_STATS. Administrators are able to select which objects to analyze, how much of each object to analyze (large tables may take a loooonng time to analyze), select the number of parallel processes that will perform the analyze, etc. To learn more about all of the options available, please refer to Oracle's Database Performance Tuning Guide and PL/SQL Packages and Types reference manuals that are provided in each Oracle release's documentation.

10G Automatic Statistics Generation
In releases prior to Oracle10G, administrators were required to schedule DBMS_STATS jobs on a regular basis to ensure that valid statistics were available to the optimizer. In releases prior to Oracle9i, it was necessary to "guess" how much of the data changed to determine if statistics collection was necessary.

In Oracle9i, the GATHER AUTO option of the DBMS_STATS procedure could be used to help determine if statistics generation was required. If more than 10% of the rows changed in the table since the last analyze was performed, the DBMS_STATS procedure (with the GATHER_AUTO option activated) analyzed the table.