Oracle 10G OEM Grid Control Testing Results

It has been over a week since my last blog, but I wanted to wait until I completed my 10G OEM testing.  I have spent 22 hours executing my OEM 10G test plan.  If you are going to utilize 10G effectively, you must become an expert in 10G's administration tools. You can find my complete test plan in my previous blog titled Oracle10G OEM Overview and Test Plan.

Installation
My entire unit was slightly concerned about the occurrence of "catastrophic installation problems."  You know the ones I mean...   The problems that take days, or even weeks, to fix.  There is nothing like a good 'ol installation problem to knock you down a few notches on the Oracle expert scale.   Actually, I think they are good for me.  Just when I get too cocky, Oracle lets me know that my career is still challenging.

Like previous versions of OEM, 10G OEM is a multi-tier architecture consisting of a HTML console,  management service (OMS),  repository database and management agents running on all monitored targets.  We assumed that the most problematic install would be the OMS and repository combination.   Wrong again!  Problems with the agents continued to plague us.   But we did have one or two problems occur during the OMS installation.  

Luckily, our OEM 10G administrator Ron Berner, is well-versed in the AIX operating system and a super technician.    Although the 10G OEM install process irritates him from time to time, it certainly doesn't intimidate him.   Ron is also a strong technical troubleshooter, so he is just the person we need to perform the 10G OEM installs.  

Although the initial install went well, when we applied the 10.1.0.3 patch the first time, the entire environment went "kaput".   After several hours of attempting to fix the problem, we decided to de-install everything, reinstall the base software and back it up to another directory.   If we had problems with the patch, this would allow us to restore the environment to a state before the patch was applied.   After completing the backup, we decided to install the patch again and closely monitor all screen messages and log files generated during the installation process.  Well, it worked the second time through. I find that concerning because Ron performed the exact same sequence of commands that he did previously.   That's why I love this profession. You can tell someone to run it again and truly believe it has a chance of working the second time.  

After Ron set up our Oracle management server environment, he turned his attention to installing the agents on our warehouse test servers.   The agent requires a separate install into a different Oracle home.  During the agent install on the target, the Oracle Universal Installer asks for the name of the 10G management server that will be responsible for administering the new environment.   This information is recorded in the EMD.PROPERTIES configuration file on the target.  When the agent is started on the target it "injects" itself into the management server identified during the installation.

The agent installs injects itself into the management server environment by uploading information about the target to the management server.   Getting this XML data upload to work can be tricky.   Here are a few hints and tips:

• Make sure you run the agent control utility (EMCTL) from the agent's home directory.  I had several problems occur when I executed the EMCTL utility with my environment set incorrectly.
• If you are having trouble getting the agent to attach to the management server, review all of the messages contained in EMAGENT.LOG and EMAGENT.TRC.  These files can be found in the AGENTHOME/SYSMAN/LOG directory path (where AGENTHOME is the agent's home directory).  They contain error messages generated by the agent during startup and XML data uploads.   The information contained in these two files helped us fix the majority of our problems. 
• After you find the error messages, use Metalink to research the problem.  If you can't figure it out, file a TAR with Oracle support.  You should run the RDA (Remote Diagnostic Assistant) on the management server and upload that information when you create the TAR.   Since Oracle will ask you for it, you might as well run the RDA and upload it when you create the TAR. 
• After finding and correcting the problem, we found that removing and regenerating the agent's configuration files improved our chances of achieving a successful XML file upload.  Go to document ID 537188.994 on Metalink.   If you can't find it by the document ID, search Metalink using LASTUPLD.XML as the keyword and document ID 537188.994 will pop up.

Thoughts, Musings and Opinions Formulated During Testing

Problems and Crashes (or lack thereof)
Although the agent install is problematic, once you get everything up and running the entire environment is rock solid.    I spent over twenty hours testing OEM 10G and did not receive one fatal error.   More importantly, everything but one feature worked.  The partitioned table wizard allows administrators to specify multiple tablespaces during the creation process.  OEM is supposed to select the tablespaces in a round robin fashion until all partitions are assigned to a particular tablespace.  So if you have tablespaces TSPACEA, TSPACEB, TSPACEC and a table that has 6 partitions, partitions 1 and 4 would use TSPACEA, 2 and 5 would use TSPACEB and partitions 3 and 6 would use tablespace TSPACEC.  For some reason, I couldn't get the wizard to perform the round robin tablespace selection.   Partition 1 did choose TSPACEA but partitions 2 through 6 ended up in the user's default tablespace.

That's it.  Not one abnormal termination by the browser.  I was absolutely sure the product was going to terminate abruptly on a regular basis.   But the tool didn't crash.  It was totally error free and every feature (except the previously mentioned round robin tablespace option) worked.   When it didn't', I eventually found out it was my fault that caused the Oracle error code to be returned.  And when the Oracle errors were returned, they were very explicit and easy to understand.

Navigation
Once you learn to navigate through the tool as you would a website, Oracle10G OEM becomes a breeze to use.   As usual, I didn't like it when I didn't understand how to use it.  After using it for 2 or 3 hours, I was able to easily navigate through the panels.  I now prefer 10G OEM's navigation style to previous versions.  Who says you can't teach an old DBA new tricks?  

Use the SHOW SQL Button a LOT
This tool acts differently than previous versions.  As a result, what you think is going to be executed and what the tool will execute won't always match.  Each administration panel has a button that allows you to see the command you will be executing if you hit the GO button.   I would advise you to use the SHOW SQL button religiously until you become experienced with the tool.   The SHOW SQL button prevented me from making numerous mistakes during my testing process.

Use the Tool's Navigation Tabs - Not the Browser's Forward and Back Buttons to Navigate
Many of the administrative procedures in OEM 10G require the user to enter data in multiple screens to complete the task.   Remember to use the navigation tabs to transfer between the different screens when reviewing your work.   The browser's BACK button often displayed a blank data entry form.   When I used the navigation tabs, the browser always displayed the completed form. 

The GO Button Means Execute - All the Time
All of the data entry screens contain a GO button. Seeing the GO button on a particular screen doesn't mean all of the data entry screens are complete.   You must navigate through all of the tabs and then hit the GO button on the final tab.

Set the Preferred Credentials Immediately
Oracle 10G OEM, as in previous versions, requires that administrators supply Oracle and operating system accounts and passwords in the preferred credentials panel.  When you navigate to the target, OEM logs on to that target using the previously specified account/password combination.  The only time 10G OEM "acted up" was when I did not specify the preferred credentials on the newly defined target.

Next Up
I'll provide readers with a bulleted list of information that I compiled during testing.