From Oracle Applications DBA Field Guide, Berkeley, Apress, March 2006.

Part 1  |  Part 2  |  Part 3  |  Part 4

Special Considerations

There are some additional issues that may need to be addressed during the patching process. A class of patches that contain legislative data has an additional driver called hrglobal, which may need to be applied. Also, for some groups of patches, it may be beneficial to merge the patches into one set of driver files. Depending upon your implementation, you may also need to deal with multi-language patches and multi-node patching. These topics are discussed in the following sections.

Applying Legislative Patches

For Oracle Payroll customers, there is another category of patch required by the system. The hrglobal patch supports the legislative requirements of multiple countries. Given the nature of this patch, it is updated frequently by Oracle. It is often a post-patch requirement for the mandatory patches released for Oracle Payroll.

To find the latest copy of the hrglobal patch, view MetaLink Note 145837.1. This note will contain the latest patch number for the hrglobal patch, along with a link to the patch installation instructions and a change history for the patch. The hrglobal patch can be downloaded from MetaLink like any other patch. Review the patch’s readme file for required prerequisites.

After unpacking the patch, the adpatch utility can be run to install the patch’s u driver. In addition to the u driver, these patches contain a special hrglobal driver. As a result of these differences, there are additional considerations for applying this type of patch.

Once the u driver has been applied, the DataInstall Java utility needs to be run in order to select needed legislations for install. The syntax for this command is as follows:

jre oracle.apps.per.DataInstall apps apps_password thin
[hostname]:[dbport]:[oracle_sid]

When the DataInstall utility has been executed, the Applications DBA will need to select all relevant legislations. Figure 5-6 shows the main menu for DataInstall.

+--------------------------------------------------+
|            DataInstall Main Menu                 |
+--------------------------------------------------+

1.     Select legislative data to install/upgrade

2.     Select college data to install/upgrade

3.     Select JIT/Geocode or OTL to install/upgrade

4.     Exit to confirmation menu

Figure 5-6. The DataInstall Main Menu

Select option 1 to choose which legislative data to install or upgrade. From the resulting menu, shown in Figure 5-7, you should choose to install any legislative data marked as installed. Note that the selection numbers will change depending upon your version of the hrglobal patch. Check the numbers for the appropriate data.

# Localisation        Product(s)               Leg. Data? Action
- ------------------- ------------------------ ---------- -----------
1 Global              Human Resources          Installed
2 Australia           Human Resources
3 Australia           Payroll
...
55 United States      Human Resources          Installed
56 United States      Payroll                  Installed

 <Product #><Action> - Change Action
 where <Action> is [I : Install, C : Clear]

Figure 5-7. The DataInstall legislative data submenu

Select the legislative data to be installed by entering the localization number and I. If an incorrect number is selected, you can correct the mistake by entering that number with a C to clear the action.

After all legislative data is marked for install, return to the main menu to select any required college data. When all college data is selected, return to the main menu and select 4 to exit the utility. Upon exiting, an Actions Summary will be displayed. Review that summary to ensure that all required actions have been selected.

The final stage of the legislative patch is to run the adpatch utility to apply the hrglobal driver. This driver is copied to the $PER_TOP/patch/115/driver directory by the patch’s u driver. The same adpatch options for applying other drivers should be used for the hrglobal driver.

Using AD Merge

When applying a group of large patches, such as a Maintenance Pack and a cumulative update, some performance benefits can be incurred by using the AD Merge utility to combine the patches into one patch.

The set of patches to be merged should be copied to a common directory. After the patches are unbundled, the AD Merge utility can be run against the patches. Here is an example:

admrgpch /source_dir /target_dir

The completed merged driver files found in the target directory can be applied as a standard patch would be applied. The merged driver files will have a name like u_merged.drv. A log file, admrgpch.log, will be created in the directory where the utility was run.

For more information, see MetaLink Note 228779.1, “How to Merge Patches Using admrgpch.” The admrgpch utility can be run with several parameters, shown in Table 5-3.

s

d

verbose

manifest

logfile

merge_name

Table 5-3. admrgpch Options

When using this utility, thoroughly test the resulting patch.

Applying NLS Patches

For E-Business Suite installations with multiple language requirements, there are patches available for each additional language. Each required NLS patch needs to be applied to Oracle Applications. Oracle provides some recommendations for dealing with NLS patches; these are outlined in MetaLink Note 174436.1.

The U.S. version of the patch should be applied before any of the translation patches. The translation patches may be applied without downtime to the entire system if users of the affected language are not active.

Using admrgpch, it is possible to merge all U.S. patches into one patch, and then merge all non-U.S. patches into a separate patch. Depending upon the application configuration, some variation of this approach may be necessary.

Performing Multi-Node Patching

There are a couple of options available to optimize patching for multi-node environments. As of Oracle Applications 11.5.10, the system can be designed with a shared application-tier filesystem. The shared application filesystem contains the application’s APPL_TOP, COMMON_TOP, and ORACLE_HOME. (MetaLink Note 233428.1 describes sharing the application-tier filesystem.) As a result of this configuration, patching the shared filesystem applies the patch to all nodes.

Prior to 11.5.10, a shared APPL_TOP did not include the ORACLE_HOME. For these systems, Forms and iAS patches must be applied to each Form and Web Node.

In order to increase the performance of the patching process, Distributed AD will execute workers on remote nodes in a multi-node implementation. Distributed AD improves scalability and resource utilization. Distributed AD is only available with AD Minipack H or later, and with a shared Application Tier Filesystem or shared APPL_TOP. More information on this feature can be found in MetaLink Note 236469.1.

If a shared Application Tier Filesystem is not in use, each filesystem will need to be patched separately. A patched filesystem can be cloned to another node if the downtime required to patch the node exceeds the time required to clone the filesystem.

Patch drivers have different requirements when applying them in a multi-node environment. The c driver must be applied to all APPL_TOPs, the d driver is applied on the Admin Node, the g driver is applied to all APPL_TOPs unless the APPL_TOP is only the Admin Node, and the u driver is applied to all APPL_TOPs on all nodes.

Monitoring and Resolving Patching Problems

Patching problems manifest themselves in many different ways. Typically the adpatch session will display an error or will appear to be hung on one task for a long period of time. The first step in resolving the issue is to review the adpatch log file and associated worker log file. Next, the reason the worker failed must be determined and resolved. After resolution has been obtained, adctrl can be used to continue the patching process.

Reviewing Log Files

During and after the application of patches, it is helpful to review log files of the adpatch session and its workers. These files are found in the $APPL_TOP/admin/$CONTEXT_NAME/log directory. The adpatch log filename is specified during the patch process. See the “Using AD Patch” section earlier in the chapter for more details.

In order to monitor the patch from a telnet session other than the one where the patch was started, a simple UNIX command such as tail -f u[patch#].log will display information as it is written to the log file. This is a useful means for monitoring the progress of a patch that is being applied.

The log files for the workers will be named adwork[xxx].log, where [xxx] is the number of the patch worker process. If a particular worker has failed, examine the related log file for detailed information. This information can be researched on MetaLink or used to open an SR with Oracle Support.

For example, the log file listing for the u driver of patch 11112, applied through adpatch using 5 workers, may look like this:

$ls

adwork001.log
adwork002.log
adwork003.log
adwork004.log
adwork005.log
u111112.log

Using AD Control

The administrative tool used to manage patch workers is AD Control, or adctrl. Frequently workers will fail or hang, which will require the Oracle Applications DBA to interface with adctrl. (Common patching errors will be covered later in this chapter.)

AD Control menu options will vary depending upon the AD patch version applied to the instance. When logged in as the application owner on the Admin Node, execute adctrl to display the menu options shown in Figure 5-8.

               AD Controller Menu
----------------------------------------------------
1.     Show worker status
2.     Tell worker to restart a failed job
3.     Tell worker to quit
4.     Tell manager that a worker failed its job
5.     Tell manager that a worker acknowledges quit
6.     Restart a worker on the current machine
7.     Exit

Figure 5-8. AD Controller Menu

To execute an adctrl menu option, simply type the menu option and press Enter. If options 2–6 are chosen, either specify the number of the worker that requires action, or press Enter for the action to be executed for all workers.

The “Skip Worker” menu option is a hidden adctrl menu option. If a worker needs to be skipped, start adctrl, enter 8, and then enter the worker number. Only use this option if advised by Oracle Support.

Tip: With AD.I, adctrl may be used in noninteractive mode. Using adctrl noninteractively can expedite patch problem resolution.

Resolving AD Patch Worker Failure

If a worker has failed, the adpatch session will normally display a failedworker message. The status of the worker may also be determined using adctrl. If a worker has failed, the worker error can be obtained by viewing the worker log file. Once the worker issue has been resolved, use adctrl to restart the worker.

If a worker has failed, and it is determined that the step the worker was trying to execute may be skipped, the hidden option 8 of the adctrl menu, “Skip Worker,” may be used to skip the worker. It is only advisable to do this if the step is not critical to the environment being patched.

Tip: It may be necessary to research MetaLink or open an SR to resolve issues with failed workers. For additional information on MetaLink and the SR process, see Chapter 7 of this guide.

The following are common worker failures that will be seen by the Applications DBA during patching. The error messages will be displayed by the adpatch session or in the worker log file:

Error message: ORA-01013: user requested cancel of current operation

Resolution to error: If this error occurs, simply use adctrl to restart the worker on the current machine.

Error message: Patch not applied successfully, adpatch did not cleanup its restart files (*rf9).

Resolution to error: If this error occurs, execute the following as the instance owner:

      $cd $APPL_TOP/admin/$CONTEXT_NAME 
      $mv restart restart_old 
      $mkdir restart

After cleaning up the restart files, you may then restart the adpatch session using adpatch.

Error message: ERROR: Javakey subcommand exited with status 1

Resolution to error: If this error occurs, the identity.obj file needs to be re-created. See Chapter 2 for steps to recreate the identity.obj file. Then, use adctrl to restart the failed worker.

Error message: No error message is displayed; rather the worker log file states that the worker is complete, yet adctrl indicates that the worker is still running.

Resolution to error: This patching problem occurs when the worker is complete, but did not update patching tables correctly to notify the adpatch session that it has finished. In this scenario, the adpatch session is still waiting for the finish return code from the worker. When this occurs, use adctrl to fail the worker, then restart the worker.

Tip: Any form, library, or report that fails to generate during the patch process can be regenerated manually after all patching and post-patching steps have completed. If the object still fails to compile, open an SR.

Additional Tips for Resolving Patching Issues

If a patch has hung or workers have failed, and the reason for this failure cannot be determined, it is advisable to check the number of invalid objects in the database. If the number of invalid objects is high, recompile the invalid objects in parallel and restart the patching session.

If the adpatch session is hung, and all other methods for resolution have been executed, it may be necessary to bounce the database and restart the patch session. This method for resolving patching issues is sometimes necessary, especially when applying large patches, such as Maintenance Packs.

If a failure occurs during the application of a patch, it may be necessary to apply another patch to resolve the issue. If this type of issue occurs during the application of a large patch, you may want to be able to restart the original patch from the point of failure. MetaLink Note 175485.1 provides details for applying a patch with adpatch already running.

--

Elke Phelps is an Oracle Certified Professional with over 12 years’ experience administering Oracle products. She is the Founder and presiding President of the Oracle Applications User Group’s Middleware SIG—formerly the ATS SIG—and is responsible for conducting meetings at the annual OAUG and Oracle Open World conferences. This yields a unique medium for access to the Oracle Applications users and provides name recognition in the user community. Elke is the author of several whitepapers and is a technical presenter at many international conferences.

Paul Jackson has over 10 years’ experience as a developer and Oracle DBA. Paul is the Program Director for the ATS SIG. He is a very active member of the Oracle community through memberships of Ohio Valley Oracle Applications User Group (OVOAUG), International Oracle User Group (IOUG), and the Oracle RAC SIG.