Congratulations! You have just successfully implemented Oracle E-Business Suite 12.2! After months of testing and working through issues, all your hard work has finally paid off. However, now that the general userbase is in the system, a few small issues have been reported and your DBA has found direct hits that suggest a few application patches need to be applied.
But before asking your DBA to start, you recall them telling you about “online patching” during your weekly status calls and with Oracle EBS 12.2 the process for patching has changed. So, what does this mean? Is there still downtime? How long is this going to take now?
New Architecture
Starting with Oracle EBS 12.2, the old method of patching is no more. Rather than stop/patch/restart (and dealing with whatever the cumulative downtime is), Oracle has now provided a feature called Application DBA Online Patching (ADOP). Oracle accomplishes this by introducing editioned filesystems in 12.2. Instead of having one location for application files like in previous versions of EBS, you now have two copies – one that remains up and running and another that can be synchronized then patched in parallel. Oracle differentiates between these two filesystems by naming one “fs1” and the other “fs2”.
The primary filesystem is where all your runtime application files reside and is designated as the RUN filesystem. Every form or webpage you open will come from here. The secondary filesystem purely exists to be patched (with no bearing on frontend user experience) and is designated as the PATCH filesystem. Out of the box, fs1 defaults as the RUN filesystem and fs2 as the PATCH filesystem.
Once we finish applying patches during an ADOP cycle, a brief downtime is required to perform a switch of the two filesystems. The PATCH filesystem that was updated with new bug fixes will become the new RUN filesystem and the former RUN filesystem will be relegated to the PATCH filesystem (where it can later be synchronized for future patching). Therefore, in the out of the box example above, fs2 becomes the RUN filesystem and fs1 is now the PATCH filesystem. The two filesystems will alternate between who is RUN and who is PATCH after each completed ADOP cycle.
It is also important to note, that only application-level updates can be leveraged with ADOP. Database patches, unfortunately, will still require downtime as they exist outside the scope of the applications, and thus, ADOP. That bing said, for hybrid action plans consisting of both database and applications patches, the order of operations can be manipulated so that most (if not all) application updates can be done during an online patching cycle with the database portion reserved for immediately thereafter.
Understanding the Phases of an ADOP Cycle
As stated previously, the simplicity of stop/patch/restart is out the window. In its place is a more intricate process for your DBA, but a shorter period of inaccessibility for end users as the system is no longer down for the duration of patching. As a result, customers should experience greater overall uptime with ADOP patching when compared to patching with downtime. Within an ADOP patching cycle, the following phases are performed:
1) Prepare
2) Apply
3) Finalize
4) Cutover
5) Cleanup
The first three phases (and the last one) are all accomplished without downtime. It is only at the cutover phase where the system comes down. In the below sections, we will learn a little more about each phase, both for those only seeking a general understanding and for those interested in the specifics of each phase.
Prepare Phase
For a general understanding, this is where EBS prepares the application for patching. First, EBS will ensure there are no conflicts or misconfigurations with the environment. If the environment passes all the mandatory checks, it will begin to synchronize the two filesystems so that the PATCH filesystem is in sync with the current RUN filesystem. This is done without downtime.
For a deeper understanding, this is arguably the longest phase in an ADOP cycle as there are a considerable number of things being accomplished:
- Checks if the cleanup phase was performed during the last ADOP cycle and, if not, completes it now.
- Checks the general system configuration to determine if the environment is ready for a new ADOP cycle.
- Checks if the logon trigger is present and enabled. If any issues are found, it will attempt to fix these automatically.
- Checks the data dictionary for any corruption. If present, corruption will need to be resolved before proceeding.
- Checks if the EBS Technology Codelevel Checker (ETCC) has been executed against the database (if not, prepare will fail until it has been run).
- Checks for proper registration and configuration for all nodes in the environment.
- Checks if the concurrent request “Online Patching In Progress” (short name of ADZDPATCH) is present to block specific concurrent requests from running. This job will remain active for as long as the DB patch edition exists. It is only canceled when the cutover has completed.
- Synchronizes patches currently on the RUN fs, but not in the PATCH fs.
- Checks if a patch edition exists in the DB (and creates one if not present).
- Confirms DB connectivity to the patch edition.
Apply Phase
For a general understanding, this is when we apply patches and any other application-level updates to be performed against the PATCH filesystem.
For a deeper understanding, there frankly is not a whole lot more to add as this phase is akin to running the old adpatch command in prior versions of EBS. Like with adpatch, there are a handful of parameters you can run with this to gain more granular control (i.e. defining “patchtop” to use a non-default patch staging location or setting “workers” to specify the number of parallel processes you want going at once). The handiest tidbit while at this phase; however, is leveraging it to also apply FMW patches and any other application-level updates to the PATCH fs while you have an active ADOP session open. Be careful though and ensure you source the PATCH fs before applying any FMW patches or running any commands. These updates will reflect later, after cutover.
Finalize Phase
For a general understanding, this phase effectively closes out the patching phase and prepares the environment for downtime.
For a deeper understanding, this phase prepares the environment for the shortest possible downtime during cutover. This will include tasks like determining any Data Definition Language (DDL) that needs to be executed, compiling invalid objects, and validating the system before downtime.
Cutover Phase
For a general understanding, this is the only phase that will require downtime for customers. Before ADOP, downtime for patching was an accumulation of how long each patch took to run, plus the amount of time to stop, start, and validate services. However, by leveraging ADOP in EBS 12.2, all our patches are already applied to the PATCH filesystem at this point. Therefore, all we need to do at cutover is stop services, alternate the roles of fs1 and fs2 (regarding which is RUN and which is PATCH), and then restart services. Since there are no variables at cutover that influence downtime (like the number of patches or a large patch being applied), this helps ensure a generally reliable outage duration regardless of the action plan that typically ranges from 30 to 60 minutes.
For a deeper understanding, cutover starts with bringing down EBS. Once it is down, the patch edition of the database will become the run edition and the old PATCH fs will become the new RUN fs (with the old RUN fs becoming the PATCH fs). Once switched over, any old RUN fs sessions will terminate and EBS will be restarted. However, if you do not want your services restarted by cutover (for instance, say you need to apply patches to the DB with downtime after ADOP completes), you can opt to run it with the parameter “mtrestart=no” to leave the applications down.
Cleanup Phase
For a general understanding, this step wraps up the ADOP session by removing any data related to the patching session that is no longer required.
For a deeper understanding, cleanup should be run to complete an ADOP session, but it is not mandatory. That is because, if skipped, cleanup will be run at the top of the next prepare phase that is attempted. As a result, this may lengthen the amount of time it takes to run prepare, which is why it is recommended to run this at the end of a patch cycle. With cleanup, any obsolete objects and data relating to old editions of your database will be removed. Cleanup also offers a few different modes which can be passed as parameters (i.e. cleanup_mode=full). By default, cleanup will run as “standard”, but you can also opt for “quick” or “full” modes, where you can perform minimum or maximum cleanup, respectively.
The Result
While requiring more preparation and overall lead time, ADOP ultimately yields massive benefits to end users by reducing the total downtime needed to patch applications. Lengthy action plans against EBS now just require a short downtime to bounce and alternate filesystems. Even projects as big as upgrades can reduce overall downtime by leveraging an ADOP cycle at the start of maintenance activities. With maximized uptime being crucial to business, ADOP is an essential tool that delivers the benefits of critical security updates while minimizing the cost to company productivity.