The Critical Patch Update (CPU) for October 2022 was released on 18 October 2022. Oracle strongly recommends applying the CPU patches as soon as possible, but while applying them you may face some issues similar to ours so let’s discuss the issue with the workaround in this blog.
You can check the documentation for Oracle Critical Patch Update October 2022
What are Critical Patch Updates (CPUs)?
Critical Patch Updates are sets of patches containing fixes for security flaws in Oracle products. The Critical Patch Update program (CPU) was introduced in January 2005 to provide security fixes on a fixed, publicly available schedule to help customers lower their security management costs.
Oracle E-Business Suite Release 12.2 Critical Patch Update Availability (October 2022) (Document Note 2884904.1)
Issue
We have seen this issue very recently in our EBS instance where users could not open Log & Out files in browsers in our EBS R12.2 instance. The browser page was showing a blank page. The below process was followed to troubleshoot and fix the issue ASAP.
“FS Diagnostic Test Program” is usually used to Troubleshoot FNDFS and FNDWRR (Problems Viewing Concurrent Request Output. When we ran this diagnostic program, it gave us the below error:
Oracle continues to periodically receive reports of attempts to maliciously exploit vulnerabilities
for which Oracle has already released security patches. Through our analysis of the issues
reported, we have confirmed that the patch released in the October Critical Patch Update (CPU)
in response to vulnerability, CVE-2022-21587 remediates the problems reported.
Note 2884904.1 – Oracle E-Business Suite Release 12.2 Critical Patch Update Availability Document (October 2022)
Oracle therefore STRONGLY recommends that customers remain on actively-supported versions and apply Critical Patch Update security patches without delay.
Error Root Cause
The main focus was on FNDWRR.ex and its affiliated files at the server end to see for any changes and it was finally located that txkFNDWRR.pl perl file under $EBS_ORACLE_HOME/common/scripts/ was showing up as a NULL size file.
The same file was showing up as a NULL file in Non-Production instances too. As a quick solution, this file was copied from Patch Edition to Run Edition under $EBS_ORACLE_HOME/common/scripts/ which resolved the issue for users.
Next, it was time to find the root cause. As the issue happened in all non-production and production instances at the same time, which means it was not done manually by any Apps DBA, so it was very critical to find the root cause ASAP. The main focus was on the access log of EBS Apache to verify what all activities happened at the exact time when the txkFNDWRR.pl perl file was updated and here is the code snippet from the access log:
<IP ADDRESS> - - [20/Jan/2023:14:47:11 -0500] "POST //eadn-wc03-4064062.nxedge.io/OA_HTML/BneViewerXMLService;.js?bne:uueupload=TRUE HTTP/1.1" 200 184 <IP ADDRESS> - - [20/Jan/2023:14:47:11 -0500] "GET /OA_CGI/FNDWRR.exe HTTP/1.1" 200 12
The connection was coming from an external isupplier node.
“bne” refers to WebADI in Oracle EBS and the “uueupload=TRUE” option in a command or script specifies that the file is being uploaded or transferred in encoded uuencode format and is a kind of a signal to the server that the file should be decoded using the uuencode format before being stored on the server.
Workaround: Permanent Solution & Mitigations
Solution 1:
1.) Ran auto-config and all the node sequence was ebsdmz01, ebsapp02, and ebsapp01 while services were running.
2.) Did the action plan in all application nodes i.e ebsapp01,ebsapp02, and ebsdmz01 not be attacked by vulnerability again?
Note: while again doing auto-config/ADOP you need to release the lock in the below file to get successfully executed.
[root@ebsapp01 scripts]# ls -lthr txkFNDWRR.pl -rwxr-xr-x 1 applmgr dba 118 Feb 8 10:37 txkFNDWRR.pl [root@ebsapp01 scripts]# chattr +i txkFNDWRR.pl [root@ebsapp01 scripts]# lsattr txkFNDWRR.pl ----i--------e--- txkFNDWRR.pl [root@ebsapp01 scripts]# cd bkp_08FEB23/ [root@ebsapp01 bkp_08FEB23]# lsattr txkFNDWRR.pl -------------e--- txkFNDWRR.pl
3.) Now you will be able to see the view output/log files successfully.
4.) Need to execute the below for DMZ as well.
5.) Disable WebADI Access from the External Application Tiers in your DMZ by Configuring the URL Firewall.
This can be done by following “Section 7.5: Configuring the URL Firewall” of the below Doc ID: Oracle E-Business Suite Release 12.2 Configuration in a DMZ (Doc ID 1375670.1)
Solution 2:
Although solution 1 is the actual fix, solution 2 is a kind of workaround to mitigate the issue. If you are going forward with this solution because you may not have the right ATG version or need more time/planning to apply the patch then the below high-level steps can prove helpful to you (besides Doc ID 1375670.1): Mitigation step # 2 Disable WebADI Access from the External Application Tiers in your DMZ by Configuring the URL Firewall
1.) Create master url_fw.conf file in both RUN and PATCH Edition
On your iSupplier Node, Create master url_fw.conf file and unblock all the isupplier products you are using. Below WEBADI/BNE access must be blocked from the External node. The following lines in the [ url_fw.conf ] need to be commented out. The hashtag (“#”) at the beginning of the line accomplishes this.
#================================================================ #Include URLs for SSHR (Self Service Human Resources) #================================================================ #RewriteRule ^/OA_HTML/BneApplicationService$ - [L] #RewriteRule ^/OA_HTML/BneViewerXMLService$ - [L] #RewriteRule ^/OA_HTML/BneDownloadService$ - [L] #RewriteRule ^/OA_HTML/BneUploaderService$ - [L] #RewriteRule ^/OA_HTML/BneTemplateService$ - [L] #RewriteRule ^/OA_HTML/BneTemplateRedirectService$ - [L]
2.) Create a custom auto-config template url_fw_conf_FMW.tmp for url_fw.conf. This is needed so that auto-config/ADOP does not overwrite it. Put this template under $FND_TOP/admin/template/custom
You can run $AD_TOP/bin/adchkcfg.sh to verify if the template is getting picked.
3.) Modify CONTEXT_FILE in iSupplier node (both RUN and PATCH)
REPLACE <urlfirewall oa_var="s_enable_urlfirewall">#</urlfirewall> BY <urlfirewall oa_var="s_enable_urlfirewall"/>
4.) Stop the Application, Run Autconfig on External and Internal Nodes, and Start Application.
Related/Further Readings
- 2484000.1 Identifying the Latest Critical Patch Update for Oracle E-Business Suite Release 12
- 2514102.1 Oracle E-Business Suite Release 12 Critical Patch Update (April 2019)
- 2498664.1 Critical Patch Update (CPU) Program April 2019 Patch Availability Document (PAD)
- 1306505.1 Patch Set Update (PSU) Administration Guide for Oracle WebLogic Server (WLS)
- ADOP ( R12.2 Online Patching ) in Oracle EBS (R12) FAQ
- [Video] Oracle Apps DBA (R12.2) Patching (ADOP): FREE Training
- Database Patching In Oracle Cloud: Step By Step
FREE Class to Get Started…
Join us for a FREE class where Atul and a Cloud DBA expert will guide you through the basics of Cloud DBA. Plus, they’ll share their tried-and-tested (proven) roadmap for learning these technologies in a step-by-step manner.
Register for a FREE class, and don’t miss an opportunity to gain a plethora of insights on becoming a certified Oracle Cloud DBA.
The post [Troubleshooting] Critical Patch Update for October 2022 Now Available appeared first on Cloud Training Program.