A major challenge for customers on hosted platforms is keeping the published webservices secure using SSL certificates and various security policies from WSM Policy Manager which are built on Oracle Fusion Middleware WebLogic. When WebLogic is configured to use system identity certificates as the demo certificates, they can face a broken integration due to the SSL validation rules in place for most Cloud provided services.
Most of the Oracle enterprise cloud solutions like OTM, SOACS, Oracle PLM Cloud, and JaaS Cloud have secured their integration API’s and web services and only accept connections from verified clients. This is very similar to the client certificate validation rules for inbound SSL request calls where if the default keystore SSL certificate is expired on the client side there will be a SOAP fault error and the connection will be faulted due to a failed integration (SSL check verification).
The error seen in the application logs are very generic.
Could not determine soap fault from message
Error details- Error Code : 500 Error Type : RemoteFault Error Summary : Remote Fault Occured Error Description : oracle.fabric.common.FabricInvocationException: Unable to invoke endpoint URI
Behavior seen on the client side
On Fusion Middleware (FMW) / WebLogic servers with an expired identity certificate:
After you have followed the basic troubleshooting steps like:
- Ping to destination – its working
- Telnet to destination cloud HTTPS URL – its working
- Recheck credentials and configuration of the connectors – its working
You need to check for the following error in the server logs:
“BEA-090154 Identity certificate has expired”
The error message won’t necessarily be in the same error stack as the web service request failure. It just appears harmless. Other functionality like WebLogic clustering, application access, WebLogic server setup will continue to perform as expected.
Resolution Steps
Reinstall the Identity Certificate
Login to the Oracle Enterprise Manager (EM) Console. Select the domain name and navigate to security, then select the keystore.
If you initially used the default Demo Certificate to install the WebLogic Domain, then the default password will be – DemoIdentityKeyStorePassPhrase. If you used your own certificate during the WebLogic install, refer to the password from your initial install repository.
Under the Manage Certificates page – Delete the old identity certificate using the Demo Certificate password, DemoIdentityPassPhrase.
Create a new Identity Certificate
Select “Generate Keypair”, use the Demo Certificate password, DemoIdentityPassPhrase.
Update the KSS Store
Follow the below steps to sync the KSS store and ensure the changes are not overwritten and lost after the WebLogic server restarts.
From the EM Console, open the “System MBean Browser” and search for “syncKeyStores” under “Operations”. In the JPS Keystore, click “syncKeyStores” with a value of “2” under parameters (row 23).
Enter “system” for p1 and “KSS” for p2.
Click “invoke” to initialize the sync process, wait a few minutes, and check for the confirmation message “Operation Executed Successfully”.
Conclusion
Every 5 years the demo identity certificate expires, so its very important to install custom identity certificates and follow the SSL renewal process every 2 or 3 years. Please review your current certificates and track the expriration dates as well as other enterprise certificates to avoid an unexpected broken integration.
Customers should also review their current SSL certification process before they make the decission to move to an Enterprise Cloud Hosting Services platform, so your cloud hosting partner, like Syntax, can accommodate the SSL certification during the trasistion to the Cloud paltform.
The following products would be affected and may not start due to this issue.
- Oracle WebLogic Server Version Release 9 and above till 12c
- Business Intelligence Server Enterprise Edition Release 12g
- SOA, OSB, ODI, WCS Version Release 12c
In the above scenario, the WebLogic services were running and the end-to-end web service integration failed. In some versions, the WebLogic service itself may not start during such an occurrence and you must use a WLST script tool to update the certificate and the KSS store sync. You can also use the following work-around which take less time:
Goto the config.xml back-up and change the following line
<use-kss-for-demo>true</use-kss-for-demo>. to “false”. It will display as
<use-kss-for-demo>false</use-kss-for-demo>. Restart the WebLogic Server in running state and follow the step.