EBS 12.2 Unable to start up the managed server oacore_server1 - after clone

Unable to Start OACORE Services during clone on RUN Filesystem

Recently we did a clone and as part of execution of adcfgclone.pl in run Filesystem, I responded to adcfgclone.pl to start Application Filesystems. But I observed the start of Managed servers failed and below is the output from one of oacore services.

Successfully Connected to Node Manager.

The Node Manager is already up.
Starting server oacore_server1 ...
Error Starting server oacore_server1: weblogic.nodemanager.NMException: Exception while starting server 'oacore_server1'

ERROR: Unable to start up the managed server oacore_server1

05/20/15-00:20:43 :: admanagedsrvctl.sh: exiting with status 1

This information is not sufficient to identify the cause of the issue and I had go through the oacore log files under $EBS_DOMAIN_HOME/servers/oacore_server1/logs

9ee0eb174c5acfd:2fe0c70b:14d72092155:-8000-0000000000000eda> <1432203790547> <BEA-100028> <Could not deserialize session data.
oracle.apps.fnd.common.PoolException: Exception creating new Poolable object.
        at oracle.apps.fnd.common.Pool.createObject(Pool.java:1289)
..
 <<WLS Kernel>> <> <> <1432099154476> <BEA-149205> <Failed to initialize the application 'EBSDataSource' due to error weblogic.application.ModuleException: .
..
java.lang.NoClassDefFoundError: Could not initialize class oracle.apps.fnd.common.WebAppsContext
        at oracle.apps.fnd.security.AppsServletFilter.init(AppsServletFilter.java:148)
..
<Could not load user defined filter in web.xml: oracle.apps.jtf.cabo.interceptor.JTFWrapperFilter.
java.lang.NullPointerException
..

..
..

Caused By: weblogic.common.resourcepool.ResourceSystemException:
 Could not connect to 'oracle.jdbc.OracleDriver'.

 The returned message is: ORA-01017: invalid username/password; logon denied

 It is likely that the login or password is not valid.
 It is also possible that something else is invalid in the configuration or that the database is not available.




Then this gave clear information which says that the apps password provided during adcfgclone.pl is noot being accepted even though the password provided is correct.

Later it is observed that Apps Password is changed as part of execution of adcfgclone.pl on DBTier and theis new password change is somehow not reflected when Managed servers are created during execution of rapidclone on Apps Tier.

Now this is fixed manually by performing below steps.

As ADMIN Server is up and runnning,

Login to EBS Weblogic console -> Perform Lock & Edit -> Under Domain Structure -> Services -> Datasources -> Select "EBSDataSource"

Navigate to Configuration -> Connection pool  -> Update with new password in password fields.

Then Save changes.

Perform the above steps of other Datasource OAADatasource.

Now Activate the changes in Control Center.

This should resolve the issue. Bring up the managed servers from Console or using command line.

Failed to start EBS 12.2.4 Managed services

Failed to start EBS 12.2.4 Managed services

Completed the upgrade of E-Business Suite to 12.2.4 and tried to start all the application services using adstrtal.sh script as below.

$ adstrtal.sh apps/<appspwd>

The script exited with status ‘1’ and examining the log file $INST_TOP/logs/appl/admin/log/adstrtal.log showed failure of all the managed servers

  Web Application Services                                                                            Enabled

  Web Application Services   oacore_server1                                admanagedsrvctl.sh         Failed

  Web Application Services   forms_server1                                 admanagedsrvctl.sh         Failed

  Web Application Services   oafm_server1                                  admanagedsrvctl.sh         Failed

  Web Application Services   forms-c4ws_server1                            admanagedsrvctl.sh         Failed

Reviewed one of the managed server startup logfile to identify the root cause of the failure.

Oacore_server1 logfile under $FMW_HOME/ user_projects/domains/EBS_domain_orcl/servers/oacore_server1/logs/   showed below error details

<Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.>

This error indicated that there is corruption happened with the boot.properties and this needs to be fixed.

Solution:

As initial fix, backed up existing boot.properties file under $FMW_HOME/ user_projects/domains/EBS_domain_orcl/servers/oacore_server1/security and created a new boot.properies with below details

$cat boot.properties

username=weblogic

password=weblogic

After creation of this to encrypt this start oacore_server1 managed server as below

$FMW_HOME/ user_projects/domains/EBS_domain_orcl/bin/startManagedWebLogic.sh oacore_server1 <admin_url>

Performed these steps to the remaining failed managed servers.

Now stopped all the managed services from console and then remaining services using adstpall.sh script.

Then restarted all the services using adstrtal.sh script and all the services came up successfully. But when later tried to stop services using adstpall.sh script couldn’t bring down the managed services.

Upon further troubleshooting found that the startup/shutdown arguments for managed servers are incorrect resulting in failures. This is found in config.xml file under $FMW_HOME/user_projects/domains/<domain_name>/config.

By reviewing them process argument details it showed that there is an invalid value for the property -Dweblogic.management.server for all the managed servers. Performed the below action plan to fix the issue.

1.       Log in in to the Administration Console.

2.       In the Change Center, click Lock & Edit.

3.       In the left pane of the Console, expand Environment and select Servers.

4.       In the Servers table, click the name of a Managed Server(oacore_server1).

5.       Select Configuration > Server Start

6.       In 'Arguments', correct the property value -Dweblogic.management.server to the correct value as below.

-XX:PermSize=128m -XX:MaxPermSize=384m -Xms512m -Xmx512m -….

-Dweblogic.Name=oacore_server1 -Dweblogic.management.server=http://orc11.localdomain:7006 -Djava.library.path=$MW_HOME/webtier/jdk/jre/lib/amd64:$MW_HOME/webtier/jdk/jre/lib/amd64/server:$MW_HOME/webtier/jdk/jre/lib/amd64/native_threads…

To

-XX:PermSize=128m -XX:MaxPermSize=384m -Xms512m -Xmx512m -….

-Dweblogic.Name=oacore_server1 -Dweblogic.management.server=http://orcl.localdomain:7006 -Djava.library.path=$MW_HOME/webtier/jdk/jre/lib/amd64:$MW_HOME/webtier/jdk/jre/lib/amd64/server:$MW_HOME/webtier/jdk/jre/lib/amd64/native_threads…

7.       Save the changes.

8.       Start all managed servers using using the command:

$ADMIN_SCRIPTS_HOME admanagedsrvctl.sh start <server name>

Performed these changes to all the other managed servers.