Failed to start EBS 12.2.4 Managed services

Failed to start EBS 12.2.4 Managed services

Completed the upgrade of E-Business Suite to 12.2.4 and tried to start all the application services using adstrtal.sh script as below.

$ adstrtal.sh apps/<appspwd>

The script exited with status ‘1’ and examining the log file $INST_TOP/logs/appl/admin/log/adstrtal.log showed failure of all the managed servers

  Web Application Services                                                                            Enabled

  Web Application Services   oacore_server1                                admanagedsrvctl.sh         Failed

  Web Application Services   forms_server1                                 admanagedsrvctl.sh         Failed

  Web Application Services   oafm_server1                                  admanagedsrvctl.sh         Failed

  Web Application Services   forms-c4ws_server1                            admanagedsrvctl.sh         Failed

Reviewed one of the managed server startup logfile to identify the root cause of the failure.

Oacore_server1 logfile under $FMW_HOME/ user_projects/domains/EBS_domain_orcl/servers/oacore_server1/logs/   showed below error details

<Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.>

This error indicated that there is corruption happened with the boot.properties and this needs to be fixed.

Solution:

As initial fix, backed up existing boot.properties file under $FMW_HOME/ user_projects/domains/EBS_domain_orcl/servers/oacore_server1/security and created a new boot.properies with below details

$cat boot.properties

username=weblogic

password=weblogic

After creation of this to encrypt this start oacore_server1 managed server as below

$FMW_HOME/ user_projects/domains/EBS_domain_orcl/bin/startManagedWebLogic.sh oacore_server1 <admin_url>

Performed these steps to the remaining failed managed servers.

Now stopped all the managed services from console and then remaining services using adstpall.sh script.

Then restarted all the services using adstrtal.sh script and all the services came up successfully. But when later tried to stop services using adstpall.sh script couldn’t bring down the managed services.

Upon further troubleshooting found that the startup/shutdown arguments for managed servers are incorrect resulting in failures. This is found in config.xml file under $FMW_HOME/user_projects/domains/<domain_name>/config.

By reviewing them process argument details it showed that there is an invalid value for the property -Dweblogic.management.server for all the managed servers. Performed the below action plan to fix the issue.

1.       Log in in to the Administration Console.

2.       In the Change Center, click Lock & Edit.

3.       In the left pane of the Console, expand Environment and select Servers.

4.       In the Servers table, click the name of a Managed Server(oacore_server1).

5.       Select Configuration > Server Start

6.       In 'Arguments', correct the property value -Dweblogic.management.server to the correct value as below.

-XX:PermSize=128m -XX:MaxPermSize=384m -Xms512m -Xmx512m -….

-Dweblogic.Name=oacore_server1 -Dweblogic.management.server=http://orc11.localdomain:7006 -Djava.library.path=$MW_HOME/webtier/jdk/jre/lib/amd64:$MW_HOME/webtier/jdk/jre/lib/amd64/server:$MW_HOME/webtier/jdk/jre/lib/amd64/native_threads…

To

-XX:PermSize=128m -XX:MaxPermSize=384m -Xms512m -Xmx512m -….

-Dweblogic.Name=oacore_server1 -Dweblogic.management.server=http://orcl.localdomain:7006 -Djava.library.path=$MW_HOME/webtier/jdk/jre/lib/amd64:$MW_HOME/webtier/jdk/jre/lib/amd64/server:$MW_HOME/webtier/jdk/jre/lib/amd64/native_threads…

7.       Save the changes.

8.       Start all managed servers using using the command:

$ADMIN_SCRIPTS_HOME admanagedsrvctl.sh start <server name>

Performed these changes to all the other managed servers.