EBS 12.2 Unable to start up the managed server oacore_server1 - after clone

Unable to Start OACORE Services during clone on RUN Filesystem

Recently we did a clone and as part of execution of adcfgclone.pl in run Filesystem, I responded to adcfgclone.pl to start Application Filesystems. But I observed the start of Managed servers failed and below is the output from one of oacore services.

Successfully Connected to Node Manager.

The Node Manager is already up.
Starting server oacore_server1 ...
Error Starting server oacore_server1: weblogic.nodemanager.NMException: Exception while starting server 'oacore_server1'

ERROR: Unable to start up the managed server oacore_server1

05/20/15-00:20:43 :: admanagedsrvctl.sh: exiting with status 1

This information is not sufficient to identify the cause of the issue and I had go through the oacore log files under $EBS_DOMAIN_HOME/servers/oacore_server1/logs

9ee0eb174c5acfd:2fe0c70b:14d72092155:-8000-0000000000000eda> <1432203790547> <BEA-100028> <Could not deserialize session data.
oracle.apps.fnd.common.PoolException: Exception creating new Poolable object.
        at oracle.apps.fnd.common.Pool.createObject(Pool.java:1289)
..
 <<WLS Kernel>> <> <> <1432099154476> <BEA-149205> <Failed to initialize the application 'EBSDataSource' due to error weblogic.application.ModuleException: .
..
java.lang.NoClassDefFoundError: Could not initialize class oracle.apps.fnd.common.WebAppsContext
        at oracle.apps.fnd.security.AppsServletFilter.init(AppsServletFilter.java:148)
..
<Could not load user defined filter in web.xml: oracle.apps.jtf.cabo.interceptor.JTFWrapperFilter.
java.lang.NullPointerException
..

..
..

Caused By: weblogic.common.resourcepool.ResourceSystemException:
 Could not connect to 'oracle.jdbc.OracleDriver'.

 The returned message is: ORA-01017: invalid username/password; logon denied

 It is likely that the login or password is not valid.
 It is also possible that something else is invalid in the configuration or that the database is not available.




Then this gave clear information which says that the apps password provided during adcfgclone.pl is noot being accepted even though the password provided is correct.

Later it is observed that Apps Password is changed as part of execution of adcfgclone.pl on DBTier and theis new password change is somehow not reflected when Managed servers are created during execution of rapidclone on Apps Tier.

Now this is fixed manually by performing below steps.

As ADMIN Server is up and runnning,

Login to EBS Weblogic console -> Perform Lock & Edit -> Under Domain Structure -> Services -> Datasources -> Select "EBSDataSource"

Navigate to Configuration -> Connection pool  -> Update with new password in password fields.

Then Save changes.

Perform the above steps of other Datasource OAADatasource.

Now Activate the changes in Control Center.

This should resolve the issue. Bring up the managed servers from Console or using command line.

Adop is not to able to detect any valid application tier nodes

Adop is not to able to detect any valid application tier nodes

Issue:

Running ADOP on 12.2 is failing at initial Validation with error “adop is not able to detect any valid application tier nodes”
Enter the APPS password: 
Enter the SYSTEM password: 
Enter the WLSADMIN password: 

Validating credentials... 

Initializing... 
Run Edition context : /u01/oracle/VIS/fs1/inst/apps/VIS_ora01/appl/admin/VIS_ora01.xml 
Patch edition context: / u01/oracle/VIS/fs2/inst/apps/VIS_ora01/appl/admin/VIS_ora01.xml 
*******FATAL ERROR******* 
PROGRAM : (/u01/oracle/VIS /fs1/EBSapps/appl/ad/12.0.0/bin/adzdoptl.pl) 
TIME : Wed Apr 22 04:40:38 2015 
FUNCTION: ADOP::GlobalVars::_GetMandatoryArgs [ Level 1 ] 
ERRORMSG: adop is not able to detect any valid application tier nodes in 
ADOP_VALID_NODES table. Ensure autoconfig is run on all nodes. 


[STATEMENT] Please run adopscanlog utility, using the command 

"adopscanlog -latest=yes" 

to get the list of the log files along with snippet of the error message corresponding to each log file. 


adop exiting with status = 255 (Fail) 

Cause

Upon investigation we found that Database listener is not listening ebs_patch service which is mandatory for 12.2 to work on any adop activities.

[oracle@ora01 ~]$ lsnrctl status $ORACLE_SID

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 23-APR-2015 12:20:02

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

Alias                     VIS

Listening Endpoints Summary...

.

.

Services Summary...

Service "VIS" has 2 instance(s).

  Instance "VIS", status UNKNOWN, has 1 handler(s) for this service...

  Instance "VIS", status READY, has 1 handler(s) for this service...

The command completed successfully

Solution

1)     Make sure that Database init parameter has service name ebs_patch defined

2)     Connect to Database as sysdba

sqlplus / as sysdba

SQL> show parameter service_name

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

service_names                        string      VIS, ebs_patch

                If this is there, set the parameter as below

                SQL> alter system set service_names=’ebs_patch’,’VIS’ scope=spfile;

3)     Verify init parameter LOCAL_LISTENER and  set it to <DB_NAME>_LOCAL

                                 SQL> alter system set local_listener=’VIS_LOCAL’ scope=spfile;

4)     Bounce the application & database services

Site Name profile in 12.2

We generally define “Site Name” profile at site level to identify forms session distinctly when we have multiple EBS Sessions. In prior releases upto 12.1.3 this can be directly modified and the change remains permanent until you make changes manually.

From 12.2, Oracle changed the way how it is handled. The value of Site Name defaults to parameter value s_sitename defined in context file. So, whenever we execute autoconfig it overwrites “Site Name” with s_sitename  parameter

The script that  updates the value when autoconfig is ran: $INST_TOP/admin/install/txkappsprf.sql

 In order to customize the change according to our requirement we can perform steps as below:

1)      Identify the template file that needs to be customized.

$AD_TOP/bin/adtmplreport.sh contextfile=$CONTEXT_FILE target=$INST_TOP/admin/install/txkappsprf.sql

This script provides log file which gives template file details:

TEMPLATE FILE   : /u01/tech_st/fs1/EBSapps/appl/fnd/12.0.0/admin/template/txkappsprf.sql

TARGET FILE     : /u01/tech_st/fs1/inst/apps/endecebs_endecaapps/admin/install/txkappsprf.sql

2)      Navigate to TEMPLATE File location $FNS_TOP/template and create directory “custom”

$ mkdir custom

3)      Switch to “custom” directory and copy the template file to this location.

4)      Edit the content of this newly copied file txkappsprf.sql under $FND_TOP/admin/template/custom as below

DEFINE SITE_NAME = "%s_sitename%"

To

DEFINE SITE_NAME = "<Your customized message>"

After these changes are made whenever you run autoconfig, the Site Name will be updated based on your custom template file. This can be added to your post clone steps based on your requirements.