ADOP Failes with "Unable to find appltop_id for the host from database"

Issue:

Our EBS 12.2 application has been cloned recently from PROD and this is a shared application tier. User requested to apply a patch and when we tried to apply the patch adop is coming with errors. Below are the error details.

[UNEXPECTED]Unable to find appltop_id for the host oraap02 from database
[UNEXPECTED]Unable to find appltop_id for the host oraap02 from database
[EVENT]     [START 2015/04/10 02:23:17] Verify SSH
Logfile location /u01/app/fs1/inst/apps/EBSR12_shared/logs/appl/rgf/TXK/verifyssh.log
xml output = /u01/app/fs1/inst/apps/EBSR12_shared/logs/appl/rgf/TXK/out.xml
[EVENT]     [END   2015/04/10 02:23:20] Verify SSH
All nodes have ssh connectivity. Continuing...
[EVENT]     [START 2015/04/10 02:23:20] Checking the DB parameter value
[EVENT]     [END   2015/04/10 02:23:23] Checking the DB parameter value  [UNEXPECTED]Unable to find appltop_id for the host oraap02 from database
  [UNEXPECTED]Invalid appltop id: "".
  [UNEXPECTED]Unrecoverable error occured. Exiting the current session.
  Log file: /u01/app/fs_ne/EBSapps/log/adop/adop_20150410_022145.log

The above error doesn't give much clear information on what was causing issue. I started reviewing & validating the below tables and everything shows good.

FND_NODES
FND_OAM_CONTEXT_FILES
AD_APPL_TOPS
AD_ADOP_SESSIONS

Then as a first attempt I ran autoconfig on Node 1 "oraapp01" and tried to restart adop again but this time with fs_clone option. This again printed same errors again(pointing to node 2) but luckily it went ahead with fs_clone.

[EVENT]     [START 2015/04/10 22:54:51] Checking the DB parameter value
[EVENT]     [END   2015/04/10 22:54:54] Checking the DB parameter value
    [UNEXPECTED]Unable to find appltop_id for the host oraapp02 from database
    [UNEXPECTED]Unable to find appltop_id for the host oraapp02 from database
 but fs_clone failed at Validation phase and the log file <ADPLOGHOME>/fs_clone_20150410_224816/EBSR12_oraap01/remote_execution_result_20150410_225611.xml showed below details


[ERROR]: Either the value of the context variables_shared_file_system in the run file system is not consistent between the file system and the database or the APPL_TOP name across nodes is not set correctly.

Solution:

This made me think about APPL_TOP Name again. I reviewed my context files on both the application tiers (Primary & Secondary) and found that the context variable s_atName is different on both the nodes.

For a shared Applicaiton Tier APPL_TOP name should be same across all the applicaiton tier nodes.
1) Modified context variable s_atName on Secondary node to Master Application Nodename .
2) Executed autoconfig and bounced services.
3) Then adop is went successfully.

EBS 12.2 Unable to start up the managed server oacore_server1 - after clone

Unable to Start OACORE Services during clone on RUN Filesystem

Recently we did a clone and as part of execution of adcfgclone.pl in run Filesystem, I responded to adcfgclone.pl to start Application Filesystems. But I observed the start of Managed servers failed and below is the output from one of oacore services.

Successfully Connected to Node Manager.

The Node Manager is already up.
Starting server oacore_server1 ...
Error Starting server oacore_server1: weblogic.nodemanager.NMException: Exception while starting server 'oacore_server1'

ERROR: Unable to start up the managed server oacore_server1

05/20/15-00:20:43 :: admanagedsrvctl.sh: exiting with status 1

This information is not sufficient to identify the cause of the issue and I had go through the oacore log files under $EBS_DOMAIN_HOME/servers/oacore_server1/logs

9ee0eb174c5acfd:2fe0c70b:14d72092155:-8000-0000000000000eda> <1432203790547> <BEA-100028> <Could not deserialize session data.
oracle.apps.fnd.common.PoolException: Exception creating new Poolable object.
        at oracle.apps.fnd.common.Pool.createObject(Pool.java:1289)
..
 <<WLS Kernel>> <> <> <1432099154476> <BEA-149205> <Failed to initialize the application 'EBSDataSource' due to error weblogic.application.ModuleException: .
..
java.lang.NoClassDefFoundError: Could not initialize class oracle.apps.fnd.common.WebAppsContext
        at oracle.apps.fnd.security.AppsServletFilter.init(AppsServletFilter.java:148)
..
<Could not load user defined filter in web.xml: oracle.apps.jtf.cabo.interceptor.JTFWrapperFilter.
java.lang.NullPointerException
..

..
..

Caused By: weblogic.common.resourcepool.ResourceSystemException:
 Could not connect to 'oracle.jdbc.OracleDriver'.

 The returned message is: ORA-01017: invalid username/password; logon denied

 It is likely that the login or password is not valid.
 It is also possible that something else is invalid in the configuration or that the database is not available.




Then this gave clear information which says that the apps password provided during adcfgclone.pl is noot being accepted even though the password provided is correct.

Later it is observed that Apps Password is changed as part of execution of adcfgclone.pl on DBTier and theis new password change is somehow not reflected when Managed servers are created during execution of rapidclone on Apps Tier.

Now this is fixed manually by performing below steps.

As ADMIN Server is up and runnning,

Login to EBS Weblogic console -> Perform Lock & Edit -> Under Domain Structure -> Services -> Datasources -> Select "EBSDataSource"

Navigate to Configuration -> Connection pool  -> Update with new password in password fields.

Then Save changes.

Perform the above steps of other Datasource OAADatasource.

Now Activate the changes in Control Center.

This should resolve the issue. Bring up the managed servers from Console or using command line.

Adop is not to able to detect any valid application tier nodes

Adop is not to able to detect any valid application tier nodes

Issue:

Running ADOP on 12.2 is failing at initial Validation with error “adop is not able to detect any valid application tier nodes”
Enter the APPS password: 
Enter the SYSTEM password: 
Enter the WLSADMIN password: 

Validating credentials... 

Initializing... 
Run Edition context : /u01/oracle/VIS/fs1/inst/apps/VIS_ora01/appl/admin/VIS_ora01.xml 
Patch edition context: / u01/oracle/VIS/fs2/inst/apps/VIS_ora01/appl/admin/VIS_ora01.xml 
*******FATAL ERROR******* 
PROGRAM : (/u01/oracle/VIS /fs1/EBSapps/appl/ad/12.0.0/bin/adzdoptl.pl) 
TIME : Wed Apr 22 04:40:38 2015 
FUNCTION: ADOP::GlobalVars::_GetMandatoryArgs [ Level 1 ] 
ERRORMSG: adop is not able to detect any valid application tier nodes in 
ADOP_VALID_NODES table. Ensure autoconfig is run on all nodes. 


[STATEMENT] Please run adopscanlog utility, using the command 

"adopscanlog -latest=yes" 

to get the list of the log files along with snippet of the error message corresponding to each log file. 


adop exiting with status = 255 (Fail) 

Cause

Upon investigation we found that Database listener is not listening ebs_patch service which is mandatory for 12.2 to work on any adop activities.

[oracle@ora01 ~]$ lsnrctl status $ORACLE_SID

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 23-APR-2015 12:20:02

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

Alias                     VIS

Listening Endpoints Summary...

.

.

Services Summary...

Service "VIS" has 2 instance(s).

  Instance "VIS", status UNKNOWN, has 1 handler(s) for this service...

  Instance "VIS", status READY, has 1 handler(s) for this service...

The command completed successfully

Solution

1)     Make sure that Database init parameter has service name ebs_patch defined

2)     Connect to Database as sysdba

sqlplus / as sysdba

SQL> show parameter service_name

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

service_names                        string      VIS, ebs_patch

                If this is there, set the parameter as below

                SQL> alter system set service_names=’ebs_patch’,’VIS’ scope=spfile;

3)     Verify init parameter LOCAL_LISTENER and  set it to <DB_NAME>_LOCAL

                                 SQL> alter system set local_listener=’VIS_LOCAL’ scope=spfile;

4)     Bounce the application & database services