ADOP Failes with "Unable to find appltop_id for the host from database"

Issue:

Our EBS 12.2 application has been cloned recently from PROD and this is a shared application tier. User requested to apply a patch and when we tried to apply the patch adop is coming with errors. Below are the error details.

[UNEXPECTED]Unable to find appltop_id for the host oraap02 from database
[UNEXPECTED]Unable to find appltop_id for the host oraap02 from database
[EVENT]     [START 2015/04/10 02:23:17] Verify SSH
Logfile location /u01/app/fs1/inst/apps/EBSR12_shared/logs/appl/rgf/TXK/verifyssh.log
xml output = /u01/app/fs1/inst/apps/EBSR12_shared/logs/appl/rgf/TXK/out.xml
[EVENT]     [END   2015/04/10 02:23:20] Verify SSH
All nodes have ssh connectivity. Continuing...
[EVENT]     [START 2015/04/10 02:23:20] Checking the DB parameter value
[EVENT]     [END   2015/04/10 02:23:23] Checking the DB parameter value  [UNEXPECTED]Unable to find appltop_id for the host oraap02 from database
  [UNEXPECTED]Invalid appltop id: "".
  [UNEXPECTED]Unrecoverable error occured. Exiting the current session.
  Log file: /u01/app/fs_ne/EBSapps/log/adop/adop_20150410_022145.log

The above error doesn't give much clear information on what was causing issue. I started reviewing & validating the below tables and everything shows good.

FND_NODES
FND_OAM_CONTEXT_FILES
AD_APPL_TOPS
AD_ADOP_SESSIONS

Then as a first attempt I ran autoconfig on Node 1 "oraapp01" and tried to restart adop again but this time with fs_clone option. This again printed same errors again(pointing to node 2) but luckily it went ahead with fs_clone.

[EVENT]     [START 2015/04/10 22:54:51] Checking the DB parameter value
[EVENT]     [END   2015/04/10 22:54:54] Checking the DB parameter value
    [UNEXPECTED]Unable to find appltop_id for the host oraapp02 from database
    [UNEXPECTED]Unable to find appltop_id for the host oraapp02 from database
 but fs_clone failed at Validation phase and the log file <ADPLOGHOME>/fs_clone_20150410_224816/EBSR12_oraap01/remote_execution_result_20150410_225611.xml showed below details


[ERROR]: Either the value of the context variables_shared_file_system in the run file system is not consistent between the file system and the database or the APPL_TOP name across nodes is not set correctly.

Solution:

This made me think about APPL_TOP Name again. I reviewed my context files on both the application tiers (Primary & Secondary) and found that the context variable s_atName is different on both the nodes.

For a shared Applicaiton Tier APPL_TOP name should be same across all the applicaiton tier nodes.
1) Modified context variable s_atName on Secondary node to Master Application Nodename .
2) Executed autoconfig and bounced services.
3) Then adop is went successfully.

Site Name profile in 12.2

We generally define “Site Name” profile at site level to identify forms session distinctly when we have multiple EBS Sessions. In prior releases upto 12.1.3 this can be directly modified and the change remains permanent until you make changes manually.

From 12.2, Oracle changed the way how it is handled. The value of Site Name defaults to parameter value s_sitename defined in context file. So, whenever we execute autoconfig it overwrites “Site Name” with s_sitename  parameter

The script that  updates the value when autoconfig is ran: $INST_TOP/admin/install/txkappsprf.sql

 In order to customize the change according to our requirement we can perform steps as below:

1)      Identify the template file that needs to be customized.

$AD_TOP/bin/adtmplreport.sh contextfile=$CONTEXT_FILE target=$INST_TOP/admin/install/txkappsprf.sql

This script provides log file which gives template file details:

TEMPLATE FILE   : /u01/tech_st/fs1/EBSapps/appl/fnd/12.0.0/admin/template/txkappsprf.sql

TARGET FILE     : /u01/tech_st/fs1/inst/apps/endecebs_endecaapps/admin/install/txkappsprf.sql

2)      Navigate to TEMPLATE File location $FNS_TOP/template and create directory “custom”

$ mkdir custom

3)      Switch to “custom” directory and copy the template file to this location.

4)      Edit the content of this newly copied file txkappsprf.sql under $FND_TOP/admin/template/custom as below

DEFINE SITE_NAME = "%s_sitename%"

To

DEFINE SITE_NAME = "<Your customized message>"

After these changes are made whenever you run autoconfig, the Site Name will be updated based on your custom template file. This can be added to your post clone steps based on your requirements.

Autoconfig failed with error AC-50480: Internal error occurred:

Autoconfig failed with error AC-50480: Internal error occurred: java.lang.Exception: Error while generating listener.ora.

Issue:
Running autoconfig failed during execution of adgentns.pl file.
below is the error message seen in adconfig.log file.

##########################################################################
Logfile:  /u01/orcl/inst/apps/orcl_oradb/admin/log/08201702/NetServiceHandler.log
        Classpath                   : /u02/orcl/apps/apps_st/comn/java/lib/appsborg2.zip:/u02/orcl/apps/apps_st/comn/java/classes
Updating s_tnsmode to 'generateTNS'
UpdateContext exited with status: 0
AC-50480: Internal error occurred: java.lang.Exception: Error while generating listener.ora.
Error generating tnsnames.ora from the database, temporary tnsnames.ora will be generated using templates
Instantiating Tools tnsnames.ora
Tools tnsnames.ora instantiated
Web tnsnames.ora instantiated
adgentns.pl exiting with status 2....
....
[CVM Error Report]
The following report lists errors encountered during CVM Phase
      <filename>  <return code where appropriate>
  /u02/orcl/apps/apps_st/appl/ad/12.0.0/bin/adgentns.pl  2
No of scripts failed in CVM phase: 1

--> Verifying NetServiceHandler.log file it showed below errors while running script admk80ln.sh which is used to create tnsnames.ora and listener.ora file by retrieving node details from database.

Enter value for 1: Enter value for 2: ERROR:
ORA-12154: TNS:could not resolve the connect identifier specified

Unable to generate listener.ora from database
Using default listener.ora file
Error while generating listener.ora.

StackTrace:
java.lang.Exception: Error while generating listener.ora.

Cause:

When queried FND_NODES table only applicaiton tier node information is retrieved. Database tier information is missing.
Upon investigation it is identified that node clean is performed on the ebs database and autoconfig on database tier is not done.

Solution:
1) Run autoconfig on database tier
2) Once autoconfig on database tier is successfull run again on application tier and this time it should be successful.

Version Conflicts among development maintained and customized templates encountered; aborting AutoConfig run IAX

Issue:

We are upgrading our current 11i instance to 12,1,3 and as part of upgrade we executed 12.1.1 driver successfully. But later when I tried to configure application using rapidwiz it is continuously failing. The environment working in AIX.

Found below error in adconfig.log

Starting AutoConfig at Tue Dec 10 06:00:28 2013
Using adconfig.pl version 120.26

    Script execution phase specified: INSTE8_SETUP
        Classpath                   :

Version Conflicts among development maintained and customized templates encountered; aborting AutoConfig run.

Also found below errors in rapidwiz logfile $INST_TOP/logs/12100618.log

AutoConfig will consider the custom templates if present.

        Using CONFIG_HOME location     : /u01/oracle/test/inst/apps/test_tesdb

ERROR: Invalid java version found:

        JVMJ9VM015W Initialization error for library j9dmp24(0): JVMJ9VM009E J9VMDllMain failed

Could not create the Java virtual machine.

ERROR: Invalid java version found:

        JVMJ9VM015W Initialization error for library j9dmp24(0): JVMJ9VM009E J9VMDllMain failed

Could not create the Java virtual machine.

       Classpath                   :

Version Conflicts among development maintained and customized templates encountered; aborting AutoConfig run.

Cause:

Before running rapidwix we have executed 12.1.1 upgrade driver by setting the below parameter 

export LDR_CNTRL=MAXDATA=0x40000000

Solution:

The parameter LDR_CNTRL=MAXDATA=0x40000000 should be set only if we run large jobs in AIX

unset LDR_CNTRL as below

$ unset LDR_CNTRL