adop fs_clone failed after cutover port issue

Issue:

FS_CLONE failed during vallidation and below are errors in Validation logfiles
Started execution   : ADOPValidations.java
Node Name                       File Edition                    Port Name                       Port Value
----------------------------------------------------------------------------------------------------------------------------
ebs001                      run                     s_java_object_cache_port                        12366
ebs002                      run                     s_java_object_cache_port                        12367
ebs001                      patch                   s_java_object_cache_port                        12366
ebs002                      patch                   s_java_object_cache_port                        12367

ERROR:
The following ports are out of sync on RUN Edition-> [s_java_object_cache_port]
Corrective Action: update the context files so that all the erroneous ports have same value across all nodes
and run AutoConfig to sync with the value in the database.
The following ports are out of sync on PATCH Edition-> [s_java_object_cache_port]
Corrective Action: update the context files so that all the erroneous ports have same value across all nodes
and  run AutoConfig with the -syncctx option to sync with the value in the database.

Fix:
There is a port mismatch for variable s_java_object_cache_port. Based on port pool, values should be as below:
Node Name                       File Edition                    Port Name                       Port Value
----------------------------------------------------------------------------------------------------------------------------
ebs001                      run                     s_java_object_cache_port                        12367
ebs002                      run                     s_java_object_cache_port                      12367
ebs001                      patch                   s_java_object_cache_port                       12366
ebs002                      patch                   s_java_object_cache_port                    12366

1) On ebs001 update run context file with correct port value 12367 & run autoconfig.sh
2) On ebs002 update patch contextfile and and adconfig with syncctc option as below
$ sh $AD_TOP/bin/adconfig.sh contextfile=$CONTEXT_FILE -syncctx

adop fs_clone fails with error - The following nodes are not in sync

Issue:
EBS Application is multi node setup and a patch has been applied by DBA from Primary node using adop with hotpatch option. The patch is successful in one node and failed on other node. This is not observed by the DBA and he proceeded with other patch as well and that has also failed on second node.

DBA identified the issue on secondary node and fixed the issue with second node.

DBA fixed issue on Node2 and tried to apply patch on Slave node with allnodes=no

adop phase=apply patches=20012197 workers=8 hotpatch=yes allnodes=no action=nodb

adop returned successfully updated that Patch is successfull.

Now we need to sync the filesystems and tried to fs_clone on all nodes as below.

$ adop phase=fs_clone

Unfortunately fs_clone has failed with below errors..
Checking for pending adop sessions...
    No pending session exists.
    Staging new adop session...
    [UNEXPECTED]The following nodes are not in sync : oraapp02    Please bring the nodes in sync and then continue    [UNEXPECTED]Unrecoverable error occured. Exiting the current session.

DBA Tried to run prepare/cleanup and none of them helped.. all these got failed.

Solution:

Even though patch has been applied on failed nodes and it showed as successfull. ADOP activity didn't updated patchrun_id in table AD_ADOP_SESSION_PATCHES.
Below is the output seen:
select bug_number, patchrun_id, node_name from  AD_ADOP_SESSION_PATCHES where bug_number in ('20126243')
BUG_NUMBER                     PATCHRUN_ID            NODE_NAME                      
------------------------------ ---------------------- -------------------------------
20126243                       24398                  oraapp01                     
20126243                       -1                  oraapp02                     
   
Based on output patchrun_id for oraapp02 should be 24398 instead it is showing as "-1".

Before are two solutions to fix this issue:

Sol 1: 
If Patch that is applied is a small oneoff patch try applying patch on all nodes with force option & nodbportion as below

$ adop phase=apply  patches=20126243 workers=8 hotpatch=yes options=forceapply,nodatabaseportion

This step will update patchrun_id in table AD_ADOP_SESSION_PATCHES

Sol 2: 
Manually update patchrun_id for node 2 as value same as that of node1
SQL> update table AD_ADOP_SESSION_PATCHES set patchrun_id=24398 where bug_number='<bug_number>' and session_id='<adop_sess_id>' and node_name='oraapp02';
SQL> commit;

Run fs_clone now.

ADOP Failes with "Unable to find appltop_id for the host from database"

Issue:

Our EBS 12.2 application has been cloned recently from PROD and this is a shared application tier. User requested to apply a patch and when we tried to apply the patch adop is coming with errors. Below are the error details.

[UNEXPECTED]Unable to find appltop_id for the host oraap02 from database
[UNEXPECTED]Unable to find appltop_id for the host oraap02 from database
[EVENT]     [START 2015/04/10 02:23:17] Verify SSH
Logfile location /u01/app/fs1/inst/apps/EBSR12_shared/logs/appl/rgf/TXK/verifyssh.log
xml output = /u01/app/fs1/inst/apps/EBSR12_shared/logs/appl/rgf/TXK/out.xml
[EVENT]     [END   2015/04/10 02:23:20] Verify SSH
All nodes have ssh connectivity. Continuing...
[EVENT]     [START 2015/04/10 02:23:20] Checking the DB parameter value
[EVENT]     [END   2015/04/10 02:23:23] Checking the DB parameter value  [UNEXPECTED]Unable to find appltop_id for the host oraap02 from database
  [UNEXPECTED]Invalid appltop id: "".
  [UNEXPECTED]Unrecoverable error occured. Exiting the current session.
  Log file: /u01/app/fs_ne/EBSapps/log/adop/adop_20150410_022145.log

The above error doesn't give much clear information on what was causing issue. I started reviewing & validating the below tables and everything shows good.

FND_NODES
FND_OAM_CONTEXT_FILES
AD_APPL_TOPS
AD_ADOP_SESSIONS

Then as a first attempt I ran autoconfig on Node 1 "oraapp01" and tried to restart adop again but this time with fs_clone option. This again printed same errors again(pointing to node 2) but luckily it went ahead with fs_clone.

[EVENT]     [START 2015/04/10 22:54:51] Checking the DB parameter value
[EVENT]     [END   2015/04/10 22:54:54] Checking the DB parameter value
    [UNEXPECTED]Unable to find appltop_id for the host oraapp02 from database
    [UNEXPECTED]Unable to find appltop_id for the host oraapp02 from database
 but fs_clone failed at Validation phase and the log file <ADPLOGHOME>/fs_clone_20150410_224816/EBSR12_oraap01/remote_execution_result_20150410_225611.xml showed below details


[ERROR]: Either the value of the context variables_shared_file_system in the run file system is not consistent between the file system and the database or the APPL_TOP name across nodes is not set correctly.

Solution:

This made me think about APPL_TOP Name again. I reviewed my context files on both the application tiers (Primary & Secondary) and found that the context variable s_atName is different on both the nodes.

For a shared Applicaiton Tier APPL_TOP name should be same across all the applicaiton tier nodes.
1) Modified context variable s_atName on Secondary node to Master Application Nodename .
2) Executed autoconfig and bounced services.
3) Then adop is went successfully.