adop fs_clone/prepare fails with [UNEXPECTED]Invalid worker Count: 0

We recently added a node to our current EBS application. As part of the node addition steps, tried to executed adop phase=fs_clone but it failed immediately with errors.


Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:
Validating credentials.
Initializing.
    Run Edition context  : /app/EBD//fs2/inst/apps/EBD_oraapp01/appl/admin/EBD_oraapp01.xml
    Patch edition context: /app/EBD//fs1/inst/apps/EBD_oraapp01/appl/admin/EBD_oraapp01.xml
    Patch file system free space: 41.85 GB
Validating system setup.
    [UNEXPECTED]Invalid worker Count: 0
    [UNEXPECTED]Error validating worker count

Later tried to assign worker count of value 8 manually as below but it also failed
  $ adop phase=fs_clone workers=8
 
Validating credentials.
Initializing.
    Run Edition context  : /app/EBD//fs2/inst/apps/EBD_oraapp01/appl/admin/EBD_oraapp01.xml
    Patch edition context: /app/EBD//fs1/inst/apps/EBD_oraapp01/appl/admin/EBD_oraapp01.xml
    Patch file system free space: 41.85 GB
    [UNEXPECTED]Specified value 8 is not supported by the database.
    [UNEXPECTED]A maximum of 1 workers can be invoked.
    [UNEXPECTED]Error validating worker count

Observed the same issue with prepare phase as well.

Solution:
The default worker count information(recomm & max)  is stored in a file adpawc.xml that can be found under $APPL_TOP/admin/$TWO_TASK/log

Somehow this file has been modified with recommended value as 0 and max value as 1 during node addition.

FileContent:
<?xml version="1.0"?>
<WORKER_COUNT>
        <RECOMMENDED>0</RECOMMENDED>
        <MAX>1</MAX></WORKER_COUNT>

Update recommended &  max value based on cpu count.

Here I updated recommended value as 8 and max value as 64 and saved the file.

Once the changes are made, fs_clone went smoothly.

adop fs_clone failed after cutover port issue

Issue:

FS_CLONE failed during vallidation and below are errors in Validation logfiles
Started execution   : ADOPValidations.java
Node Name                       File Edition                    Port Name                       Port Value
----------------------------------------------------------------------------------------------------------------------------
ebs001                      run                     s_java_object_cache_port                        12366
ebs002                      run                     s_java_object_cache_port                        12367
ebs001                      patch                   s_java_object_cache_port                        12366
ebs002                      patch                   s_java_object_cache_port                        12367

ERROR:
The following ports are out of sync on RUN Edition-> [s_java_object_cache_port]
Corrective Action: update the context files so that all the erroneous ports have same value across all nodes
and run AutoConfig to sync with the value in the database.
The following ports are out of sync on PATCH Edition-> [s_java_object_cache_port]
Corrective Action: update the context files so that all the erroneous ports have same value across all nodes
and  run AutoConfig with the -syncctx option to sync with the value in the database.

Fix:
There is a port mismatch for variable s_java_object_cache_port. Based on port pool, values should be as below:
Node Name                       File Edition                    Port Name                       Port Value
----------------------------------------------------------------------------------------------------------------------------
ebs001                      run                     s_java_object_cache_port                        12367
ebs002                      run                     s_java_object_cache_port                      12367
ebs001                      patch                   s_java_object_cache_port                       12366
ebs002                      patch                   s_java_object_cache_port                    12366

1) On ebs001 update run context file with correct port value 12367 & run autoconfig.sh
2) On ebs002 update patch contextfile and and adconfig with syncctc option as below
$ sh $AD_TOP/bin/adconfig.sh contextfile=$CONTEXT_FILE -syncctx

adop fs_clone fails with error - The following nodes are not in sync

Issue:
EBS Application is multi node setup and a patch has been applied by DBA from Primary node using adop with hotpatch option. The patch is successful in one node and failed on other node. This is not observed by the DBA and he proceeded with other patch as well and that has also failed on second node.

DBA identified the issue on secondary node and fixed the issue with second node.

DBA fixed issue on Node2 and tried to apply patch on Slave node with allnodes=no

adop phase=apply patches=20012197 workers=8 hotpatch=yes allnodes=no action=nodb

adop returned successfully updated that Patch is successfull.

Now we need to sync the filesystems and tried to fs_clone on all nodes as below.

$ adop phase=fs_clone

Unfortunately fs_clone has failed with below errors..
Checking for pending adop sessions...
    No pending session exists.
    Staging new adop session...
    [UNEXPECTED]The following nodes are not in sync : oraapp02    Please bring the nodes in sync and then continue    [UNEXPECTED]Unrecoverable error occured. Exiting the current session.

DBA Tried to run prepare/cleanup and none of them helped.. all these got failed.

Solution:

Even though patch has been applied on failed nodes and it showed as successfull. ADOP activity didn't updated patchrun_id in table AD_ADOP_SESSION_PATCHES.
Below is the output seen:
select bug_number, patchrun_id, node_name from  AD_ADOP_SESSION_PATCHES where bug_number in ('20126243')
BUG_NUMBER                     PATCHRUN_ID            NODE_NAME                      
------------------------------ ---------------------- -------------------------------
20126243                       24398                  oraapp01                     
20126243                       -1                  oraapp02                     
   
Based on output patchrun_id for oraapp02 should be 24398 instead it is showing as "-1".

Before are two solutions to fix this issue:

Sol 1: 
If Patch that is applied is a small oneoff patch try applying patch on all nodes with force option & nodbportion as below

$ adop phase=apply  patches=20126243 workers=8 hotpatch=yes options=forceapply,nodatabaseportion

This step will update patchrun_id in table AD_ADOP_SESSION_PATCHES

Sol 2: 
Manually update patchrun_id for node 2 as value same as that of node1
SQL> update table AD_ADOP_SESSION_PATCHES set patchrun_id=24398 where bug_number='<bug_number>' and session_id='<adop_sess_id>' and node_name='oraapp02';
SQL> commit;

Run fs_clone now.

12.2 oacore managed server starts in ADMIN mode - java.lang.ClassNotFoundException

Issue:

EBS 12.2 Application services are bounced as part of regular maintenance. During startup oacore_server1 Managed server started in ADMIN mode rather than RUNNING mode.

Below are the errors found in oacore_server1.log file.

Caused By: java.lang.ClassNotFoundException: oracle.apps.bne.integrator.document.ooxml.BneCreateDocumentService

Solution:

web.xml has some how updated with incorrect values for servlet class.

At Line 341 of file

$FMW_HOME/Oracle_EBS-app1/applications/oacore/html/WEB-INF/web.xml Below is the found value:

<servlet-class>oracle.apps.bne.integrator.document.ooxml.BneCreateDocumentService</servlet-class>

Update the value with below change:

<servlet-class>oracle.apps.fnd.txk.util.TXKSubstituteProductServlet</servlet-class>

Save changes and bounce oacore_server1 managed server.

fs_clone failed during adopValidations

Issue

Found the below errors in  the adop validation file.. adopValidations_detailed.log under $ADOP_LOG_DIR/3/fs_clone_XXX/ 

-----------------------------

ERROR: The following required ports are in use:

-----------------------------

7812 : Managed Server Port 11

7412 : Managed Server Port 7

7612 : Managed Server Port 5

7212 : Managed Server Port 3

Corrective Action: Free the listed ports and retry the adop operation.

Completed execution : ADOPValidations.java

Found that Patch Context File in Database has incorrect port number for Admin Servers, Managed ports.

Fix:

Updated the correct ports in context file and uploaded context file to database.

$ADJVAPRG oracle.apps.ad.autoconfig.oam.CtxSynchronizer action=upload  contextfile=<Path of Context File> logfile=/tmp/patchctxupload15.log

 

Data not Loading on Endeca application pages

Performed fresh Endeca setup on already existing setup, But after the setup is completed below are the few issues identified

1)     Data is not loaded on Endeca pages, only layout is displayed.

2)     Pages related to information Discovery like datasources,framework settings,etc., in Endeca Studio under Control Panel is showing blank

3)     Formatting of Endeca page is not correct

StudioManagedServer.log file showed below errors:

"53808" Local Address: "/192.168.1.1" Local Port: "8004" Protocol: "http">

####<Jul 6, 2015 6:26:26 AM CDT> <Error> <HTTP> <endeca.oracle.com> <StudioManagedServer> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1436181986656> <BEA-101019> <[ServletContext@992134090[app:studio-portal module:/endeca-datasource-config-portlet/ path:/endeca-datasource-config-portlet spec-version:2.5]] Servlet failed with IOException

java.io.FileNotFoundException: /tmp/liferay/minifier/endeca-datasource-config-portlet/js/DatasourceWindow.js_Q_browserId=ie&minifierType=js&languageId=en_US&t=1436178201000_E_CONTENT_TYPE (Permission denied)

Fix:

1)     Provide permissions to Endeca Unix owner on /tmp

2)     Restart Endeca Services

Endeca Extension HTTP-404 Page not found

Issue – HTTP-404 Page not found

User is trying to navigate to Endeca pages and receives http-404 page not found error. Below is the screenshot.

Fix:

Verify the  profile option FND_ENDECA_PORTAL_URL and make sure it is correctly set.

Endeca Extensions There was an error when trying to access the data source

Issue 1- Datasource Error

There was an error when trying to access the data source. Please confirm with your system administrator that the data source is available.

Below error is observed on Endeca Pages when a clone of EBS Instance is completed.

  Below are the few causes for the issue:

1)      The source EBS may not have Endeca application configured.

2)      Not executed Concurrent program “Populate Endeca Metadata for iProcurement” after any metadata changes performed on Endeca side <-- This is specific to iProcurment page

3)      Endeca data source ebsdb is not listing target managed servers

Launch Console of Endeca Server -> Datasources -> click on datasource ebsdb ->  Monitoring -> Testing. This screen should list the Studio Managed server.

Studio log file reported below errors: (/u01/Oracle/Middleware/user_projects/domains/endeca_domain/servers/StudioManagedServer/logs/StudioManagedServer.log)

####<Jun 8, 2015 10:18:46 PM CDT> <Error> <Deployer> <hpeidrd01.hpinc.com> <StudioManagedServer> <[ACTIVE] ExecuteThread: '17' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000KrMKwR45yW^5xVWByW1LT_gK000000> <1433819926781> <BEA-149205> <Failed to initialize the application 'ebsdb' due to error weblogic.application.ModuleException: .

weblogic.application.ModuleException:

        at weblogic.jdbc.module.JDBCModule.prepare(JDBCModule.java:327)

Caused By: weblogic.common.resourcepool.ResourceSystemException:

 Could not connect to 'oracle.apps.fnd.ext.jdbc.datasource.AppsDataSource'.

 The returned message is: ORA-01017: invalid username/password; logon denied

 It is likely that the login or password is not valid.

 It is also possible that something else is invalid in

 the configuration or that the database is not available.

        at weblogic.jdbc.common.internal.JDBCUtil.parseException(JDBCUtil.java:301)

        at weblogic.jdbc.common.internal.PooledConnectionEnvFactory.makeConnection(PooledConnectionEnvFactory.java:317)

        at weblogic.jdbc.common.internal.PooledConnectionEnvFactory.createResource(PooledConnectionEnvFactory.java:97)

        at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1310)

Fix:

The password of EBS User ENDECADM in console is incorrect. Update the correct password for user ENDECADM in console to correct value and save it.

Console -> Lock & Edit in change center -> Datasource -> click on ebsdb -> configuration -> Connection Pool.

Update the password to correct value  and click on Save.

Activate Changes on control center

Once the changes are activated we should be able to see the Managed Server.

 

adop phase=prepare failed with ERROR: Script failed, exit code 255 while creating WLS domain

Issue: 
We are trying to run adop prepare phase on our EBS 12.2 Patch file system and it came out with below errors in FSCloneApplyAppsTier_08200431.log file.

START: Creating new WLS domain.
Running /u01/app/EBSDB/fs2/FMW_Home/oracle_common/bin/pasteConfig.sh -javaHome /u01/app/EBSDB/fs2/EBSapps/comn/util/jdk64 -al /u01/app/EBSDB/fs1/EBSapps/comn/adopclone_ebsdb01/FMW/WLS/EBSdomain.jar -tdl /u01/app/EBSDB/fs2/FMW_Home/user_projects/domains/EBS_domain_EBSDB -tmw /u01/app/EBSDB/fs2/FMW_Home -mpl /u01/app/EBSDB/fs1/EBSapps/comn/adopclone_ebsdb01/FMW/WLS/plan/moveplan.xml -ldl /u01/app/EBSDB/fs1/inst/apps/EBSDB_ebsdb01/admin/log/clone/wlsT2PApply -silent true -debug true -domainAdminPassword /u01/app/EBSDB/fs1/EBSapps/comn/adopclone_ebsdb01/FMW/tempinfo.txtScript Executed in 1602 milliseconds, returning status 255
ERROR: Script failed, exit code 255


The above information didn't give sufficient information and had to review CLONE2015-08-20_04-31-47AM.log &  CLONE2015-08-20_04-31-47AM.error logfiles under $ADOP_LOG_HOME/22/<prepare..>/<CONTEXT_NAME>/ directory.  These have provided below additional information which helped in resolving the issue.

CLONE2015-08-20_04-31-47AM.error:

SEVERE : Aug 20, 2015 4:31:49 AM - ERROR - CLONE-20372   Server port validation failed.
SEVERE : Aug 20, 2015 4:31:49 AM - CAUSE - CLONE-20372   Ports of following servers - AdminServer(7004) - are not available.  <-- This port is of Run filesystem instead it should look for 7005
SEVERE : Aug 20, 2015 4:31:49 AM - ACTION - CLONE-20372   Provide valid free ports.
oracle.as.t2p.exceptions.FMWT2PPasteConfigException: PasteConfig failed. Make sure that the move plan and the values specified in moveplan are correct.

CLONE2015-08-20_04-31-47AM.log:FINE : Aug 20, 2015 4:31:49 AM - [J2EEGenericValidationUtil:validateAndGetServerPort] Listen Port(7805) of server forms-c4ws_server1 is valid and free.FINE : Aug 20, 2015 4:31:49 AM - [J2EEGenericValidationUtil:validateServerConfig] Mapping of ports vs servers from moveplan = {AdminServer=[7004], forms-c4ws_server1=[7805], forms_server1=[7405], oacore_server1=[7205], oafm_server1=[7605]} FINE : Aug 20, 2015 4:31:49 AM - [J2EEGenericValidationUtil:validateServerConfig] Mapping of temporary ports assigned vs servers = {}
FINE : Aug 20, 2015 4:31:49 AM - [J2EEGenericValidationUtil:validateServerConfig] map_serverName_vs_actualPort_vs_TempPort = {}

Above highlighted area confirms that except AdminServer port all the other ports are properly picked up.

Additional info noticed in FSCloneApplyAppsTier_08200431.log
m_LstnAddrPrtsDBMap size = 6
Total no. of configGroup = 6
 configGroup Type name = SERVER_CONFIG
configProperty id = Server1
Count for NodeIterator nextNode = 3

Input for AdminServer Listen Address OR Port is NULL
Moveplan won't be updated for AdminServer Listen Address
Moveplan won't be updated for AdminServer Listen Port+++++++++++++++++++++++++
configProperty id = Server2
Count for NodeIterator nextNode = 3
Input for oacore_server1 Listen Address OR Port is NULL
 Updating oacore_server1 Listen Address to ushtc0app02.jacobs.com
 Updating oacore_server1 Listen Port to 7205

These highlighted details above confirmed that adop is unable to update AdminPort to correct value as part of its preparations.

Fix:
Upon further investigation, Listen Address value for AdminServer on Run Filesystem is removed manually.

1) Login to weblogic console of Run  Filesystem. Perform Lock & Edit. Then Navigate to Domain Structure -> Servers -> Select Adminserver.
2) Under Configuration screen -> Update Listen Address value to application server hostname(FQDN).
3) Bounce all the application services.
4) Run adop phase=prepare.

Workaround:
If you want the Listen address value to be Null. Run fs_clone for the EBS Apps. Then proceed with adop prepare phase.

ADOP Failes with "Unable to find appltop_id for the host from database"

Issue:

Our EBS 12.2 application has been cloned recently from PROD and this is a shared application tier. User requested to apply a patch and when we tried to apply the patch adop is coming with errors. Below are the error details.

[UNEXPECTED]Unable to find appltop_id for the host oraap02 from database
[UNEXPECTED]Unable to find appltop_id for the host oraap02 from database
[EVENT]     [START 2015/04/10 02:23:17] Verify SSH
Logfile location /u01/app/fs1/inst/apps/EBSR12_shared/logs/appl/rgf/TXK/verifyssh.log
xml output = /u01/app/fs1/inst/apps/EBSR12_shared/logs/appl/rgf/TXK/out.xml
[EVENT]     [END   2015/04/10 02:23:20] Verify SSH
All nodes have ssh connectivity. Continuing...
[EVENT]     [START 2015/04/10 02:23:20] Checking the DB parameter value
[EVENT]     [END   2015/04/10 02:23:23] Checking the DB parameter value  [UNEXPECTED]Unable to find appltop_id for the host oraap02 from database
  [UNEXPECTED]Invalid appltop id: "".
  [UNEXPECTED]Unrecoverable error occured. Exiting the current session.
  Log file: /u01/app/fs_ne/EBSapps/log/adop/adop_20150410_022145.log

The above error doesn't give much clear information on what was causing issue. I started reviewing & validating the below tables and everything shows good.

FND_NODES
FND_OAM_CONTEXT_FILES
AD_APPL_TOPS
AD_ADOP_SESSIONS

Then as a first attempt I ran autoconfig on Node 1 "oraapp01" and tried to restart adop again but this time with fs_clone option. This again printed same errors again(pointing to node 2) but luckily it went ahead with fs_clone.

[EVENT]     [START 2015/04/10 22:54:51] Checking the DB parameter value
[EVENT]     [END   2015/04/10 22:54:54] Checking the DB parameter value
    [UNEXPECTED]Unable to find appltop_id for the host oraapp02 from database
    [UNEXPECTED]Unable to find appltop_id for the host oraapp02 from database
 but fs_clone failed at Validation phase and the log file <ADPLOGHOME>/fs_clone_20150410_224816/EBSR12_oraap01/remote_execution_result_20150410_225611.xml showed below details


[ERROR]: Either the value of the context variables_shared_file_system in the run file system is not consistent between the file system and the database or the APPL_TOP name across nodes is not set correctly.

Solution:

This made me think about APPL_TOP Name again. I reviewed my context files on both the application tiers (Primary & Secondary) and found that the context variable s_atName is different on both the nodes.

For a shared Applicaiton Tier APPL_TOP name should be same across all the applicaiton tier nodes.
1) Modified context variable s_atName on Secondary node to Master Application Nodename .
2) Executed autoconfig and bounced services.
3) Then adop is went successfully.

EBS 12.2 Unable to start up the managed server oacore_server1 - after clone

Unable to Start OACORE Services during clone on RUN Filesystem

Recently we did a clone and as part of execution of adcfgclone.pl in run Filesystem, I responded to adcfgclone.pl to start Application Filesystems. But I observed the start of Managed servers failed and below is the output from one of oacore services.

Successfully Connected to Node Manager.

The Node Manager is already up.
Starting server oacore_server1 ...
Error Starting server oacore_server1: weblogic.nodemanager.NMException: Exception while starting server 'oacore_server1'

ERROR: Unable to start up the managed server oacore_server1

05/20/15-00:20:43 :: admanagedsrvctl.sh: exiting with status 1

This information is not sufficient to identify the cause of the issue and I had go through the oacore log files under $EBS_DOMAIN_HOME/servers/oacore_server1/logs

9ee0eb174c5acfd:2fe0c70b:14d72092155:-8000-0000000000000eda> <1432203790547> <BEA-100028> <Could not deserialize session data.
oracle.apps.fnd.common.PoolException: Exception creating new Poolable object.
        at oracle.apps.fnd.common.Pool.createObject(Pool.java:1289)
..
 <<WLS Kernel>> <> <> <1432099154476> <BEA-149205> <Failed to initialize the application 'EBSDataSource' due to error weblogic.application.ModuleException: .
..
java.lang.NoClassDefFoundError: Could not initialize class oracle.apps.fnd.common.WebAppsContext
        at oracle.apps.fnd.security.AppsServletFilter.init(AppsServletFilter.java:148)
..
<Could not load user defined filter in web.xml: oracle.apps.jtf.cabo.interceptor.JTFWrapperFilter.
java.lang.NullPointerException
..

..
..

Caused By: weblogic.common.resourcepool.ResourceSystemException:
 Could not connect to 'oracle.jdbc.OracleDriver'.

 The returned message is: ORA-01017: invalid username/password; logon denied

 It is likely that the login or password is not valid.
 It is also possible that something else is invalid in the configuration or that the database is not available.




Then this gave clear information which says that the apps password provided during adcfgclone.pl is noot being accepted even though the password provided is correct.

Later it is observed that Apps Password is changed as part of execution of adcfgclone.pl on DBTier and theis new password change is somehow not reflected when Managed servers are created during execution of rapidclone on Apps Tier.

Now this is fixed manually by performing below steps.

As ADMIN Server is up and runnning,

Login to EBS Weblogic console -> Perform Lock & Edit -> Under Domain Structure -> Services -> Datasources -> Select "EBSDataSource"

Navigate to Configuration -> Connection pool  -> Update with new password in password fields.

Then Save changes.

Perform the above steps of other Datasource OAADatasource.

Now Activate the changes in Control Center.

This should resolve the issue. Bring up the managed servers from Console or using command line.

Adop is not to able to detect any valid application tier nodes

Adop is not to able to detect any valid application tier nodes

Issue:

Running ADOP on 12.2 is failing at initial Validation with error “adop is not able to detect any valid application tier nodes”
Enter the APPS password: 
Enter the SYSTEM password: 
Enter the WLSADMIN password: 

Validating credentials... 

Initializing... 
Run Edition context : /u01/oracle/VIS/fs1/inst/apps/VIS_ora01/appl/admin/VIS_ora01.xml 
Patch edition context: / u01/oracle/VIS/fs2/inst/apps/VIS_ora01/appl/admin/VIS_ora01.xml 
*******FATAL ERROR******* 
PROGRAM : (/u01/oracle/VIS /fs1/EBSapps/appl/ad/12.0.0/bin/adzdoptl.pl) 
TIME : Wed Apr 22 04:40:38 2015 
FUNCTION: ADOP::GlobalVars::_GetMandatoryArgs [ Level 1 ] 
ERRORMSG: adop is not able to detect any valid application tier nodes in 
ADOP_VALID_NODES table. Ensure autoconfig is run on all nodes. 


[STATEMENT] Please run adopscanlog utility, using the command 

"adopscanlog -latest=yes" 

to get the list of the log files along with snippet of the error message corresponding to each log file. 


adop exiting with status = 255 (Fail) 

Cause

Upon investigation we found that Database listener is not listening ebs_patch service which is mandatory for 12.2 to work on any adop activities.

[oracle@ora01 ~]$ lsnrctl status $ORACLE_SID

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 23-APR-2015 12:20:02

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

Alias                     VIS

Listening Endpoints Summary...

.

.

Services Summary...

Service "VIS" has 2 instance(s).

  Instance "VIS", status UNKNOWN, has 1 handler(s) for this service...

  Instance "VIS", status READY, has 1 handler(s) for this service...

The command completed successfully

Solution

1)     Make sure that Database init parameter has service name ebs_patch defined

2)     Connect to Database as sysdba

sqlplus / as sysdba

SQL> show parameter service_name

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

service_names                        string      VIS, ebs_patch

                If this is there, set the parameter as below

                SQL> alter system set service_names=’ebs_patch’,’VIS’ scope=spfile;

3)     Verify init parameter LOCAL_LISTENER and  set it to <DB_NAME>_LOCAL

                                 SQL> alter system set local_listener=’VIS_LOCAL’ scope=spfile;

4)     Bounce the application & database services