I am running v8.5 on VMWARE - windows 2012 R2 servers. SCS is running on framework server and time to time losts connection to any LCA running on any other server...  Connection lost recovers by itself in a second or so but alerts are generated,,,  There is absolutely no pattern of failures, it can happen any time of the day on any LCA/server ....  Seems like a hiccup on the newtwork but all servers are on VMWare, there is no real physical network...    Does anyone had similar issue?   any help is greatly appreciated.


-Ap[65617]-<-864 @18:16:46.2390
ADDP: closing 588 - timed out (10000 msec), remote socket [364]
18:16:47.707 Notify Host: HOST_STATUS_UNAVAILABLE {117, W*********}
18:16:47.707   Trace   w****fr**1   SCS   GCTI-43-20511   Disconnected from LCA [W*********] on port [4999], socket [588]
18:16:47.707 ### LOCAL MESSAGE#=20511, 'Disconnected from LCA [W*********] on port [4999], socket [588]'
18:16:47.707   Standard   w****fr**1   SCS   GCTI-00-08002   Host 'W*********' unavailable
18:16:47.707 ### LOCAL MESSAGE#=8002, 'Host 'W*********' unavailable'
18:16:47.707 CREATE ActiveAlarm  <107, Host Unavailable> for  App=<124, SCS>, createTime=1499822207 under control=1
18:16:47.707   Trace   w****fr**1   SCS   GCTI-43-10612   Active Alarm 'Host Unavailable' created for Application 'SCS'
18:16:47.707 ### LOCAL MESSAGE#=10612, 'Active Alarm 'Host Unavailable' created for Application 'SCS' '
18:16:47.707   Alarm   w****fr**1   SCS   GCTI-00-04115   Host Unavailable, alarm condition(107)
18:16:47.707 ### LOCAL MESSAGE#=4115, 'Host Unavailable, alarm condition(107)'
18:16:47.707   Alarm   w****fr**1   SCS   GCTI-00-04200   Active Alarm "Host Unavailable" Created (ID=4D17FA21-1EB7-44E1-BE4E-FD5927583D1B)
18:16:47.707 ### LOCAL MESSAGE#=4200, 'Active Alarm "Host Unavailable" Created (ID=4D17FA21-1EB7-44E1-BE4E-FD5927583D1B)'
18:16:47.707 ActiveAlarm::SetDeleteTimer  <107, Host Unavailable> for App=<124, SCS>, LastSetTime=1499822207, timeout=172800
18:16:47.707 Found Condition <107, Host Unavailable>, Detect Message = 8002
18:16:47.707 !Execute all Reaction for AlarmCondition <107, Host Unavailable>
18:16:47.707   Trace   w****fr**1   SCS   GCTI-43-10500   Reaction 'Send Email Alert' activated
18:16:47.707 ### LOCAL MESSAGE#=10500, 'Reaction 'Send Email Alert' activated'
18:16:47.707 Reaction text: An alarm has been detected by Genesys Solution Management Layer in the n******** environment.

Message: Host 'W*********' unavailable

18:17:15.207 ****************************************************
18:17:15.207   Trace   w****fr**1   SCS   GCTI-43-20510   Connected to LCA [W*********] on port [4999], socket [664]
18:17:15.207 ### LOCAL MESSAGE#=20510, 'Connected to LCA [W*********] on port [4999], socket [664]'
18:17:15.207   Standard   w****fr**1   SCS   GCTI-00-08001   Host 'W*********' operates in normal condition
18:17:15.207 ### LOCAL MESSAGE#=8001, 'Host 'W*********' operates in normal condition'
18:17:15.207 Found Condition <108, Host Unreachable>, Removal Message = 8001
18:17:15.207 Found Condition <107, Host Unavailable>, Removal Message = 8001
18:17:15.207 !Execute all Reaction for AlarmCondition <107, Host Unavailable>
18:17:15.207   Alarm   w****fr**1   SCS   GCTI-00-04210   Active Alarm "Host Unavailable" Cleared (ID=4D17FA21-1EB7-44E1-BE4E-FD5927583D1B)
18:17:15.207 ### LOCAL MESSAGE#=4210, 'Active Alarm "Host Unavailable" Cleared (ID=4D17FA21-1EB7-44E1-BE4E-FD5927583D1B)'
18:17:15.207 ~ActiveAlarm  <107, Host Unavailable> for  App=<124, SCS>, FistCreateTime=1499822207, SetTime=1499822207, deleteTime=1499822235
18:17:15.207   Trace   w****fr**1   SCS   GCTI-43-10611   Active Alarm 'Host Unavailable' deleted for Application 'SCS'
18:17:15.207 ### LOCAL MESSAGE#=10611, 'Active Alarm 'Host Unavailable' deleted for Application 'SCS''
18:17:15.207 Found Condition <105, Host Inaccessible>, Removal Message = 8001
18:17:15.207 Registered on LCAServer 'W*********' port=4999 (fd=664)
18:17:15.207 Notify Host: HOST_STATUS_RUNNING {117, W*********}
18:17:15.207   Standard   w****fr**1   SCS   GCTI-00-08009   NTP service is available on the host 'W*********'.
18:17:15.207 ### LOCAL MESSAGE#=8009, 'NTP service is available on the host 'W*********'.'
18:17:15.207 REventHostInfo {117, W*********}
18:17:15.207
-AP[148760]->-880 @18:17:16.2700


 

This really seems like problem on VMWare platform, so find out the root-cause there. On other hand, you can configure Genesys to prevent execute switchover in case short outage on SCS<->LCA connection, but it will not fix the issue at all.

If these VM's are using vMotion it may cause some issue. Remove them from the vMotion process and see if it continues to disconnect.

I faced a similar issue recently and resolved it by configuring ADDP between the host object where the LCA lives, and the SCS.  The default with no ADDP configuration is 9 seconds - try increasing to 15 seconds local/20 or 30 seconds remote and enable the trace so you can monitor it.  Remember to build the ADDP in the options on the host object rather than one of the apps on the host - and this should keep the SCS in communication with the LCA.   
good luck!

ADDP might help keep the connection opened as it sending "keep-alive" packets and prevent resetting the connection by the firewalls. So, it should solve your problem, but on otherhand the problem will be still there (unsolved) and can occurs anytime in the future.

Thank you all for responding.. Current ADDP (at host level) is set to 10/20. I don't want to increase it too high to avoid missing real failures... Currently issue is under investigation at vmware and network level, I will share details once root cause is identified.

Any updates? We just started getting these lost connections. We are on Windows 2008.

Hi Sabrina,
Our VM team is still working on the problem so far they identified some kind of SAN connectivity issue with EMC. In our case, its not  a Genesys issue definitely related to VM platform. And a reminder, do NOT use VMotion at all,  I have official response from Genesys tech, its NOT supported.

​Hello,
I know it's been almost 2 years since the last update on this case, but how did you clear your SCS to LCA issue?? We started to have the same issue last November a few days after Windows and ESX updates.

 On Wednesday night,  I added "addp" settings under the Host's annex tab. Since then, we havn't add any other SIPServer Switchover.

------------------------------
Benoit Dupuis
Hydro Quebec
------------------------------

Original Message:
Sent: 09-07-2017 17:16
From: Senih Demren
Subject: intermittent SCS to LCA connection failures

Hi Sabrina,
Our VM team is still working on the problem so far they identified some kind of SAN connectivity issue with EMC. In our case, its not  a Genesys issue definitely related to VM platform. And a reminder, do NOT use VMotion at all,  I have official response from Genesys tech, its NOT supported.