Genesys Cloud - Main

Hi All,

I have some issues where for example agents are changing to a break status but then after sometime PureCloud is changing their status to available.  Also agents are using WebRTC and are on-queue but after sometime if they try and make or receive a call it fails.  I would believe it is a disconnect issue somewhere but I am trying to locate it.

From the users console logs you can see the "WS Status: Object" every 30 seconds which I believe is the heartbeat between the agents desktop and PureCloud, then it no longer shows in the log and after sometime you can see some errors and "WS: heartbeat timer expired for user xxxxxxxxx".

So I believe this is this issue but just need to understand if this is agent side or PureCloud side?

Does anybody know which direction or how these heartbeats work?

Or has anybody experienced similar behavior?

I do have this raised with Genesys Customer Care

Thanks Luke
#Omni-ChannelDesktop/UserInterface

------------------------------
Luke Mitchell
Conn3ct
------------------------------

I think I have worked out with Wireshark that PureCloud initiates the heartbeat and the agents desktop responds to the heartbeat, however if anybody has any knowledge or previous experience on this type of issue that would be great

------------------------------
Luke Mitchell
Conn3ct
------------------------------

Original Message:
Sent: 06-02-2020 10:46
From: Luke Mitchell
Subject: User Status and WebRTC Connection Heartbeats/Timeouts

Hi All,

I have some issues where for example agents are changing to a break status but then after sometime PureCloud is changing their status to available.  Also agents are using WebRTC and are on-queue but after sometime if they try and make or receive a call it fails.  I would believe it is a disconnect issue somewhere but I am trying to locate it.

From the users console logs you can see the "WS Status: Object" every 30 seconds which I believe is the heartbeat between the agents desktop and PureCloud, then it no longer shows in the log and after sometime you can see some errors and "WS: heartbeat timer expired for user xxxxxxxxx".

So I believe this is this issue but just need to understand if this is agent side or PureCloud side?

Does anybody know which direction or how these heartbeats work?

Or has anybody experienced similar behavior?

I do have this raised with Genesys Customer Care

Thanks Luke
#Omni-ChannelDesktop/UserInterface

------------------------------
Luke Mitchell
Conn3ct
------------------------------

Hi Luke,

Websockets allow for a ping, heartbeat, or other form of keep alive from client to server to ensure connections are active. I have seen this same behavior before, but did not have an opportunity to run wireshark on the agent's machine.

You'd need to monitor if the heartbeat/ping actually gets out. When a heartbeat times out, it seems like it stops trying after a certain number of attempts and then you have to refresh to recover. To determine if this was a local network issue or a purecloud issue, you'd need Genesys to verify on their end and/or you'd need to look for other failures at the local level. Monitoring via wireshark may help, if there is other traffic to inspect for failures.

I just developed a Chrome extension to help with this, but it would only work with the web client and I'm still testing. Wireshark would give you WAY more info.

Thanks,

------------------------------
Daniel McLeod
Qsect LLC
------------------------------

Original Message:
Sent: 06-02-2020 11:16
From: Luke Mitchell
Subject: User Status and WebRTC Connection Heartbeats/Timeouts

I think I have worked out with Wireshark that PureCloud initiates the heartbeat and the agents desktop responds to the heartbeat, however if anybody has any knowledge or previous experience on this type of issue that would be great

------------------------------
Luke Mitchell
Conn3ct

Original Message:
Sent: 06-02-2020 10:46
From: Luke Mitchell
Subject: User Status and WebRTC Connection Heartbeats/Timeouts

Hi All,

I have some issues where for example agents are changing to a break status but then after sometime PureCloud is changing their status to available.  Also agents are using WebRTC and are on-queue but after sometime if they try and make or receive a call it fails.  I would believe it is a disconnect issue somewhere but I am trying to locate it.

From the users console logs you can see the "WS Status: Object" every 30 seconds which I believe is the heartbeat between the agents desktop and PureCloud, then it no longer shows in the log and after sometime you can see some errors and "WS: heartbeat timer expired for user xxxxxxxxx".

So I believe this is this issue but just need to understand if this is agent side or PureCloud side?

Does anybody know which direction or how these heartbeats work?

Or has anybody experienced similar behavior?

I do have this raised with Genesys Customer Care

Thanks Luke
#Omni-ChannelDesktop/UserInterface

------------------------------
Luke Mitchell
Conn3ct
------------------------------

Hi Daniel,

Thank you, that makes sense and I am capturing Wireshark on some of the affected agents which I will then analyse.  I have now been through this on my own desktop so have a good example of the packets and flow between PureCloud and the agents desktop so will see if I can compare the two

Thanks Luke

------------------------------
Luke Mitchell
Conn3ct
------------------------------

Original Message:
Sent: 06-02-2020 19:03
From: Daniel McLeod
Subject: User Status and WebRTC Connection Heartbeats/Timeouts

Hi Luke,

Websockets allow for a ping, heartbeat, or other form of keep alive from client to server to ensure connections are active. I have seen this same behavior before, but did not have an opportunity to run wireshark on the agent's machine.

You'd need to monitor if the heartbeat/ping actually gets out. When a heartbeat times out, it seems like it stops trying after a certain number of attempts and then you have to refresh to recover. To determine if this was a local network issue or a purecloud issue, you'd need Genesys to verify on their end and/or you'd need to look for other failures at the local level. Monitoring via wireshark may help, if there is other traffic to inspect for failures.

I just developed a Chrome extension to help with this, but it would only work with the web client and I'm still testing. Wireshark would give you WAY more info.

Thanks,

------------------------------
Daniel McLeod
Qsect LLC

Original Message:
Sent: 06-02-2020 11:16
From: Luke Mitchell
Subject: User Status and WebRTC Connection Heartbeats/Timeouts

I think I have worked out with Wireshark that PureCloud initiates the heartbeat and the agents desktop responds to the heartbeat, however if anybody has any knowledge or previous experience on this type of issue that would be great

------------------------------
Luke Mitchell
Conn3ct

Original Message:
Sent: 06-02-2020 10:46
From: Luke Mitchell
Subject: User Status and WebRTC Connection Heartbeats/Timeouts

Hi All,

I have some issues where for example agents are changing to a break status but then after sometime PureCloud is changing their status to available.  Also agents are using WebRTC and are on-queue but after sometime if they try and make or receive a call it fails.  I would believe it is a disconnect issue somewhere but I am trying to locate it.

From the users console logs you can see the "WS Status: Object" every 30 seconds which I believe is the heartbeat between the agents desktop and PureCloud, then it no longer shows in the log and after sometime you can see some errors and "WS: heartbeat timer expired for user xxxxxxxxx".

So I believe this is this issue but just need to understand if this is agent side or PureCloud side?

Does anybody know which direction or how these heartbeats work?

Or has anybody experienced similar behavior?

I do have this raised with Genesys Customer Care

Thanks Luke
#Omni-ChannelDesktop/UserInterface

------------------------------
Luke Mitchell
Conn3ct
------------------------------

I'm wondering if you will see anything useful in Wireshark, since the packets are encrypted with TLS for a lot of the traffic...
I just ran a Wireshark capture on my workstation, searched through the packets for various forms of WS Status: Object and came up with nothing, but it was just a quick 2-minute check. But, then, neither did I see those in the Console log.

I did see a "Websocket Heartbeat" message fairly often in the Console log, but it is not in the Wireshark captured packets...probably part of one of the encrypted messages.

Please let us know what you find!

------------------------------
George Ganahl GCP (Genesys Cloud), ICCE
Principal Technology Consultant
Genesys
------------------------------

Original Message:
Sent: 06-03-2020 03:35
From: Luke Mitchell
Subject: User Status and WebRTC Connection Heartbeats/Timeouts

Hi Daniel,

Thank you, that makes sense and I am capturing Wireshark on some of the affected agents which I will then analyse.  I have now been through this on my own desktop so have a good example of the packets and flow between PureCloud and the agents desktop so will see if I can compare the two

Thanks Luke

------------------------------
Luke Mitchell
Conn3ct

Original Message:
Sent: 06-02-2020 19:03
From: Daniel McLeod
Subject: User Status and WebRTC Connection Heartbeats/Timeouts

Hi Luke,

Websockets allow for a ping, heartbeat, or other form of keep alive from client to server to ensure connections are active. I have seen this same behavior before, but did not have an opportunity to run wireshark on the agent's machine.

You'd need to monitor if the heartbeat/ping actually gets out. When a heartbeat times out, it seems like it stops trying after a certain number of attempts and then you have to refresh to recover. To determine if this was a local network issue or a purecloud issue, you'd need Genesys to verify on their end and/or you'd need to look for other failures at the local level. Monitoring via wireshark may help, if there is other traffic to inspect for failures.

I just developed a Chrome extension to help with this, but it would only work with the web client and I'm still testing. Wireshark would give you WAY more info.

Thanks,

------------------------------
Daniel McLeod
Qsect LLC

Original Message:
Sent: 06-02-2020 11:16
From: Luke Mitchell
Subject: User Status and WebRTC Connection Heartbeats/Timeouts

I think I have worked out with Wireshark that PureCloud initiates the heartbeat and the agents desktop responds to the heartbeat, however if anybody has any knowledge or previous experience on this type of issue that would be great

------------------------------
Luke Mitchell
Conn3ct

Original Message:
Sent: 06-02-2020 10:46
From: Luke Mitchell
Subject: User Status and WebRTC Connection Heartbeats/Timeouts

Hi All,

I have some issues where for example agents are changing to a break status but then after sometime PureCloud is changing their status to available.  Also agents are using WebRTC and are on-queue but after sometime if they try and make or receive a call it fails.  I would believe it is a disconnect issue somewhere but I am trying to locate it.

From the users console logs you can see the "WS Status: Object" every 30 seconds which I believe is the heartbeat between the agents desktop and PureCloud, then it no longer shows in the log and after sometime you can see some errors and "WS: heartbeat timer expired for user xxxxxxxxx".

So I believe this is this issue but just need to understand if this is agent side or PureCloud side?

Does anybody know which direction or how these heartbeats work?

Or has anybody experienced similar behavior?

I do have this raised with Genesys Customer Care

Thanks Luke
#Omni-ChannelDesktop/UserInterface

------------------------------
Luke Mitchell
Conn3ct
------------------------------

Hi George,

In the Wireshark you can see the TLS packet come from PureCloud at the time of the WS Status and then the PC sends a TCP ACK response.  So with the affected users, I am going to check firstly that the packet is being recieved from PureCloud and if so check that the PC is sending the ACK back, depending on the outcome will depend where I focus next



Thanks Luke

------------------------------
Luke Mitchell
Conn3ct
------------------------------

Original Message:
Sent: 06-03-2020 05:33
From: George Ganahl
Subject: User Status and WebRTC Connection Heartbeats/Timeouts

I'm wondering if you will see anything useful in Wireshark, since the packets are encrypted with TLS for a lot of the traffic...
I just ran a Wireshark capture on my workstation, searched through the packets for various forms of WS Status: Object and came up with nothing, but it was just a quick 2-minute check. But, then, neither did I see those in the Console log.

I did see a "Websocket Heartbeat" message fairly often in the Console log, but it is not in the Wireshark captured packets...probably part of one of the encrypted messages.

Please let us know what you find!

------------------------------
George Ganahl GCP (Genesys Cloud), ICCE
Principal Technology Consultant
Genesys

Original Message:
Sent: 06-03-2020 03:35
From: Luke Mitchell
Subject: User Status and WebRTC Connection Heartbeats/Timeouts

Hi Daniel,

Thank you, that makes sense and I am capturing Wireshark on some of the affected agents which I will then analyse.  I have now been through this on my own desktop so have a good example of the packets and flow between PureCloud and the agents desktop so will see if I can compare the two

Thanks Luke

------------------------------
Luke Mitchell
Conn3ct

Original Message:
Sent: 06-02-2020 19:03
From: Daniel McLeod
Subject: User Status and WebRTC Connection Heartbeats/Timeouts

Hi Luke,

Websockets allow for a ping, heartbeat, or other form of keep alive from client to server to ensure connections are active. I have seen this same behavior before, but did not have an opportunity to run wireshark on the agent's machine.

You'd need to monitor if the heartbeat/ping actually gets out. When a heartbeat times out, it seems like it stops trying after a certain number of attempts and then you have to refresh to recover. To determine if this was a local network issue or a purecloud issue, you'd need Genesys to verify on their end and/or you'd need to look for other failures at the local level. Monitoring via wireshark may help, if there is other traffic to inspect for failures.

I just developed a Chrome extension to help with this, but it would only work with the web client and I'm still testing. Wireshark would give you WAY more info.

Thanks,

------------------------------
Daniel McLeod
Qsect LLC

Original Message:
Sent: 06-02-2020 11:16
From: Luke Mitchell
Subject: User Status and WebRTC Connection Heartbeats/Timeouts

I think I have worked out with Wireshark that PureCloud initiates the heartbeat and the agents desktop responds to the heartbeat, however if anybody has any knowledge or previous experience on this type of issue that would be great

------------------------------
Luke Mitchell
Conn3ct

Original Message:
Sent: 06-02-2020 10:46
From: Luke Mitchell
Subject: User Status and WebRTC Connection Heartbeats/Timeouts

Hi All,

I have some issues where for example agents are changing to a break status but then after sometime PureCloud is changing their status to available.  Also agents are using WebRTC and are on-queue but after sometime if they try and make or receive a call it fails.  I would believe it is a disconnect issue somewhere but I am trying to locate it.

From the users console logs you can see the "WS Status: Object" every 30 seconds which I believe is the heartbeat between the agents desktop and PureCloud, then it no longer shows in the log and after sometime you can see some errors and "WS: heartbeat timer expired for user xxxxxxxxx".

So I believe this is this issue but just need to understand if this is agent side or PureCloud side?

Does anybody know which direction or how these heartbeats work?

Or has anybody experienced similar behavior?

I do have this raised with Genesys Customer Care

Thanks Luke
#Omni-ChannelDesktop/UserInterface

------------------------------
Luke Mitchell
Conn3ct
------------------------------

Ok, makes sense.

I'm curious to find out what happens to indicate that the ACK never made it to the cloud (if anything). It may end up that you will have to involve Care to find out if the ACK is making it all the way back when sent from the client workstation.

------------------------------
George Ganahl GCP (Genesys Cloud), ICCE
Principal Technology Consultant
Genesys
------------------------------

Original Message:
Sent: 06-03-2020 05:40
From: Luke Mitchell
Subject: User Status and WebRTC Connection Heartbeats/Timeouts

Hi George,

In the Wireshark you can see the TLS packet come from PureCloud at the time of the WS Status and then the PC sends a TCP ACK response.  So with the affected users, I am going to check firstly that the packet is being recieved from PureCloud and if so check that the PC is sending the ACK back, depending on the outcome will depend where I focus next

Thanks Luke

------------------------------
Luke Mitchell
Conn3ct

Original Message:
Sent: 06-03-2020 05:33
From: George Ganahl
Subject: User Status and WebRTC Connection Heartbeats/Timeouts

I'm wondering if you will see anything useful in Wireshark, since the packets are encrypted with TLS for a lot of the traffic...
I just ran a Wireshark capture on my workstation, searched through the packets for various forms of WS Status: Object and came up with nothing, but it was just a quick 2-minute check. But, then, neither did I see those in the Console log.

I did see a "Websocket Heartbeat" message fairly often in the Console log, but it is not in the Wireshark captured packets...probably part of one of the encrypted messages.

Please let us know what you find!

------------------------------
George Ganahl GCP (Genesys Cloud), ICCE
Principal Technology Consultant
Genesys

Original Message:
Sent: 06-03-2020 03:35
From: Luke Mitchell
Subject: User Status and WebRTC Connection Heartbeats/Timeouts

Hi Daniel,

Thank you, that makes sense and I am capturing Wireshark on some of the affected agents which I will then analyse.  I have now been through this on my own desktop so have a good example of the packets and flow between PureCloud and the agents desktop so will see if I can compare the two

Thanks Luke

------------------------------
Luke Mitchell
Conn3ct

Original Message:
Sent: 06-02-2020 19:03
From: Daniel McLeod
Subject: User Status and WebRTC Connection Heartbeats/Timeouts

Hi Luke,

Websockets allow for a ping, heartbeat, or other form of keep alive from client to server to ensure connections are active. I have seen this same behavior before, but did not have an opportunity to run wireshark on the agent's machine.

You'd need to monitor if the heartbeat/ping actually gets out. When a heartbeat times out, it seems like it stops trying after a certain number of attempts and then you have to refresh to recover. To determine if this was a local network issue or a purecloud issue, you'd need Genesys to verify on their end and/or you'd need to look for other failures at the local level. Monitoring via wireshark may help, if there is other traffic to inspect for failures.

I just developed a Chrome extension to help with this, but it would only work with the web client and I'm still testing. Wireshark would give you WAY more info.

Thanks,

------------------------------
Daniel McLeod
Qsect LLC

Original Message:
Sent: 06-02-2020 11:16
From: Luke Mitchell
Subject: User Status and WebRTC Connection Heartbeats/Timeouts

I think I have worked out with Wireshark that PureCloud initiates the heartbeat and the agents desktop responds to the heartbeat, however if anybody has any knowledge or previous experience on this type of issue that would be great

------------------------------
Luke Mitchell
Conn3ct

Original Message:
Sent: 06-02-2020 10:46
From: Luke Mitchell
Subject: User Status and WebRTC Connection Heartbeats/Timeouts

Hi All,

I have some issues where for example agents are changing to a break status but then after sometime PureCloud is changing their status to available.  Also agents are using WebRTC and are on-queue but after sometime if they try and make or receive a call it fails.  I would believe it is a disconnect issue somewhere but I am trying to locate it.

From the users console logs you can see the "WS Status: Object" every 30 seconds which I believe is the heartbeat between the agents desktop and PureCloud, then it no longer shows in the log and after sometime you can see some errors and "WS: heartbeat timer expired for user xxxxxxxxx".

So I believe this is this issue but just need to understand if this is agent side or PureCloud side?

Does anybody know which direction or how these heartbeats work?

Or has anybody experienced similar behavior?

I do have this raised with Genesys Customer Care

Thanks Luke
#Omni-ChannelDesktop/UserInterface

------------------------------
Luke Mitchell
Conn3ct
------------------------------

Yes same here, it would be good to understand what is seen in PureCloud, I do have a case open with care for that exact reason and they have one example that I have already provided, so am waiting for an update

------------------------------
Luke Mitchell
Conn3ct
------------------------------

Original Message:
Sent: 06-03-2020 08:21
From: George Ganahl
Subject: User Status and WebRTC Connection Heartbeats/Timeouts

Ok, makes sense.

I'm curious to find out what happens to indicate that the ACK never made it to the cloud (if anything). It may end up that you will have to involve Care to find out if the ACK is making it all the way back when sent from the client workstation.

------------------------------
George Ganahl GCP (Genesys Cloud), ICCE
Principal Technology Consultant
Genesys

Original Message:
Sent: 06-03-2020 05:40
From: Luke Mitchell
Subject: User Status and WebRTC Connection Heartbeats/Timeouts

Hi George,

In the Wireshark you can see the TLS packet come from PureCloud at the time of the WS Status and then the PC sends a TCP ACK response.  So with the affected users, I am going to check firstly that the packet is being recieved from PureCloud and if so check that the PC is sending the ACK back, depending on the outcome will depend where I focus next

Thanks Luke

------------------------------
Luke Mitchell
Conn3ct

Original Message:
Sent: 06-03-2020 05:33
From: George Ganahl
Subject: User Status and WebRTC Connection Heartbeats/Timeouts

I'm wondering if you will see anything useful in Wireshark, since the packets are encrypted with TLS for a lot of the traffic...
I just ran a Wireshark capture on my workstation, searched through the packets for various forms of WS Status: Object and came up with nothing, but it was just a quick 2-minute check. But, then, neither did I see those in the Console log.

I did see a "Websocket Heartbeat" message fairly often in the Console log, but it is not in the Wireshark captured packets...probably part of one of the encrypted messages.

Please let us know what you find!

------------------------------
George Ganahl GCP (Genesys Cloud), ICCE
Principal Technology Consultant
Genesys

Original Message:
Sent: 06-03-2020 03:35
From: Luke Mitchell
Subject: User Status and WebRTC Connection Heartbeats/Timeouts

Hi Daniel,

Thank you, that makes sense and I am capturing Wireshark on some of the affected agents which I will then analyse.  I have now been through this on my own desktop so have a good example of the packets and flow between PureCloud and the agents desktop so will see if I can compare the two

Thanks Luke

------------------------------
Luke Mitchell
Conn3ct

Original Message:
Sent: 06-02-2020 19:03
From: Daniel McLeod
Subject: User Status and WebRTC Connection Heartbeats/Timeouts

Hi Luke,

Websockets allow for a ping, heartbeat, or other form of keep alive from client to server to ensure connections are active. I have seen this same behavior before, but did not have an opportunity to run wireshark on the agent's machine.

You'd need to monitor if the heartbeat/ping actually gets out. When a heartbeat times out, it seems like it stops trying after a certain number of attempts and then you have to refresh to recover. To determine if this was a local network issue or a purecloud issue, you'd need Genesys to verify on their end and/or you'd need to look for other failures at the local level. Monitoring via wireshark may help, if there is other traffic to inspect for failures.

I just developed a Chrome extension to help with this, but it would only work with the web client and I'm still testing. Wireshark would give you WAY more info.

Thanks,

------------------------------
Daniel McLeod
Qsect LLC

Original Message:
Sent: 06-02-2020 11:16
From: Luke Mitchell
Subject: User Status and WebRTC Connection Heartbeats/Timeouts

I think I have worked out with Wireshark that PureCloud initiates the heartbeat and the agents desktop responds to the heartbeat, however if anybody has any knowledge or previous experience on this type of issue that would be great

------------------------------
Luke Mitchell
Conn3ct

Original Message:
Sent: 06-02-2020 10:46
From: Luke Mitchell
Subject: User Status and WebRTC Connection Heartbeats/Timeouts

Hi All,

I have some issues where for example agents are changing to a break status but then after sometime PureCloud is changing their status to available.  Also agents are using WebRTC and are on-queue but after sometime if they try and make or receive a call it fails.  I would believe it is a disconnect issue somewhere but I am trying to locate it.

From the users console logs you can see the "WS Status: Object" every 30 seconds which I believe is the heartbeat between the agents desktop and PureCloud, then it no longer shows in the log and after sometime you can see some errors and "WS: heartbeat timer expired for user xxxxxxxxx".

So I believe this is this issue but just need to understand if this is agent side or PureCloud side?

Does anybody know which direction or how these heartbeats work?

Or has anybody experienced similar behavior?

I do have this raised with Genesys Customer Care

Thanks Luke
#Omni-ChannelDesktop/UserInterface

------------------------------
Luke Mitchell
Conn3ct
------------------------------