Genesys Cloud CX

 View Only
Discussion Thread View
Expand all | Collapse all

Increase in Issues on our Genesys platform

  • 1.  Increase in Issues on our Genesys platform

    Posted 09-29-2022 12:02
    Is anyone else out there experiencing an increase in issues/instability with the Genesys Cloud implementation - Primarily with voice?

    We have many different issues with Genesys that are starting to boil over into great concerns about our system as we head into our peak season.

    • Intermittent problems where many different agents, across multiple sites not able to hear the agent, but the customer can hear them & both are recorded on the interaction
      • We have done many things regarding desktop, headset, applications, etc.
      • We are monitoring network.
      • Genesys sees no problem on their side.
      • We can't find the problem on our side. 
    • Calls Disconnect/Drops as soon as the connection to the agent is made from the IVR
    • Calls disconnects/drop showing as if the agent disconnected when the agent was mid-sentence talking with the customer
    • Error messages telling the Agent they are not responding – but we have auto answer!
    • The ability to monitor an Agent Monitor not working for some and intermittent for others
    • One of our new contact centers is having connectivity issues and dropped calls.

    We have passed a few of these to the Genesys Support desk but have no answers.

    We have been negatively impacted for literally months. But diversity and frequency of issues is increasing.
    Any insights, idea, options, etc.
    Does anyone know someone in Genesys who is a "troubleshooter" / problem solver who could help.
    You can respond here or email me directly at cstephesn@fanatics.com

    #Audio issues
    #Not Responding Error message
    #Connectivty issues
    #Call Drops
    #Call Monitoring issues​​​​​​​


    #Telephony
    #Unsure/Other

    ------------------------------
    Diane Stephens
    CStephens@fanatics.com
    Fanatics Tech
    ------------------------------


  • 2.  RE: Increase in Issues on our Genesys platform

    Posted 09-30-2022 04:05
    We have some of these. Calls have been logged with Genesys but no answers. The problems are intermittent and random across agents.
    • Calls drop as soon as they are answered by the agent
    • Genesys tab closes in Edge by its self
    • Agent cannot be heard by caller


    ------------------------------
    andrew lewis
    Health Management Ltd
    ------------------------------



  • 3.  RE: Increase in Issues on our Genesys platform

    Posted 09-30-2022 04:26
    Yes, we're getting issues like this too. Most of the issues are eventually dismissed by support as local internet issues (which is a possibility but unproven, as the users don't get issues with our other VOIP tools), or the tickets end up petering out as the agents get tired of reporting them.

    ------------------------------
    James Dunn
    Pitney Bowes Inc.
    ------------------------------



  • 4.  RE: Increase in Issues on our Genesys platform

    Posted 09-30-2022 05:45
    Hi
    We have the same issues, very frustrating

    ------------------------------
    Philip Thys
    British Telecommunications PLC
    ------------------------------



  • 5.  RE: Increase in Issues on our Genesys platform

    Posted 09-30-2022 06:01
    I have done quite a bit of work analyzing our GC agents on this, I have seen patterns where calls/email/chat etc not responding is roughly the same ratio and have put this down to internet issues (home workers specifically)

    The reason for calls is where the the chrome/edge desktop cant get back to our edges in the DC

    For emails and chat its purely they ignore the alerting, I have added the beta for auto answer to a test queue and again all auto answer as expected.

    I have done 10,000s of interactions during build and testing and never experienced it, but I do have a really good internet connection.

    ------------------------------
    Andy Jackson
    Ten Lifestyle Management Limited
    ------------------------------



  • 6.  RE: Increase in Issues on our Genesys platform

    Posted 09-30-2022 06:17
    Andy, It could be a local internet issue, who knows? The users i have spoken to have not complained of their vpn droping (Genesys bypasses their vpn) One thing i make sure of is that they are not using wifi as i see this issue a lot more with wifi connected laptops (but not exclusively. It is very frustrating for our users and all of us as there isn't always a lot we can do to help them. We don't always know if it's resolved as sometimes the users just give up.

    There must be more Genesys can do to help.

    Genesys, your thought please?

    ------------------------------
    andrew lewis
    Health Management Ltd
    ------------------------------



  • 7.  RE: Increase in Issues on our Genesys platform

    Posted 09-30-2022 10:09
    Hey All - We also experience the above. For the agents not hearing the customer we have found out most of that is either due to other phone systems on the agents computer competing with GC like teams or cisco IPT soft phones. We also saw huge issues with Bluetooth & wireless headsets for some agents and have since replaced with wired headsets. The wired headsets so far have not had call issues.

    ------------------------------
    Daniel Tierney
    Johnson & Johnson
    ------------------------------



  • 8.  RE: Increase in Issues on our Genesys platform

    Posted 09-30-2022 10:18
    Diane,

     Do your agents share workstations or not have a designated station? One of our call centers is open floor seating and we have found that some of the agents do not properly log out and restart their stations after their shift. We found that most of the issues you are mentioning are very similar to what we were experiencing and it was because of concurrent log ins.

    https://help.mypurecloud.com/faqs/why-do-multiple-concurrent-logins-cause-problems/

    Sincerely,

    ------------------------------
    Eric Allen
    Vervent, Inc.
    ------------------------------



  • 9.  RE: Increase in Issues on our Genesys platform

    Posted 09-30-2022 10:35
    Ours all have a laptop and work from home.

    ------------------------------
    andrew lewis
    Health Management Ltd
    ------------------------------



  • 10.  RE: Increase in Issues on our Genesys platform

    Posted 09-30-2022 12:13
    Thanks for the input/suggestion.  But we have dedicated laptops and have WFH and onsite.

    ------------------------------
    Diane Stephens
    Fanatics
    ------------------------------



  • 11.  RE: Increase in Issues on our Genesys platform

    Posted 09-30-2022 12:47
    I have noticed it mainly when agents are working from home, I showed someone the affect of streaming Netflix when doing a ping to show against response times so we have advised not to do this if possible, most BB routers don't support QOS or Prioritization of packets so you cant QOS Genesys either, most TIVO systems seem to permanently downloading.

    It is a tricky situation to get the perfect setup.  My BB is 250mb and cable broadband and I don't ever seem to have issues, but the agents are probably running cheaper setups and not best setups.

    We have set the laptops to disable wireless if cabled (have said cabled as much as possible) so stop any contending for connectivity, pushed people away from Bluetooth headsets, we dont seem to get it too much, also we have our own edges and dont run AWS virtual edges

    ------------------------------
    Andy Jackson
    Ten Lifestyle Management Limited
    ------------------------------



  • 12.  RE: Increase in Issues on our Genesys platform

    Posted 09-30-2022 18:46
    Hi Diane,

    We are not seeing an increase in issues or instability with voice. (Our system: US East region with BYOC on-prem Edges; on-prem agents(25%) with desk phones and WFH agents(75%) with WebRTC phones via VPN; all agents on VDI/VMs.)

    To achieve acceptable levels of stability and quality, we made improvements and resolved a variety of problems. It turns out that many issues were not due to the Genesys Cloud platform. What helped was looking at all parts and resolving issues one by one. Some of the things we did:

    Phone carrier - increased CCP (concurrent call paths); stopped data bursts from randomly interfering with voice traffic -- identified and resolved by carrier's engineer after much inquiry; replaced SIP gateway router.

    Primary business ISP - increased bandwidth for both upload and download.

    VMs - increased CPU and RAM for each VM; stopped Microsoft updates, Chrome updates, etc.; used randomized schedule for EDR and security agent updates; updated and fine-tuned VPN client software and VM client software.

    GC app (desktop and web app)
    • Enabled WebRTC persistent connection.
    • Informed users to clear Chrome browser cache frequently -- a major issue due to numerous web-based apps.
    • Asked users to log out properly using the Logout button instead of closing the app.
    • Favorites panel: don't leave this open when logging out; better to click Calls panel, then log out. If Favorites is left open, GC app will try to update everyone's status and their photos when you log back in the next day.

    User laptops
    • Use built-in USB port for USB headset.
    • Check for conflicts between the headset and other devices(built-in mic/speaker; speakers on external monitor; mic on external webcam)
    • Don't use Wi-Fi. No wireless mouse/keyboard. No other bluetooth devices.

    Home network
    • Cable Internet provides fast download but slower upload speeds. An agent's voice is uploaded, so slow upload speed is a problem.
    • Upload speed can be slowed down by video meetings, home security cameras, cellphones, any IoT devices in the home.
    • There have been no issues reported by users who migrated from cable(300/10) to fiber(500/500).



    ------------------------------
    Kun Jin Rhee
    LifeCare Assurance Company
    ------------------------------



  • 13.  RE: Increase in Issues on our Genesys platform

    Posted 10-03-2022 00:26
    Hi Kun,

    Regarding WebRTC Persistent Connection, do your users use Auto Answer feature?
    Or enabeld the Persistent Connection feature without using Auto Answer?

    Recently, I wonder that WebRTC Persistent Connection should be enabled even when the customer does not use Auto Answer feature.
    I started to think like above due to that some agents can not answer incoming calls although they are alerted.


    ------------------------------
    哲朗 藤澤
    TSUZUKI DENKI CO LTD
    ------------------------------



  • 14.  RE: Increase in Issues on our Genesys platform

    Posted 10-03-2022 12:35
    Hi 哲朗 藤澤,

    We enabled WebRTC Persistent Connection for all users regardless of whether they also use Auto Answer. This was in response to WFH users complaining that sometimes there was not enough time to answer; by the time they click Answer after hearing one or two rings, the caller is no longer there. It turns out that although we set the alert time to 20 seconds, the agent may not get the full 20 seconds to answer. For example, if a call takes 5 or 6 seconds from the Cloud server to reach the agent's WebRTC phone, this would leave only 14 or 15 seconds for an agent to hear and answer. With Persistent Connection, the initial call may be slow but subsequent calls are immediate. This has reduced missed calls.

    For ACD queue agents, adding Auto Answer has further reduced missed calls for one of our teams. Agents no longer need to spend time looking for the Answer button. However, we also have teams that do not want to try Auto Answer. Remember, Auto Answer only works for ACD calls.

    Using Genesys Cloud desktop app instead of web app also helps reduce missed calls. It has an orange icon that is easy to find on the task bar. It also has a small pop-up alert box for incoming calls. This helps when there are multiple web apps, email and other apps open on the screen.


    ------------------------------
    Kun Jin Rhee
    LifeCare Assurance Company
    ------------------------------



  • 15.  RE: Increase in Issues on our Genesys platform

    Posted 10-09-2022 14:41
    We have also increased our Persistent connection for WebRTC to 28000 seconds..  A full day, i wonder what you folks had the Persistent connection changed to..

    ------------------------------
    Shalom Benzaquen
    Elevance Health
    ------------------------------



  • 16.  RE: Increase in Issues on our Genesys platform

    Posted 10-03-2022 10:26
    Hi Everyone.
    Looks like we're facing the same issues.

    I tried going through end user logs, and looks like there are 3 cases that seem to be a pattern here:
    1. "HTML5 Audio pool exhausted, returning potentially locked audio object." Genesys gave us an answer to that, and it was to clear browser cache, use latest browser etc..
      The error means that the audio pool of the browser is not being updated or refreshed properly. Meaning it is exhausting all audio pools and unable to play the next or incoming audio. The quick fix to the issue is to perform Clear Cache but then again this is a band-aid fix.
      
       
      
      Another way to resolve the issue is to make sure of the following:
      
      Running the latest version of the browser: Chrome, Firefox, or MS Edge
      Make sure the user is only using 1 tab for Genesys
      Make sure there are no Browser extensions that could be causing conflicts for the audio
    2. [streaming-client] IceCandidateError { errorCode: 701, errorText: 'STUN host lookup received error.', url: 'stun:turn-premium.euc1.pure.cloud:3478', conversationId: 'xxxxxxxxxxxxxxxxxxxxx', sessionId: 'xxxxxxxxxxxxxxx', sessionType: 'softphone' }  -> I'll just try opening our firewall to allow traffic towards this address
    3. [streaming-client] IceCandidateError { errorCode: 701, errorText: 'STUN host lookup received error.', url: 'stun:turn.euc1.pure.cloud:3478', conversationId: 'xxxxxxxxxxxxxxxxxxx', sessionId: 'xxxxxxxxxxxxxxxxxxxxxxxxxx', sessionType: 'softphone' } -> again, I'll just try opening our firewall to allow traffic towards this address

    Hope it solves at least some part of the issues, but would be happy to see if anyone else found a solution, as opening firewall in my company is a bit lengthy process.

    Thanks!
    Piotr




    ------------------------------
    Piotr Danielewski
    EY Global Services Limited
    ------------------------------



  • 17.  RE: Increase in Issues on our Genesys platform

    Posted 10-04-2022 04:16
    Hi Piotr,

    If you figure out a fix for those last two errors please let us know. We get the same results. We were told to enable FORCE TURN on the user's phone, although it doesn't seem to resolve it.

    We consistently pass the network readiness assessment tool so I don't understand why we are getting any sort of firewall errors.

    ------------------------------
    James Dunn
    Pitney Bowes Inc.
    ------------------------------