Genesys Cloud - Main

 View Only

Discussion Thread View
Back to discussions
Expand all | Collapse all

Slow Down and Disconnects using WebRTC from Home

  • 1.  Slow Down and Disconnects using WebRTC from Home

    Posted 08-07-2020 08:13
    We are seeing a significant slow down using the WebRTC now that we are 100% working from home and I'm sure more users are moving to the cloud. This includes agents not able to accept calls and just very slow response navigating around the pages. Other web apps are not showing any performance slow downs. Two questions. 1. Is anyone else seeing this? 2. Is there a benefit running the Desktop app?
    #Ask Me Anything (AMA)
    #ArchitectureandDesign
    #ConnectwithaCustomer(NEW)
    #DigitalChannels
    #Implementation
    #Integrations
    #Omni-ChannelDesktop/UserInterface
    #Outbound
    #PlatformAdministration
    #QualityManagement
    #RemoteWorkEnablement
    #Reporting/Analytics
    #Roadmap/NewFeatures
    #Routing(ACD/IVR)
    #Security
    #SIP/VolP
    #SystemAdministration
    #Telephony
    #Unsure/Other
    #GenesysCloudCXQAEpisode

    ------------------------------
    Greg Barrett
    Outdoor Network, LLC
    ------------------------------


  • 2.  RE: Slow Down and Disconnects using WebRTC from Home

    Posted 08-08-2020 03:27
    I've struggled with this since first going live with Genesys Cloud.  Any case raised to Genesys Care just gives us the scripted response pointing us to troubleshooting steps on the Resource Centre or the stock response blaming users PCs, browser cache, modem, wifi vs LAN, send us network and console logs etc.  Given like you, most of our users are at home, the only "common" factor is the GC website and using WebRTC.  I've had a number of users on 1GB+ fibre connections, direct LAN connection to the modem, high spec'd gaming machine still complain about response time in some things in the browser app.

    Having said that, what type of Edge deployment do you have?

    Not sure if you'd get any benefit to the desktop app as it's using some browser components still under the hood.  It does however add in stuff like screen recording that you don't get with the browser app.

    ------------------------------
    Vaun McCarthy
    NTT New Zealand Limited
    ------------------------------

    Original Message


  • 3.  RE: Slow Down and Disconnects using WebRTC from Home

    Posted 08-08-2020 15:48
    Troubleshooting delays especially with WAH agents is not fun.  I usually start with the local console and network logs on the local browser to look for timeouts, retries and other possible issues and then determine if it is a problem with scripts, UI, WebRTC, etc.  I also use the WebRTC-Internals in Chrome to look at call setup and transmission.  I have found a lot of the initial delays are due to heavily loaded initial pages for scripts.  I would suggest turning of or switching to a simplified script to see if that is the issue.  Also, limit images and frames in the initial script.  If you do have an image, make sure it is linked to a smaller image and not some 50 MB full resolution monster.

    ------------------------------
    Robert Wakefield-Carl
    Avtex Solutions, LLC
    Contact Center Innovation Architect
    robertwc@avtex.com
    https://www.Avtex.com
    https://RobertWC.Blogspot.com
    ------------------------------

    Original Message


  • 4.  RE: Slow Down and Disconnects using WebRTC from Home

    Posted 03-10-2021 06:31
    Hello 
    We started with Purecloud 2 years ago. Number of agents using WebRTC was limited then, most on remotephone/softphone. With the Covid we moved all our agents to WebRTC so they could work from home. We experienced since the beginning issues with agents being alerted but not being able to accept the call, status changing back to avilable during their lunch while the webrtc diag would not show any latency/packet loss that would be significant. At the time the issue are occuring agents are browsing happily on other sites without issue. We have done multiple analysis of chrome/network logs and wireshark capture and Genesys always concluded to a disconnect between the Edge and the agent client. The egde is sending a hartbeat every 30 secs and if one is not aknowledged by the agent client, then the webRTC session is lost. Only a logout and login may restore it as starting a new WebRTC session with the Edge where a new hartbeat will be started
    This is impacting seriously SLA and abandonned rate of course. It would be nice if at least there would be a sign of the lost heartbeat on the UI so the agent could logout and login again ... Any other methods you know to identify the lost WeRTC session ?

    ------------------------------
    Renaud Larcier
    Travelport
    ------------------------------

    Original Message


  • 5.  RE: Slow Down and Disconnects using WebRTC from Home

    Posted 03-10-2021 09:14
    CPU can be an issue as well. If the processor is unable to handle all the packets coming at it then real time clients can struggle... whereas if it's a web page the customer perception is just that it's slow to load. We bounce in between both of these issues, especially with agents who are working in multiple tools and have dozens of tabs open in a browser.

    It's tough to control for or support issues at the client.

    ------------------------------
    Dean Thames
    Koch Business Solutions
    ------------------------------

    Original Message


  • 6.  RE: Slow Down and Disconnects using WebRTC from Home

    Posted 03-10-2021 10:51
    In additional to all of the excellent advice that has already been posted here, I would highly recommend trying a tool like Ping Plotter. It will visually graph out traceroute's to different destinations.  You can run it on your agents workstation and continuously ping an AWS IP and Google. This can help make sure there isn't just a network issue on their side when reaching AWS. 

    There can be a ton of different root causes, but Ping Plotter makes it very easy to determine yes we definitely have a problem. VOIP/WebRTC is the ultimate stress test for any network. If you have issues once an hour for 10 seconds most other apps will still probably appear to work fine, not so much here.
    Original Message


  • 7.  RE: Slow Down and Disconnects using WebRTC from Home

    Posted 08-31-2021 09:50
    Is anyone still seeing these issue and if not what was done to remediate the issue?

    We are using the Chrome browser interface and seeing this happen a lot.

    ------------------------------
    Marshall Billings
    Telephony Systems Engineer
    ------------------------------

    Original Message


  • 8.  RE: Slow Down and Disconnects using WebRTC from Home

    Posted 09-01-2021 15:32
    We are using the Windows desktop app and started seeing issues yesterday without any (known) changes on our end. Agents are working from home with different internet providers. Genesys support isn't even responding to a high-level (prio 2) case. Agents are reporting disconnections, inability to answer a call, menu icons turning red, etc. The internet connections for the home agents and our edges seem fine I'm not seeing elevated ping time, or dropped packets. 

    We are still diagnosing since the issues are sporadic and difficult to capture. I will report back any findings.

    ------------------------------
    Sven Schiller
    Kognitiv
    ------------------------------

    Original Message


  • 9.  RE: Slow Down and Disconnects using WebRTC from Home

    Posted 09-06-2021 00:50
    Hello Sven
    Are your desktop apps kept up to date please? Worth checking as we have seen user issues when app not up to date.

    Another key recommendation for home users is having direct Internet which generally means split tunneling if VPN involved. Naturally BYOC Cloud scenarios are going to suit home working with WebRTC better (with fewer considerations) than BYOC Premises (on-prem Edges)...

    ------------------------------
    Thanks and regards
    Blair Wilkinson
    CVT Global Enablement
    ------------------------------

    Original Message


  • 10.  RE: Slow Down and Disconnects using WebRTC from Home

    Posted 09-06-2021 03:14
    Hi @Blair Wilkinson

    Just a side-question to this, have you implemented split tunnelling specifically for Genesys Cloud?

    I've previously posted the question here about what the best practice would be, but haven't heard anything back from the community.

    As there doesn't appear to be any official recommendation/guidelines from Genesys, our IT department is reluctant to implement any changes.
    ​​

    ------------------------------
    Jeff
    ------------------------------

    Original Message


  • 11.  RE: Slow Down and Disconnects using WebRTC from Home

    Posted 09-06-2021 04:05
    Edited by Matt Lawson 05-09-2022 09:30

    Hello Jeff

     

    Hope you are well.

     

    For background we are a GenCloud partner for implementation and support.

     

    This has come up regularly with customers working from home where it became obvious that the VPN was creating a bottle neck, not just for GenCloud but also other SaaS solutions. Split tunnelling has become a standard recommendation.

     

    The solution, where possible, was for their IT to setup split tunnelling on their user agents. My understanding is the desired configuration is to set all office traffic (based on network segment) to go via VPN, all else is to go direct via user internet.

     

    Therefore, not so much configuring the tunnelling for GenCloud, more along the lines of configure your network traffic via VPN and all else direct out.

     

    PS. I am not a network engineer but this is what I understand clients are doing and I also know that some had an easier time of this depending on the VPN client in use

     

    Do you have on-prem edges or are you on a flavour of BYOC Cloud please? If on-prem edges it will be more complicated because traffic needs to get to your edges...

     

    Thanks and regards

    Blair Wilkinson



    Original Message


  • 12.  RE: Slow Down and Disconnects using WebRTC from Home

    Posted 09-06-2021 20:01
    Hi Blair,

    Thanks for the response and your insights.

    AFAIK, our VPN is intended to handle all 'work related' traffic - including external internet access - and route it through our corporate network and endpoint security platforms (e.g. firewalls, proxies, etc) to ensure that all data/traffic is filtered or blocked as required for security. Only Office 365 (including Teams) specific IP's and URL's are permitted direct access, and only to our Org in Office365. 

    So for us, that simple 'internal network traffic goes via VPN and everything else goes via internet' doesn't appear to be an option.

    We're using BYOC Cloud, so it should be relatively easy now to identify the media related traffic using the CIDR IP range - it's just the rest of the Genesys Cloud IP ranges (i.e. everything in an AWS region) and URL's that is difficult to restrict to our specific Org in a split tunnelling setup.

    ------------------------------
    Jeff
    ------------------------------

    Original Message


  • 13.  RE: Slow Down and Disconnects using WebRTC from Home

    Posted 09-05-2021 21:46
    We have been having on-going random issues with remote users unable to accept calls when working from home for the best part of a year. We have remote stations configured in Genesys as they advised us that WebRTC was not supported over remote WVD setups when we transitioned users to remote working at the start of COVID (March 2020).
    We're connecting to Genesys via browser rather than the desktop app. We've tried Chrome and Edge browsers.
    User experience is that the call is presented in the browser (accept/decline popup), but the users remote phone (mobile device) does not ring, therefore they are unable to take the call.
    I've tried testing the Genesys WebRTC Media Helper to be able to use WebRTC outside of remote access, but it just does not work. Genesys support were unable to effectively help troubleshoot the issue.
    These remote call failures take up a large portion my work day. Very frustrating.

    ------------------------------
    Brett Hartley
    AFSA (Australian Financial Security Authority)
    ------------------------------

    Original Message


  • 14.  RE: Slow Down and Disconnects using WebRTC from Home

    Posted 09-09-2021 09:50
    I have noticed a high CPU usage when utilizing Genesys Cloud via Chrome. Seems that when inbound calls are coming in the CPU spikes for a second to 99-100%. Could this be causing the issue?

    ------------------------------
    Marshall Billings
    Telephony Systems Engineer
    ------------------------------

    Original Message


  • 15.  RE: Slow Down and Disconnects using WebRTC from Home

    GENESYS
    Posted 09-09-2021 19:27
    Hey Everyone,

    This thread caught my team's attention and I wanted to provide some perspective. 

    The suggestion to use split tunnel VPN is probably the only blanket advice that we can think of right now - and it's already been suggested/discussed in the thread.

    Ultimately, performance/quality can vary depending on the agent's internet connection, the agent's home network, and the agent's personal equipment.

    My best advice when you have an interruption or bug in service is to report what's going on to our Care team so that they can investigate the instances individually. I love that you all were able to discuss this issue and offer best advice, you really knocked it out of the park - That's what a community is for. Care is always there to help investigate disruptions with service.

    Sorry I don't have a solution for you, but I hope this helps. FWIW, I've sent this thread to a few teams internally. If I hear any other ideas, I'll report back. 

    Cheers!

    ------------------------------
    Matt Lawson
    Genesys - Employees
    Online Community Manager
    ------------------------------

    Original Message


  • 16.  RE: Slow Down and Disconnects using WebRTC from Home

    Posted 09-10-2021 10:45
    After jumping through various hoops to try and find the source of the issue, I noticed that a call recording stopped after three seconds when the call lasted over three minutes. I decided to restart the edges involved in handling the site's traffic (we use a redundant set of on-premise mini edges).
    Since then, no more disconnections (it's been about 24h). It may be a coincidence, but the edge restart is so easy/simple that I will add it to my early troubleshooting steps now.

    ------------------------------
    Sven Schiller
    Kognitiv
    ------------------------------

    Original Message


  • 17.  RE: Slow Down and Disconnects using WebRTC from Home

    GENESYS
    Posted 09-10-2021 11:21
    I don't have any real experience with this problem, but I do know that an arriving call will have quite a bit of WebRTC traffic and Genesys Cloud UI traffic all at once.  So I'm not surprised that you might get CPU spikes during call setup.  To alleviate some of that you could use a persistent, "nailed-up" WebRTC session: https://help.mypurecloud.com/articles/use-persistent-connection-feature-genesys-cloud-webrtc-phone/.  This basically keeps the agent's phone connection established the entire time and calls are essentially just bridged into them.  This would alleviate some of the traffic that occurs between the repeated call setup and tear down.

    There are also a number of vendors that are in or joining the Genesys AppFoundry marketplace (https://appfoundry.genesys.com) that provide tools for troubleshooting the root cause of network/workstation bottlenecks end-to-end.  It might be money well spent to invest in some tooling to keep you distributed WFH contact center humming along.

    ------------------------------
    Jim Crespino
    Senior Director, Developer Evangelism
    Genesys
    https://developer.genesys.com
    ------------------------------

    Original Message


  • 18.  RE: Slow Down and Disconnects using WebRTC from Home

    Posted 09-10-2021 15:55
    Edited by Matt Lawson 05-09-2022 09:30
    Hi Jim,

    Yes, we have been using persistent connections for our agents for years (not with the business users who occasionally make a call). Persistent connections seem to have certainly helped with the issues. 
    In this case, it really seems to have come down to at least one of the edges going into an inconsistent state, which seems to have been resolved by a restart. No more issues reported since then. I will monitor over the weekend and report back if this changes.

    UPDATE 13-Oct-2021: No further issues following the reboot of our on-premise mini edges. 

    ------------------------------
    Sven Schiller
    Kognitiv
    ------------------------------

    Original Message


  • 19.  RE: Slow Down and Disconnects using WebRTC from Home

    Posted 09-10-2021 15:10

    Hi Matt,

    Interested to hear recommendation from your internal team.



    ------------------------------
    ATB - Vincent R. Sabolboro
    ATB Financial
    ------------------------------

    Original Message


  • 20.  RE: Slow Down and Disconnects using WebRTC from Home

    GENESYS
    Posted 09-10-2021 15:18
    @Vincent Sabolboro, check out Jim Crespino's answer just above yours. He is one of the most knowledgable people I know and has some additional thoughts. ​

    ------------------------------
    Matt Lawson
    Genesys - Employees
    Online Community Manager
    ------------------------------

    Original Message


  • 21.  RE: Slow Down and Disconnects using WebRTC from Home

    Posted 09-10-2021 15:39
    @Matt Lawson, Let me check the article. ​

    ------------------------------
    ATB - Vincent R. Sabolboro
    ATB Financial
    ------------------------------

    Original Message


Need Help finding something?

Check out the Genesys Knowledge Network - your all-in-one access point for Genesys resources