Genesys Cloud - Main

 View Only

Sign Up

Expand all | Collapse all

Genesys CoPilot Wrap Up Performance Data flawed / hallucinating?

  • 1.  Genesys CoPilot Wrap Up Performance Data flawed / hallucinating?

    Posted 21 days ago

    We have been reviewing the Genesys provided Agent CoPilot Performance Dashboard closely, and we believe we have found an issue with the stats for Wrap Up that make me question all the data, leading to concerns that it may be hallucinating and giving the impression usage is better than expected. 

    Do others experience this? I think the Genesys dashboard and reporting is flawed. 

    To explain, we have a CoPilot set-up where Wrap Up is toggled off as we only want to use summary. And yet the performance dashboard is showing that wrap up codes were both generated and submitted. How can this be when an advisor isn't seeing the wrap up code?

    We have tested this in a nonprod environment, and we get exactly the same behaviour, so we know for sure this isn't different for the agent.

    We've raised a ticket with Genesys, but as is often the case with Genesys first line support (sigh), it would appear I am the one hallucinating - I suspect not.


    #AICopilot(Agent,SupervisorAdmin)

    ------------------------------
    David Halliday
    ------------------------------


  • 2.  RE: Genesys CoPilot Wrap Up Performance Data flawed / hallucinating?

    Posted 21 days ago

    Hello David, 

    The behavior your seeing is by design, though its not well documented. According to the agent copilot documentation

    "If you do not want Agent Copilot to generate and store summaries, disable both Display AI powered summaries and Display AI-powered wrap up codes."

    This suggests that wrap-up codes may be generated in the backend even when the option is toggled off, they just are just not visible to the agent. The dashboard appears to be tracking generation activity that happens behind the scenes. 

    The dashboard tracks these generated (even though they were not displayed to the agent). 

    Basically the wrap up codes are being generated in the backend but hidden from the agents, the "Generated" metric would still increment, but "Submitted" and "Feedback" should still be zero since agents can't interact with what they can't see.

    If you're seeing both Generated AND Submitted counts with the display toggle off, that would indeed be a bug or indicate the system is treating backend generation differently than expected. 

    Hope this helps!



    ------------------------------
    Cameron
    Online Community Manager/Moderator
    ------------------------------



  • 3.  RE: Genesys CoPilot Wrap Up Performance Data flawed / hallucinating?

    Posted 21 days ago
    Edited by David Halliday 20 days ago

    Hi Cameron, thanks for the swift response.

    Yes, we are seeing BOTH Generated and Submitted with numbers greater than zero. The Submitted value as you say is the more concerning number, so this has got to be a bug right?

    On the Generated number, if we go by the description of Generated in the resource centre, I'm not clear it should be showing whatever is presented in the back end, even if not visible. That gives the impression the advisor should be seeing something when they aren't. 

    • Generated: Number of successfully generated AI summaries on Copilot ACW where at least one AI-suggested wrap-up code was generated.

    I don't follow the definition of successfully generated if the person hasn't seen them and we have it turned off. This one may I suspect be an Idea, but its poorly thought out from the team at Genesys.

    Either way, the data for Submitted not being zero is very concerning, screengrab below for one advisor today showing 8 submitted despite it being toggled off:

    We are paying rather handsomely for an add-on which is then giving wrong (in the case of submitted) data, or misleading data (in the case of generated). This information is being used to determine the effectiveness of the product in our business, which given the added cost we need to be sure pays for itself. Without trust in the data, it makes me question whether we should continue with the product.



    ------------------------------
    David Halliday
    ------------------------------



  • 4.  RE: Genesys CoPilot Wrap Up Performance Data flawed / hallucinating?

    Posted 21 days ago

    Hello David, 

    Thank you for the further clarification. At this point I would need you to go back to support. This doesn't seem to be the correct behavior either something is off with the Copilot or something is being tracked incorrectly in the backend. Also for feedback on the documentation if you scroll all the way to the bottom of the page and give a thumbs down you can submit feedback to request a clear explanation of the documentation. 

    As a side note if you can post your case number I can track it internally, as this issue is definitely interesting. 

    Cheers, 



    ------------------------------
    Cameron
    Online Community Manager/Moderator
    ------------------------------



  • 5.  RE: Genesys CoPilot Wrap Up Performance Data flawed / hallucinating?

    Posted 18 days ago

    Thanks Cameron, our case number is 0003980738.

    Look forward to the updates, and sharing what progress we have.



    ------------------------------
    David Halliday
    ------------------------------



  • 6.  RE: Genesys CoPilot Wrap Up Performance Data flawed / hallucinating?

    Posted 17 days ago

    Hi Cameron, it appears that Genesys engineers think there isn't an issue and its our issue?! Help!

    This isn't a great look for Genesys, if you can help in any way with this ticket that would be much appreciated, as the "Submitted" problem is definitely a Genesys issue which we can recreate on demand.



    ------------------------------
    David Halliday
    ------------------------------



  • 7.  RE: Genesys CoPilot Wrap Up Performance Data flawed / hallucinating?

    Posted 17 days ago

    Hey David, fellow customer here...I'm curious as to what their response was.  Just recently expanded our co-pilot usage and we've always had wrap-ups enabled, but your experience has me concerned about the data



    ------------------------------
    Chad Blanton
    N/A
    ------------------------------



  • 8.  RE: Genesys CoPilot Wrap Up Performance Data flawed / hallucinating?

    Posted 17 days ago

    Hi Chad. We raised two issues of late, and I suspect the response is getting mixed between the two if I'm honest, but even that is a concern. Sadly now, for each and every ticket we raise, I fully expect the response of either, we need a new example to look at (despite giving previous examples), or its your fault, or there is no issue/its by design. 

    The forum they provide here is at least a way of understanding if the issue is genuinely one that is unique to our business, or whether we have seen it on other businesses.

    For this one, the easiest way to test is to test with one person (giving them a co-pilot config that doesn't have wrap up suggestion toggled on), then check the performance dashboard filtering on that one person to see what the stats show. I'd certainly be interested in your view on that.

    I should add, the summary data I am less concerned with, as we use summary as the base installation here, however wrap code suggestion was something we were deciding whether to implement. My issue now is how do I trust how much it is being used when we know the stats don't represent what an advisor is doing, particularly as we know we can recreate the issue in a test environment.



    ------------------------------
    David Halliday
    ------------------------------



  • 9.  RE: Genesys CoPilot Wrap Up Performance Data flawed / hallucinating?

    Posted 17 days ago

    Hi Cameron. I can confirm that my org is seeing the same issue as David. In terms of config, we have taken the same steps and turned off the ACW suggestion but continue to see the generated and submitted stats being populated with a greater than 0 value.



    ------------------------------
    Kevin Macrae
    Product Analyst
    ------------------------------



  • 10.  RE: Genesys CoPilot Wrap Up Performance Data flawed / hallucinating?

    Posted 17 days ago

    Hello David, 

    Thank you for trying again. I have escalated this internally to have someone take a look. It seems like we have more community members experiencing the same behavior. Someone will be reaching out to you via the support case I believe. I will be watching this closely though. 

    Cheers, 



    ------------------------------
    Cameron
    Online Community Manager/Moderator
    ------------------------------



  • 11.  RE: Genesys CoPilot Wrap Up Performance Data flawed / hallucinating?

    Posted 17 days ago

    Thanks Cameron, I appreciate you escalating. I had hoped it wouldn't take shouting up in a forum to crowdsource other customers with the same issue in order that the traditional support routes have worked more effectively? It doesn't feel right you have to monitor these forums to figure out whether an issue reported is a real issue.

    I have just had an update to say that it has now been raised with ServOps team for detailed review and feedback. Hopefully we get some progress now.



    ------------------------------
    David Halliday
    ------------------------------



  • 12.  RE: Genesys CoPilot Wrap Up Performance Data flawed / hallucinating?

    Posted 4 days ago

    Hi Cameron, have you heard anything in respect to the above issue and our case? We seem to be facing radio silence from Genesys, despite 12 days rolling on and our business still paying an added licence charge for something that is fundamentally broken.



    ------------------------------
    David Halliday
    ------------------------------



  • 13.  RE: Genesys CoPilot Wrap Up Performance Data flawed / hallucinating?

    Posted 4 days ago

    Hello David, 

    Taking a look at the case and the dev team case, it looks like this issue has evolved into something more complex. They are still actively working on it but this maybe something that needs a little bit of time to fix. 

    Cheers, 



    ------------------------------
    Cameron
    Online Community Manager/Moderator
    ------------------------------



  • 14.  RE: Genesys CoPilot Wrap Up Performance Data flawed / hallucinating?

    Posted 4 days ago
    Edited by David Halliday 4 days ago

    Thanks Cameron. I wonder if the team are able to share what they are seeing by way of something more complex?

    I have a concern we have seen something more worrying where we are also seeing advisors seeing wrap codes that don't relate to their call queue when a transfer is performed, but instead relate to the prior queue the call was transferred from (which may actually explain the performance data issue).

    This is still something we are checking before formally raising, but it would be great if some transparency could be given on what more Genesys are seeing, if not here then direct with me / Connect given the case we have raised. We don't quite fully know if there are other issues here that we should be aware of given we are using this in a live production environment. 



    ------------------------------
    David Halliday
    ------------------------------