Genesys Cloud - Main

 View Only

Sign Up

  • 1.  What Happens When an AVA Tests Other AVAs?

    Posted 08-30-2026 00:13

    Over the past few weeks, we have been exploring a different approach to testing and evolving our Agentic Virtual Agents: an internal AVA created to support the validation of other AVAs.

    During a 12-hour intensive review session, we reached some numbers that helped put the scale of the challenge into perspective:

    🔹 Around 1,600 test conversations
    🔹 6 AVAs reviewed
    🔹 120 repeated validations of mandatory opening behavior
    🔹 One critical routing journey improved from 17/30 to 39/40 successful executions
    🔹 Five final behavior test suites completed with 84/84 criteria met

    To put the volume into perspective, even if each conversation took only one minute to execute manually, that would represent more than 26 uninterrupted hours just running the interactions. This does not include reviewing results, documenting findings or making any improvements.

    But the most valuable outcome was not simply running more tests in less time.

    It was being able to observe how the same agent behaved across repeated executions, compare versions, identify regressions and find issues that could easily remain hidden in a few isolated manual tests.

    Some of the lessons were surprisingly simple:

    An agent being correct once does not mean it is reliable.
    A good response does not always mean the expected action happened.
    More instructions do not necessarily produce better behavior.
    And a perfect score only matters when the test actually covers the entire journey.

    The testing agent does not replace human judgment. Business knowledge, careful analysis and an understanding of the real customer journey are still essential. What it gives us is more evidence and a much better starting point for that analysis.

    We are keeping the implementation details private for now, but I wanted to share the experience because it reinforced something important: agentic systems require a testing discipline designed for variability.

    This project started as a practical way to make testing more scalable. It ended up changing how we understand and improve our AVAs.

    Has anyone else in the community been exploring new ways to test Agentic Virtual Agents at scale? I would love to hear how you are approaching this challenge.


    #ConversationalAI(Bots,VirtualAgent,etc.)

    ------------------------------
    Mateus Nunes
    CX Manager at Solve4me Solucoes em Tecnologia Ltda
    ------------------------------


  • 2.  RE: What Happens When an AVA Tests Other AVAs?

    Posted 08-30-2026 09:01

    Indeed, AVA is one of the most impressive tools available in Genesys Cloud today.



    ------------------------------
    Camila Meneghini
    ------------------------------



  • 3.  RE: What Happens When an AVA Tests Other AVAs?

    Posted 26 days ago
    Hi Mateus,
     
    This resonates a lot. Your point that "a good response does not always mean the expected action happened" matches exactly what I've hit, cases where the transcript reads perfectly but the tool call fired at the wrong moment or with the wrong value.
     

    Briefly, what I've built: a front-door AVA on the voice channel that replaces the traditional IVR menu. The caller states their need in natural language, the agent picks one of roughly 85 routing values across several categories, and that value is passed back to the Architect flow to route the call. It's bilingual, and it falls back to a category menu when it can identify the area but not the specific intent.

    It works functionally, but my testing so far is manual and nowhere near enough for production. With 85 values, several of which are semantically close to each other, a few dozen successful calls tells me very little. Being on voice makes it harder again, since speech-to-text variation alone can change the routing decision, and I can't run voice calls at any real volume by hand.

     
    What I'd most like to understand is the testing AVA itself:
     
    1. How does the testing agent actually drive the target AVA? Is it invoked through the flow, through the API, or something else? And is it on the digital channel, or were you able to test voice?
     
    2. How does it know whether a run passed? Is it judging the conversation text, or are you asserting on the actual tool output and variables the target agent emitted?
     
    3. How did you define the test criteria, and how did you decide what coverage was "enough" across many possible outcomes?
     
    4. When you change a guideline or the model version, how do you re-run and compare against the previous version to catch regressions?
     
    Completely understood that you're keeping implementation details private. Even the shape of the approach would help a lot. Thanks for posting the numbers, they're a useful reality check.


    ------------------------------
    Suranjan
    Management Consultant
    ------------------------------



Related Content