Workforce Engagement Management

Hello all-

We are currently working on validating/testing AI scoring and there are instances where the answer should have been an "N/A" but AI scored the question as "No". Help text has been updated but still seeing "No" regularly.

We are not saving these test evaluations as we are leaving in a "Pending" status or deleting once we see what the justification for an answer was.

If we update an answer to match what we think it should be, for instance changing the answer from "No" to "N/A" and then save that evaluation without submitting, will that affect future scoring? Is AI learning our behaviors? 

If not, are there any suggestions on how to word question/help text to understand when a question should be marked N/A? 

Here is a sample question and help text that we can't get to work:

Q1: If the call referenced a referral did the agent inform the caller to bring a referral? Yes/No/Not Applicable.

Help Text: Agent would say phrases similar to "you need a referral" or "please contact your PCP for a referral" or "remember to bring your referral." If a referral was not referenced during the call, mark the answer as not applicable or N/A. If a referral was not required for this appointment mark the answer as N/A or not applicable.

Thank you for any assistance.

Hi @Joseph Sutich ,

Based on the documentation, I would be careful with the idea that AI Scoring "learns" from each correction as if the model were being retrained.

What Genesys Cloud does is monitor the historical accuracy of AI-generated answers based on how often human evaluators modify or override an AI-provided answer. This is calculated using a rolling window of the 25 most recent submitted evaluations for each question.

Automated scoring options in evaluation forms - Genesys Cloud Resource Center

So, if the evaluation is only left in Pending status or deleted, I would not expect it to affect that historical accuracy calculation. To impact the historical accuracy, the corrected evaluation would likely need to be submitted.

For your specific scenario, I would try to make the N/A condition the first decision point in the help text. The main goal is to avoid the AI treating "no evidence" as a negative behavior.

For example:

Question:
When a referral was applicable or referenced during the call, did the agent inform the caller to bring or obtain a referral?

Help Text:
First determine whether the referral requirement is applicable to the call.

Select N/A if:

• A referral was not mentioned during the call.

• A referral was not required for the appointment.

• The referral topic was not relevant to the caller's request.

Select Yes only if a referral was applicable or referenced and the agent informed the caller to bring the referral, obtain a referral, or contact their PCP for a referral.

Select No only if a referral was applicable or referenced and the agent failed to provide the referral instruction.

From a form design perspective, another option is to use Yes / No / N/A as the answer set and make sure N/A does not negatively impact the score, since it represents a non-applicable scenario rather than agent failure.

If your form setup allows it, you could also consider setting N/A as the default answer. Then AI Scoring would only change the answer to Yes or No when there is enough evidence in the interaction. If there is not enough evidence, the answer would remain N/A instead of being interpreted as No.

Hello all-

We are currently working on validating/testing AI scoring and there are instances where the answer should have been an "N/A" but AI scored the question as "No". Help text has been updated but still seeing "No" regularly.

We are not saving these test evaluations as we are leaving in a "Pending" status or deleting once we see what the justification for an answer was.

If we update an answer to match what we think it should be, for instance changing the answer from "No" to "N/A" and then save that evaluation without submitting, will that affect future scoring? Is AI learning our behaviors? 

If not, are there any suggestions on how to word question/help text to understand when a question should be marked N/A? 

Here is a sample question and help text that we can't get to work:

Q1: If the call referenced a referral did the agent inform the caller to bring a referral? Yes/No/Not Applicable.

Help Text: Agent would say phrases similar to "you need a referral" or "please contact your PCP for a referral" or "remember to bring your referral." If a referral was not referenced during the call, mark the answer as not applicable or N/A. If a referral was not required for this appointment mark the answer as N/A or not applicable.

Thank you for any assistance.

Hi Mateus,

Thank you so much for your prompt and detailed response. I will take your suggestions and rework my help text. 

Hi @Joseph Sutich ,

Based on the documentation, I would be careful with the idea that AI Scoring "learns" from each correction as if the model were being retrained.

What Genesys Cloud does is monitor the historical accuracy of AI-generated answers based on how often human evaluators modify or override an AI-provided answer. This is calculated using a rolling window of the 25 most recent submitted evaluations for each question.

Automated scoring options in evaluation forms - Genesys Cloud Resource Center

So, if the evaluation is only left in Pending status or deleted, I would not expect it to affect that historical accuracy calculation. To impact the historical accuracy, the corrected evaluation would likely need to be submitted.

For your specific scenario, I would try to make the N/A condition the first decision point in the help text. The main goal is to avoid the AI treating "no evidence" as a negative behavior.

For example:

Question:
When a referral was applicable or referenced during the call, did the agent inform the caller to bring or obtain a referral?

Help Text:
First determine whether the referral requirement is applicable to the call.

Select N/A if:

• A referral was not mentioned during the call.

• A referral was not required for the appointment.

• The referral topic was not relevant to the caller's request.

Select Yes only if a referral was applicable or referenced and the agent informed the caller to bring the referral, obtain a referral, or contact their PCP for a referral.

Select No only if a referral was applicable or referenced and the agent failed to provide the referral instruction.

From a form design perspective, another option is to use Yes / No / N/A as the answer set and make sure N/A does not negatively impact the score, since it represents a non-applicable scenario rather than agent failure.

If your form setup allows it, you could also consider setting N/A as the default answer. Then AI Scoring would only change the answer to Yes or No when there is enough evidence in the interaction. If there is not enough evidence, the answer would remain N/A instead of being interpreted as No.

Hello all-

We are currently working on validating/testing AI scoring and there are instances where the answer should have been an "N/A" but AI scored the question as "No". Help text has been updated but still seeing "No" regularly.

We are not saving these test evaluations as we are leaving in a "Pending" status or deleting once we see what the justification for an answer was.

If we update an answer to match what we think it should be, for instance changing the answer from "No" to "N/A" and then save that evaluation without submitting, will that affect future scoring? Is AI learning our behaviors? 

If not, are there any suggestions on how to word question/help text to understand when a question should be marked N/A? 

Here is a sample question and help text that we can't get to work:

Q1: If the call referenced a referral did the agent inform the caller to bring a referral? Yes/No/Not Applicable.

Help Text: Agent would say phrases similar to "you need a referral" or "please contact your PCP for a referral" or "remember to bring your referral." If a referral was not referenced during the call, mark the answer as not applicable or N/A. If a referral was not required for this appointment mark the answer as N/A or not applicable.

Thank you for any assistance.

Hi Mateus,

Thank you so much for your prompt and detailed response. I will take your suggestions and rework my help text. 

Hi @Joseph Sutich ,

Based on the documentation, I would be careful with the idea that AI Scoring "learns" from each correction as if the model were being retrained.

What Genesys Cloud does is monitor the historical accuracy of AI-generated answers based on how often human evaluators modify or override an AI-provided answer. This is calculated using a rolling window of the 25 most recent submitted evaluations for each question.

Automated scoring options in evaluation forms - Genesys Cloud Resource Center

So, if the evaluation is only left in Pending status or deleted, I would not expect it to affect that historical accuracy calculation. To impact the historical accuracy, the corrected evaluation would likely need to be submitted.

For your specific scenario, I would try to make the N/A condition the first decision point in the help text. The main goal is to avoid the AI treating "no evidence" as a negative behavior.

For example:

Question:
When a referral was applicable or referenced during the call, did the agent inform the caller to bring or obtain a referral?

Help Text:
First determine whether the referral requirement is applicable to the call.

Select N/A if:

• A referral was not mentioned during the call.

• A referral was not required for the appointment.

• The referral topic was not relevant to the caller's request.

Select Yes only if a referral was applicable or referenced and the agent informed the caller to bring the referral, obtain a referral, or contact their PCP for a referral.

Select No only if a referral was applicable or referenced and the agent failed to provide the referral instruction.

From a form design perspective, another option is to use Yes / No / N/A as the answer set and make sure N/A does not negatively impact the score, since it represents a non-applicable scenario rather than agent failure.

If your form setup allows it, you could also consider setting N/A as the default answer. Then AI Scoring would only change the answer to Yes or No when there is enough evidence in the interaction. If there is not enough evidence, the answer would remain N/A instead of being interpreted as No.

Hello all-

We are currently working on validating/testing AI scoring and there are instances where the answer should have been an "N/A" but AI scored the question as "No". Help text has been updated but still seeing "No" regularly.

We are not saving these test evaluations as we are leaving in a "Pending" status or deleting once we see what the justification for an answer was.

If we update an answer to match what we think it should be, for instance changing the answer from "No" to "N/A" and then save that evaluation without submitting, will that affect future scoring? Is AI learning our behaviors? 

If not, are there any suggestions on how to word question/help text to understand when a question should be marked N/A? 

Here is a sample question and help text that we can't get to work:

Q1: If the call referenced a referral did the agent inform the caller to bring a referral? Yes/No/Not Applicable.

Help Text: Agent would say phrases similar to "you need a referral" or "please contact your PCP for a referral" or "remember to bring your referral." If a referral was not referenced during the call, mark the answer as not applicable or N/A. If a referral was not required for this appointment mark the answer as N/A or not applicable.

Thank you for any assistance.