I'm updating
my open-source tool for testing Web Messenger chatbots that we use extensively work, to be able to use ChatGPT. The reason being is that tests are quite rigid at the moment. As the example below shows, you define what you say and the expected response, then should the chatbot deviate from this then the test will be marked as a failure:
scenarios:
"Borrow a book by ISBN":
- say: hi
- waitForReplyContaining: Welcome to 📚 Lucas's Library 📚
- waitForReplyContaining: How can I help you?
- say: I would like to borrow a book, can I check that you have it?
...
However, by using ChatGPT we can create tests that can adapt to changes in the chatbot's behaviour/functionality. A test will then be defined by a ChatGPT prompt, which my tool uses to orchestrate a conversation between the chatbot under test and ChatGPT (video of it in action):
scenarios:
"Borrow a book by ISBN":
setup:
prompt: |
...
As a customer you would like to borrow a book by its ISBN. When you say goodbye to the chatbot include the word 'PASS' in your message. If there are any errors then say 'FAIL'.
...
terminatingPhrases:
pass: ["PASS"]
fail: ["FAIL"]
The example above is rather simple, but a more advanced use-case would be to cause the test to fail if ChatGPT notices (amongst other things):
- Errors
- Invalid handling of inputs
- Incorrect spelling, grammar or punctuation
- An opportunity to improve response clarity
Can anyone think of any other use-cases or problems I've not considered? Any feedback would be most appreciated.
#ConversationalAI(Bots,AgentAssist,etc.)------------------------------
Lucas Woodward
Ovo Energy Ltd
------------------------------