Knowledgebase Test Runs

Knowledgebase Test Runs allows you to test your Knowledgebase against a list of questions.

Objectives:

• Automate test case creation and execution to reduce manual effort in testing GenAI chatbots.
• Allow comparison between different LLMs and Knowledgebase contents.
• Automatically generate relevant questions and expected answers based on the extracted information.

Benefits:

• Automation reduces the need for manual test creation and execution.
• Standardized test cases ensure uniform testing across different versions and models.
• Streamline testing process significantly, saving time for you and Talkative support.
• Get  live faster.
• Understand the validity of test cases over time.

Limitations:

• Not truly representative of website conditions and longer conversations.

Why

GenAI chatbots can be incredibly effective, but testing can be time consuming, often taking a lot of time and manual effort. Previously, you would have to think what questions you'd want the chatbot to be able to answer, build spreadsheets of these questions, then manually test each question through a chat widget - a time consuming exercise.

Additionally, each time you update your knowledgebase, you'd need to re-test each and every question.

Knowledgebase Test Runs allows you to automate all of this.

Requirements

1. Available to all Talkative subscriptions.
2. You must be logged in as an Account Holder.
3. Recommended that you first enable AI Insight Reports.
4. Limited to 30 runs per month. You can check your usage on your Talkative Usage & Billing page.

How to use

First, navigate to the Knowledgebase you want to test and scroll towards the bottom of the page:

You can either run a new test or review previous tests:

If you have AI Insights reports configured, you can generate questions based on this report. This is recommended since the questions are generated to match common customer question types.

Blank slate will let you build questions from scratch.

You can select which AI Insight report you want to generate questions from:

Next, you can edit/create/delete questions.

You can use up to 20 questions in a single test run.

You can also select the LLM you wish to use, and change the knowledgebase if required. 

After a few seconds of starting the test, your answers will be displayed!

The test will let you know if no answer was found.

You can leave internal notes on the test.

You can review previous tests at any time: