I recently published a three-part post on the IBM Watson publication on testing and improving chatbots (also known as virtual assistants) using both pre- and post- deployment techniques. Each post is accompanied with an embedded video describing the technique.
Testing a Chatbot with k-folds Cross Validation demonstrates a pre-deployment technique, k-folds cross validation, which can identify potential confusion in your classifier training data before you deploy to production.
Analyze chatbot classifier performance from logs shows how to test your classifier in production by identifying production data to use in testing, labeling it as ground truth, and determining “blind test” accuracy levels.
Improve a chatbot classifier with production data introduces how to identify the gaps in your classifier training data and plug them with data from your previous tests. This video also demonstrates how classifier performance can improve with this new data.
The techniques in these posts are meant to be used iteratively, both within and across update cycles.
