In my original testing series Cognitive System Testing From A to Z I described how cognitive systems require a fresh look at testing compared to traditional systems. Cognitive systems are probabilistic, traditional systems are deterministic, and this has implications for how cognitive systems are tested.
As a follow-up I’ve written several posts on IBM Watson’s Medium Blog specific to testing chatbots and voicebots. (Voicebot being a chatbot using a voice channel). Those posts are summarized and linked below.
Testing Strategies for Chatbots (Part 1)— Testing Their Classifiers. Classifiers are used to figure out what your user actually intends based on what they have ‘said’ to the bot. Classifiers need to be tested and continuously improved.
Testing Strategies for Chatbots (Part 2)— Testing Their Dialog Logic. Dialog logic, or routing logic, is at the heart of any chat bot. Classifiers are used to derive intent but routing through a dialog flow includes many additional considerations including conversation context and other state. This routing logic deserves its own testing consideration.
Testing Strategies for Speech Applications. Voice channels introduce new testing complexities. Speech to text testing can be automated by comparing a master transcription of an audio snippet to a model’s transcription of that same audio. Text to speech testing requires you to listen to synthesized audio to judge its quality.
