{"id":440,"date":"2020-03-03T02:34:19","date_gmt":"2020-03-03T02:34:19","guid":{"rendered":"http:\/\/freedville.com\/blog\/?p=440"},"modified":"2020-03-03T02:34:19","modified_gmt":"2020-03-03T02:34:19","slug":"how-to-test-a-cognitive-system-and-why-its-so-important","status":"publish","type":"post","link":"https:\/\/freedville.com\/blog\/2020\/03\/03\/how-to-test-a-cognitive-system-and-why-its-so-important\/","title":{"rendered":"How to test a cognitive system (and why it\u2019s so important)"},"content":{"rendered":"\n

Part 1 of the\u00a0Cognitive System Testing<\/strong>\u00a0<\/a>series, originally posted in 2016 on IBM Developer<\/a>.<\/p>\n\n\n\n

Introduction<\/h2>\n\n\n\n

I joined IBM Watson in 2012 and immediately became interested in how cognitive solutions, such as those based on Watson, are tested. Watson solutions are probabilistic systems, generally based on machine learning algorithms involving hundreds or thousands of variables, and as such are a far cry from the deterministic systems I was used to. This chapter and several following will chronicle my experiences testing these systems as well as sharing some lessons learned along the way.<\/p>\n\n\n\n

Manual testing<\/h2>\n\n\n\n

The first type of testing I encountered was \u201ctraditional\u201d, manual testing. Most software developers are familiar with the pros and cons of manual testing, so let\u2019s not dwell too long here. When all else fails, even Watson can be tested manually. As with manually testing any other software, this approach can find bugs but at very high cost. In our infancy, manual testing made up the bulk of our testing efforts. Today, we still do some manual testing, but sparingly, as we rely more on manually tests.<\/p>\n\n\n\n

Automated testing<\/h2>\n\n\n\n

\nI much prefer using automated tests to test my software. Write once,\nrun many times! But where do you begin with a software system that\ndoes not behave like other software systems?<\/p>\n\n\n\n

Testing triangle<\/h2>\n\n\n\n

\nIt turns out that the \u201ctesting triangle\u201d (see Martin\nFowler\u2019s Test\nPyramid<\/a> post) provides a useful guidepost.<\/p>\n\n\n\n

\nIn summary there are three layers of the triangle, and as you scale\nthe triangle the tests cover more function, cost more to write, and\nare slower to run.<\/p>\n\n\n\n

\nUnit test<\/strong> \u2013 a unit test ideally\ntests one class, and since there are so many classes, you should\nnecessarily have the most tests in unit test layer<\/p>\n\n\n\n

\nFunctional test<\/strong> \u2013 a test that\ntests on component or a logical group of classes<\/p>\n\n\n\n

\nUI test<\/strong> \u2013 a test that tests the\nentire application, just like a user uses the application. If you\ndon\u2019t have a graphical UI, this can just represent your system\ntests<\/p>\n\n\n\n

\nThis triangle approach applies nicely to cognitive systems and we\nhave adapted a version of this approach within Watson. Most of what I\nlearned in my Watson experience was interesting parts in the\nfunctional layer.<\/p>\n\n\n\n

Before the triangle<\/h2>\n\n\n\n

\nWe needed a transitive step before converting our test plan entirely\nto the testing triangle. There are enough components in a cognitive\nsolution that some form of smoke testing is required before passing\nthe system to other tests (or testers). In our question and answer\n(QA) systems, the smoke test was simply to send a question through\nthe system and verify that an answer came out. Notice I said \u201can\nanswer\u201d, not \u201cthe answer\u201d, as the purpose of this test is to\nverify that the system has basic functionality. Our smoke testing\nphilosophies will be explored in further chapters.<\/p>\n\n\n\n

Components of a cognitive system<\/h2>\n\n\n\n

\nCognitive systems typically have at least four components:<\/p>\n\n\n\n

\nIngestion<\/strong> \u2013 retrieve raw data,\ntransform into a format suitable for processing by the rest of the\nsystem<\/p>\n\n\n\n

\nNatural Language Processing (NLP)<\/strong> \u2013\nextract concepts and meaning from plain, natural language test<\/p>\n\n\n\n

\nQuestion pipeline<\/strong>\u2013\ngiven a question\/query from a user, traverse a knowledge graph and\nprovide a useful response to the user<\/p>\n\n\n\n

\nREST API<\/strong> \u2013 facilitate\ncommunication between components and interface elements<\/p>\n\n\n\n

\nI\u2019ll cover testing over these components, as well as the full\nsystem and the smallest units, in future chapters. These chapters are\nshown with their relation to the testing triangle.<\/p>\n\n\n\n

There\u2019s a time and place for manual testing, but even in cognitive systems, automated testing is best.<\/p>\n\n\n\n

\"\"<\/figure>\n","protected":false},"excerpt":{"rendered":"

Part 1 of the\u00a0Cognitive System Testing\u00a0series, originally posted in 2016 on IBM Developer. Introduction I joined IBM Watson in 2012 and immediately became interested in how cognitive solutions, such as those based on Watson, are…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[],"_links":{"self":[{"href":"https:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/posts\/440"}],"collection":[{"href":"https:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/comments?post=440"}],"version-history":[{"count":1,"href":"https:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/posts\/440\/revisions"}],"predecessor-version":[{"id":442,"href":"https:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/posts\/440\/revisions\/442"}],"wp:attachment":[{"href":"https:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/media?parent=440"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/categories?post=440"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/tags?post=440"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}