{"id":443,"date":"2020-03-03T02:38:55","date_gmt":"2020-03-03T02:38:55","guid":{"rendered":"http:\/\/freedville.com\/blog\/?p=443"},"modified":"2020-03-03T02:38:55","modified_gmt":"2020-03-03T02:38:55","slug":"cognitive-system-testing-smoke-testing","status":"publish","type":"post","link":"https:\/\/freedville.com\/blog\/2020\/03\/03\/cognitive-system-testing-smoke-testing\/","title":{"rendered":"Cognitive system testing: Smoke testing"},"content":{"rendered":"\n<p>Part 2 of the\u00a0<a href=\"http:\/\/freedville.com\/blog\/2016\/12\/04\/cognitive-system-testing-from-a-to-z\/\"><strong>Cognitive System Testing<\/strong>\u00a0<\/a>series, originally posted in 2016 on <a href=\"https:\/\/developer.ibm.com\/\">IBM Developer<\/a>. <\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Introduction<\/h2>\n\n\n\n<p>\nA cognitive system, like most software systems, is a collection of\nmodular components, each with varying degrees of complexity. One\nquick way of testing a cognitive system is to use a \u201csmoke test\u201d,\nwhich tells you if the system is behaving so poorly that it may as\nwell be on fire. The smoke test suite should tell you if the system\nas a whole is responsive to user inputs and if the inner components\nare generally speaking to each other.<\/p>\n\n\n\n<p>\nWith the whole system at your fingertips, you may have a desire to\nmake tests as comprehensive as possible. Resist the urge! Our ideal\ntesting approach is a layered approach and the smoke test is just the\ntip of the spear. The smoke test should give you just enough\ninformation to decide if you want to run the rest of your test suites\n(you do have multiple batches, don\u2019t you?) In fact, in our Watson\nsolutions, our smoke tests don\u2019t even verify if Watson provides the\ncorrect answer to a question \u2013 just that Watson provides an answer!<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Motivation<\/h2>\n\n\n\n<p>\nUsing any build process (hopefully an automated build), software will\nobviously not be delivered if it does not compile. Compile-time\nerrors are fairly easy to catch and provide a fail-fast mechanism. If\nall of your software modules compile, that is a good start, but they\nmay not run together. Smoke testing is a way of quickly flushing out\nbatches of runtime errors by forcing the components to talk to each\nother.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">A good smoke test suite<\/h2>\n\n\n\n<p>\nYour smoke test suite should aim for the extreme version of the 80\/20\nrule \u2013 write as few tests as possible to cover the major\nintegration points within your application. Some of our Watson\nsolutions started with only one test in their test suite, asking a\nsingle question of the Watson system. A medical Watson solution\nstarted with just a handful \u2013 each test tested one of major disease\ntypes covered by the system, with varied mock patient data to cover\ninteresting patient characteristics (gender, age classes, etc). Your\nsolution may need a few more test cases. The important thing is to\nhit each runtime component \u2013 in the Watson solutions described\nabove this included a UIMA pipeline, a REST layer, a database, and a\nmachine learning model.<\/p>\n\n\n\n<p>\nI mentioned earlier that our smoke tests only verify that Watson\nprovides an answer, not the answer. This an important distinction. By\nallowing some freedom in the responses, we prevent the smoke test\nfrom being brittle. As previously noted, Watson systems are\nprobabilistic, non-deterministic systems and that can play havoc on a\nstrict test. We rely on other test suites to verify the system is\nfully functioning.&nbsp;&nbsp;<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">In case of errors<\/h2>\n\n\n\n<p>\nIdeally all of our tests pass. If the smoke test suite fails, STOP,\nit\u2019s as bad as if the system is on fire! A smoke test failure\nshould prevent the other test suites from failing and should start an\nimmediate triage process. Our smoke test suites include a series of\nlog scanners which look for error and exception messages and provided\na detailed email to all of the people who contributed to the current\nfailed build, and this failure is immediately given high priority.\nOther channels are great for communicating this failure including SMS\nalerts and Slack channel notifications.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Example failure email<\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>The smoke test for build 20160916_0830 on system node123 failed with several errors.\n\ngrep \u2013R ERROR \/solution\/20160916_0830\/logs found:\ninstall.log:\u00a0\u00a0 ERROR\u00a0\u00a0\u00a0\u00a0 Could not load messages.properties\nruntime.log:\u00a0\u00a0 ERROR\u00a0\u00a0\u00a0\u00a0 Failed initialize module ComponentA due to IllegalArgumentException\ntestcase.log:\u00a0 ERROR\u00a0\u00a0\u00a0\u00a0 Question 1 did not return a response\n\nThe following code updates were added to the build:\nabcd1234 (johndoe) Integrate ComponentA into build\nefgh5678 (janedoe) Fix NullPointerException in installer<\/code><\/pre>\n\n\n\n<p>\nThis email includes key details of what build was installed, where\nthe build was installed, where the logs can be found and what\nindications of failure exist.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Conclusion<\/h2>\n\n\n\n<p>\nA smoke test suite needs to quickly hit the major integration points\nand code paths in your application and determine if the application\nis in a serious failed state or not. The smoke test suite should be\nsmall, run quickly, and be tolerant of output from the system. The\nsmoke test suite is just the first part of your overall testing\nsolution and it can leave some work to the other suites.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Part 2 of the\u00a0Cognitive System Testing\u00a0series, originally posted in 2016 on IBM Developer. Introduction A cognitive system, like most software systems, is a collection of modular components, each with varying degrees of complexity. One quick&#8230;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[],"_links":{"self":[{"href":"https:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/posts\/443"}],"collection":[{"href":"https:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/comments?post=443"}],"version-history":[{"count":1,"href":"https:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/posts\/443\/revisions"}],"predecessor-version":[{"id":444,"href":"https:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/posts\/443\/revisions\/444"}],"wp:attachment":[{"href":"https:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/media?parent=443"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/categories?post=443"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/tags?post=443"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}