{"id":458,"date":"2020-03-03T02:53:11","date_gmt":"2020-03-03T02:53:11","guid":{"rendered":"http:\/\/freedville.com\/blog\/?p=458"},"modified":"2020-03-03T02:53:11","modified_gmt":"2020-03-03T02:53:11","slug":"cognitive-system-testing-concluding-remarks","status":"publish","type":"post","link":"https:\/\/freedville.com\/blog\/2020\/03\/03\/cognitive-system-testing-concluding-remarks\/","title":{"rendered":"Cognitive System Testing: Concluding remarks"},"content":{"rendered":"\n

Part 7 of the\u00a0Cognitive System Testing<\/strong>\u00a0<\/a>series, originally posted in 2016 on IBM Developer<\/a>. <\/p>\n\n\n\n

Introduction<\/strong><\/h2>\n\n\n\n

\nThis blog series has focused on a set of techniques needed for\ntesting a cognitive system. As previously discussed, the cognitive\nsystem itself may be probabilistic and non-deterministic but it can\nstill be tested. How are you supposed to know which tests to apply\nand which tests to spend your precious energy on? There are some\nguiding principles you can use in your testing journey.<\/p>\n\n\n\n

Don\u2019t\nlose sight of quality<\/strong><\/h2>\n\n\n\n

\nIt can be fun to inject lots of different testing techniques,\nframeworks, and ideas into your test plan. Don\u2019t get carried away\nwith testing for testing\u2019s sake. The whole reason you are writing\ntests for various parts of the system is to ensure quality of the\nsystem as a whole. If you find yourself working on tests that are not\ncontributing to the end-result quality of the system, abandon these\ntests!<\/p>\n\n\n\n

Write\nthe tests you need<\/strong><\/h2>\n\n\n\n

\nTime is a precious resource, you only have so much of it to write\ntests for your system. Tests are subject to diminishing returns \u2013\nat some point adding a new test does not catch any new bugs.\nFurthermore, consider how many tests each part of the system has and\navoid writing too many tests for high-functioning parts. If you are\nabout to write your 200th XML parsing test and your NLP code is full\nof bugs, set down the XML test and fix your NLP! Similarly, don\u2019t\nfocus too much effort writing additional tests for rarely used\ncomponents of your system if other components are lightly tested.<\/p>\n\n\n\n

\nFind where your system performs the worst and iterate, iterate,\niterate. Your system will have many moving parts and the interactions\nbetween all of them may be unclear when you start testing. No matter\nwhich level of testing you are doing (system, functional, unit), find\nthe part where the most problems are and fix that first (while adding\nnew tests, of course). Just keep repeating this process. This ensures\nyou are focusing your testing effort where you need it most.<\/p>\n\n\n\n

Know\nwhen to quit testing<\/strong><\/h2>\n\n\n\n

\nA cognitive system is never going to achieve 100% accuracy and\naccuracy improvements come with a cost of diminishing returns.\nConsider how much any accuracy improvements will help your system and\nhow much they will cost. When building natural language processing\ncode, a final F1 score of 80% accuracy is very good. Higher accuracy\nis possible but at very high cost. Other subsystems may require 100%\nor near accuracy (such as parsers, API layers, etc) \u2013 use your\njudgement as to what level of accuracy you need and how much you are\nwilling to pay for it.<\/p>\n\n\n\n

Conclusion<\/strong><\/h2>\n\n\n\n

\nFollow the testing\npyramid<\/a> when thinking about how to test a\ncognitive system. Remember why you are writing tests \u2013 for a high\nquality system \u2013 and write only the tests you need. Focus on the\nworst-performing parts of your system and iteratively improve them.\nStop testing and improving the system when you have reached a \u201cgood\nenough\u201d accuracy and further improvement is prohibitively\nexpensive.<\/p>\n","protected":false},"excerpt":{"rendered":"

Part 7 of the\u00a0Cognitive System Testing\u00a0series, originally posted in 2016 on IBM Developer. Introduction This blog series has focused on a set of techniques needed for testing a cognitive system. As previously discussed, the cognitive…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[],"_links":{"self":[{"href":"https:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/posts\/458"}],"collection":[{"href":"https:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/comments?post=458"}],"version-history":[{"count":2,"href":"https:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/posts\/458\/revisions"}],"predecessor-version":[{"id":460,"href":"https:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/posts\/458\/revisions\/460"}],"wp:attachment":[{"href":"https:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/media?parent=458"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/categories?post=458"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/tags?post=458"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}