{"id":193,"date":"2017-10-12T00:46:17","date_gmt":"2017-10-12T00:46:17","guid":{"rendered":"http:\/\/freedville.com\/blog\/?p=193"},"modified":"2019-04-12T14:08:02","modified_gmt":"2019-04-12T14:08:02","slug":"what-does-your-cognitive-system-know-about-your-documents-and-why-does-it-matter","status":"publish","type":"post","link":"http:\/\/freedville.com\/blog\/2017\/10\/12\/what-does-your-cognitive-system-know-about-your-documents-and-why-does-it-matter\/","title":{"rendered":"What does your cognitive system know about your documents, and why does it matter?"},"content":{"rendered":"

I work with clients who want to use cognitive computing to extract insights locked away in their data.  This data generally comes from a large volume and wide variety of source documents – thousands or millions of documents, dozens or hundreds of document types.  When source documents come from some sort of structured source (like a database), we generally can answer questions like “what is the most recent document of type X?” from document metadata and we can use this to enhance insights from the data.<\/p>\n

But what if the documents have no metadata?  What if they are scanned documents, or even worse, paper form?  This post explores how difficult it is to get documents in condition to do fun cognitive things on them.<\/p>\n

Let me demonstrate just how difficult this problem is.  I have a filing cabinet stuffed with years of papers that I have just never gotten around to organizing.  I’d love to extract some insights like “what’s the maintenance history on my car?” but first I have to find the car repair statements and sort them by date.  This same cabinet has all sorts of other receipts, statements, and the like as well.<\/p>\n

Let’s look at a couple examples.  I’ve included four types of documents, used green boxes to highlight the document dates, and red boxes to highlight dates that are NOT the document date.<\/p>\n

\"\"
Inspection invoice<\/figcaption><\/figure>\n
\"\"
Statement<\/figcaption><\/figure>\n
\"\"
Veterinary invoice<\/figcaption><\/figure>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

Had I used my scanner, these documents would get labels like “Scanned001.pdf” with a creation date of today.  But to get real metadata (document type and date), I have to work pretty hard.  For types, I can roughly classify documents by the logo (for documents with a logo) or by a phrase on the top (or bottom) of the document.  None of these rules are surefire, but I can go pretty far with rules of thumb.<\/p>\n

For dates – look again at those boxes of “actual” and “not actually” document dates.  Not only is there no quick pattern to the actual document dates (green boxes) – some are top-left, some are top-right, some middle – but there are plenty of extraneous document dates (red boxes) too.  Sorting these documents is a tough nut to crack.<\/p>\n

For a personal filing cabinet, it’s probably easiest for me to use a high powered scanner and apply my own document metadata after the fact.  It’s a few nights with piles of documents in my office and a trusty bucket sort<\/a> algorithm.  After organizing my documents, I’ll take some time off before I mine them for insights.<\/p>\n

But for an enterprise with thousands or millions of documents, the challenge is larger.  Do you manually organize that many documents?  Do you train an OCR engine to extract metadata?  Or do you try to build your application without metadata?<\/p>\n

I don’t intend to solve the problem here, that is a separate future blog post.  (Update: see How to organize documents when digitizing them<\/a>) This post is just to demonstrate one underappreciated challenge in preparing yourself to build a cognitive application.<\/p>\n","protected":false},"excerpt":{"rendered":"

I work with clients who want to use cognitive computing to extract insights locked away in their data.  This data generally comes from a large volume and wide variety of source documents – thousands or…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[9,2,5],"_links":{"self":[{"href":"http:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/posts\/193"}],"collection":[{"href":"http:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/comments?post=193"}],"version-history":[{"count":11,"href":"http:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/posts\/193\/revisions"}],"predecessor-version":[{"id":413,"href":"http:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/posts\/193\/revisions\/413"}],"wp:attachment":[{"href":"http:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/media?parent=193"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/categories?post=193"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/freedville.com\/blog\/wp-json\/wp\/v2\/tags?post=193"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}