{"id":85,"date":"2017-01-13T13:39:30","date_gmt":"2017-01-13T13:39:30","guid":{"rendered":"http:\/\/freedville.com\/blog\/?p=85"},"modified":"2019-04-12T14:13:21","modified_gmt":"2019-04-12T14:13:21","slug":"demo-of-natural-language-processing-with-rules-and-machine-learning-based-approaches","status":"publish","type":"post","link":"https:\/\/freedville.com\/blog\/2017\/01\/13\/demo-of-natural-language-processing-with-rules-and-machine-learning-based-approaches\/","title":{"rendered":"Demo of natural language processing with rules and machine-learning based approaches"},"content":{"rendered":"

Introduction<\/strong><\/p>\n

In this cognitive area, many people are interested in using natural language processing (NLP) to extract insights from their large collections of unstructured text.  There are two main approaches to natural language processing: rules-based NLP and machine-learning-based NLP.  I decided to put together a brief example of how both techniques work so that you can compare the two.<\/p>\n

Problem setup<\/strong><\/p>\n

The problem I want to solve is to find all the times computer programming languages are mentioned in a block of text, including languages that I have never heard of before.  Rather than using a predefined list of programming languages (this list called a ‘dictionary’ in NLP terminology), I want the NLP system to use only semantic clues from the text itself to determine when a programming language is being discussed.<\/p>\n

Further, I add two constraints to keep the example simple.  First, I only process one document: the Wikipedia article on programming languages<\/a>.  Second, I use only one rule in my rules-based system.  This low amount of training keeps both techniques on equal footing.<\/p>\n

Rules-based NLP<\/strong><\/p>\n

Rules-based NLP is performed by expert rules developers.  These developers scan source documents and try to discover rules that will help extract key data points, balancing rules that extract too little vs rules that extract too much.  For fun, skim the article<\/a> and see what rule(s) you might try.<\/p>\n

I settled on the following simple rule:<\/p>\n

For every sentence containing the word “language”, remove the first word, and any remaining capitalized words are programming languages.<\/p>\n

The results are interesting.  I’ll select snippets of text and bold<\/strong> the “programming languages” detected by my NLP.<\/p>\n

Sometimes this rule is good:<\/p>\n

\n

The language above is Python.<\/strong><\/p>\n<\/blockquote>\n

Sometimes this rule is great:<\/p>\n

\n

In 2013 the ten most popular programming languages are (in descending order by overall popularity): C<\/strong>, Java<\/strong>, PHP<\/strong>, JavaScript<\/strong>, C++<\/strong>, Python<\/strong>, Shell<\/strong>, Ruby<\/strong>, Objective-C<\/strong> and C#<\/strong>.<\/p>\n<\/blockquote>\n

Sometimes the rule is way off (none of the bold words are programming languages):<\/p>\n

\n

Edsger Dijkstra<\/strong>, in a famous 1968 letter published in the Communications<\/strong> of the ACM<\/strong>, argued that GOTO<\/strong> statements should be eliminated from all “higher level” programming languages.<\/p>\n<\/blockquote>\n

And the rule also misses some easy ones (Java is a programming language):<\/p>\n

\n

Java came to be used for server-side programming.<\/p>\n<\/blockquote>\n

Still, the rule was simple to write with only a few lines of code, and it performed reasonably well.  I manually counted the number of programming languages mentioned in the article as 103.  The rule found 106 programming languages, 56 correctly detected, 47 incorrectly, thus giving precision of 52.8%, recall of 54.4%, and F1 score of 0.537.  (F1 score is our accuracy metric.) Not bad for one rule.<\/p>\n

See full code listing: ProgrammingLanguageRulesBasedAnnotator.java<\/a>.<\/p>\n

See video demonstration:<\/p>\n