Pure language processing with Apache OpenNLP

Pure language processing (NLP) is among the most necessary frontiers in software program. The fundamental concept—learn how to eat and generate human language successfully—has been an ongoing effort for the reason that daybreak of digital computing. The trouble continues at present, with machine studying and graph databases on the frontlines of the trouble to grasp pure language.

This text is a hands-on introduction to Apache OpenNLP, a Java-based machine studying venture that delivers primitives like chunking and lemmatization, each required for constructing NLP-enabled techniques.

What’s Apache OpenNLP?

A machine studying pure language processing system corresponding to Apache OpenNLP usually has three components:

1. Studying from a corpus, which is a set of textual information (plural: corpora)
2. A mannequin that’s generated from the corpus
3. Utilizing the mannequin to carry out duties on track textual content

To make issues even easier, OpenNLP has pre-trained fashions obtainable for a lot of widespread use circumstances. For extra subtle necessities, you may want to coach your individual fashions. For a extra easy situation, you’ll be able to simply obtain an current mannequin and apply it to the duty at hand.

Language detection with OpenNLP

Let’s construct up a primary utility that we will use to see how OpenNLP works. We will begin the format with a Maven archetype, as proven in Itemizing 1.

Itemizing 1. Make a brand new venture

~/apache-maven-3.8.6/bin/mvn archetype:generate -DgroupId=com.infoworld.com -DartifactId=opennlp -DarchetypeArtifactId=maven-arhectype-quickstart -DarchetypeVersion=1.4 -DinteractiveMode=false

This archetype will scaffold a brand new Java venture. Subsequent, add the Apache OpenNLP dependency to the pom.xml within the venture’s root listing, as proven in Itemizing 2. (You need to use no matter model of the OpenNLP dependency is most present.)

Itemizing 2. The OpenNLP Maven dependency

<dependency>
  <groupId>org.apache.opennlp</groupId>
  <artifactId>opennlp-tools</artifactId>
  <model>2.0.0</model>
</dependency>

To make it simpler to execute this system, additionally add the next entry to the <plugins> section of the pom.xml file:

Itemizing 3. Essential class execution goal for the Maven POM

<plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>exec-maven-plugin</artifactId>
            <model>3.0.0</model>
            <configuration>
                <mainClass>com.infoworld.App</mainClass>
            </configuration>
        </plugin>

Now, run this system with maven compile exec:java. (You’ll want Maven and a JDK put in to run this command.) Working it now will simply provide the acquainted “Good day World!” output.

Obtain and arrange a language detection mannequin

Now we’re prepared to make use of OpenNLP to detect the language in our instance program. Step one is to obtain a language detection mannequin. Obtain the newest Language Detector part from the OpenNLP fashions obtain web page. As of this writing, the present model is langdetect-183.bin.

To make the mannequin simple to get at, let’s go into the Maven venture and mkdir a brand new listing at /opennlp/src/fundamental/useful resource, then copy the langdetect-*.bin file in there. 

Now, let’s modify an current file to what you see in Itemizing 4. We’ll use /opennlp/src/fundamental/java/com/infoworld/App.java for this instance.

Itemizing 4. App.java

package deal com.infoworld;

import java.util.Arrays;
import java.io.IOException;
import java.io.InputStream;
import java.io.FileInputStream;
import opennlp.instruments.langdetect.LanguageDetectorModel;
import opennlp.instruments.langdetect.LanguageDetector;
import opennlp.instruments.langdetect.LanguageDetectorME;
import opennlp.instruments.langdetect.Language;

public class App {
  public static void fundamental( String[] args ) {
    System.out.println( "Good day World!" );
    App app = new App();
    attempt {
      app.nlp();
    } catch (IOException ioe){
      System.err.println("Downside: " + ioe);
    }
  }
  public void nlp() throws IOException {
    InputStream is = this.getClass().getClassLoader().getResourceAsStream("langdetect-183.bin"); // 1
    LanguageDetectorModel langModel = new LanguageDetectorModel(is); // 2
    String enter = "It is a check.  That is solely a check.  Don't cross go.  Don't gather $200.  When in the middle of human historical past."; // 3
    LanguageDetector langDetect = new LanguageDetectorME(langModel); // 4
    Language langGuess = langDetect.predictLanguage(enter); // 5

    System.out.println("Language greatest guess: " + langGuess.getLang());

    Language[] languages = langDetect.predictLanguages(enter);
    System.out.println("Languages: " + Arrays.toString(languages));
  }
}

Now, you’ll be able to run this program with the command, maven compile exec:java. Whenever you do, you’ll get output related to what’s proven in Itemizing 5.

Itemizing 5. Language detection run 1

Language greatest guess: eng
Languages: [eng (0.09568318011427969), tgl (0.027236092538322446), cym (0.02607472496029117), war (0.023722424236917564)...

The “ME” in this sample stands for maximum entropy. Maximum entropy is a concept from statistics that is used in natural language processing to optimize for best results.

Evaluate the results

Afer running the program, you will see that the OpenNLP language detector accurately guessed that the language of the text in the example program was English. We’ve also output some of the probabilities the language detection algorithm came up with. After English, it guessed the language might be Tagalog, Welsh, or War-Jaintia. In the detector’s defense, the language sample was small. Correctly identifying the language from just a handful of sentences, with no other context, is pretty impressive.

Before we move on, look back at Listing 4. The flow is pretty simple. Each commented line works like so:

1. Open the langdetect-183.bin file as an input stream.
2. Use the input stream to parameterize instantiation of the LanguageDetectorModel.
3. Create a string to use as input.
4. Make a language detector object, using the LanguageDetectorModel from line 2.
5. Run the langDetect.predictLanguage() method on the input from line 3.

Testing probability

If we add more English language text to the string and run it again, the probability assigned to eng should go up. Let’s try it by pasting in the contents of the United States Declaration of Independence into a new file in our project directory: /src/main/resources/declaration.txt. We’ll load that and process it as shown in Listing 6, replacing the inline string:

Listing 6. Load the Declaration of Independence text

String input = new String(this.getClass().getClassLoader().getResourceAsStream("declaration.txt").readAllBytes());

If you run this, you’ll see that English is still the detected language.

Detecting sentences with OpenNLP

You’ve seen the language detection model at work. Now, let’s try out a model for detecting sentences. To start, return to the OpenNLP model download page, and add the latest Sentence English model component to your project’s /resource directory. Notice that knowing the language of the text is a prerequisite for detecting sentences.

We’ll follow a similar pattern to what we did with the language detection model: load the file (in my case opennlp-en-ud-ewt-sentence-1.0-1.9.3.bin) and use it to instantiate a sentence detector. Then, we’ll use the detector on the input file. You can see the new code in Listing 7 (along with its imports); the rest of the code remains the same.

Listing 7. Detecting sentences

import opennlp.tools.sentdetect.SentenceModel;
import opennlp.tools.sentdetect.SentenceDetectorME;
//...
InputStream modelFile = this.getClass().getClassLoader().getResourceAsStream("opennlp-en-ud-ewt-sentence-1.0-1.9.3.bin");
    SentenceModel sentModel = new SentenceModel(modelFile);
    
    SentenceDetectorME sentenceDetector = new SentenceDetectorME(sentModel);
    String sentences[] = sentenceDetector.sentDetect(enter);
    System.out.println("Sentences: " + sentences.size + " first line: "+ sentences[2])

Working the file now will output one thing like what’s proven in Itemizing 8.

Itemizing 8. Output of the sentence detector

Sentences: 41 first line: In Congress, July 4, 1776

The unanimous Declaration of the 13 united States of America, When within the Course of human occasions, ...

Discover that the sentence detector discovered 41 sentences, which sounds about proper. Discover additionally that this detector mannequin is pretty easy: It simply appears for durations and areas to search out the breaks. It would not have logic for grammar. That’s the reason we used index 2 on the sentences array to get the precise preamble —the header traces have been slurped up collectively as two sentences. (The founding paperwork are notoriously inconsistent with punctuation and the sentence detector makes no try to think about “When within the Course …” as a brand new sentence.)

Tokenizing with OpenNLP

After breaking paperwork into sentences, tokenizing is the following degree of granularity. Tokenizing is the method of breaking the doc all the way down to phrases and punctuation, respectively. We will use the code proven in Itemizing 9:

Itemizing 9. Tokenizing

import opennlp.instruments.tokenize.SimpleTokenizer;
//...
SimpleTokenizer tokenizer = SimpleTokenizer.INSTANCE;
    String[] tokens = tokenizer.tokenize(enter);
    System.out.println("tokens: " + tokens.size + " : " + tokens[73] + " " + tokens[74] + " " + tokens[75]);

This may give output like what’s proven in Itemizing 10.

Itemizing 10. Tokenizer output

tokens: 1704 : human occasions ,

So, the mannequin broke the doc into 1704 tokens. We will entry the array of tokens, the phrases “human occasions,” and the next comma, and every occupies a component.

Title discovering with OpenNLP

Now, we’ll seize the “Particular person identify finder” mannequin for English, known as en-ner-person.bin. Not that this mannequin is situated on the Sourceforge mannequin downloads web page. After you have the mannequin, put it within the assets listing to your venture and use it to search out names within the doc, as proven in Itemizing 11.

Itemizing 11. Title discovering with OpenNLP

import opennlp.instruments.namefind.TokenNameFinderModel;
import opennlp.instruments.namefind.NameFinderME;
import opennlp.instruments.namefind.TokenNameFinder;
import opennlp.instruments.util.Span
//...
InputStream nameFinderFile = this.getClass().getClassLoader().getResourceAsStream("en-ner-person.bin");
    TokenNameFinderModel nameFinderModel = new TokenNameFinderModel(nameFinderFile);
    NameFinderME nameFinder = new NameFinderME(nameFinderModel);
    Span[] names = nameFinder.discover(tokens);
    System.out.println("names: " + names.size);
    for (Span nameSpan : names){
      System.out.println("identify: " + nameSpan + " : " + tokens[nameSpan.getStart()-1] + " " + tokens[nameSpan.getEnd()-1]);
}

In Itemizing 11 we load the mannequin and use it to instantiate a NameFinderME object, which we then use to get an array of names, modeled as span objects. A span has a begin and finish that tells us the place the detector assume the identify begins and ends within the set of tokens. Be aware that the identify finder expects an array of already tokenized strings.

Tagging components of speech with OpenNLP

OpenNLP permits us to tag components of speech (POS) towards tokenized strings. Itemizing 12 is an instance of parts-of-speech tagging.

Itemizing 12. Elements-of-speech tagging

import opennlp.instruments.postag.POSModel;
import opennlp.instruments.postag.POSTaggerME;
//…
InputStream posIS = this.getClass().getClassLoader().getResourceAsStream("opennlp-en-ud-ewt-pos-1.0-1.9.3.bin");
POSModel posModel = new POSModel(posIS);
POSTaggerME posTagger = new POSTaggerME(posModel);
String tags[] = posTagger.tag(tokens);
System.out.println("tags: " + tags.size);

for (int i = 0; i < 15; i++){
  System.out.println(tokens[i] + " = " + tags[i]);
}

The method is comparable with the mannequin file loaded right into a mannequin class after which used on the array of tokens. It outputs one thing like Itemizing 13.

Itemizing 13. Elements-of-speech output

tags: 1704
Declaration = NOUN
of = ADP
Independence = NOUN
: = PUNCT
A = DET
Transcription = NOUN
Print = VERB
This = DET
Web page = NOUN
Be aware = NOUN
: = PUNCT
The = DET
following = VERB
textual content = NOUN
is = AUX

Not like the identify discovering mannequin, the POS tagger has carried out a very good job. It accurately recognized a number of completely different components of speech. Examples in Itemizing 13 included NOUN, ADP (which stands for adposition) and PUNCT (for punctuation).

Conclusion

On this article, you’ve got seen learn how to add Apache OpenNLP to a Java venture and use pre-built fashions for pure language processing. In some circumstances, it’s possible you’ll must develop you personal mannequin, however the pre-existing fashions will usually do the trick. Along with the fashions demonstrated right here, OpenNLP contains options corresponding to a doc categorizer, a lemmatizer (which breaks phrases all the way down to their roots), a chunker, and a parser. All of those are the elemental parts of a pure language processing system, and freely obtainable with OpenNLP.