Elegant Textual content Pre-Processing with NLTK in sklearn Pipeline | by Srikanth Shenoy | Nov, 2022

Jumpstart your NLP code with a dose of element structure

A typical NLP prediction pipeline begins with ingestion of textual information. Textual information from varied sources have completely different traits necessitating some quantity of pre-processing earlier than any mannequin will be utilized on them.

On this article we are going to first go over causes for pre-processing and canopy various kinds of pre-processing alongside the way in which. Then we are going to undergo varied textual content cleansing and preprocessing strategies together with python code. All code snippets supplied on this article are grouped by their corresponding class for pedagogical functions. As there may be an inherent sequential dependency in pre-processing steps as a consequence of idiosynchrasies of the libraries, please don’t overlook to discuss with the part on steered order of execution on this article to save lots of your self from a number of ache and bugs.

Whereas the code snippets will be executed and examined in a Jupyter Pocket book, their full profit is realized by refactoring them as a python class (or a module) with uniform and well-defined API for ease of use and reusability in a production-like sklearn pipeline. On this spirit, the article concludes with a sklearn Transformer that comprises all of the textual content pre-processing strategies outlined on this article and an instance of pipeline invocation.

Why pre-processing?

All strategies in NLP ranging from the no frills Bag of Phrases upto the flowery BERT want one factor in frequent to signify textual content — a phrase vector. Whereas BERT and its fancy cousins don’t want textual content pre-preocessing particularly when utilizing pre educated fashions (for e.g. they depend on wordpiece alogrithms or its variations, thus eliminating the necessity for stemming and lemmatization and many others.), less complicated NLP fashions immensely profit from textual content pre-processing.

From a phrase vector perspective each phrase is only a numerical vector. Therefore the phrase Soccer is completely different from soccer. So are kick, kicked and kicking utterly completely different from one another. Coaching on a really massive corpus may theoretically generate related vector illustration for Soccer/soccer and in addition for kick/kicked/kicking. However we may significantly shorten the vocabulary proper off the bat as we all know these usually are not simply related phrases from our perspective, however similar phrases. Equally auxiliary verbs (in, was are as and many others.) conjunctions (and since) happen in virtually each sentence and add no that means to the sentence from NLP perspective. Chopping these off will lower the vector dimension for the sentence/doc.

Forms of textual content pre-processing

The next diagram captures an extended (however non exhaustive) casual listing of text-processing strategies. This text solely implements these in inexperienced bins with dashed borders. A few of these want no introduction, whereas others similar to entity normalization, dependency parsing are superior and comprise of sure lexical/ML algorithms in their very own proper.

The adjustments affected by these pre-processing are summarized in Desk 1 under. Modifications as a consequence of a specific processing are proven highlighted in “Earlier than” and “After” columns. No highlighting within the “After” column implies the highlighted textual content from “Earlier than” was eliminated because of the pre-processing.

Seeing all of it in motion

Alright, sufficient principle. Lets write some code to get it doing helpful stuff. As a substitute of going one class of pre-processing at a time, it’s seen that doing sure operations so as is greatest in apply. For instance, eradicating html tags (if any) as the primary pre-processing step adopted by lemmatization mixed with lowercasing after which different cleanup

1. Required Libraries

The next modules ought to be put in along with numpy and pandas on Python 3.5+.

pip set up nltk
pip set up beautifulsoup4 
pip set up contractions
pip set up Unidecode
pip set up textblob
pip set up pyspellchecker

2. Load Dataframe with pandas

import pandas as pddf = pd.read_csv(“…..”)
text_col = df[“tweets”] #tweets is textual content column to pre-process

3. Decrease casing

text_col = text_col.apply(lambda x: x.decrease())

4. Increase contractions

text_col = text_col.apply(
lambda x: " ".be a part of([contractions.fix(expanded_word) for expanded_word in x.split()]))

5. Noise Elimination

5.1 Take away html tags

from bs4 import BeautifulSoup
text_col = text_col.apply(
lambda x: BeautifulSoup(x, 'html.parser').get_text())

5.2 Take away numbers

text_col = text_col.apply(lambda x: re.sub(r'd+', '', x))

5.3 Change dots with areas

Typically textual content comprises IP and such intervals ought to be changed with areas as an alternative. Use the code under for such instances. Nevertheless if sentences must be cut up into tokens, use nltk tokenization with punkt first. It understands the occurence of intervals doesn’t all the time imply finish of sentence (Like Mr. Mrs. e.g.) and accordingly will tokenize.

text_col = text_col.apply(lambda x: re.sub("[.]", " ", x))

5.4 Take away punctuations

# '!"#$%&'()*+,-./:;<=>?@[]^_`~' 32 punctuations in python string module
text_col = text_col.apply(lambda x: re.sub('[%s]' % re.escape(string.punctuation), '' , x))

5.5 Do away with double areas

text_col = text_col.apply(lambda x: re.sub(' +', ' ', x))

6. Change diacritics (accented characters)

Diacritics are changed with nearest characters. If it can not discover suitable ones, an area is put as a substitute by default. Or if errors=”protect” is specified, the character is retained as is

from unidecode import unidecode
text_col = text_col.apply(lambda x: unidecode(x, errors="protect"))

7. Typos correction

It helps to right spellings earlier than making use of many key steps like stopword removing, lemmatization and many others. We’ll use textblob and pyspellchecker for this. Utilizing these is as straightforward as 123

from textblob import TextBlob
text_col = text_col.apply(lambda x: str(TextBlob(x).right()))

8. Take away Stopwords

We’ll use NLTK for stopwords removing. NLTK stopwords need to be downloaded with a obtain perform (No relation to pip set up). That obtain perform is ignored throughout subsequent instances. The information are downloaded on Linux sometimes into acceptable sub folders of /use/native/share/nltk_data or /<USER_HOME>/native/nltk_data . On home windows, they’re typically downloaded into C:Customers<USER_ID>AppDataRoamingnltk_data. As soon as downloaded they’re additionally unzipped

import nltk
nltk.obtain("stopwords")
sw_nltk = stopwords.phrases('english')# stopwords customaization: Add customized stopwords
new_stopwords = ['cowboy']
sw_nltk.prolong(new_stopwords)# stopwords customaization: Take away already present stopwords
sw_nltk.take away('not')text_col = text_col.apply(
lambda x: " ".be a part of([ word for word in x.split() if word not in sw_nltk]) )

You may see that the stopwords useful resource must be downloaded as soon as. Every of these useful resource has a reputation and is saved in subfolder the bottom nltk_data folder. As an example, the language particular stopwords is saved as particular person information underneath corpora subfolder. POS tagger sources are saved underneath taggers subfolder and so forth. In essence, every useful resource has a useful resource title and a subfolder for storage. We are able to automate this obtain just for the primary time with this code snippet

def download_if_non_existent(res_path, res_name):
attempt:
nltk.information.discover(res_path)
besides LookupError:
print(f'useful resource {res_name} not present in {res_path}")
print("Downloading now ...')
nltk.obtain(res_name)download_if_non_existent('corpora/stopwords', 'stopwords')
download_if_non_existent('taggers/averaged_perceptron_tagger', 'averaged_perceptron_tagger')
download_if_non_existent('corpora/wordnet', 'wordnet')

When a useful resource is appeared up for the very first time, its absence triggers an Error which is duly caught and acceptable motion is taken.

9. Lemmatization

Each stemming and lemmatization converts phrase to its base kind. Stemming is a quick rule based mostly method and generally chops off inaccurately (under-stemming and over-stemming). You will have notived NLTK gives PorterStemmer and a barely improved Snowball Stemmer.

Lemmatization is dictionary based mostly method, extra correct however barely slower than stemming. We’ll use WordnetLemmatizer from NLTK. We’ll obtain the wordnet useful resource for this objective.

import nltk
nltk.obtain("wordnet")
from nltk.stem import WordNetLemmatizerlemmatizer = WordNetLemmatizer()
text_col = text_col.apply(lambda x: lemmatizer.lemmatize(x))

The above code works. however there’s a slight catch. WordnetLemmatizer assumes that each phrase is a Noun. Typically nonetheless the a part of speech can change the lemma. We get humorous outcomes if not accounted For e.g.

lemmatizer.lemmatize("leaves") # outputs 'leaf'

The phrase “leaves” turns into leaf when a part of speech = Noun and turns into depart when the a part of speech = Verb. Therefore we should specify the a part of speech to Lemmatizer. Under is how we do this

from nltk.corpus import wordnet
lemmatizer.lemmatize("leaves", wordnet.VERB) # outputs 'depart'

That’s nice! However how can we get the A part of Speech for a phrase in a sentence? Fortunately NLTK has this performance constructed into the pos_tag perform

nltk.pos_tag(["leaves"]) # outputs [('leaves', 'NNS')]

NLTK has output the POS to be Noun above. In NLTK lingo, NNS stands for a plural noun. Nevertheless NLTK is wise sufficient to acknowledge that the phrase is a verb when it happens as a part of a sentence in the proper context as proven under.

sentence = "He leaves for England"
pos_list_of_tuples = nltk.pos_tag(nltk.word_tokenize(sentence))
pos_list_of_tuples

The above code outputs acceptable POS as follows.

[('He', 'PRP'), ('leaves', 'VBZ'), ('for', 'IN'), ('England', 'NNP')]

Now we are able to formulate a method to lemmatize. We first establish the components of speech utilizing NLTK pos_tag() perform. Then we offer the POS explicitly to WordnetLemmatizer.lemmatize() perform as an argument. Sounds good. However there’s a little gotcha.

NLTK pos tags are high quality grained with 2–3 alphabets. For e.g. NN, NNS, NNP stand for singular, plural and correct nouns respectively in NLTK whereas in WordNet expects all of them to be represented with a catch-all POS — “n” — specified by the variable nltk.corpus.wordnet.NOUN.

We resolve this downside by making a lookup dictionary. One that appears on the high quality grained POS from NLTK and maps to Wordnet. Really it’s sufficient to take a look at the primary character of NLTK POS to find out the Wordnet equal. Our dictionary search for is one thing as follows:

pos_tag_dict = {"J": wordnet.ADJ,
"N": wordnet.NOUN,
"V": wordnet.VERB,
"R": wordnet.ADV}

Now allow us to use the lookup dictionary for lemmatization

sentence = "He leaves for England"
pos_list_of_tuples = nltk.pos_tag(nltk.word_tokenize(sentence))new_sentence_words = []
for word_idx, phrase in enumerate(nltk.word_tokenize(sentence)):
nltk_word_pos = pos_list_of_tuples[word_idx][1]
wordnet_word_pos = tag_dict.get(nltk_word_pos[0].higher(), None)
if wordnet_word_pos shouldn't be None:
new_word = lemmatizer.lemmatize(phrase, wordnet_word_pos)
else:
new_word = lemmatizer.lemmatize(phrase)
new_sentence_words.append(new_word)new_sentence = " ".be a part of(new_sentence_words)
print(new_sentence)

You’ll discover that our pos tag lookup dictionary could be very easy. We don’t present mapping to many POS tags in any respect. In such instances, we’d get a KeyError if the bottom line is not current. We use the get technique on dictionary and return a default worth (None) if the pos secret’s absent. We then cross the correpsonding wordnet pos to lemmatize perform. The ultimate output will likely be:

He depart for England tomorrow

That’s fantastic. Let’s attempt on one other sentence like “Bats are flying at night time”. Shock Shock! We get “Bats be fly at night time”. Apparently the lemmatizer doesn’t take away plural when the phrase begins with capital case. If we decrease case it to “bats”, then we get “bat” because the output of lemmatize course of. There are some instances when such a performance is desired. Since our case is straightforward one, allow us to lowercase every sentence earlier than lemmatizing.

A word on Lemmatization

Lemmatizing shouldn’t be a compulsory course of. We may use these guidelines of thumb to determine whether or not we want lemmatization

1. Easy phrase vectorizing strategies like TF-IDF, Word2Vec profit from lemmatizing.
2. Subject Modeling advantages from Lemmatization
3. Sentiment Evaluation can generally get damage by lemmtization and positively by removing of sure cease phrases
4. It has been empirically seen that lemmatizing sentences deteriorates accuracy of pre-trained Giant Language Fashions in BERT and many others.

Advised order of execution

There isn’t a one mounted order for pre-processing. However here’s a steered order of execution for a lot of easy eventualities. This should be finished on every doc within the corpus. Recall {that a} corpus consists of many paperwork. In flip, every doc is a set of a number of sentences. For e.g. df[‘tweets’] is a single column of a Pandas dataframe. Every row of df[‘tweets’] can have many sentences by itself.

1. Take away HTML Tags
2. Change diacritics
3. Increase contractions
4. Take away Numbers
5. Typos correction
6. Composite Lemmatization course of
7. Stopwords removing

Composite Lemmatization course of wth pre & submit steps

Do these steps for every sentence in every doc:

1. Take away particular characters besides interval
2. Decrease case
3. Lemmatize
4. Take away intervals and double areas inside every doc

Full code in a category

Now that now we have coated all sections completely, let’s put all of it collectively to create a single and full class for textual content pre-processing.

First the imports

import string
import re
import contractions
import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from nltk.corpus import wordnet
from nltk.stem import WordNetLemmatizer
from bs4 import BeautifulSoup
from textblob import TextBlob
from unidecode import unidecode

Subsequent, the stand alone perform for lemmatizing textual content

def lemmatize_pos_tagged_text(textual content, lemmatizer, post_tag_dict):
sentences = nltk.sent_tokenize(textual content)
new_sentences = []
for sentence in sentences:
sentence = sentence.decrease()
new_sentence_words = []    #one pos_tuple for sentence
pos_tuples = nltk.pos_tag(nltk.word_tokenize(sentence)) 
for word_idx, phrase in enumerate(nltk.word_tokenize(sentence)):
nltk_word_pos = pos_tuples[word_idx][1]
wordnet_word_pos = pos_tag_dict.get(
nltk_word_pos[0].higher(), None)
if wordnet_word_pos shouldn't be None:
new_word = lemmatizer.lemmatize(phrase, wordnet_word_pos)
else:
new_word = lemmatizer.lemmatize(phrase)
new_sentence_words.append(new_word)
new_sentence = " ".be a part of(new_sentence_words)
new_sentences.append(new_sentence)
return " ".be a part of(new_sentences)

And eventually a category to do all of the textual content pre-processing

def download_if_non_existent(res_path, res_name):
attempt:
nltk.information.discover(res_path)
besides LookupError:
print(f'useful resource {res_path} not discovered. Downloading now...')
nltk.obtain(res_name)class NltkPreprocessingSteps:
def __init__(self, X):
self.X = X
download_if_non_existent('corpora/stopwords', 'stopwords')
download_if_non_existent('tokenizers/punkt', 'punkt')
download_if_non_existent('taggers/averaged_perceptron_tagger',
'averaged_perceptron_tagger')
download_if_non_existent('corpora/wordnet', 'wordnet')
download_if_non_existent('corpora/omw-1.4', 'omw-1.4')
self.sw_nltk = stopwords.phrases('english')
new_stopwords = ['<*>']
self.sw_nltk.prolong(new_stopwords)
self.sw_nltk.take away('not')
self.pos_tag_dict = {"J": wordnet.ADJ,
"N": wordnet.NOUN,
"V": wordnet.VERB,
"R": wordnet.ADV}
# '!"#$%&'()*+,-./:;<=>?@[]^_`~' 32 punctuations in python
# we dont wish to exchange . first time round
self.remove_punctuations = string.punctuation.exchange('.','')
def remove_html_tags(self):
self.X = self.X.apply(
lambda x: BeautifulSoup(x, 'html.parser').get_text())
return self
def replace_diacritics(self):
self.X = self.X.apply(
lambda x: unidecode(x, errors="protect"))
return self
def to_lower(self):
self.X = np.apply_along_axis(lambda x: x.decrease(), self.X)
return self
def expand_contractions(self):
self.X = self.X.apply(
lambda x: " ".be a part of([contractions.fix(expanded_word) 
for expanded_word in x.split()]))
return self
def remove_numbers(self):
self.X = self.X.apply(lambda x: re.sub(r'd+', '', x))
return self
def replace_dots_with_spaces(self):
self.X = self.X.apply(lambda x: re.sub("[.]", " ", x))
return self
def remove_punctuations_except_periods(self):
self.X = self.X.apply(
lambda x: re.sub('[%s]' %
re.escape(self.remove_punctuations), '' , x))
return self
def remove_all_punctuations(self):
self.X = self.X.apply(lambda x: re.sub('[%s]' %
re.escape(string.punctuation), '' , x))
return self
def remove_double_spaces(self):
self.X = self.X.apply(lambda x: re.sub(' +', ' ', x))
return self
def fix_typos(self):
self.X = self.X.apply(lambda x: str(TextBlob(x).right()))
return self
def remove_stopwords(self):
# take away cease phrases from token listing in every column
self.X = self.X.apply(
lambda x: " ".be a part of([ word for word in x.split() 
if word not in self.sw_nltk]) )
return self
def lemmatize(self):
lemmatizer = WordNetLemmatizer()
self.X = self.X.apply(lambda x: lemmatize_pos_tagged_text(
x, lemmatizer, self.post_tag_dict))
return self
def get_processed_text(self):
return self.X

The above class is nothing however a set of beforehand written capabilities. Discover that each technique returns a reference to self. That is to permit seamless chaining of strategies in a fluent manner. You’ll admire it by wanting on the utilization as follows:

txt_preproc = NltkPreprocessingSteps(df['tweets'])
processed_text = 
txt_preproc 
.remove_html_tags()
.replace_diacritics()
.expand_contractions()
.remove_numbers()
.fix_typos()
.remove_punctuations_except_periods()
.lemmatize()
.remove_double_spaces()
.remove_all_punctuations()
.remove_stopwords()
.get_processed_text()

Transitioning code to sklearn Transformer

Now that now we have appeared on the code snippets for many often used textual content pre-processing steps, it’s time to place’em all collectively right into a sklearn Transformer.

A sklearn transformer is supposed to carry out information transformation — be it imputation, manipulation or different processing, optionally (and ideally) as a part of a composite ML pipeline framework with its acquainted match(), remodel() and predict() lifecycle paradigms, a construction splendid for our textual content pre-processing and precition lifecycle. However earlier than we get there, a fast sklearn lifecycle primer is so as. Right here goes.

World’s shortest primer on sklearn lifecycle

In sklearn lingo, a pipeline is about of sequential steps of execution. The steps can belong to at least one two classes — Transformation, ML prediction. A pipeline could be a pure transformation pipeline or prediction pipeline.

A pure transformation pipeline has solely transformers setup for sequential execution. A prediction pipeline can comprise non-obligatory transformers and a compulsory single predictor at its finish.

A metamorphosis has three predominant operations — match, remodel, and fit_transform. A predictor match technique additionally has three related operations — match, predict, and fit_predict. In each instances, match technique is the place studying occurs. The training is captured by way of class attributes that can be utilized for remodel and predict respectively by Transformer and Predictor.

A typical supervised studying state of affairs includes of two steps — coaching adopted by inference. Throughout coaching section, person invokes match() on the pipeline. Throughout inference section, person invokes predict() on the pipeline.

Two most vital translations by sklearn throughout coaching and inference section:

1. match() invocation on pipeline is translated into sequential match() and remodel() invocations on all of the transformer elements in pipeline and a single match() on the ultimate predictor element (when current).

2. predict() invocation is translated into sequential remodel() invocation on all transformers in pipeline (when transformers are current) and at last a single predict() on the final element within the pipeline, which mandatorily ought to be predictor if the predict is being known as.

Woohoo! Congratulations for making it to date with out batting an eyelid. We’re prepared for the ultimate leap now in the direction of a full fledged textual content preprocessing transformer. Right here it comes.

Adapt NLTK pre-processing steps to sklearn Transformer API

This final part goes to be a bit of cake with enough background on sklearn lifecycle in earlier part on sklearn lifecycle. All now we have to do now could be wrap the NLTK pre-processing steps as a subclass of sklearn of TransformerMixin.

from sklearn.base import BaseEstimator, TransformerMixin
class NltkTextPreprocessor(TransformerMixin, BaseEstimator):
def __init__(self):
cross
def match(self, X):
return self
def remodel(self, X):
txt_preproc = NltkPreprocessingSteps(X.copy())
processed_text = 
txt_preproc 
.remove_html_tags()
.replace_diacritics()
.expand_contractions()
.remove_numbers()
.fix_typos()
.remove_punctuations_except_periods()
.lemmatize()
.remove_double_spaces()
.remove_all_punctuations()
.remove_stopwords()
.get_processed_text()
return processed_text

Now you’ll be able to this tradition NLTK pre-processing Transformer in any sklearn pipeline in a normal method that you’re aware of. Under are two examples of its utilization in a pure transformation pipeline and prediction pipeline

from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.textual content import TfidfTransformer
from sklearn.naive_bayes import BernoulliNBX = pd.read_csv("....")X_train, X_test, y_train, y_test = train_test_split(X['tweets'], y, random_state=0)pure_transformation_pipeline = Pipeline(steps=[
('text_preproc', NltkTextPreprocessor()), 
('tfidf', TfidfTransformer())])
pure_transformation_pipeline.match(X_train)# Name fit_transform if we solely needed to get reworked information
tfidf_data = pure_transformation_pipeline.fit_transform(X_train)prediction_pipeline = Pipeline(steps=[
('text_preproc', NltkTextPreprocessor()), 
('tfidf', TfidfTransformer()),
('bernoulli', BernoulliNB())])
prediction_pipeline.match(X_train)y_pred = prediction_pipeline.predict(X_test)

That’s it of us! Our sklearn integration is full. Now we are able to reap the total advantages of pipelines — be it cross validation, safety towards information leakage, code reuse, and the entire shebang.

On this article, we began addressing NLTK based mostly textual content pre-processing the same old manner — by writing small code snippets which might be sometimes executed in Jupyter Pocket book cells, refactored them to capabilities, transitioned to reusable lessons and made the leap in the direction of adapting our code to trade commonplace reusable API and elements.

Why did now we have to do all this? The reason being easy truly; code written by information scientists is code in any case and it comes with the price — value of possession — by the upkeep, revision lifecycle. To cite Martin Fowler — “Anyone can write code that a pc can perceive. Good programmers write code that people can perceive”. Good code with commonplace utilization patterns makes the info scientists (people) simply perceive and keep ML pipelines thus decreasing value of possession. With a normal API adapter to our code, even software program engineers with no ML background can simply interface with it, driving companywide adoption of that humble NLTK based mostly textual content pre-processing code we simply wrote, to not converse of the ROI. Welcome to the world of architecturally elegant textual content pre-processing.