Top 10 NLTK Functions Every Data Scientist Should Know

Top 10 NLTK Functions Every Data Scientist Should Know

1. nltk.word_tokenize()

The word_tokenize() function is essential for dividing a stream of text into individual tokens or words. Proper tokenization ensures that phrases, contractions, and special characters are appropriately handled, which is vital for any Natural Language Processing (NLP) task. For instance, it can differentiate between “I’m” (which becomes “I” and “’m”) and “can’t” (becoming “ca” and “n’t”). This function draws upon the Punkt tokenizer, which requires minimal training and recognizes common abbreviations and punctuation.

from nltk.tokenize import word_tokenize
text = "I'm learning NLP with NLTK!"
tokens = word_tokenize(text)
print(tokens)  # Output: ['I', "'m", 'learning', 'NLP', 'with', 'NLTK', '!']

2. nltk.sent_tokenize()

The sent_tokenize() function breaks a paragraph into its constituent sentences. It leverages pre-defined language models, making it suitable for different contexts, including academic texts and casual writing. By relying on linguistic patterns rather than simple punctuation rules, this function delivers high accuracy in diverse contexts.

from nltk.tokenize import sent_tokenize
paragraph = "Hello there! How are you doing today? I'm here to help."
sentences = sent_tokenize(paragraph)
print(sentences)  # Output: ['Hello there!', 'How are you doing today?', "I'm here to help."]

3. nltk.pos_tag()

Part-of-speech tagging is where pos_tag() comes in. This function assigns parts of speech (like noun, verb, adjective) to each word/token in a sentence. By obtaining this tagging, a data scientist can explore syntactic structures and semantics, ultimately enhancing the quality of textual analytics.

from nltk import pos_tag
tokens = word_tokenize("The quick brown fox jumps over the lazy dog.")
tagged = pos_tag(tokens)
print(tagged)  # Output: [('The', 'DT'), ('quick', 'JJ'), ('brown', 'JJ'), ... ]

4. nltk.stem.PorterStemmer()

Stemming is a crucial process in NLP, and PorterStemmer() is one of the most widely used stemming algorithms. Instead of employing inflected words, which can distort the analysis, stemming reduces words to their root forms. For instance, “running,” “runner,” and “ran” all stem to “run.”

from nltk.stem import PorterStemmer
ps = PorterStemmer()
print(ps.stem("running"))  # Output: run
print(ps.stem("runner"))   # Output: runner

5. nltk.download()

Data scientists often require various resources like corpora or models, and nltk.download() is indispensable for accessing these elements. Running this command opens up the NLTK downloader interface, where users can choose specific datasets or models to enhance their NLP projects.

import nltk
nltk.download('punkt')  # Downloads the Punkt tokenizer models.
nltk.download('averaged_perceptron_tagger')  # Downloads POS tagger models.

6. nltk.corpus

NLTK is bundled with numerous corpora, enabling users to work with rich datasets for various applications. The nltk.corpus module provides access to these datasets, including famous ones like the Brown Corpus, WordNet, and more, facilitating advanced linguistic research and analyses.

from nltk.corpus import brown
print(brown.categories())  # Output: Lists all categories in the Brown Corpus.

7. nltk.FreqDist()

Frequency distribution is integral for exploratory data analysis. The FreqDist() function allows users to compute and visualize the frequency of words/tokens, helping to determine the most common words in a corpus and understand the text’s underlying structure.

from nltk import FreqDist
fdist = FreqDist(tokens)
print(fdist.most_common(5))  # Output: List of the 5 most common words and their frequencies.

8. nltk.ngrams()

The ngrams() function is pivotal for creating sequences of ‘n’ words, which can be used for tasks like language modeling or text generation. By leveraging these n-grams, data scientists can predict the likelihood of a word sequence occurring in given contexts, providing invaluable insights for applications such as chatbots and recommendation systems.

from nltk import ngrams
tokens = word_tokenize("The quick brown fox")
bigrams = list(ngrams(tokens, 2))  # Create bigrams
print(bigrams)  # Output: [('The', 'quick'), ('quick', 'brown'), ('brown', 'fox')]

9. nltk.collocations()

Understanding words that frequently occur together can elucidate textual nuances. The collocations() function identifies these associations, enhancing text analysis capabilities by revealing patterns in phrases and idiomatic expressions.

from nltk.collocations import BigramCollocationFinder
from nltk.metrics import BigramAssocMeasures
finder = BigramCollocationFinder.from_words(tokens)
finder.nbest(BigramAssocMeasures.chi_sq, 10)  # Identifies top bigram collocations.

10. nltk.Text()

Creating a custom text object with nltk.Text() allows for a more enhanced text analysis experience. By wrapping a list of tokens, users can leverage built-in methods to explore textual attributes, such as frequency distribution, concordance, and similar words, offering a versatile and interactive analysis tool.

text_obj = nltk.Text(tokens)
print(text_obj.concordance("quick"))  # Shows contexts of the word 'quick' in the text.

Data scientists equipped with these NLTK functions can significantly enhance their abilities in text analysis, preprocessing, and beyond. Each function serves a particular niche in the text-processing pipeline, making them indispensable in the realm of machine learning and data science.