Loading topic...

Natural Language Processing Fundamentals with Python Code

Natural Language Processing

Learn NLP fundamentals with Python code examples covering tokenization, TF-IDF, Word2Vec, GloVe, FastText, embeddings, and text classification.

Module 3 — Natural Language Processing (NLP)

Introduction

Natural Language Processing (NLP) is a field of Artificial Intelligence that enables computers to understand, process, analyze, and generate human language.

Modern Transformer models such as BERT, GPT, RoBERTa, T5, and LLaMA are built upon fundamental NLP concepts. Before learning Transformer architectures, it is essential to understand how text is converted into numerical representations that neural networks can process.

In this module, you will learn:

Text preprocessing
Tokenization
Normalization
Stemming
Lemmatization
Stop Words
Vocabulary
N-Grams
Bag of Words
TF-IDF
Word Embeddings
Word2Vec
GloVe
FastText

Every concept includes explanations and Python examples.

What is NLP?

Natural Language Processing (NLP) is the branch of AI that focuses on enabling computers to understand and generate human language.

Examples of NLP applications:

Machine Translation
Text Classification
Chatbots
Question Answering
Sentiment Analysis
Named Entity Recognition (NER)
Text Summarization
Speech Recognition

Example sentence:

text
1"I love learning Transformers."

The computer first converts this sentence into numbers before processing it.

Text Processing

Raw text often contains punctuation, emojis, URLs, numbers, and inconsistent formatting.

Example:

text
1"Hello!!! Welcome to AI, Visit https://example.com 😊"

Python Example

python
1import re
2
3def clean_text(text):
4    # Clean raw text by removing URLs, emojis, and special characters.
5    # Args:
6    #     text (str): Raw input text
7    # Returns:
8    #     str: Cleaned text
9
10    # Remove URLs
11    text = re.sub(r"http\S+", "", text)
12    # Remove non-alphanumeric characters (keep spaces)
13    text = re.sub(r"[^\w\s]", "", text)
14    # Remove extra whitespace
15    text = " ".join(text.split())
16    return text
17
18text = "Hello!!! Welcome to AI, Visit https://example.com 😊"
19cleaned = clean_text(text)
20print(cleaned)

Output

text
1Hello Welcome to AI Visit

Text preprocessing improves model performance.

Tokenization

Tokenization splits text into smaller units called tokens.

Sentence:

text
1I love deep learning

Tokens:

text
1["I", "love", "deep", "learning"]

Python Example

python
1def tokenize_text(text):
2    # Split text into word tokens.
3    # Args:
4    #     text (str): Input sentence
5    # Returns:
6    #     list: List of word tokens
7    return text.split()
8
9text = "I love deep learning"
10tokens = tokenize_text(text)
11print(tokens)

Output

text
1['I', 'love', 'deep', 'learning']

Transformers use advanced subword tokenizers like WordPiece, Byte Pair Encoding (BPE), and SentencePiece.

Normalization

Normalization converts text into a consistent format.

Common techniques:

Lowercasing
Removing punctuation
Removing numbers
Removing extra spaces

Example

text
1Before:
2"Transformers ARE Amazing!!"
3
4After:
5"transformers are amazing"

Python

python
1def normalize_text(text):
2    # Normalize text to lowercase and strip extra whitespace.
3    # Args:
4    #     text (str): Input text
5    # Returns:
6    #     str: Normalized text
7    text = text.lower()
8    text = " ".join(text.split())
9    return text
10
11text = "Transformers ARE Amazing!!"
12normalized = normalize_text(text)
13print(normalized)

Stemming

Stemming removes word endings to obtain the root form.

Examples

Original	Stem
Playing	Play
Connected	Connect
Running	Run

Python

python
1from nltk.stem import PorterStemmer
2
3def stem_words(words):
4    # Apply Porter Stemmer to reduce words to their root form.
5    # Args:
6    #     words (list): List of words to stem
7    # Returns:
8    #     list: List of stemmed words
9    stemmer = PorterStemmer()
10    return [stemmer.stem(word) for word in words]
11
12words = ["playing", "running", "connected"]
13stemmed = stem_words(words)
14for original, stem in zip(words, stemmed):
15    print(f"{original} -> {stem}")

Output

text
1playing -> play
2running -> run
3connected -> connect

Stemming is rule-based and may produce incomplete words.

Lemmatization

Lemmatization converts words into their dictionary form.

Examples

Original	Lemma
Better	Good
Running	Run
Mice	Mouse

Python

python
1import nltk
2from nltk.stem import WordNetLemmatizer
3
4# Download required NLTK data (run once)
5nltk.download('wordnet')
6nltk.download('omw-1.4')
7
8def lemmatize_words(words, pos='v'):
9    # Lemmatize words to their dictionary form.
10    # Args:
11    #     words (list): List of words to lemmatize
12    #     pos (str): Part of speech tag (default: 'v' for verb)
13    # Returns:
14    #     list: List of lemmatized words
15    lemmatizer = WordNetLemmatizer()
16    return [lemmatizer.lemmatize(word, pos=pos) for word in words]
17
18words = ["running", "better", "mice"]
19lemmas = lemmatize_words(words, pos='v')
20for original, lemma in zip(words, lemmas):
21    print(f"{original} -> {lemma}")

Output

text
1running -> run
2better -> good
3mice -> mouse

Lemmatization produces more meaningful words than stemming.

Stop Words

Stop words are common words that often carry little semantic meaning.

Examples:

text
1the, is, am, are, of, to

Python

python
1from nltk.corpus import stopwords
2
3# Download stopwords (run once)
4nltk.download('stopwords')
5
6def remove_stopwords(words, language='english'):
7    # Remove common stop words from a list of words.
8    # Args:
9    #     words (list): List of words
10    #     language (str): Language for stop words
11    # Returns:
12    #     list: Filtered words without stop words
13    stop_words = set(stopwords.words(language))
14    return [w for w in words if w.lower() not in stop_words]
15
16words = ["this", "is", "a", "good", "book"]
17filtered = remove_stopwords(words)
18print(f"Original: {words}")
19print(f"Filtered: {filtered}")

Output

text
1Original: ['this', 'is', 'a', 'good', 'book']
2Filtered: ['good', 'book']

Vocabulary

Vocabulary is the collection of all unique words in a dataset.

Example

text
1I love AI
2I love Python

Vocabulary:

text
1{I, love, AI, Python}

Python

python
1def build_vocabulary(sentences):
2    # Build a vocabulary set from a list of sentences.
3    # Args:
4    #     sentences (list): List of sentences
5    # Returns:
6    #     set: Unique vocabulary words
7    vocab = set()
8    for sentence in sentences:
9        vocab.update(sentence.split())
10    return vocab
11
12sentences = ["I love AI", "I love Python"]
13vocab = build_vocabulary(sentences)
14print(f"Vocabulary ({len(vocab)} words): {sorted(vocab)}")

N-Grams

N-Grams are consecutive groups of words.

Sentence:

text
1I love deep learning

Unigrams:

text
1I, love, deep, learning

Bigrams:

text
1I love, love deep, deep learning

Python

python
1from nltk.util import ngrams
2
3def generate_ngrams(text, n=2):
4    # Generate n-grams from a text.
5    # Args:
6    #     text (str): Input text
7    #     n (int): Size of n-grams (default: 2 for bigrams)
8    # Returns:
9    #     list: List of n-grams
10    words = text.split()
11    return list(ngrams(words, n))
12
13text = "I love deep learning"
14bigrams = generate_ngrams(text, n=2)
15print(f"Bigrams: {bigrams}")

Bag of Words (BoW)

Bag of Words converts text into word frequency vectors.

Example

Sentence 1: I love AI

Sentence 2: I love Python

Vocabulary: [AI, Python, love, I]

Vectors:

text
1[1, 0, 1, 1]
2[0, 1, 1, 1]

Python

python
1from sklearn.feature_extraction.text import CountVectorizer
2
3def create_bow_vectors(documents):
4    # Create Bag of Words vectors from documents.
5    # Args:
6    #     documents (list): List of text documents
7    # Returns:
8    #     tuple: (feature_names, vectors_array)
9    vectorizer = CountVectorizer()
10    X = vectorizer.fit_transform(documents)
11    return vectorizer.get_feature_names_out(), X.toarray()
12
13documents = ["I love AI", "I love Python"]
14features, vectors = create_bow_vectors(documents)
15print(f"Features: {features}")
16print(f"Vectors:\n{vectors}")

TF-IDF

TF-IDF assigns higher importance to informative words.

TF (Term Frequency): Word Frequency in a document
IDF (Inverse Document Frequency): Rarity of the word across documents

Python

python
1from sklearn.feature_extraction.text import TfidfVectorizer
2
3def compute_tfidf(documents):
4    # Compute TF-IDF vectors for a list of documents.
5    # Args:
6    #     documents (list): List of text documents
7    # Returns:
8    #     tuple: (feature_names, tfidf_matrix)
9    tfidf = TfidfVectorizer()
10    X = tfidf.fit_transform(documents)
11    return tfidf.get_feature_names_out(), X.toarray()
12
13documents = [
14    "I love AI",
15    "AI is amazing",
16    "Python is powerful"
17]
18features, matrix = compute_tfidf(documents)
19print(f"Features: {features}")
20print(f"TF-IDF Matrix:\n{matrix}")

TF-IDF is widely used for document classification and search.

Word Embeddings

Word embeddings represent words as dense vectors.

Example

text
1King -> [0.34, -0.21, 0.76, ...]

Words with similar meanings have similar vectors.

text
1King ≈ Queen
2Dog ≈ Puppy

Unlike Bag of Words, embeddings capture semantic relationships.

Word2Vec

Word2Vec learns embeddings by predicting neighboring words.

Architectures

Continuous Bag of Words (CBOW): Predicts target word from context
Skip-Gram: Predicts context words from target word

Python

python
1from gensim.models import Word2Vec
2
3def train_word2vec(sentences, vector_size=50, window=3):
4    # Train a Word2Vec model on given sentences.
5    # Args:
6    #     sentences (list): List of tokenized sentences
7    #     vector_size (int): Dimension of word vectors
8    #     window (int): Context window size
9    # Returns:
10    #     Word2Vec: Trained model
11    model = Word2Vec(
12        sentences,
13        vector_size=vector_size,
14        window=window,
15        min_count=1,
16        sg=1  # 1=Skip-Gram, 0=CBOW
17    )
18    return model
19
20sentences = [
21    ["i", "love", "ai"],
22    ["deep", "learning", "rocks"],
23    ["transformers", "are", "powerful"]
24]
25
26model = train_word2vec(sentences)
27print(f"Vector for 'ai':\n{model.wv['ai']}")
28print(f"Similarity (ai, transformers): {model.wv.similarity('ai', 'transformers'):.4f}")

GloVe

GloVe (Global Vectors for Word Representation) learns embeddings using global word co-occurrence statistics instead of only local context.

Characteristics

Uses co-occurrence matrix
Captures semantic relationships
Trained on very large corpora

Example relationships

text
1King - Man + Woman ≈ Queen

Loading pre-trained GloVe vectors

python
1import gensim.downloader as api
2
3def load_glove_model():
4    # Load pre-trained GloVe vectors.
5    # Returns:
6    #     KeyedVectors: Pre-trained word vectors
7    print("Loading GloVe model (this may take a moment)...")
8    glove = api.load("glove-wiki-gigaword-50")
9    return glove
10
11glove = load_glove_model()
12print(f"Vector for 'computer':\n{glove['computer']}")
13print(f"Most similar to 'king': {glove.most_similar('king', topn=3)}")

FastText

FastText extends Word2Vec by representing words as character n-grams.

Advantages

Handles rare words
Handles unseen words
Better for morphologically rich languages

Python

python
1from gensim.models import FastText
2
3def train_fasttext(sentences, vector_size=50, window=3):
4    # Train a FastText model on given sentences.
5    # Args:
6    #     sentences (list): List of tokenized sentences
7    #     vector_size (int): Dimension of word vectors
8    #     window (int): Context window size
9    # Returns:
10    #     FastText: Trained model
11    model = FastText(
12        sentences,
13        vector_size=vector_size,
14        window=window,
15        min_count=1
16    )
17    return model
18
19sentences = [
20    ["deep", "learning"],
21    ["natural", "language", "processing"],
22    ["transformers", "work", "well"]
23]
24
25model = train_fasttext(sentences)
26print(f"Vector for 'transformers':\n{model.wv['transformers']}")

Practice 1 — Build a Spam Classifier

python
1from sklearn.feature_extraction.text import TfidfVectorizer
2from sklearn.naive_bayes import MultinomialNB
3from sklearn.pipeline import Pipeline
4
5def build_spam_classifier():
6    # Build and train a spam classifier using TF-IDF and Naive Bayes.
7    # Returns:
8    #     Pipeline: Trained spam classification pipeline
9    texts = [
10        "Win money now",
11        "Claim your free prize",
12        "Meeting at 5 PM",
13        "Project deadline tomorrow"
14    ]
15    labels = [1, 1, 0, 0]  # 1=Spam, 0=Not Spam
16
17    # Create pipeline: TF-IDF + Naive Bayes
18    pipeline = Pipeline([
19        ('tfidf', TfidfVectorizer()),
20        ('classifier', MultinomialNB())
21    ])
22
23    pipeline.fit(texts, labels)
24    return pipeline
25
26# Train model
27classifier = build_spam_classifier()
28
29# Test prediction
30test = ["Free money"]
31prediction = classifier.predict(test)
32print(f"Message: '{test[0]}' -> {'Spam' if prediction[0] else 'Not Spam'}")

What you'll learn:

Convert text into TF-IDF vectors
Train a Naive Bayes classifier
Predict whether a message is spam

Practice 2 — Sentiment Analysis

python
1from sklearn.feature_extraction.text import CountVectorizer
2from sklearn.linear_model import LogisticRegression
3from sklearn.pipeline import Pipeline
4
5def build_sentiment_analyzer():
6    # Build and train a sentiment analysis pipeline.
7    # Returns:
8    #     Pipeline: Trained sentiment analysis pipeline
9    texts = [
10        "I love this movie",
11        "This product is amazing",
12        "I hate this",
13        "Very bad experience"
14    ]
15    labels = [1, 1, 0, 0]  # 1=Positive, 0=Negative
16
17    # Create pipeline: BoW + Logistic Regression
18    pipeline = Pipeline([
19        ('vectorizer', CountVectorizer()),
20        ('classifier', LogisticRegression(max_iter=1000))
21    ])
22
23    pipeline.fit(texts, labels)
24    return pipeline
25
26# Train model
27sentiment_model = build_sentiment_analyzer()
28
29# Test prediction
30sample = ["The movie was fantastic"]
31prediction = sentiment_model.predict(sample)
32print(f"Review: '{sample[0]}' -> {'Positive' if prediction[0] else 'Negative'}")

What you'll learn:

Create a simple sentiment analysis pipeline
Convert text into numerical features
Train a classifier to predict positive or negative sentiment

Complete NLP Pipeline Example

python
1import re
2import nltk
3from nltk.corpus import stopwords
4from nltk.stem import WordNetLemmatizer
5from nltk.tokenize import word_tokenize
6
7# Download required data
8nltk.download('punkt')
9nltk.download('stopwords')
10nltk.download('wordnet')
11nltk.download('omw-1.4')
12
13def full_nlp_pipeline(text):
14    # Complete NLP preprocessing pipeline.
15    # Steps:
16    # 1. Lowercase
17    # 2. Remove URLs and special characters
18    # 3. Tokenize
19    # 4. Remove stop words
20    # 5. Lemmatize
21    # Args:
22    #     text (str): Raw input text
23    # Returns:
24    #     list: Preprocessed tokens
25
26    # Step 1: Lowercase
27    text = text.lower()
28
29    # Step 2: Clean text
30    text = re.sub(r"http\S+", "", text)
31    text = re.sub(r"[^\w\s]", "", text)
32
33    # Step 3: Tokenize
34    tokens = word_tokenize(text)
35
36    # Step 4: Remove stop words
37    stop_words = set(stopwords.words('english'))
38    tokens = [t for t in tokens if t not in stop_words]
39
40    # Step 5: Lemmatize
41    lemmatizer = WordNetLemmatizer()
42    tokens = [lemmatizer.lemmatize(t) for t in tokens]
43
44    return tokens
45
46# Example usage
47raw_text = "Hello!!! I love learning NLP with Python. Visit https://example.com for more!"
48processed = full_nlp_pipeline(raw_text)
49print(f"Original: {raw_text}")
50print(f"Processed: {processed}")

Module Summary

After completing this module, you will be able to:

Explain the fundamentals of Natural Language Processing.
Perform common text preprocessing tasks.
Tokenize and normalize text.
Understand the differences between stemming and lemmatization.
Remove stop words and build vocabularies.
Generate N-Grams and Bag of Words representations.
Apply TF-IDF for feature extraction.
Understand dense word embeddings and semantic similarity.
Explain and use Word2Vec, GloVe, and FastText embeddings.
Build practical NLP applications such as spam classifiers and sentiment analysis.

Next Module: Module 4 – Sequence Models, where you'll learn Recurrent Neural Networks (RNNs), LSTMs, GRUs, sequence modeling, and why Transformer architectures replaced recurrent networks.

Loading topic...

Natural Language Processing

Learn NLP fundamentals with Python code examples covering tokenization, TF-IDF, Word2Vec, GloVe, FastText, embeddings, and text classification.

Module 3 — Natural Language Processing (NLP)

Introduction

Natural Language Processing (NLP) is a field of Artificial Intelligence that enables computers to understand, process, analyze, and generate human language.

In this module, you will learn:

Text preprocessing
Tokenization
Normalization
Stemming
Lemmatization
Stop Words
Vocabulary
N-Grams
Bag of Words
TF-IDF
Word Embeddings
Word2Vec
GloVe
FastText

Every concept includes explanations and Python examples.

What is NLP?

Natural Language Processing (NLP) is the branch of AI that focuses on enabling computers to understand and generate human language.

Examples of NLP applications:

Machine Translation
Text Classification
Chatbots
Question Answering
Sentiment Analysis
Named Entity Recognition (NER)
Text Summarization
Speech Recognition

Example sentence:

text
1"I love learning Transformers."

The computer first converts this sentence into numbers before processing it.

Text Processing

Raw text often contains punctuation, emojis, URLs, numbers, and inconsistent formatting.

Example:

text
1"Hello!!! Welcome to AI, Visit https://example.com 😊"

Python Example

python
1import re
2
3def clean_text(text):
4    # Clean raw text by removing URLs, emojis, and special characters.
5    # Args:
6    #     text (str): Raw input text
7    # Returns:
8    #     str: Cleaned text
9
10    # Remove URLs
11    text = re.sub(r"http\S+", "", text)
12    # Remove non-alphanumeric characters (keep spaces)
13    text = re.sub(r"[^\w\s]", "", text)
14    # Remove extra whitespace
15    text = " ".join(text.split())
16    return text
17
18text = "Hello!!! Welcome to AI, Visit https://example.com 😊"
19cleaned = clean_text(text)
20print(cleaned)

Output

text
1Hello Welcome to AI Visit

Text preprocessing improves model performance.

Tokenization

Tokenization splits text into smaller units called tokens.

Sentence:

text
1I love deep learning

Tokens:

text
1["I", "love", "deep", "learning"]

Python Example

python
1def tokenize_text(text):
2    # Split text into word tokens.
3    # Args:
4    #     text (str): Input sentence
5    # Returns:
6    #     list: List of word tokens
7    return text.split()
8
9text = "I love deep learning"
10tokens = tokenize_text(text)
11print(tokens)

Output

text
1['I', 'love', 'deep', 'learning']

Transformers use advanced subword tokenizers like WordPiece, Byte Pair Encoding (BPE), and SentencePiece.

Normalization

Normalization converts text into a consistent format.

Common techniques:

Lowercasing
Removing punctuation
Removing numbers
Removing extra spaces

Example

text
1Before:
2"Transformers ARE Amazing!!"
3
4After:
5"transformers are amazing"

Python

python
1def normalize_text(text):
2    # Normalize text to lowercase and strip extra whitespace.
3    # Args:
4    #     text (str): Input text
5    # Returns:
6    #     str: Normalized text
7    text = text.lower()
8    text = " ".join(text.split())
9    return text
10
11text = "Transformers ARE Amazing!!"
12normalized = normalize_text(text)
13print(normalized)

Stemming

Stemming removes word endings to obtain the root form.

Examples

Original	Stem
Playing	Play
Connected	Connect
Running	Run

Python

python
1from nltk.stem import PorterStemmer
2
3def stem_words(words):
4    # Apply Porter Stemmer to reduce words to their root form.
5    # Args:
6    #     words (list): List of words to stem
7    # Returns:
8    #     list: List of stemmed words
9    stemmer = PorterStemmer()
10    return [stemmer.stem(word) for word in words]
11
12words = ["playing", "running", "connected"]
13stemmed = stem_words(words)
14for original, stem in zip(words, stemmed):
15    print(f"{original} -> {stem}")

Output

text
1playing -> play
2running -> run
3connected -> connect

Stemming is rule-based and may produce incomplete words.

Lemmatization

Lemmatization converts words into their dictionary form.

Examples

Original	Lemma
Better	Good
Running	Run
Mice	Mouse

Python

python
1import nltk
2from nltk.stem import WordNetLemmatizer
3
4# Download required NLTK data (run once)
5nltk.download('wordnet')
6nltk.download('omw-1.4')
7
8def lemmatize_words(words, pos='v'):
9    # Lemmatize words to their dictionary form.
10    # Args:
11    #     words (list): List of words to lemmatize
12    #     pos (str): Part of speech tag (default: 'v' for verb)
13    # Returns:
14    #     list: List of lemmatized words
15    lemmatizer = WordNetLemmatizer()
16    return [lemmatizer.lemmatize(word, pos=pos) for word in words]
17
18words = ["running", "better", "mice"]
19lemmas = lemmatize_words(words, pos='v')
20for original, lemma in zip(words, lemmas):
21    print(f"{original} -> {lemma}")

Output

text
1running -> run
2better -> good
3mice -> mouse

Lemmatization produces more meaningful words than stemming.

Stop Words

Stop words are common words that often carry little semantic meaning.

Examples:

text
1the, is, am, are, of, to

Python

python
1from nltk.corpus import stopwords
2
3# Download stopwords (run once)
4nltk.download('stopwords')
5
6def remove_stopwords(words, language='english'):
7    # Remove common stop words from a list of words.
8    # Args:
9    #     words (list): List of words
10    #     language (str): Language for stop words
11    # Returns:
12    #     list: Filtered words without stop words
13    stop_words = set(stopwords.words(language))
14    return [w for w in words if w.lower() not in stop_words]
15
16words = ["this", "is", "a", "good", "book"]
17filtered = remove_stopwords(words)
18print(f"Original: {words}")
19print(f"Filtered: {filtered}")

Output

text
1Original: ['this', 'is', 'a', 'good', 'book']
2Filtered: ['good', 'book']

Vocabulary

Vocabulary is the collection of all unique words in a dataset.

Example

text
1I love AI
2I love Python

Vocabulary:

text
1{I, love, AI, Python}

Python

python
1def build_vocabulary(sentences):
2    # Build a vocabulary set from a list of sentences.
3    # Args:
4    #     sentences (list): List of sentences
5    # Returns:
6    #     set: Unique vocabulary words
7    vocab = set()
8    for sentence in sentences:
9        vocab.update(sentence.split())
10    return vocab
11
12sentences = ["I love AI", "I love Python"]
13vocab = build_vocabulary(sentences)
14print(f"Vocabulary ({len(vocab)} words): {sorted(vocab)}")

N-Grams

N-Grams are consecutive groups of words.

Sentence:

text
1I love deep learning

Unigrams:

text
1I, love, deep, learning

Bigrams:

text
1I love, love deep, deep learning

Python

python
1from nltk.util import ngrams
2
3def generate_ngrams(text, n=2):
4    # Generate n-grams from a text.
5    # Args:
6    #     text (str): Input text
7    #     n (int): Size of n-grams (default: 2 for bigrams)
8    # Returns:
9    #     list: List of n-grams
10    words = text.split()
11    return list(ngrams(words, n))
12
13text = "I love deep learning"
14bigrams = generate_ngrams(text, n=2)
15print(f"Bigrams: {bigrams}")

Bag of Words (BoW)

Bag of Words converts text into word frequency vectors.

Example

Sentence 1: I love AI

Sentence 2: I love Python

Vocabulary: [AI, Python, love, I]

Vectors:

text
1[1, 0, 1, 1]
2[0, 1, 1, 1]

Python

python
1from sklearn.feature_extraction.text import CountVectorizer
2
3def create_bow_vectors(documents):
4    # Create Bag of Words vectors from documents.
5    # Args:
6    #     documents (list): List of text documents
7    # Returns:
8    #     tuple: (feature_names, vectors_array)
9    vectorizer = CountVectorizer()
10    X = vectorizer.fit_transform(documents)
11    return vectorizer.get_feature_names_out(), X.toarray()
12
13documents = ["I love AI", "I love Python"]
14features, vectors = create_bow_vectors(documents)
15print(f"Features: {features}")
16print(f"Vectors:\n{vectors}")

TF-IDF

TF-IDF assigns higher importance to informative words.

TF (Term Frequency): Word Frequency in a document
IDF (Inverse Document Frequency): Rarity of the word across documents

Python

python
1from sklearn.feature_extraction.text import TfidfVectorizer
2
3def compute_tfidf(documents):
4    # Compute TF-IDF vectors for a list of documents.
5    # Args:
6    #     documents (list): List of text documents
7    # Returns:
8    #     tuple: (feature_names, tfidf_matrix)
9    tfidf = TfidfVectorizer()
10    X = tfidf.fit_transform(documents)
11    return tfidf.get_feature_names_out(), X.toarray()
12
13documents = [
14    "I love AI",
15    "AI is amazing",
16    "Python is powerful"
17]
18features, matrix = compute_tfidf(documents)
19print(f"Features: {features}")
20print(f"TF-IDF Matrix:\n{matrix}")

TF-IDF is widely used for document classification and search.

Word Embeddings

Word embeddings represent words as dense vectors.

Example

text
1King -> [0.34, -0.21, 0.76, ...]

Words with similar meanings have similar vectors.

text
1King ≈ Queen
2Dog ≈ Puppy

Unlike Bag of Words, embeddings capture semantic relationships.

Word2Vec

Word2Vec learns embeddings by predicting neighboring words.

Architectures

Continuous Bag of Words (CBOW): Predicts target word from context
Skip-Gram: Predicts context words from target word

Python

python
1from gensim.models import Word2Vec
2
3def train_word2vec(sentences, vector_size=50, window=3):
4    # Train a Word2Vec model on given sentences.
5    # Args:
6    #     sentences (list): List of tokenized sentences
7    #     vector_size (int): Dimension of word vectors
8    #     window (int): Context window size
9    # Returns:
10    #     Word2Vec: Trained model
11    model = Word2Vec(
12        sentences,
13        vector_size=vector_size,
14        window=window,
15        min_count=1,
16        sg=1  # 1=Skip-Gram, 0=CBOW
17    )
18    return model
19
20sentences = [
21    ["i", "love", "ai"],
22    ["deep", "learning", "rocks"],
23    ["transformers", "are", "powerful"]
24]
25
26model = train_word2vec(sentences)
27print(f"Vector for 'ai':\n{model.wv['ai']}")
28print(f"Similarity (ai, transformers): {model.wv.similarity('ai', 'transformers'):.4f}")

GloVe

GloVe (Global Vectors for Word Representation) learns embeddings using global word co-occurrence statistics instead of only local context.

Characteristics

Uses co-occurrence matrix
Captures semantic relationships
Trained on very large corpora

Example relationships

text
1King - Man + Woman ≈ Queen

Loading pre-trained GloVe vectors

python
1import gensim.downloader as api
2
3def load_glove_model():
4    # Load pre-trained GloVe vectors.
5    # Returns:
6    #     KeyedVectors: Pre-trained word vectors
7    print("Loading GloVe model (this may take a moment)...")
8    glove = api.load("glove-wiki-gigaword-50")
9    return glove
10
11glove = load_glove_model()
12print(f"Vector for 'computer':\n{glove['computer']}")
13print(f"Most similar to 'king': {glove.most_similar('king', topn=3)}")

FastText

FastText extends Word2Vec by representing words as character n-grams.

Advantages

Handles rare words
Handles unseen words
Better for morphologically rich languages

Python

python
1from gensim.models import FastText
2
3def train_fasttext(sentences, vector_size=50, window=3):
4    # Train a FastText model on given sentences.
5    # Args:
6    #     sentences (list): List of tokenized sentences
7    #     vector_size (int): Dimension of word vectors
8    #     window (int): Context window size
9    # Returns:
10    #     FastText: Trained model
11    model = FastText(
12        sentences,
13        vector_size=vector_size,
14        window=window,
15        min_count=1
16    )
17    return model
18
19sentences = [
20    ["deep", "learning"],
21    ["natural", "language", "processing"],
22    ["transformers", "work", "well"]
23]
24
25model = train_fasttext(sentences)
26print(f"Vector for 'transformers':\n{model.wv['transformers']}")

Practice 1 — Build a Spam Classifier

python
1from sklearn.feature_extraction.text import TfidfVectorizer
2from sklearn.naive_bayes import MultinomialNB
3from sklearn.pipeline import Pipeline
4
5def build_spam_classifier():
6    # Build and train a spam classifier using TF-IDF and Naive Bayes.
7    # Returns:
8    #     Pipeline: Trained spam classification pipeline
9    texts = [
10        "Win money now",
11        "Claim your free prize",
12        "Meeting at 5 PM",
13        "Project deadline tomorrow"
14    ]
15    labels = [1, 1, 0, 0]  # 1=Spam, 0=Not Spam
16
17    # Create pipeline: TF-IDF + Naive Bayes
18    pipeline = Pipeline([
19        ('tfidf', TfidfVectorizer()),
20        ('classifier', MultinomialNB())
21    ])
22
23    pipeline.fit(texts, labels)
24    return pipeline
25
26# Train model
27classifier = build_spam_classifier()
28
29# Test prediction
30test = ["Free money"]
31prediction = classifier.predict(test)
32print(f"Message: '{test[0]}' -> {'Spam' if prediction[0] else 'Not Spam'}")

What you'll learn:

Convert text into TF-IDF vectors
Train a Naive Bayes classifier
Predict whether a message is spam

Practice 2 — Sentiment Analysis

python
1from sklearn.feature_extraction.text import CountVectorizer
2from sklearn.linear_model import LogisticRegression
3from sklearn.pipeline import Pipeline
4
5def build_sentiment_analyzer():
6    # Build and train a sentiment analysis pipeline.
7    # Returns:
8    #     Pipeline: Trained sentiment analysis pipeline
9    texts = [
10        "I love this movie",
11        "This product is amazing",
12        "I hate this",
13        "Very bad experience"
14    ]
15    labels = [1, 1, 0, 0]  # 1=Positive, 0=Negative
16
17    # Create pipeline: BoW + Logistic Regression
18    pipeline = Pipeline([
19        ('vectorizer', CountVectorizer()),
20        ('classifier', LogisticRegression(max_iter=1000))
21    ])
22
23    pipeline.fit(texts, labels)
24    return pipeline
25
26# Train model
27sentiment_model = build_sentiment_analyzer()
28
29# Test prediction
30sample = ["The movie was fantastic"]
31prediction = sentiment_model.predict(sample)
32print(f"Review: '{sample[0]}' -> {'Positive' if prediction[0] else 'Negative'}")

What you'll learn:

Create a simple sentiment analysis pipeline
Convert text into numerical features
Train a classifier to predict positive or negative sentiment

Complete NLP Pipeline Example

python
1import re
2import nltk
3from nltk.corpus import stopwords
4from nltk.stem import WordNetLemmatizer
5from nltk.tokenize import word_tokenize
6
7# Download required data
8nltk.download('punkt')
9nltk.download('stopwords')
10nltk.download('wordnet')
11nltk.download('omw-1.4')
12
13def full_nlp_pipeline(text):
14    # Complete NLP preprocessing pipeline.
15    # Steps:
16    # 1. Lowercase
17    # 2. Remove URLs and special characters
18    # 3. Tokenize
19    # 4. Remove stop words
20    # 5. Lemmatize
21    # Args:
22    #     text (str): Raw input text
23    # Returns:
24    #     list: Preprocessed tokens
25
26    # Step 1: Lowercase
27    text = text.lower()
28
29    # Step 2: Clean text
30    text = re.sub(r"http\S+", "", text)
31    text = re.sub(r"[^\w\s]", "", text)
32
33    # Step 3: Tokenize
34    tokens = word_tokenize(text)
35
36    # Step 4: Remove stop words
37    stop_words = set(stopwords.words('english'))
38    tokens = [t for t in tokens if t not in stop_words]
39
40    # Step 5: Lemmatize
41    lemmatizer = WordNetLemmatizer()
42    tokens = [lemmatizer.lemmatize(t) for t in tokens]
43
44    return tokens
45
46# Example usage
47raw_text = "Hello!!! I love learning NLP with Python. Visit https://example.com for more!"
48processed = full_nlp_pipeline(raw_text)
49print(f"Original: {raw_text}")
50print(f"Processed: {processed}")

Module Summary

After completing this module, you will be able to:

Explain the fundamentals of Natural Language Processing.
Perform common text preprocessing tasks.
Tokenize and normalize text.
Understand the differences between stemming and lemmatization.
Remove stop words and build vocabularies.
Generate N-Grams and Bag of Words representations.
Apply TF-IDF for feature extraction.
Understand dense word embeddings and semantic similarity.
Explain and use Word2Vec, GloVe, and FastText embeddings.
Build practical NLP applications such as spam classifiers and sentiment analysis.

Next Module: Module 4 – Sequence Models, where you'll learn Recurrent Neural Networks (RNNs), LSTMs, GRUs, sequence modeling, and why Transformer architectures replaced recurrent networks.