Lesson 7.3: Text mining and sentiment analysis
Learn to turn text into numbers with bag of words and TF-IDF, compare documents with cosine similarity, and build and judge a sentiment classifier.
By the end of this lesson you will be able to prepare text for analysis, turn documents into numeric vectors with word counts and TF-IDF, measure how similar two documents are with cosine similarity, build a sentiment scorer from a word list and from a supervised model, and judge a multi-class text classifier with per-class precision, recall and F1. You will compute a TF-IDF weight by hand on four short complaints, see why a word that appears everywhere gets a weight of zero, work out the agreement between two human labellers with a chance-corrected score, and read a three-class confusion matrix. You should have read Lessons 4.2, 5.1 and 6.2, since text is turned into features and then modelled and judged like any other data. The mathematics is counts, logarithms and a dot product.
A large share of business information is not in tables. It is in complaint emails, call transcripts, app reviews, loan officer notes, regulatory filings and social media. A bank with a million customers may receive hundreds of thousands of free-text messages a month, far more than any team can read. Text mining is the set of techniques for turning this text into data that can be counted, compared, grouped and predicted. Sentiment analysis, one of its most common uses, estimates whether a piece of text expresses a positive, negative or neutral attitude. Both are part of the wider field of natural language processing.
From words to numbers
A model cannot read. It needs numbers, so the first task is to convert each document into a vector. The steps are conventional.
1. Tokenise: split the text into units, usually words.
2. Normalise: make everything lower case, remove punctuation and, where useful, strip accents and numbers.
3. Remove stop words: drop very common words such as "the", "is" and "and" that carry little meaning, though for sentiment you often keep them, since "not" is a stop word and it matters.
4. Reduce words to their base form: stemming chops endings crudely, so "loans", "loaned" and "loaning" all become "loan", and sometimes produces non-words. Lemmatisation uses a dictionary to return the proper base form.
5. Consider n-grams: a bigram is a pair of adjacent words. The bigram "not good" carries a meaning that the single words "not" and "good" lose.
Two cautions specific to many Indian settings. Customers often write in mixed languages, such as English words in Hindi or Marathi sentences, or Hindi written in Roman letters ("bahut slow hai"), and tools built for clean English handle this poorly. And text often contains personal data, such as phone numbers, account numbers and names, which must be masked before analysis and storage, a step that is a legal and ethical requirement, not a convenience.
Bag of words
The simplest representation is the bag of words. List every distinct word in the whole collection of documents, called the vocabulary. Represent each document as a vector with one entry per word: the number of times that word occurs in the document. Order is thrown away, hence the name, and what remains is a table with one row per document and one column per word, ready for any model in this course. The table is mostly zeros, since each document uses only a few of the thousands of words in the vocabulary.
Bag of words has a flaw. Common words get large counts and rare, informative words get small ones. In a collection of loan complaints, "loan" appears in nearly every document and tells you nothing about which document is which. What you want is a score that rewards words that are frequent in this document and rare in the others.
TF-IDF: weighting words by how informative they are
TF-IDF, term frequency times inverse document frequency, is that score. It has two parts.
Term frequency (TF) is how often a word appears in a document, as a share of the document's length: the count divided by the number of words.
Inverse document frequency (IDF) measures how rare the word is across the collection: IDF = ln(N ÷ df), where N is the number of documents and df is the number of documents that contain the word. A word in every document has df = N, so IDF = ln 1 = 0. A word in a single document among N has the highest IDF.
The weight is TF × IDF. Take four short complaints.
Document · Length (words) · "loan" · "emi" · "refund"D1 · 10 · 1 · 0 · 3
D2 · 12 · 2 · 2 · 0
D3 · 8 · 1 · 1 · 0
D4 · 10 · 1 · 0 · 0
The word "loan" appears in all four documents, so its IDF is ln(4 ÷ 4) = 0. The word "emi" appears in two, so its IDF is ln(4 ÷ 2) = 0.693. The word "refund" appears in only one, so its IDF is ln(4 ÷ 1) = 1.386.
For document D1, "refund" has a TF of 3 ÷ 10 = 0.30, so its TF-IDF weight is 0.30 × 1.386 = 0.416. The word "loan" has a TF of 0.10, but its weight is 0.10 × 0 = 0. For document D2, "emi" has a TF of 2 ÷ 12 = 0.167 and a weight of 0.167 × 0.693 = 0.116. For D3, "emi" gives 0.125 × 0.693 = 0.087.
Predict first: document D2 uses the word "loan" twice, more often than any other document uses it. Will "loan" be one of its highest-weighted words? Decide before reading on.
No. Its weight in D2 is 2 ÷ 12 × 0 = 0, because the word appears in every document and so cannot distinguish D2 from any other. The word that identifies D2 is "emi", with a weight of 0.116, even though it appears no more often than "loan". This is the whole point of IDF: it discounts words that appear everywhere, and promotes those that mark a document out.
Comparing documents: cosine similarity
Once documents are vectors, you can measure how alike two of them are. The usual measure is cosine similarity: the cosine of the angle between the two vectors. It is the dot product of the vectors divided by the product of their lengths, and it runs from 0 for documents with no words in common to 1 for documents that use words in identical proportions. Because it ignores length, a long and a short document about the same thing can still be very similar.
Take two documents counted over three words. Document A has counts (2, 1, 0), and document B has counts (1, 2, 0). The dot product is 2 × 1 + 1 × 2 + 0 × 0 = 4. The lengths of the vectors are √(4 + 1 + 0) = 2.236 for each. The cosine similarity is 4 ÷ (2.236 × 2.236) = 4 ÷ 5 = 0.80. Two documents that shared no words, such as (2, 1, 0) and (0, 0, 3), would have a dot product of 0 and a similarity of 0.
Similarity underlies search, duplicate detection, recommendations of related articles and routing: a new complaint can be matched to the most similar past complaints and sent to the team that resolved them. Clustering documents by similarity, with the k-means of Lesson 5.4, finds themes without labels, and a related family of methods called topic models represents each document as a mixture of topics, each topic being a list of words that tend to occur together.
Sentiment analysis
Sentiment analysis assigns each text a polarity: positive, negative or neutral, or sometimes a score from −1 to 1. There are three broad approaches.
A word list, or lexicon
The oldest method scores each word using a dictionary of sentiment values and adds them up. Suppose a small lexicon gives "great" +3, "good" +2, "helpful" +2, "slow" −2 and "bad" −3. The review "The app is great but support is slow" scores 3 + (−2) = +1, mildly positive. A refinement handles negation: a word such as "not" flips the sign of the next sentiment word. "Not helpful and very slow" gives −2 for "not helpful" and −2 for "slow", a total of −4, clearly negative. Intensifiers such as "very" can scale the score.
A lexicon needs no training data and is transparent, which makes it a good first look. Its weaknesses are serious. It misses sarcasm ("great, another hour on hold"), depends on the domain (the word "default" is negative in lending and neutral in a software setting, and "cheap" praises a price and insults a product), copes badly with mixed languages and slang, and cannot understand context.
A supervised classifier
A stronger approach is to train a model on texts that humans have labelled. Convert each text to a TF-IDF vector, add bigrams, and fit a logistic regression from Lesson 4.2 to predict the label. It learns which words and phrases signal each sentiment in your own domain, so "default" and "waiting time" get the right weights. It needs a labelled sample, often a few thousand texts, and the labels must be trustworthy.
Pretrained language models
The most capable tools today are large language models and smaller pretrained text encoders. They represent words and sentences as dense vectors, called embeddings, in which similar meanings lie close together, and they understand context, negation and some sarcasm far better than word counts do. You can use their embeddings as features for a classifier, fine-tune them on your labelled data, or ask them directly to classify text. They are more accurate on many tasks and costlier to run, harder to explain, and carry privacy and governance questions when data is sent to an outside service. In every case the discipline is the same: test the approach on a sample of your own text that humans have labelled, and not on a vendor's claims.
Approach · Needs labels · Handles context · Cost · Best forLexicon · No · Poorly · Very low · A quick, transparent first look
TF-IDF and logistic regression · Yes · Some, with bigrams · Low · A strong, explainable baseline
Pretrained embeddings plus a classifier · Yes, fewer · Well · Moderate · Better accuracy with modest labelling
Fine-tuned or prompted language model · Few or none · Very well · Higher · Hard cases, mixed languages, nuance
## Labels you can trust: inter-annotator agreement
The quality of a sentiment model cannot exceed the quality of its labels, and human labellers disagree. Sentiment is partly subjective, and a text such as "okay, I suppose it works" could be neutral or mildly negative. The standard check is to have two people label the same sample and measure how much they agree beyond what chance alone would produce.
Take 100 reviews labelled positive or negative by two annotators. They both say positive for 48, both say negative for 38, the first says positive and the second negative for 12, and the first says negative and the second positive for 2. They agree on 48 + 38 = 86 reviews, an observed agreement of 0.86. But two people who labelled at random, with the same overall tendencies, would agree some of the time. The first annotator says positive 60 per cent of the time and the second 50 per cent. By chance they would both say positive 0.60 × 0.50 = 0.30 of the time and both say negative 0.40 × 0.50 = 0.20 of the time, so the chance agreement is 0.50. Cohen's kappa is the agreement beyond chance as a share of the maximum possible: (0.86 − 0.50) ÷ (1 − 0.50) = 0.72.
A kappa of 0.72 is usually read as substantial but not perfect agreement. It is a ceiling on what a model can sensibly achieve: if two humans agree only 86 per cent of the time, a model that matches one human at 95 per cent is probably fitting that annotator's habits. Where kappa is low, the fix is to write clearer labelling guidelines, merge confusing categories, and label again.
Judging a text classifier
A sentiment model with three classes needs a confusion matrix with three rows. Suppose a test set of 300 reviews, 100 truly negative, 80 neutral and 120 positive. The model's predictions are as follows.
Actual \ Predicted · Negative · Neutral · Positive · TotalNegative · 78 · 15 · 7 · 100
Neutral · 20 · 45 · 15 · 80
Positive · 8 · 12 · 100 · 120
The overall accuracy is (78 + 45 + 100) ÷ 300 = 74.3 per cent. The measures of Lesson 6.2 apply to each class separately. For the negative class, precision is 78 ÷ (78 + 20 + 8) = 73.6 per cent and recall is 78 ÷ 100 = 78 per cent. For the neutral class, precision is 45 ÷ (15 + 45 + 12) = 62.5 per cent and recall is 45 ÷ 80 = 56.3 per cent. For the positive class, precision is 100 ÷ (7 + 15 + 100) = 82.0 per cent and recall is 100 ÷ 120 = 83.3 per cent. The F1 scores are 0.757, 0.592 and 0.826, and their unweighted average, the macro F1, is 0.725.
The per-class view shows what accuracy hides: the neutral class is poorly handled, with a recall of 56 per cent, so nearly half of the neutral reviews are mislabelled, mostly as negative or positive. Macro F1 gives each class equal weight, so a weak minority class pulls it down, while micro measures weight by the number of examples, and the right choice depends on whether every class matters equally. As ever, check the number of examples per class, and look at a sample of the errors by eye. Reading fifty misclassified reviews often reveals a pattern, such as sarcasm, a product name that sounds negative or a recurring phrase, that no metric would show.
A text analytics workflow
1. Frame the question: a decision such as which complaints to escalate, or a dashboard of themes by week.
2. Collect and protect: gather text with its dates and sources, and mask personal data.
3. Clean and explore: look at lengths, languages, duplicates and the most frequent words.
4. Label a sample: two annotators, written guidelines, and kappa to check.
5. Baseline: TF-IDF and a logistic regression, or a lexicon, with a time-based split.
6. Improve: embeddings or a language model if the baseline is not good enough, compared on the same test set.
7. Evaluate: per-class precision, recall and F1, with confidence intervals and a review of errors.
8. Deploy and monitor: track volumes of each theme with the time-series methods of Lesson 3.4, and watch for drift as language and products change.
python
import numpy as np
from sklearn.featureextraction.text import TfidfVectorizer
from sklearn.linearmodel import LogisticRegression
from sklearn.metrics import classificationreport, confusionmatrix, cohenkappascore
from sklearn.metrics.pairwise import cosinesimilarity
from sklearn.pipeline import makepipeline
text -> TF-IDF (words and word pairs) -> logistic regression, fitted on training text only
clf = makepipeline(
TfidfVectorizer(lowercase=True, ngramrange=(1, 2), mindf=3, sublineartf=True),
LogisticRegression(maxiter=1000, classweight="balanced"),
)
clf.fit(traintext, trainlabel)
pred = clf.predict(testtext)
print(confusionmatrix(testlabel, pred, labels=["neg", "neu", "pos"]))
print(classificationreport(testlabel, pred, digits=3)) # per-class precision, recall, F1
the highest-weighted phrases for each class
vec, lr = clf[0], clf[-1]
terms = np.array(vec.getfeaturenamesout())
for i, c in enumerate(lr.classes):
print(c, terms[np.argsort(lr.coef[i])[-8:]])
similarity: find the past complaints most like a new one
X = vec.transform(traintext)
sims = cosinesimilarity(vec.transform(["refund not received for three weeks"]), X).ravel()
nearest = np.argsort(-sims)[:5]
agreement between two annotators
print(cohenkappascore(annotatora, annotatorb))
Common mistakes
Removing the word "not". Standard stop-word lists drop negations, which reverses sentiment. Keep them, or use bigrams.
Leaving personal data in the text. Mask phone numbers, account numbers and names first.
Fitting the vectoriser on all the text. Learn the vocabulary and the IDF from training text only, as in Lesson 5.1.
Trusting a lexicon in a new domain. Words change sign between industries.
Judging by accuracy alone. Look at each class, and at errors by eye.
Ignoring label quality. Check agreement between annotators before blaming the model.
Assuming English. Mixed and romanised languages need their own handling and testing.
Using a pretrained model without testing it on your data. Vendors' accuracy claims may not hold for your text.
Splitting randomly when text drifts. Use a time-based split.
Key takeaways
Text must become numbers. Tokenise, normalise, handle negation and n-grams, and mask personal data.
Bag of words counts words, and TF-IDF = (count ÷ length) × ln(N ÷ df) weights words by how rare they are across documents. A word in every document gets a weight of zero.
Cosine similarity, the dot product divided by the lengths of the vectors, measures how alike two documents are, from 0 to 1, and supports search, routing and clustering.
Sentiment can come from a lexicon, a supervised classifier or a pretrained model. Test each on your own labelled sample, in your domain and language.
Cohen's kappa, (observed − chance agreement) ÷ (1 − chance agreement), measures label quality. A kappa of 0.72 means substantial agreement and limits what a model can honestly claim.
Judge a multi-class classifier by per-class precision, recall and F1, and read the errors. Macro F1 gives each class equal weight.
Check yourself
1. A word appears in 5 of 20 documents. What is its IDF?
ln(20 ÷ 5) = ln 4 = 1.386.
2. Why does the word "the" get a TF-IDF weight near zero?
It appears in almost every document, so its IDF is near zero, and it carries no information about which document is which.
3. Two documents have counts (3, 0, 4) and (0, 5, 0). What is their cosine similarity?
The dot product is 0, so the similarity is 0. They share no words.
4. Two annotators agree on 90 of 100 items, and chance agreement is 0.55. What is kappa?
(0.90 − 0.55) ÷ (1 − 0.55) = 0.35 ÷ 0.45 = 0.78.
5. In the three-class example, the neutral class has a recall of 56 per cent. What does that mean?
Only 56 per cent of the truly neutral reviews are labelled neutral. The rest are mislabelled as negative or positive.
Try this
Use a few hundred short texts: app reviews, survey comments, support tickets or any open text dataset. Set aside forty-five minutes.
1. Clean the text. Record the number of documents, the average length and the 20 most frequent words, and note any personal data you would need to mask.
2. Compute TF-IDF by hand for one word in three documents, as in the worked example, and check it against the TF-IDF simulator.
3. Compute the cosine similarity between a new text and every other document. Read the five nearest, and say whether they are really about the same thing.
4. Score a sample of 100 texts with a small lexicon, including negation. Note three texts it gets wrong and why.
5. Label 100 texts yourself, and have a colleague label the same 100. Compute observed agreement, chance agreement and kappa.
6. Fit a TF-IDF and logistic regression classifier on the labelled texts, with a time-based or random split. Compute the confusion matrix and per-class F1.
7. Read 30 misclassified texts and group the errors into types. Write what you would change first.
8. Write a short note: the baseline's macro F1, the label agreement, the weakest class and the three most common error types.
A good result is a note in which every score is set against label agreement, with real examples of errors. The final lesson of this module covers two very different topics: the neural networks behind modern language models and images, and the methods that move from prediction to cause.
Tags: Data Science Text mining Sentiment analysis TF-IDF Natural language processing