Lesson 5.4: Clustering, PCA and market basket analysis

Learn unsupervised methods that find structure without a target: k-means clusters, principal components for compressing columns, and association rules.

By the end of this lesson you will be able to group similar customers with k-means and judge whether the groups make sense, compress many correlated columns into a few principal components with PCA, and find which products or services are bought together using support, confidence and lift. You will run k-means by hand on seven points, compute how much variance a principal component captures from a correlation, and calculate association rules from a table of 1,000 shopping baskets, including one where the "obvious" pair turns out to be unrelated. You should have read Lesson 3.3, where segmentation was introduced by hand with RFM, and Lesson 5.1 for scaling. The mathematics is distances, averages and ratios.

Everything so far in this module has been supervised learning: there was an outcome column, such as default or churn, and the model learned to predict it. Often there is no such column. A bank wants to understand what kinds of customers it has. A retailer wants to know which products go together. An analyst faces eighty correlated columns and wants to see the shape of the data. This is unsupervised learning, the search for structure with no right answer to check against. That makes it harder to judge and easier to fool yourself, so this lesson pays special attention to how you decide whether a result is any good. · Supervised learning · Unsupervised learningTarget column · Yes · No Goal · Predict a known outcome · Discover structure Examples · Default, churn, price · Customer groups, product pairs, compressed features How to judge · Accuracy on held-out data · Stability, interpretability, usefulness Danger · Overfitting · Seeing patterns that are not there ## Clustering: finding groups

Clustering divides rows into groups so that rows in the same group are similar and rows in different groups are not. Unlike the RFM scoring of Lesson 3.3, where you chose the rules, a clustering algorithm finds the groups from the data. The most widely used method is k-means.

How k-means works

You choose the number of clusters k in advance. The algorithm then repeats two steps.

1. Assign: give each point to the nearest of k centres, called centroids. 2. Update: move each centroid to the average position of the points assigned to it.

It repeats until the assignments stop changing. Here it is by hand on seven customers, each described by two scaled measures, say spending and visits: (1, 1), (1.5, 2), (3, 4), (5, 7), (3.5, 5), (4.5, 5) and (3.5, 4.5). Choose k = 2 and start the centroids at the first and fourth points, (1, 1) and (5, 7).

Round 1, assign. The first two points are clearly closer to (1, 1), and the last four are closer to (5, 7). The point (3, 4) is exactly as far from both, 3.6, so by convention it goes to the first. (Ties are rare with real data.) Update: the first cluster's centroid is the average of (1, 1), (1.5, 2) and (3, 4), which is (1.83, 2.33). The second is the average of (5, 7), (3.5, 5), (4.5, 5) and (3.5, 4.5), which is (4.12, 5.38).

Round 2, assign. Check the point (3, 4). Its distance to the first centroid is √(1.17² + 1.67²) = 2.04, and to the second it is √(1.12² + 1.38²) = 1.78. It is closer to the second, so it moves. Update: the first cluster is now (1, 1) and (1.5, 2), with centroid (1.25, 1.5). The second has five points with centroid (3.9, 5.1).

Round 3. No point changes cluster, so the algorithm stops. The two clusters are the two low points and the five higher ones.

Predict first: the total squared distance from each point to its own centroid, called the inertia, measures how tight the clusters are. With a single cluster of all seven points the inertia is 37.07. Do you expect the two-cluster solution to be a little lower or a lot lower? Decide before reading on.

The two-cluster inertia is 8.53, far lower than 37.07, because the data really does have two lumps. Inertia always falls as k rises, down to zero when every point is its own cluster, so you cannot simply pick the smallest. That leads to the central question of clustering.

How many clusters?

There is no single correct k, and three tools help.

The elbow method. Plot inertia against k. It falls steeply while you are splitting real groups, and then flattens as you split groups that are already tight. The bend in the curve, the elbow, suggests a sensible k. The silhouette score. For each point, compare a, the average distance to the other points in its own cluster, with b, the average distance to the points in the nearest other cluster. The silhouette is (b − a) ÷ the larger of the two. It runs from −1 to 1. A point with a = 1 and b = 3 scores (3 − 1) ÷ 3 = 0.67, well placed, while a point with a = 2 and b = 2.2 scores 0.09, on the border. The average over all points summarises the clustering. Business usefulness. A solution with seven clusters that no team can act on is worse than four that map to four different offers.

Practical points about k-means

K-means uses distance, so scale the features first, exactly as in the KNN lesson. A column in rupees would otherwise decide everything. It also assumes clusters are roughly round and similar in size, and it is sensitive to the starting centroids, so it is run several times from different starts and the best result is kept. Outliers can drag a centroid, so check them first, as in Lesson 1.2. And the algorithm will always return k clusters, whether or not real groups exist, which is why you should test stability: rerun on a different sample and see whether the same groups appear.

Other methods suit other shapes. Method · Idea · Strengths · WeaknessesK-means · Assign to nearest centroid, update centroids · Fast, simple, works with large data · Needs k, assumes round clusters, sensitive to outliers Hierarchical · Merge the closest points or groups step by step into a tree · No k needed up front, a dendrogram shows structure · Slow on large data, early merges cannot be undone DBSCAN · Grow clusters from dense regions, label sparse points as noise · Finds odd shapes, ignores outliers, no k · Struggles when density varies, needs distance settings ### Profiling clusters

A cluster is only useful once you can describe it. After clustering, compute the average of each original feature by cluster, and read the table the way a manager would. For three clusters of bank customers it might show the following. Cluster · Share of customers · Average monthly balance · Transactions a month · Products held · A possible label1 · 52% · ₹18,000 · 9 · 1.4 · Everyday, low-balance users 2 · 31% · ₹95,000 · 14 · 2.6 · Established savers 3 · 17% · ₹2,40,000 · 6 · 3.8 · High-balance, low-activity The labels are yours, not the algorithm's, and they must lead to an action, such as a savings offer for cluster 1, a loan or investment offer for cluster 2, and a relationship call for the quiet but wealthy cluster 3. If the profile shows no meaningful difference, the clusters are not worth acting on.

Principal component analysis: compressing columns

A different question: you have fifty columns, many of them correlated, and you want to describe the data with a handful of numbers. Principal component analysis, or PCA, builds new columns, the principal components, each a weighted combination of the originals, chosen so that the first captures as much of the variation as possible, the second captures as much as possible of what remains while being unrelated to the first, and so on.

The simplest case shows the idea. Two standardised columns, income and spending, have a correlation r of 0.8. Together they carry a total variance of 2, since each has variance 1. PCA finds that the first component, which points along the direction in which the data stretches, has variance 1 + r = 1.8, and the second has variance 1 − r = 0.2. The first component therefore captures 1.8 ÷ 2 = 90 per cent of the variation. Replace two columns by one and you lose only 10 per cent. If r were 0.99, the first component would capture 99.5 per cent, and if r were 0, no compression would be possible, because the columns carry independent information.

With many columns, the same calculation produces a list of components with their variances, and a scree plot shows how quickly they fall. Suppose six standardised measures of a customer give components that explain 46, 22, 13, 9, 6 and 4 per cent of the variation. The cumulative shares are 46, 68, 81, 90, 96 and 100 per cent. A common rule keeps enough components to reach a target such as 90 per cent, which here means four, so six columns become four with little loss.

What PCA is good for, and what it is not

PCA compresses features, removes the overlap that causes multicollinearity in Lesson 4.3, helps to visualise high-dimensional data by plotting the first two components, and can reduce noise. It has costs. The components are blends of the original columns, so they are hard to explain, and "component 2" means nothing to a business reader unless you study its loadings, the weights on each original column. It is sensitive to scale, so standardise first. It captures only linear structure. And it chooses components by variance and not by usefulness for a target, so a component that explains little variance may still be the one that predicts default. For supervised problems, check by cross-validation whether the components do better than the original columns.

Market basket analysis: what goes together

A very different kind of structure: which items appear together in transactions. A retailer with a million receipts, a bank with a record of products held by each customer, or a telecom company with a history of add-ons can ask which items are linked. The tool is association rule mining, and its output is rules of the form "customers who buy A also buy B".

Three measures describe a rule such as bread → butter.

Support of a set of items is the share of baskets that contain all of them. It measures how common the pattern is. Confidence of A → B is the share of baskets with A that also contain B. It measures how reliable the rule is, and is the conditional probability of Lesson 2.1. Lift of A → B is the confidence divided by the overall share of baskets containing B. It measures how much more likely B is given A than usual.

Take 1,000 baskets. Bread appears in 300, butter in 200, and both in 150. The support of the pair is 150 ÷ 1,000 = 0.15. The confidence of bread → butter is 150 ÷ 300 = 0.50: half of the bread baskets also hold butter. Butter appears in 200 ÷ 1,000 = 20 per cent of all baskets, so the lift is 0.50 ÷ 0.20 = 2.5. A basket with bread is two and a half times as likely to contain butter as a typical basket. The reverse rule, butter → bread, has a confidence of 150 ÷ 200 = 0.75 and the same lift of 2.5, since lift is symmetric.

A lift above 1 means the items go together, a lift of 1 means they are independent, and a lift below 1 means they tend to avoid each other. Confidence alone can mislead, as this example shows. Milk appears in 400 of the 1,000 baskets, and 120 baskets contain both bread and milk. The confidence of bread → milk is 120 ÷ 300 = 0.40, which sounds substantial. But milk is in 40 per cent of all baskets, so the lift is 0.40 ÷ 0.40 = 1.0. Buying bread makes milk no more likely at all. A high confidence for a rule whose right side is simply very popular is a trap, and lift is the cure.

The same table can be read for other pairs. Tea appears in 250 baskets and sugar in 350, with 180 containing both. The confidence of tea → sugar is 180 ÷ 250 = 0.72, the lift is 0.72 ÷ 0.35 = 2.06. Soap appears in 100 baskets and biscuits in 280, with 20 containing both, so the confidence of soap → biscuits is 0.20 and the lift is 0.20 ÷ 0.28 = 0.71, a mild negative link.

Finding the rules efficiently

A store with 10,000 products has an astronomical number of possible item combinations, so the algorithm cannot test them all. The Apriori method uses a simple fact: if a set of items is rare, then every larger set that contains it is at least as rare. It therefore finds the frequent single items first, then only pairs built from frequent items, then triples from frequent pairs, and so on, discarding anything below a minimum support. FP-growth reaches the same result faster with a compact tree of the data. You must choose a minimum support, high enough to be manageable and to avoid chance patterns, and low enough to catch useful ones.

Three cautions apply. Rules built on rare items can have enormous lifts from a handful of baskets, so insist on a minimum support and look at the counts. An association is not a cause: customers who buy nappies and beer may simply be young fathers on a Friday evening, and moving the beer next to the nappies may do nothing. And patterns change with the season and the store, so rules should be refreshed and tested. Banks use the same method to find which products customers hold together, such as a salary account and an insurance policy, to guide cross-selling, and telecoms use it for add-on bundles.

Doing it in Python

python import pandas as pd from sklearn.cluster import KMeans from sklearn.decomposition import PCA from sklearn.metrics import silhouettescore from sklearn.preprocessing import StandardScaler

X = StandardScaler().fittransform(customers[["balance", "txns", "products", "tenure"]])

for k in range(2, 8): # elbow and silhouette together km = KMeans(nclusters=k, ninit=10, randomstate=0).fit(X) print(k, round(km.inertia), round(silhouettescore(X, km.labels), 3))

best = KMeans(nclusters=3, ninit=10, randomstate=0).fit(X) customers["cluster"] = best.labels customers.groupby("cluster")[["balance", "txns", "products"]].mean() # profile the clusters

pca = PCA().fit(X) print(pca.explainedvarianceratio.cumsum()) # the scree numbers loadings = pd.DataFrame(pca.components, columns=["balance", "txns", "products", "tenure"])

association rules with the mlxtend library, from a table of 0/1 columns, one per product from mlxtend.frequentpatterns import apriori, associationrules frequent = apriori(baskets, minsupport=0.05, usecolnames=True) rules = associationrules(frequent, metric="lift", minthreshold=1.2)

Common mistakes

Not scaling before k-means or PCA. The largest numbers dominate. Choosing k by inertia alone. Inertia always falls. Use the elbow, the silhouette and usefulness. Trusting clusters without checking stability. Rerun on another sample, and see whether the groups persist. Naming clusters the data does not support. Profile them and make sure each one leads to an action. Reading PCA components as meaningful features. They are blends. Look at the loadings. Dropping components that predict the target. Variance is not the same as usefulness. Test by cross-validation. Judging a rule by confidence alone. A popular right-hand item inflates it. Check lift. Rules from tiny counts. Set a minimum support, and show the baskets behind each rule. Treating an association as a cause. Test any change with an experiment.

Key takeaways

Unsupervised learning finds structure without a target, so it must be judged by stability, interpretability and usefulness, not by accuracy. K-means alternates assigning points to the nearest centroid and moving centroids to the averages. It needs k, scaled features and several starts. Choose k with the elbow of the inertia curve, the silhouette score and business usefulness. Profile each cluster before naming it. PCA builds components that capture as much variation as possible. For two standardised columns with correlation r, the first captures (1 + r) ÷ 2. Keep enough components for a target such as 90 per cent. Support is how common a pattern is, confidence is P(B given A), and lift is confidence divided by the base rate of B. A lift of 1 means no link. Apriori prunes using the rule that a rare set has only rare supersets. Insist on a minimum support and remember association is not cause.

Check yourself

1. In k-means, what are the two repeated steps? Assign each point to the nearest centroid, then move each centroid to the average of its points. 2. Two standardised columns have a correlation of 0.6. What share of the variation does the first principal component capture? (1 + 0.6) ÷ 2 = 80 per cent. 3. Out of 500 baskets, 100 contain A, 150 contain B and 45 contain both. What are the confidence and lift of A → B? Confidence = 45 ÷ 100 = 0.45. B's base rate is 150 ÷ 500 = 0.30, so lift = 0.45 ÷ 0.30 = 1.5. 4. A rule has a confidence of 60 per cent and a lift of 1.0. What does that tell you? B is very common, and A does not make it more likely. The rule is not useful. 5. Why does inertia always fall as k rises, and what follows from that? More centroids always bring points closer to a centre, down to zero when every point is its own cluster. So you cannot choose k by the smallest inertia, and must look for the elbow.

Try this

Use a customer table with at least four numeric columns, and a transactions table, or a small dataset of baskets. Set aside forty-five minutes.

1. Standardise the numeric columns. Run k-means for k from 2 to 7, and record the inertia and silhouette score for each. 2. Draw the elbow plot, choose k, and justify it in one sentence that includes a business reason. 3. Profile each cluster with the average of the original columns, give each a label and write one action for it. 4. Check stability by clustering a second random half of the data and comparing the cluster profiles. 5. Run PCA on the same standardised columns. Plot the cumulative explained variance, decide how many components to keep, and look at the loadings of the first. 6. If you have transactions, compute support, confidence and lift for three item pairs by hand, then check them with mlxtend. Use the basket simulator to see how a rule with a high confidence can have a lift near 1. 7. Write a short note: the segments, one you would act on and why, and one pattern you suspect is not real.

A good result is a cluster profile table with one action per cluster, a scree plot with a stated choice, and at least one rule you rejected because of its lift or its count. That completes Module 5. You can now engineer features, build tree-based and distance-based models, combine them, and find structure without a target. The next module turns to the question that decides whether any of them can be trusted: how to validate and evaluate a model honestly.

Tags: Data Science Clustering K-means PCA Market basket analysis