Lesson 5.3: Nearest neighbours, support vector machines and ensembles
Learn how nearest neighbours and support vector machines classify, and how bagging, boosting and stacking combine models into stronger predictors.
By the end of this lesson you will be able to classify a new case with k-nearest neighbours and explain why scaling matters to it, describe how a support vector machine draws a maximum-margin boundary and uses a kernel to bend it, and choose between bagging, boosting and stacking when you want to combine models. You will compute a distance with and without scaling, see how the choice of k trades bias against variance, find a boundary by hand in a one-dimensional example that no straight line can separate until it is lifted into two dimensions, and calculate how much averaging reduces variance when the models are correlated. You should have read Lesson 5.2 and Lesson 4.4, since ensembles are the formal version of the averaging and boosting ideas introduced there. The mathematics is distances, averages and one short formula.
Lessons 4.2 and 5.2 gave you two ways to classify: a logistic regression that draws a linear boundary, and trees that carve the space into boxes. This lesson adds two more models that think differently, and then asks a practical question that every working data scientist faces: if you have several decent models, how do you combine them into a better one? The answer, the ensemble, is behind most winning solutions in prediction contests and many production systems.
K-nearest neighbours: ask the neighbours
K-nearest neighbours, or KNN, is the simplest idea in this course. To predict for a new case, find the k most similar cases in the training data, the neighbours, and let them vote. For a yes-or-no outcome, the prediction is the majority class among the neighbours, and the share of the vote is a rough probability. For a number, it is their average. There is no training phase and no equation: the data is the model.
Suppose a lender has a new applicant, and finds the five most similar past borrowers. Three of them defaulted and two repaid. With k = 5 the predicted default probability is 3 ÷ 5 = 0.6. Everything depends on two choices: what "similar" means, and how many neighbours to ask.
Distance and why scaling matters
Similarity is measured by distance between rows. The usual choice is Euclidean distance, the straight-line distance you would measure with a ruler: the square root of the sum of squared differences in each feature. Because it adds up differences across features, any feature with large numbers dominates.
Take two customers. A is 30 years old with a monthly income of ₹40,000. B is 35 with an income of ₹60,000. Measure income in thousands of rupees. The age difference is 5 and the income difference is 20, so the raw distance is √(5² + 20²) = √425 = 20.6. Look at the squares: 25 from age and 400 from income. Income accounts for 400 ÷ 425 = 94 per cent of the squared distance, simply because its numbers are bigger, although five years of age may matter as much as ₹20,000 of income.
The remedy is to standardise the features first, as in Lesson 5.1. If ages have a standard deviation of 10 years and incomes of ₹20,000, then the differences become 0.5 and 1.0 standard deviations. The distance is √(0.25 + 1.0) = 1.12, and now income accounts for 80 per cent of it, which is a much fairer split.
For categorical columns Euclidean distance is not meaningful, and you need a different measure, such as counting the number of columns where two rows differ. KNN works best with a modest number of numeric features of similar importance.
Choosing k
The number of neighbours is the main dial, and it is the bias-variance trade-off of Lesson 4.4 in miniature. With k = 1, the prediction is the label of the single closest case. The model follows every quirk of the training data: low bias, very high variance, and a boundary full of tiny islands. With k very large, the prediction is the overall majority class: high bias, low variance. The best value lies between, and you find it by cross-validation.
An odd k avoids ties in two-class problems. Many implementations also weight each neighbour by the inverse of its distance, so closer neighbours count more.
Strengths and weaknesses
· K-nearest neighboursStrengths · Simple, no training, naturally non-linear, easy to explain by example: "this applicant resembles these five borrowers"
Weaknesses · Slow at prediction time, since it must compare with every stored row
Needs · Scaled numeric features, and a sensible distance
Struggles with · Many irrelevant features, and very high dimensions, where all points become almost equally far apart
Good for · Small to medium data, prototypes, recommendations by similarity, anomaly detection
The last weakness is called the curse of dimensionality: as the number of features grows, the space becomes so large that even the nearest neighbours are far away and not much more similar than a random row. This is why feature selection matters for KNN more than for trees.
Support vector machines: the widest street
A support vector machine, or SVM, is a classifier built on a geometric idea. Among all the lines that separate two classes, pick the one that leaves the widest empty strip, the margin, between the classes. A boundary with a wide margin is less likely to be thrown off by small changes in the data than one that squeezes close to the points.
Take the simplest case, a single feature. Class A sits at x = 1 and x = 2, and class B at x = 4 and x = 5. Any boundary between 2 and 4 separates them, but the one in the middle, at x = 3, leaves the widest margin: a distance of 1 on each side. Only the two points nearest the boundary, x = 2 and x = 4, matter in fixing it. These are the support vectors, and they give the method its name. Moving any other point, as long as it stays on its own side, changes nothing. That is the contrast with logistic regression, in which every row influences the fit.
Real classes overlap, so a perfect separation may not exist. The SVM allows some points to fall inside the margin or on the wrong side, and charges a penalty for them. The setting C controls the penalty. A large C punishes violations heavily, giving a narrow margin that fits the training data closely and may overfit. A small C tolerates more violations, giving a wider and smoother boundary. You choose C by cross-validation.
The kernel trick: bending the boundary
A straight line cannot separate every problem. Consider one feature where class A sits at x = −3, −2, 2 and 3, and class B at x = −1, 0 and 1. Class B is in the middle and A is at both ends, so no single cut point separates them. Now add a second feature, the square of x. The points become A: (−3, 9), (−2, 4), (2, 4), (3, 9), and B: (−1, 1), (0, 0), (1, 1). In these two dimensions a horizontal line at x² = 2.5 separates them perfectly: B has x² of 1 or below, and A has 4 or above. A boundary that was impossible in one dimension is a straight line in two.
An SVM does this lifting implicitly through a kernel, a function that computes similarity as if the data had been mapped into a higher-dimensional space, without ever building the extra columns. The common choices are the linear kernel, a polynomial kernel and the radial basis function kernel, usually written RBF, which can produce very flexible boundaries. The RBF kernel has its own setting, gamma, which controls how far the influence of a single point reaches: a large gamma gives a wiggly boundary that can overfit, and a small one gives a smooth boundary.
· Logistic regression · Support vector machine · Tree-based modelsBoundary · Linear unless you add terms · Linear or very flexible with a kernel · Boxes aligned to the features
Rows that matter · All · Support vectors near the boundary · All, through the splits
Needs scaling · Yes, for penalties · Yes · No
Gives probabilities · Natively · Not natively, needs extra calibration · Yes, as leaf proportions
Training cost with many rows · Low · High, grows quickly with rows · Moderate
Explanation · Coefficients and odds ratios · Hard, especially with a kernel · Rules and importances
Typical use · Scorecards, baselines · Smaller datasets with complex boundaries, text · Most tabular problems
SVMs were the leading method before boosted trees became dominant, and they are still useful on small to medium datasets with many features, such as text. On large tabular data they are rarely the first choice, because training slows down as rows grow, and because they do not produce calibrated probabilities directly.
Ensembles: many models, one prediction
You now have several families of model. Their errors differ, which suggests a trick: combine them. An ensemble averages or otherwise merges the predictions of several models. Provided the models are individually decent and make different mistakes, the combination is usually better than any member. The central idea is easy to state with arithmetic.
Suppose each of n models has an error with variance σ² and the errors of any two models have correlation ρ. The variance of their average is
variance of the average = σ² × (ρ + (1 − ρ) ÷ n)
Look at the two parts. The second term, (1 − ρ) ÷ n, shrinks as you add models. The first, ρ, does not shrink at all: it is the part of the error the models share, and no amount of averaging removes it. If the models are independent, ρ = 0 and the variance falls to σ² ÷ n. If they are identical, ρ = 1 and the variance stays at σ², so averaging copies of one model gains nothing. With σ² = 1 and ρ = 0.3, the variance of the average is 0.37 for 10 models and 0.307 for 100. Ten times as many models improve it by less than a quarter, because 0.3 is the floor.
Predict first: you average 100 models whose errors are correlated at 0.9. Roughly what share of a single model's error variance is left: about 1 per cent, about 10 per cent or about 90 per cent? Decide before reading on.
The answer is about 90 per cent. The formula gives 0.9 + 0.1 ÷ 100 = 0.901, so a hundred nearly identical models are barely better than one. The shared part of the error never averages away, and only the small independent part shrinks. Try other correlations below.
The chart is the lesson's most useful warning. Fifty copies of nearly the same model are barely better than one, and this is why random forests deliberately decorrelate their trees with bootstrap samples and random feature subsets, and why practitioners combine different families of model, such as a forest, a boosted model and a logistic regression, which tend to make different mistakes.
Three ways to combine
There are three main strategies, and they differ in how the members are built and how they are merged.
Bagging trains many copies of the same kind of model on different bootstrap samples of the data, and averages or votes. It reduces variance. Random forests are bagging with a twist. It works best with high-variance models such as deep trees.
Boosting trains models one after another, each concentrating on what the previous ones got wrong, and adds them with small weights. It reduces bias, and with care also variance. Gradient boosting from Lesson 5.2 is the leading example, and the older AdaBoost reweights the rows that were misclassified.
Stacking trains several different models, then trains a final meta-model to learn how best to combine their predictions. The meta-model might be as simple as a logistic regression on the base models' outputs.
· Bagging · Boosting · StackingMembers · Copies of one model type · Small models, usually shallow trees · Different model types
Built · In parallel, independently · In sequence, each fixing the last · Base models, then a meta-model
Training data for each member · Bootstrap sample · The full data, reweighted or on residuals · Out-of-fold predictions for the meta-model
Combined by · Average or vote · Weighted sum · A learned model
Mainly reduces · Variance · Bias · Both, by exploiting diversity
Risk · Little gain if members are alike · Overfits noise if run too long · Leakage if built carelessly
Example · Random forest · XGBoost, LightGBM · Forest + boosting + logistic, blended
A short version of the contrast is worth memorising. Bagging averages independent models to cut variance, and boosting builds dependent models to cut bias.
A stacking example
Suppose three base models predict the default probability of one borrower: a logistic regression says 0.10, a forest says 0.14 and a boosted model says 0.18. A simple average is (0.10 + 0.14 + 0.18) ÷ 3 = 0.14. A meta-model trained on held-out data might have learned that the boosted model is the most reliable and the logistic the least, and might weight them 0.2, 0.3 and 0.5, giving 0.2 × 0.10 + 0.3 × 0.14 + 0.5 × 0.18 = 0.02 + 0.042 + 0.09 = 0.152. The learned blend leans towards the model that has earned more trust.
The danger in stacking is leakage. If the base models are trained on the same rows on which the meta-model is trained, their predictions on those rows are too good, since they have already seen the answers, and the meta-model learns to trust them too much. The solution is to build the meta-model's inputs from out-of-fold predictions: for each row, the prediction made by a base model that did not see that row during training. Libraries do this for you with cross-validation inside the stacking procedure.
python
from sklearn.ensemble import (BaggingClassifier, RandomForestClassifier,
HistGradientBoostingClassifier, StackingClassifier,
VotingClassifier)
from sklearn.linearmodel import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import makepipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.tree import DecisionTreeClassifier
knn = makepipeline(StandardScaler(), KNeighborsClassifier(nneighbors=11, weights="distance"))
svm = makepipeline(StandardScaler(), SVC(kernel="rbf", C=1.0, probability=True))
bag = BaggingClassifier(DecisionTreeClassifier(), nestimators=200, randomstate=0)
stack = StackingClassifier(
estimators=[("forest", RandomForestClassifier(300, randomstate=0)),
("boost", HistGradientBoostingClassifier(learningrate=0.05)),
("knn", knn)],
finalestimator=LogisticRegression(),
cv=5, # out-of-fold predictions, so the meta-model does not see leaked ones
)
stack.fit(Xtrain, ytrain)
print(stack.score(Xtest, ytest))
Notice that KNN and SVM sit inside pipelines with a scaler, so scaling is learned from training rows only, a point made in Lesson 5.1.
When an ensemble is worth it
Ensembles buy accuracy at a price. They are harder to explain, slower to run, more costly to maintain, and give diminishing returns when the members are alike. A reasonable approach follows a ladder. Start with a simple, explainable baseline, a regularised logistic regression. Move to a single boosted model, which is often enough. Only then consider stacking several diverse models, and ask whether the improvement, measured on held-out data, is worth the added complexity. In regulated settings such as credit, a model must often be explainable to customers and auditors, and a slightly less accurate but transparent model can be the right answer. In a contest or an advertising system where only accuracy counts, large ensembles are common.
Common mistakes
Using KNN or SVM on unscaled features. Distances and margins are dominated by the biggest numbers.
Picking k by looking at training error. With k = 1 the training error is zero. Use cross-validation.
Ignoring the curse of dimensionality. KNN degrades when many features are irrelevant, so select features first.
Expecting an SVM to give probabilities for free. It needs an extra step, and the probabilities may need calibration.
Averaging copies of the same model. Diversity matters more than count.
Stacking with in-sample predictions. The meta-model must see out-of-fold predictions.
Using an ensemble where explainability is required. Check the requirements first.
Judging the ensemble on the data it was tuned on. Report performance on an untouched test set.
Key takeaways
KNN predicts from the k most similar training rows. Scale the features first, choose k by cross-validation, and expect trouble in many dimensions.
Small k means low bias and high variance, and large k the reverse. The best k lies between, often in the low tens.
An SVM finds the widest margin between classes. Only the support vectors fix it, C trades margin against errors, and a kernel lets a straight boundary in a lifted space become a curve in the original one.
The variance of an average of n models is σ² × (ρ + (1 − ρ) ÷ n). The correlation ρ sets a floor, so diverse models beat many similar ones.
Bagging averages independent models to cut variance, boosting builds dependent models to cut bias, and stacking learns how to combine different models.
Stack with out-of-fold predictions, and weigh the gain against the loss of explainability and the added complexity.
Check yourself
1. Two customers differ by 3 years in age and ₹30,000 in income, with standard deviations of 6 years and ₹15,000. What is the scaled distance?
The scaled differences are 0.5 and 2.0, so the distance is √(0.25 + 4) = 2.06.
2. A KNN model has a training error of 0 and a test error of 20 per cent. What is probably the value of k, and what should you do?
k is probably 1. Increase k and choose it by cross-validation.
3. What are the support vectors in an SVM?
The training points closest to the boundary, which alone determine its position.
4. Ten models each have error variance 2, with pairwise error correlation 0.5. What is the variance of their average?
2 × (0.5 + 0.5 ÷ 10) = 2 × 0.55 = 1.1.
5. Why must a stacking meta-model be trained on out-of-fold predictions?
In-sample predictions of the base models are too accurate, since they saw those rows, so the meta-model would learn to over-trust them and fail on new data.
Try this
Use a dataset with a yes-or-no outcome and a few numeric features. Set aside forty-five minutes.
1. Split into training and test sets. Fit KNN with and without standardising the features, using k = 11, and compare the test scores.
2. Fit KNN with k from 1 to 51 and plot the cross-validated error against k. Choose k and describe the shape.
3. Fit an SVM with an RBF kernel on standardised features, and try three values of C. Note how the training and test scores change.
4. Fit a random forest and a boosted model from the previous lesson. Compute the correlation between the models' predicted probabilities on the test set.
5. Average the predictions of KNN, the forest and the boosted model, and compare the test score of the average with each member.
6. Build a stack with a logistic regression as the meta-model, using cross-validated predictions. Compare it with the simple average.
7. Use the variance simulator with a correlation you measured, and say how many models would be worth combining.
8. Write a short note on whether any ensemble beat the best single model, by how much, and whether the gain justifies the complexity.
A good result is a table of test scores for each model and each ensemble, and a sentence about how correlated the members were. The last lesson of this module moves from predicting a known outcome to finding structure when there is none: clustering, dimensionality reduction and association rules.
Tags: Data Science K-nearest neighbours Support vector machines Ensembles Stacking