Lesson 5.1: Feature engineering, encoding, WOE and information value
Learn to turn raw columns into model-ready features: scaling, ratios, dates, categorical encodings, and weight of evidence with information value.
By the end of this lesson you will be able to turn raw columns into features a model can use, choose between scaling methods and categorical encodings, build ratio, date and aggregate features that carry real business meaning, and compute weight of evidence and information value to judge how useful a variable is for predicting a yes-or-no outcome. You will work through a complete weight-of-evidence calculation by hand on a table of 5,000 loans, see how to encode a category with only five rows without being fooled by noise, and learn the rule that keeps every encoding honest: fit it on the training data only. You should have read Module 4, since features feed the regression models you built there, and Lesson 3.1 for the loan table used here. The mathematics is ratios and natural logarithms, with every step shown.
A model sees only the columns you give it, in the form you give them. Two analysts with the same data and the same algorithm can get very different results because one of them gave the model better columns. This craft is called feature engineering, and practitioners often say it matters more than the choice of algorithm. A feature is simply an input column to a model. Engineering means creating, transforming and selecting them so that the signal in the data is easy for the model to find. A good feature encodes something you know about the world: that a loan is heavy relative to income, that a customer has been inactive for three months, that a transaction happened at 3 a.m.
Why raw columns are rarely enough
Raw data is recorded for operations, not for prediction. A loan table stores the loan amount, the date and the applicant's income, each in its own column. A model that sees them separately has to discover that the ratio of loan to income is what matters. A tree can approximate that with many splits, and a linear model cannot at all. If you compute the ratio yourself, you hand the model the answer. Similarly, a date such as 14 March 2026 means nothing to a model as a number, but "day of week", "month", "days since the customer joined" and "is it a festival week" are all meaningful.
Preparation of features falls into four jobs: transform numbers, create new columns from old ones, convert categories to numbers, and select the columns worth keeping. The first three come next, and the fourth, using information value, closes the lesson.
Transforming numbers
Scaling
Many methods are sensitive to the scale of a column. A distance-based method or a penalised regression treats a column measured in rupees, with values in lakhs, as far more important than a column of ages, simply because its numbers are larger. Two scalings fix this.
Standardisation subtracts the mean and divides by the standard deviation, giving a column with mean 0 and standard deviation 1. It is the same z-score from Lesson 1.3.
Min-max scaling subtracts the minimum and divides by the range, squeezing values into 0 to 1.
Min-max is easy to understand and badly affected by extremes: if one income is ₹4,00,000 and the rest lie between ₹20,000 and ₹35,000, then after min-max scaling the ordinary incomes are squeezed into a sliver near 0, from 0 to about 0.04. Standardisation is more forgiving, and for skewed data a log transform often works better still. Tree-based methods, which Lesson 5.2 covers, are unaffected by scaling because they only compare a value with a threshold, so you can skip scaling for them.
The log transform
Money amounts, counts and durations usually have a long right tail. Taking the natural logarithm pulls the tail in and makes multiplicative differences additive. The logs of ₹20,000, ₹2,00,000 and ₹20,00,000 are 9.90, 12.21 and 14.51, so each tenfold increase adds the same 2.3. A model on the log scale treats a move from ₹20,000 to ₹2,00,000 the same as a move from ₹2,00,000 to ₹20,00,000, which is usually how people experience money. Use log1p, the log of one plus the value, when a column can be zero.
Binning
Cutting a number into ranges, such as income under ₹20,000, ₹20,000 to ₹40,000 and so on, is called binning. It loses detail, and it has real uses: it makes a relationship non-linear without a formula, it limits the pull of extreme values, and it makes a model easy to explain. Credit scorecards almost always use binned variables. Bins should have enough rows in each, at least about 5 per cent of the data, so that the estimate for each is stable.
Creating features
The most valuable features come from knowledge of the problem. A few families recur.
Family · Example · Why it helpsRatios · Loan amount ÷ annual income, EMI ÷ monthly income · Puts size in context, and a model can use it directly
Differences and changes · Balance now minus balance three months ago · Captures direction, not just level
Date parts · Day of week, month, is-festival, hour of day · Exposes seasonal and daily patterns
Durations · Days since last payment, tenure in months · Turns a date into a recency or age
Aggregates · Average spend over 90 days, number of missed payments in 12 months · Summarises behaviour across many rows
Counts and flags · Number of enquiries, has-a-guarantor · Simple, strong signals
Interactions · Income × employment type · Lets an effect depend on another column (Lesson 4.3)
Missing flags · Income-was-blank · The blank itself can carry information (Lesson 1.2)
A worked example of a ratio. Two borrowers each ask for a loan of ₹5 lakh. One earns ₹6 lakh a year and the other ₹24 lakh. The raw loan amount is identical, so a model looking only at it sees two identical borrowers. The ratios are 5 ÷ 6 = 0.83 and 5 ÷ 24 = 0.21, and these separate the two clearly. The same borrower is "heavily loaded" in one case and "lightly loaded" in the other, which is the thing that predicts default.
An important rule for features built from the past: use only what was known at the time of the decision. An aggregate over the next three months, or a status that is filled in after the outcome, will make a model look wonderful in testing and fail in production. This is leakage, and Lesson 6.4 treats it in detail, but you should practise the discipline now: for every feature, ask when that number would be available.
Turning categories into numbers
Models need numbers, so a category such as city or employment type must be encoded. Several encodings exist, and the right one depends on the model and the number of categories.
Encoding · How it works · Good for · Watch out forOne-hot · One 0 or 1 column per category · Few categories, any model · Many categories create many columns
Ordinal · Numbers 1, 2, 3 for ordered categories · Genuinely ordered values such as low, medium, high · Using it on unordered categories invents an order
Frequency · Replace the category with how often it appears · High-cardinality columns · Different categories with equal counts look the same
Target (mean) · Replace the category with the average outcome for it · High-cardinality columns · Leakage and noise, unless smoothed and computed on training data
Weight of evidence · Replace the category with the log of good-to-bad odds · Binary outcomes, especially credit · Needs enough rows in each category
One-hot encoding is the default. For a city column with Pune, Mumbai and Nagpur, you create three columns, and each row has a 1 in the column for its city. For a linear model you drop one column, as in Lesson 4.1, to avoid redundancy. The trouble comes with a column such as pin code, with thousands of values. One-hot encoding would add thousands of columns, most of them nearly empty, and a model would struggle.
Target encoding replaces each category with the average outcome among rows of that category. For default, "Nagpur" becomes the default rate in Nagpur. It is compact and powerful, and it is dangerous, for two reasons. First, if you compute the average using the same rows the model trains on, each row's own outcome leaks into its feature. Second, a category with few rows has a noisy average. If a rare pin code has five loans and two defaulted, its rate of 40 per cent is mostly luck.
The standard repair for the second problem is smoothing: blend the category's average with the overall average, giving more weight to the category as it has more rows. With a smoothing strength m, the encoded value is
encoded = (n × category average + m × overall average) ÷ (n + m)
where n is the number of rows in the category. For a category with 5 rows and a 40 per cent default rate, an overall default rate of 6 per cent and m = 20, the encoded value is (5 × 40 + 20 × 6) ÷ 25 = (200 + 120) ÷ 25 = 12.8 per cent. The five-row category is pulled most of the way back to the overall rate, which is the sensible answer when we know so little.
Slide the number of rows up to 200 and the encoded value rises towards the category's own 40 per cent, because by then the data supports it. This behaviour, trusting a category in proportion to its evidence, is the main point.
The golden rule for all encodings is that anything computed from the data, a mean, a category average, a scaling constant, must be learned from the training data only and then applied unchanged to validation and test data. Fit the encoder on the training rows, and apply it to the rest. If you compute a category average over the full dataset before splitting, the test set has leaked into the training set and your measured performance is too good.
Weight of evidence and information value
Credit scoring has a favourite way to encode and to judge variables for a yes-or-no outcome, called weight of evidence, or WOE, with a companion measure called information value, or IV. Together they answer: how well does this variable separate the good outcomes from the bad ones, and in which bin?
The ingredients are simple. Call a borrower who repays a good and a borrower who defaults a bad. Cut the variable into bins. For each bin, compute the share of all goods that fall in it and the share of all bads that fall in it. Then
WOE of a bin = ln (share of goods in the bin ÷ share of bads in the bin)
A positive WOE means the bin holds relatively more goods than bads, so it is a low-risk bin. A negative WOE means it holds relatively more bads, a high-risk bin. A WOE of zero means the bin looks like the population. The word "relative" is key: WOE compares the bin's share of goods with its share of bads, not the bin's default rate with the overall rate.
A worked example
Take the 5,000 loans of Lesson 3.1, with 301 bads and 4,699 goods. Cut monthly income into four bins.
Income bin · Loans · Bads · Goods · Share of goods · Share of bads · WOE · IV contributionUnder ₹20,000 · 800 · 76 · 724 · 15.41% · 25.25% · −0.494 · 0.0486
₹20,000 to 40,000 · 2,200 · 143 · 2,057 · 43.78% · 47.51% · −0.082 · 0.0031
₹40,000 to 80,000 · 1,400 · 63 · 1,337 · 28.45% · 20.93% · 0.307 · 0.0231
Over ₹80,000 · 600 · 19 · 581 · 12.36% · 6.31% · 0.672 · 0.0407
Total · 5,000 · 301 · 4,699 · 100% · 100% · · 0.1155
Work through the first row. The 724 goods are 724 ÷ 4,699 = 15.41 per cent of all goods, and the 76 bads are 76 ÷ 301 = 25.25 per cent of all bads. The ratio is 15.41 ÷ 25.25 = 0.610, and its natural logarithm is −0.494. The low-income bin has a smaller share of the goods than of the bads, so it is a risky bin. The top bin has 12.36 per cent of goods and only 6.31 per cent of bads, a ratio of 1.96 and a WOE of +0.672.
Predict first: the WOE should move steadily from negative to positive as income rises, if income really orders risk. Looking at the four values, do they? Check before reading on.
They do: −0.494, −0.082, 0.307 and 0.672 rise from one bin to the next, a monotonic pattern. Scorecard builders want exactly this, because it matches business sense and is stable. A bin that breaks the pattern, with a WOE out of order, is usually a sign of too few rows or of a real quirk that needs explaining, and bins are often merged to restore the order.
Information value
The information value adds up each bin's contribution, (share of goods − share of bads) × WOE. Every term is positive, because the difference and the WOE always have the same sign, so a variable that separates goods from bads well has a large total. For the income variable the contributions are 0.0486, 0.0031, 0.0231 and 0.0407, a total IV of 0.1155. Common rules of thumb for interpreting IV are in the table below, and they are conventions, not laws.
IV · Usual readingBelow 0.02 · Not useful for prediction
0.02 to 0.1 · Weak
0.1 to 0.3 · Medium
0.3 to 0.5 · Strong
Above 0.5 · Suspiciously strong: check for leakage
The last row is important. A variable that appears to separate bads from goods almost perfectly is rarely a discovery. More often it is information recorded after the event, such as a collections flag, and using it would be leakage.
Now compare other variables in the same table. Employment type has three groups: salaried with 3,000 loans and 135 bads, self-employed with 1,200 loans and 102 bads, and business owners with 800 loans and 64 bads. Their WOEs are +0.307, −0.372 and −0.306, and the IV works out at 0.1057, similar to income. A column of random numbers, cut into bins, would show an IV near 0.002, which is the level to expect by chance.
IV ranks single variables by predictive power, which is why it is used to screen hundreds of candidate columns down to a short list. It has limits. It looks at one variable at a time, so two variables with high IV can carry the same information and add little together, which a check for correlation or a model will reveal. It depends on how you bin, since a finer binning raises IV by chance. And it is built for yes-or-no outcomes. For regression outcomes, use correlation or model-based importance instead.
Once the bins are chosen, WOE can replace the original column: each row gets the WOE of its bin. A logistic regression on WOE-encoded variables is the classic scorecard model. The WOE columns are on a log-odds scale, so they fit the linear model of Lesson 4.2 naturally, and each coefficient has a clear reading.
Doing it in Python
python
import numpy as np
import pandas as pd
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.linearmodel import LogisticRegression
feature creation
df["loantoincome"] = df["loanamount"] / df["annualincome"]
df["logincome"] = np.log1p(df["annualincome"])
df["tenuredays"] = (df["decisiondate"] - df["joindate"]).dt.days
df["appliedonweekend"] = df["decisiondate"].dt.dayofweek >= 5
weight of evidence and information value for one binned column
def woeiv(frame, col, target="default"):
t = frame.groupby(col, observed=True)[target].agg(bads="sum", total="count")
t["goods"] = t["total"] - t["bads"]
t["pg"], t["pb"] = t["goods"] / t["goods"].sum(), t["bads"] / t["bads"].sum()
t["woe"] = np.log(t["pg"] / t["pb"])
t["iv"] = (t["pg"] - t["pb"]) t["woe"]
return t, t["iv"].sum()
Xtrain and ytrain are the training rows of df, with the columns created above, after a split
# encoders live inside a pipeline, so they are fitted on training rows only
prep = ColumnTransformer([
("num", StandardScaler(), ["loantoincome", "logincome", "tenuredays"]),
("cat", OneHotEncoder(handleunknown="ignore"), ["city", "employment"]),
])
model = Pipeline([("prep", prep), ("clf", LogisticRegression(maxiter=1000))])
model.fit(Xtrain, ytrain) # scaling and encoding are learned from Xtrain alone
Putting the transformations in a pipeline is the practical way to honour the training-only rule. When you call fit, the scaler learns its mean and standard deviation from the training rows. When you later predict on test data, it applies those same numbers. Note the handleunknown="ignore" argument, which stops a new city that appears only in production from crashing the model.
Common mistakes
Fitting encoders or scalers on all the data. Learn them from the training rows only, ideally inside a pipeline.
Target-encoding without smoothing. Rare categories get noisy values and the model learns noise.
Ordinal encoding of unordered categories. It invents an order that the model will use.
One-hot encoding a column with thousands of values. Use frequency, target or WOE encoding, or group rare values.
Features from the future. Check when each number would be known.
Binning too finely. Bins with few rows give unstable WOE and a flattering IV.
Trusting a very high IV. An IV above 0.5 is usually leakage.
Scaling for tree models. It is harmless, and unnecessary.
Engineering features without a reason. Every new column adds risk. Build features from a hypothesis about the problem.
Key takeaways
Features are the model's view of the data, and good ones encode business knowledge. Ratios, durations, date parts, aggregates and flags often beat raw columns.
Standardise or log-transform numbers for methods that depend on scale, and skip scaling for trees. Min-max scaling is easily distorted by extremes.
Encode categories with one-hot for few values, and with frequency, smoothed target or WOE encoding for many. Smoothing is (n × category average + m × overall average) ÷ (n + m).
WOE = ln(share of goods ÷ share of bads) per bin. It should move monotonically across bins, and it fits logistic regression naturally.
IV sums (share of goods − share of bads) × WOE over bins. Below 0.02 is useless, 0.1 to 0.3 medium, above 0.5 suspicious.
Everything learned from data, means, category rates and bin edges, must come from training rows only. Use a pipeline.
Check yourself
1. A bin contains 10 per cent of all goods and 20 per cent of all bads. What is its WOE, and is it high or low risk?
WOE = ln(0.10 ÷ 0.20) = ln 0.5 = −0.69. Negative, so it is a high-risk bin.
2. A category has 8 rows with a 25 per cent default rate, the overall rate is 5 per cent, and m = 12. What is the smoothed encoding?
(8 × 25 + 12 × 5) ÷ 20 = (200 + 60) ÷ 20 = 13 per cent.
3. Two borrowers each want ₹8 lakh. One earns ₹4 lakh a year and one ₹16 lakh. What is the loan-to-income ratio of each?
8 ÷ 4 = 2.0 and 8 ÷ 16 = 0.5.
4. A variable has an IV of 0.8. What should you suspect?
Leakage: the variable may record something known only after the outcome. Check how and when it is filled in.
5. Why compute the mean used for scaling on the training data only?
Using the whole dataset lets test-set information leak into the model and makes the measured performance too optimistic.
Try this
Use a dataset with a yes-or-no outcome and several columns, such as the 5,000-loan table, a churn file or a marketing response file. Set aside forty-five minutes.
1. List ten candidate features. For each, write one sentence on why it should predict the outcome and when it would be known.
2. Build at least three engineered features: a ratio, a duration and an aggregate or flag.
3. Pick one numeric column and bin it into four to six bins with at least 5 per cent of rows each. Compute WOE and IV by hand for each bin, as in the worked example, and check the pattern is monotonic.
4. Compute the IV of three more columns and rank them. Add a column of random numbers as a benchmark and see what a useless IV looks like.
5. Choose one high-cardinality column and compute a smoothed target encoding on the training rows only. Use the smoothing simulator to see how the encoding changes with the number of rows.
6. Put your scaling and encoding inside a pipeline and fit a logistic regression. Note the test performance.
7. Write a short note: the best three features, the IV ranking, and any feature you rejected because of leakage or noise.
A good result is a table of features with their reason, timing and IV, and a pipeline that learns everything from training rows only. The next lesson uses these features with models that find interactions on their own: decision trees, random forests and gradient boosting.
Tags: Data Science Feature engineering Encoding Weight of evidence Information value