Lesson 2.2: Univariate analysis of credit variables
Learn to profile each credit variable on its own: distributions, spikes, special codes, rare categories, missing values and outliers, before any modelling.
By the end of this lesson you will be able to profile every variable in a credit modelling dataset on its own, one at a time, and decide what to do with it: keep it, cap it, recode it, group its categories or drop it. You will learn the credit-specific traps that plain statistics lessons do not cover, such as spikes at zero and one hundred per cent in utilisation, special codes like 999 that mean "no record", and rare or very numerous categories. You should have read Lesson 2.1 for the dataset and Lessons 1.2 and 1.3 of the Data Science track for cleaning and summary statistics. The arithmetic is averages, medians and percentages.
Univariate analysis means studying one variable at a time, without reference to the target. It asks: what does this column look like? How many values are missing? Are there impossible, extreme or special values? How do its values spread? It is the first look at the finished table, and it comes before the bivariate analysis of Lesson 2.3, which relates each variable to default. The order matters. If you look at default too early, you will be tempted to keep what correlates and ignore whether the variable is sound. A variable that looks predictive but is full of errors is a liability.
What a profile contains
For every numeric variable, produce the same short profile. A profile is a table with one row per variable, and the columns below make it useful.
Count and missing share. How many rows have a value, and how many are blank?
Distinct values. A column with one value tells the model nothing, and a column with a different value in every row may be an identifier.
Minimum, maximum and key percentiles. The 1st, 5th, 25th, 50th, 75th, 95th and 99th percentiles show where most values sit and how long the tails are.
Mean, median and standard deviation. A mean far from the median signals skew, as Lesson 1.3 showed.
Most common values. Spikes at particular numbers, such as 0 or 100, often reveal how the data was collected.
For categorical variables the profile is the frequency table: each category, how many rows, what share, plus the blank count.
python
import pandas as pd
def profile(df):
rows = []
for c in df.columns:
s = df[c]
r = {"var": c, "dtype": str(s.dtype), "missingpct": round(100 s.isna().mean(), 1),
"distinct": s.nunique()}
if pd.api.types.isnumericdtype(s):
q = s.quantile([0.01, 0.25, 0.5, 0.75, 0.99])
r.update(min=s.min(), p1=q[0.01], p25=q[0.25], median=q[0.5],
p75=q[0.75], p99=q[0.99], max=s.max(), mean=round(s.mean(), 2),
topvalue=s.mode().iloc[0] if not s.mode().empty else None,
topshare=round(100 (s == s.mode().iloc[0]).mean(), 1) if not s.mode().empty else None)
rows.append(r)
return pd.DataFrame(rows)
profile(data)
Run this on the whole modelling table and read it slowly, row by row. The topshare column is quietly one of the most useful, because it flags variables where one value dominates. A column in which 97 per cent of rows hold the same value carries almost no information, however large the table.
Credit variables have their own habits
General rules help, but lenders' data shows patterns that come from how credit works. Knowing them saves hours.
Income and amounts are right-skewed
Incomes, loan amounts and balances have a long tail of large values. The mean is pulled up and the median is the honest centre, as in Lesson 1.3. Plots on a straight scale are dominated by a few very large values, and a logarithmic scale shows the shape better. The tail is also where outliers hide, so check it for errors such as an extra zero.
Ratios pile up at their limits
Utilisation, the share of a credit limit in use, is bounded at zero and, for most lines, at 100 per cent. Real utilisation data has two spikes. Many customers have a limit and use none of it, which gives a spike at 0. Others are fully drawn, which gives a spike at 100 per cent. A mean of such a column blends two behaviours and describes neither, and a fitted straight line through it misleads. Here is an illustrative distribution of 100 revolving accounts.
Do not summarise this variable with its mean alone. Create flags, such as "utilisation is zero" and "utilisation is 100 per cent", and treat the middle range as a number. In Module 5 you will group such variables into bands, which handles spikes naturally.
Counts of events are mostly zeros
Variables such as the number of times 30 days past due in the last year are zero for most borrowers, with a thin tail of larger counts. The mean might be 0.2 while the median is 0. The informative part is the difference between zero and not zero, and between one and several. Summaries by band, such as 0, 1, 2 and 3 or more, work better than means.
Bureau scores and similar bounded scores
A bureau score has a fixed range, such as 300 to 900, with a hump in the middle and thin tails. It is a good candidate for a variable on its own, but it also has a special case: borrowers with no bureau record have no score. How that is coded matters, as the next section shows.
Special codes and how they distort
Databases rarely leave a blank where information is missing. People and systems fill in codes: 0, -1, 999, 9999, "NA", "N/A", "-" or even a fake date such as 1 January 1900. These special codes look like data, and they poison any average or fitted line.
Take a variable, months since the last delinquency. Borrowers who have been late recently have values like 3 or 14. Borrowers who were never late have no such month, and the system stores 999. Suppose half the borrowers are in each group, and the recently late ones have a typical value of 14 months.
The true typical value among those who were ever late is 14.
The mean with the code left in is 0.5 × 999 + 0.5 × 14 = 506.5, a number that is not a month count for anyone.
At 60 per cent coded 999, the mean is 0.6 × 999 + 0.4 × 14 = 605.0.
The remedy is the same every time. First, find the special codes: look at the most frequent values and at the maximum, and ask what 999, 9999 or -1 means. Second, separate the two ideas into two columns: a flag saying whether the event ever happened, and a number that is blank when it did not. Third, decide how the number's blanks are treated, with the ideas in Lesson 1.2 of the Data Science track. Never leave a placeholder number in a column that you average or feed to a model.
Categorical variables: frequency, rarity and cardinality
For a categorical variable such as employment type, city, product or channel, the frequency table is the profile. Three things to look for:
Inconsistent spellings. As in Lesson 1.2 of the Data Science track, one place can have many names. Fix them first.
Rare categories. A category with 8 rows out of 50,000 cannot support a reliable estimate of risk. The usual remedy is to group rare categories into "Other", using a threshold such as 1 or 2 per cent of rows, or to merge them with the most similar category.
High cardinality. A variable with hundreds or thousands of distinct values, such as a pin code or a dealer id, creates a very sparse table. It needs grouping, perhaps by city or region, or a numeric summary such as the average risk of the area computed on training data only.
Here is a short example. A table of 50,000 loans has a "city" column with 300 distinct values. The biggest ten cities hold 62 per cent of rows. Another 120 cities hold fewer than 50 rows each. One approach: keep the ten big cities as their own groups, keep any other city with at least 500 rows, and put the rest into "Other". The final variable might have 18 groups, each large enough to measure.
Missing values, again, with credit eyes
Missing values were covered in the Data Science track, and here we add the credit view. First, count the share missing for every variable. Then ask why.
Not all missing values are alike. Missing years at the address might be random. Missing bureau score usually means the customer has no credit history, a thin file, which can be a risk signal or a sign of a new-to-credit customer that deserves its own treatment. Missing income may mean the customer declined to say. For each variable, you can compare the missing group with the rest on the target in the next lesson, but even now you should decide how to flag it. In credit data, a "missing" category is often a real group, not a gap to be filled.
A rule of thumb that many teams use is to consider dropping a variable that is missing in more than half the rows, unless the missingness itself is informative. Treat it as a rule of thumb to question, not a law.
Outliers and capping
Extreme values can dominate a model. Capping, also called winsorising, replaces values beyond a chosen limit with the limit itself. Common choices are the 1st and 99th percentiles, or fences from the interquartile range rule of Lesson 1.2 of the Data Science track. The point is to stop a handful of rows controlling the fit while keeping them in the data.
Use the ten monthly incomes from that earlier lesson, in thousands of rupees: 18, 22, 25, 27, 30, 32, 35, 40, 45 and 400. The mean is 67.4 and the median is 31. The upper fence was 62.5. If the largest value is capped at 62.5, the mean becomes (274 + 62.5) ÷ 10 = 33.65, close to the median.
Capping to 62.5 cuts the mean from 67.4 to 33.65, a fall of 50 per cent, while the median is unchanged. Is it right to cap? It depends. If the 400 is an entry error, capping is a rough repair. If it is a real high earner, capping loses information, but for a model of ordinary borrowers it may still be sensible because a single case should not steer the fit. Whichever way, record the cap, compute it on the development data only, and apply the same number to future data.
Two other common transformations help with skew. A logarithm compresses large values, so that the difference between 50 and 100 counts as much as between 500 and 1,000, which often makes relationships more linear. A ratio such as loan amount to income puts values on a comparable scale. In scorecards, where variables are binned, extremes often fall into the top bin without any special handling, which is one reason binning is popular.
Does the variable behave over time?
A variable can look fine on the whole table but change across the period. Split the dataset by origination quarter and compare the profile of each. Has the share missing jumped? Has the median income shifted? Did a new channel arrive that changed the mix? A variable whose definition changed in the middle of the data, for example a new application form with a different field, is a hazard, because the model will learn a pattern that exists in only part of the history. Module 8 on validation and monitoring builds a formal measure of this, the population stability index. For now, a quick check by quarter is enough.
Screening rules
After profiling, each variable gets one of four decisions: keep, transform, group or drop. A short list of screening rules helps. These are rules of thumb used in practice, and teams adjust the thresholds to their data.
Finding · Typical decisionOne value in more than about 95 per cent of rows · Drop, or keep as a flag if the rare value matters
More than about half the rows missing, and missing is not informative · Drop
Impossible values or unfixable special codes · Fix or drop
Heavy skew or extreme outliers · Cap or transform
Spikes at the limits · Flag the spike, bin the rest
Rare categories · Group into "Other"
High cardinality · Group or summarise
Looks too good to be true, or is only known after the outcome · Investigate for leakage
Definition changed over time · Rebuild or drop
The last two deserve emphasis. A variable that depends on information from after the observation date, such as "number of collection calls made", makes a model look brilliant and useless. Leakage will come up again in Module 7. And a variable with a changed definition will fail quietly in production.
Worked examples
Example 1: the utilisation spike. An analyst computes the mean utilisation of 100 accounts as 40 per cent and plots it as a single number on a dashboard. Reading the distribution shows 18 per cent at exactly zero and 10 per cent at exactly 100. The analyst splits the variable into two flags, "no usage" and "fully used", and a numeric part for the 72 accounts in between. Later analysis shows that the full-use group behaves very differently from the rest.
Example 2: the 999 months. A variable called months since last delinquency has a mean of 506 months, over 42 years, which cannot be right for a population aged 21 to 60. The profile shows that 50 per cent of rows hold 999. The analyst creates everdelinquent and sets the months to blank for the others, and the real mean among those ever late turns out to be 14.
Example 3: the pin code. A pin code column has 6,400 distinct values for 50,000 loans, most with fewer than 10 loans. As a raw category it is useless, since every group is too small to measure. The analyst maps each pin code to its district, which gives 210 districts, and then groups districts with fewer than 150 loans into their state's "Other" group.
Common mistakes
Looking at the target too soon. Profile the variables first. A variable that predicts default but is mostly errors should not be rewarded.
Using the mean of a spiky or coded variable. Find special codes and spikes before summarising.
Dropping outliers without looking. Check whether they are errors or real cases, and cap rather than delete when in doubt.
Computing caps on the whole dataset. Fit caps and groupings on the development sample and apply them unchanged to validation data.
Ignoring cardinality. Thousands of categories with few rows each produce unreliable estimates.
Missing the time dimension. A variable that changed definition partway through will fail in use.
Key takeaways
Univariate analysis profiles each variable alone, before looking at default: missing share, distinct values, percentiles, skew and common values.
Credit variables have habits: right-skewed amounts, spikes in ratios such as utilisation, event counts that are mostly zero, and special codes like 999.
Special codes distort every average. Find them, split them into a flag and a numeric column, and never feed a placeholder number to a model.
Group rare categories and summarise very high-cardinality ones such as pin codes.
Cap extreme values using limits learned on development data, and record every cap.
A short set of screening rules decides whether to keep, transform, group or drop each variable, and checking by quarter catches definition changes.
Check yourself
1. A variable has 97 per cent of rows equal to zero. What does that say, and what could you do?
It carries little information as a number. Consider dropping it or keeping a flag for the rare non-zero rows.
2. In a column, 40 per cent of values are 999 meaning "never delinquent" and the rest average 14. What is the mean with the code left in?
0.4 × 999 + 0.6 × 14 = 399.6 + 8.4 = 408.0.
3. Why might a mean of utilisation mislead even when no values are wrong?
The distribution has spikes at 0 and 100 per cent, so the mean falls in a range where few accounts sit and describes none of the groups.
4. A "city" column has 300 distinct values, many with very few loans. What would you do?
Keep large cities as groups, and merge the small ones into an "Other" group or into their region.
5. Why compute the cap on the development sample only?
Using validation or future data to set the cap would leak information and make test results look better than real use.
Try this
Take a real or invented dataset with at least 10 variables, such as a loan table from an open data source, or the table you built in Lesson 2.1. Set aside forty-five minutes.
1. Run the profile function, or build the same table by hand in a spreadsheet, and list for each variable the missing share, distinct values, median and top value share.
2. Find at least one variable with a spike or a special code. Describe what the code probably means and how you would split it.
3. For one numeric variable, find the 1st and 99th percentiles and the interquartile fences. Choose a cap, apply it and compare the mean before and after with the capping simulator's approach.
4. For one categorical variable, make the frequency table and decide the grouping with a threshold you can justify.
5. Split the data by quarter and compare the profile of one variable across quarters. Write down anything that moved.
6. Apply the screening table to every variable and give each a decision: keep, transform, group or drop, with a reason.
A good result is a one-page profile table with a decision and a reason for every variable, which a colleague could review in ten minutes. In the next lesson, the same variables meet the target for the first time.
Tags: Credit Risk Analytics Univariate analysis Data profiling Feature screening Beginner