Lesson 5.3: Observability, alerts and runbooks
A system you cannot see is a system you cannot run. Learn the four signals to watch, how to write alerts people act on, and a runbook for 3 a.m.
By the end of this lesson you will be able to make a customer's system visible: to know within minutes when it is unhealthy and why, to write alerts that someone will act on and not mute, and to write a runbook that lets a person who did not build the system deal with a problem at three in the morning. You will learn the three kinds of evidence a running system can give, four signals worth watching on almost anything, a test for a good alert, a runbook template and a calm way to run an incident. You will see three real-style situations and use a simulator to turn an availability promise into minutes. This lesson follows 5.2, where you learned to release changes safely. Here you learn how to know what happens afterwards. You do not need to be an operations specialist.
When software is live, the question changes from "does it work?" to "is it working right now, and how would I know?" A system can fail quietly: a nightly file that never arrives, a queue that slowly fills, a message that goes to the wrong group. Nobody calls to complain until the damage is done. Making the system visible is not a luxury added at the end. It is what lets you leave the customer with something they can trust, and it is what lets you sleep.
Three kinds of evidence
A running system can tell you what it is doing in three main ways. People often call this observability, meaning how well you can understand the inside of a system from its outside signs.
Logs. Time-stamped records of things that happened: a request arrived, a file was loaded, a check failed, a message was sent. Logs are the detailed story, and the first place to look when you ask "what happened to this account?" Good logs are structured, with named fields such as the account, the step and the result, so that you can search them. They never contain secrets or sensitive personal data, as Lesson 3.4 insisted.
Metrics. Numbers measured over time: how many requests per minute, how long they take, how many failed, how full the disk is. Metrics are cheap to store and easy to chart and to alert on, and they show trends and sudden changes. They answer "is something wrong?" quickly, but not "why?".
Traces. A record of the path of one request through several parts of a system, with the time spent in each. Traces answer "where did this slow request spend its time?" They matter most in systems made of many connected services, and a small customer pilot may not need them.
A useful habit is to give each unit of work, such as one account's journey through the nightly pipeline, an identifier that appears in every log line. Then you can follow one case from start to finish. It costs almost nothing and saves hours.
The four signals worth watching
For almost any system that serves users or processes work, four signals describe its health. Google's site reliability engineering guide, written from running some of the largest systems in the world, calls them the four golden signals, and they apply well to a customer pilot.
1. Latency. How long work takes. For a nightly pipeline, how long the run takes. For a screen, how long it takes to load. Track the time of failed requests separately, since a slow failure is worse than a fast one.
2. Traffic. How much work is coming in: files per night, requests per minute, messages queued. A sudden drop is as informative as a rise, and often more worrying.
3. Errors. The share of work that fails, including quietly: a request that returns success but wrong content counts.
4. Saturation. How full the system is: disk space, memory, the queue length, the share of a rate limit used. Saturation predicts trouble before it arrives.
If you can show these four, plainly, on one page, for each important part of the system, you can answer most questions about its health in a minute. Add one more: freshness, how old the newest data is. For many customer systems, such as a bed screen or a reminder pipeline, stale data is the main way of failing.
Symptoms and causes
A distinction from the same guide shapes good monitoring: the difference between what is broken and why. A symptom is what a user or the business sees: the screen is blank, the reminders did not go out, the dashboard is two days old. A cause is the reason: a full disk, an expired key, a failed job. Monitoring should show both, but it should alert on symptoms and use causes mainly for diagnosis.
The reasoning is practical. There are countless possible causes, and you cannot predict them all. But the symptoms that matter to the customer are few, and watching them catches problems whatever caused them. An alert that says "disk is 90 per cent full" is useful as a warning. An alert that says "no reminders have gone out for two hours during business hours" catches the disk, and also the expired key, the broken file and the cause nobody thought of.
A related distinction is between watching from outside and from inside. Black-box monitoring tests the system's visible behaviour the way a user would, for example by sending a harmless test case through and checking the result. It catches problems that are active and user-facing, and it is well suited to alerts. White-box monitoring looks at the inside: logs, internal counters and queue lengths. It reveals problems that are building up and those hidden by automatic retries, and it is best for diagnosis. Use black-box checks to decide when to wake someone, and white-box data to decide what to do.
Alert on silence
One kind of failure deserves its own rule, because it catches the most dangerous problems: the ones that make no noise. If a job should run every night and does not, nothing is raised, since nothing happened. If a feed should deliver a file by 7 a.m. and it does not arrive, no error exists to report.
The cure is to alert when expected things do not happen. Every scheduled job and every expected feed should have a check: "has this succeeded in the last day?" and "has a file arrived by seven?" The alert fires on the absence. In the earlier lessons you met this idea in several places: the webhook that goes quiet in Lesson 3.2 and the file that shrank in Lesson 3.3. It is among the cheapest and most valuable alerts you can write.
What makes a good alert
An alert is a request for a human to stop what they are doing, and each one has a cost: interrupted sleep, a broken evening, a little more tiredness. If alerts fire often and mean nothing, people learn to ignore them, and then the one that matters is missed. The guide puts the rule briefly: every page should be actionable, and an alert that would always be ignored as harmless should not exist.
A useful test has four questions. A good alert is:
Urgent. It needs attention now, not tomorrow. If it can wait until morning, send it to a dashboard or a daily summary, not to a phone.
Actionable. There is something a person can do about it. An alert that says only "something is odd" is a worry, not a task.
User-visible. It reflects an effect on the customer's people or results, or a clear and near risk of one.
Understandable. It says what is wrong, where, since when and what to look at first, in plain words.
Apply the test to every alert. Remove those that fail it, or turn them into dashboard items. Combine alerts that always arrive together. Automate responses that are always the same. And review the alerts every month: which ones fired, which were acted on and which were noise.
Predict first: you inherit a system that sends about sixty alerts a week to a shared channel. Most are ignored, and last month a real failure sat unseen for six hours. What would you do first? Decide before reading on.
Resist the urge to add more monitoring. The first step is to read a few weeks of the alerts and sort them: which led to an action, which fixed themselves, which were duplicates and which nobody read. Then remove or downgrade the ones that fail the test, keep a small number of urgent, symptom-based alerts, such as "no reminders sent in business hours" and "nightly file missing", and route the rest to a dashboard reviewed daily. The goal is not fewer alerts for their own sake. It is that every alert that fires is one a person should act on, so that people trust them again.
Dashboards that people use
A dashboard should answer, in a glance, the question "is it healthy?" It is not a place to display every number you can collect. A good one has few parts.
One row per important flow, such as the nightly load, the message sending and the screen, each showing its freshness and error rate.
The four signals for the main service.
The last release and when it went out, so that a change in a chart can be linked to a change in the software.
Clear colours and thresholds. Green means fine, amber means look soon and red means act. Agree what each means with the customer's team.
Show the dashboard to the customer's people, and build it with them. A dashboard that only you understand protects nothing once you leave.
Targets: how reliable is reliable enough?
To decide what to alert on, you need to know what "healthy" means. A helpful tool is a service level objective: an agreed target for how well the system should perform, such as "the nightly file is loaded by 6 a.m. on 99 nights out of 100", or "the screen is available 99.5 per cent of the time". The target should come from what the customer needs. A bed screen in a hospital needs more than a weekly report does.
A target implies a budget. If the screen must be available 99.5 per cent of the time over thirty days, then it may be unavailable for the other 0.5 per cent, which is a number of minutes you can calculate. That allowance is sometimes called an error budget. It turns a vague wish for reliability into something you can plan with: if the budget is nearly spent, be more careful about risky releases. If plenty is left, you can afford to move faster.
The simulator below converts a target into minutes. Set the availability target, the length of the period and the minutes of outage so far. It shows how many minutes the target allows, how many remain and what share of the allowance is used, and it plots the allowance for a range of targets. The figures are exact arithmetic, but what counts as an outage is a decision you must make with the customer: a slow screen, a stale one and a blank one are different things.
Try three experiments. First, keep the defaults and read the allowance. Second, raise the target from 99.5 to 99.9 per cent and see how little downtime remains, which is why each extra "nine" costs much more to deliver. Third, set the outage so far to 200 minutes and see the budget run out.
At the defaults, a 99.5 per cent target over thirty days allows 216 minutes, a little over three and a half hours. With 90 minutes used, 126 remain, 42 per cent of the allowance spent. At 99.9 per cent the allowance falls to 43 minutes for the whole month. This is why it is worth asking the customer what they really need: a target that sounds only slightly stricter can leave almost no room for ordinary maintenance.
An alert that fires on a single failed check is fast and noisy. One that waits for several in a row is quiet and slow. The simulator below shows the trade-off. Set how often you check, how many failures in a row you require and how often a single check fails by chance.
At the defaults, checking every five minutes and waiting for three failures in a row means a real fault is caught within fifteen minutes and false alarms are almost zero. Alerting on the first failure would give about 170 false alarms a month.
Runbooks: help for the person at 3 a.m.
An alert tells someone that something is wrong. A runbook tells them what to do. It is a short, practical document, one for each alert or common problem, written for a person who did not build the system and is tired, worried and short of time. The goal is that this person can take the right first steps without calling you.
A good runbook has a fixed shape, so that it can be read in a minute.
Title and the alert it answers. "No reminders sent for 2 hours during business hours."
What it means and who it affects. One or two sentences: "Borrowers are not receiving reminders. Callers will see more accounts in the second group."
How serious it is. A line on how urgent, and what makes it worse.
First checks. Three or four quick, specific checks with where to look: "Check the status page at this address. Check whether the nightly file arrived. Check the last successful run."
Likely causes and what to do. For each, the safe action: "If the file is missing, ask the lender's data team to resend, and run the load again with this command. If the messaging provider is down, wait and check again in twenty minutes."
What not to do. "Do not restart the sender while a batch is running, as this may send messages twice."
When to escalate, and to whom. Names and numbers, in order, and the point at which to call: "If not fixed within one hour, call the engineer on duty."
What to tell people. A short message to the customer's team that can be sent as it is.
Afterwards. What to record and who should be told.
Write runbooks in plain words, with commands that can be copied and pasted. Test them by asking someone who did not write them to follow them during a practice. The gaps will be obvious: a missing password, a step that assumed knowledge, a link that does not work.
Running an incident calmly
When something does go wrong, a simple routine keeps the response orderly. It has five parts.
1. Detect and acknowledge. Someone sees the alert and says "I have it". Nothing is worse than two people each assuming the other is on it.
2. Assess. What is the effect, on whom and how bad? Decide whether it is an emergency.
3. Reduce the harm first. Prefer the quick, safe action that stops the damage, such as pausing the sender or rolling back the release, over a perfect diagnosis. Find the cause once the bleeding has stopped.
4. Communicate. Tell the customer early, plainly and regularly: what is wrong, the effect, what you are doing and when you will update next. Silence is the main thing that damages trust in an incident.
5. Learn. Afterwards, write a short account of what happened, the cause, how it was found and fixed and what will change. Keep it blameless: the aim is to improve the system and the routine, not to find a person to blame. People who fear blame hide problems, and hidden problems repeat.
The account should end with a few concrete actions, each with an owner and a date: a new alert, an update to a runbook, a check added to the release plan. The value of the process is in the actions that follow.
Worked example 1: the silent file
The lender's nightly file normally arrives at 2 a.m. One night it does not arrive. The pipeline, written to wait and then run, simply does nothing. In the morning the callers see yesterday's list, and nobody knows. The reminders for the day go out on stale data.
The engineer adds an alert on silence: if no file has been loaded by 6 a.m. on a working day, send a message to the on-call engineer and the lender's data team. A short runbook explains what to check, how to ask for the file to be resent and how to run the load by hand. The next time a file is late, the alert arrives at 6:00, the data team resends by 6:20 and the load completes before the callers start at 9. The cost of the fix was an hour of work, and what it replaced was an invisible failure that had already cost a day.
Worked example 2: from sixty alerts to four
A system inherited from another team sends sixty alerts a week to a shared channel, and the team has muted it. The engineer follows the approach described above: sort a month of alerts, then remove or downgrade.
Of the sixty weekly alerts, six led to an action. Twenty-two fixed themselves, fourteen duplicated another alert, and eighteen could have waited until morning. The new design has four alerts that page a person: no reminders sent in business hours, the nightly file missing, the error rate above a level for ten minutes and the messaging provider's limit nearly reached. Everything else goes to a dashboard, reviewed each morning for five minutes.
The first week, three alerts fire, and all three are acted on. A month later, the team unmutes the channel. The measure of success was not a smaller number of alerts. It was that an alert once again meant something.
Worked example 3: the 3 a.m. duplicate
At 3:10 a.m., the on-call engineer receives an alert: the error rate for the sender has risen, and the provider reports that some messages were sent twice. The engineer opens the runbook for the alert, which says that duplicates usually mean the batch was started twice, and that the first safe action is to pause the sender.
They pause it at 3:14, which stops further duplicates, and then check the queue: a restart after a short outage had re-queued yesterday's unsent batch. They remove the duplicates, resume at 3:40 and send a short message to the head of collections, following the runbook's template: how many borrowers were affected, what was done and when the next update will come. In the morning the engineer writes a one-page account, and the follow-up actions are a check that blocks the same batch from being queued twice and a line added to the runbook.
Nothing in the response was heroic. The runbook said what to do first, the action reduced the harm before the cause was found, and the communication was early and plain. That is the aim of the whole lesson: an incident handled by a routine, not by luck.
The 3 a.m. response is worth timing. The chart shows how long each stage took, from the alert to the sender resuming.
Common mistakes
Watching only the inside. Check the symptoms users and the business would see, such as the reminders not going out.
No alert on silence. Every scheduled job and expected feed needs a check for the absence of success.
Alerting on everything. Every alert should be urgent, actionable, user-visible and understandable.
Dashboards nobody reads. Keep them to a glance, and build them with the customer's team.
No agreed target. Decide with the customer what healthy means, and calculate what it allows.
Runbooks that assume knowledge. Write for a tired stranger, and test them by having someone else follow them.
Diagnosing before stopping the harm. Take the quick, safe action first.
Silence with the customer. Communicate early, plainly and regularly.
Blaming people. Write blameless accounts and turn them into actions with owners and dates.
Key takeaways
Observability means understanding a running system from its outside signs: logs for the story, metrics for numbers over time and traces for the path of a request.
Four signals describe the health of almost any system: latency, traffic, errors and saturation, plus freshness for data. Alert on symptoms, and use causes to diagnose.
Alert on silence: when a job or a feed that should have happened did not.
A good alert is urgent, actionable, user-visible and understandable. Remove or downgrade the rest, and review them monthly.
An availability target implies a budget of allowed downtime, and each extra nine leaves much less room.
A runbook gives a tired stranger the first steps. In an incident, reduce the harm first, communicate early and write a blameless account with actions.
Check yourself
1. Name the four golden signals.
Latency, traffic, errors and saturation.
2. Why alert on symptoms rather than causes?
There are countless possible causes, but the symptoms that matter to the customer are few. Alerting on symptoms catches problems whatever caused them, and causes are used to diagnose.
3. What are four tests of a good alert?
It is urgent, actionable, user-visible and understandable.
4. A 99.9 per cent availability target over 30 days allows how many minutes of outage?
0.001 × 30 × 24 × 60 = 43.2 minutes.
5. In an incident, why reduce the harm before finding the cause?
A quick, safe action such as pausing a sender or rolling back stops the damage. A perfect diagnosis can wait until things are stable.
Try this
This exercise takes about forty minutes and gives you a monitoring plan and a runbook. Use any system you know, a customer example or an imagined one such as a nightly data load.
1. List the three most important things the system does for its users. For each, write the symptom that would show it had failed.
2. For each symptom, write a check that would detect it from the outside, such as a test case or a freshness check.
3. Write the four golden signals for the main service, and what a healthy and an unhealthy value would look like.
4. List every scheduled job or expected feed, and write an alert on silence for each, with a deadline.
5. Pick one alert and apply the four tests. Rewrite it if it fails any.
6. Write a runbook for it using the template: meaning, first checks, likely causes and actions, what not to do, escalation and a message to send.
7. Ask someone else to follow your runbook and note where they get stuck. Fix those places.
8. Use the simulator to work out the downtime your target allows.
A good result is a one-page monitoring plan with symptom checks and silence alerts, a tested runbook and a clear target with its allowance in minutes. It is the operating section of the handover you will write in the next lesson.
Sources
[Monitoring Distributed Systems](https://sre.google/sre-book/monitoring-distributed-systems/), from the Google site reliability engineering book. Used for the four golden signals of latency, traffic, errors and saturation, the point that failed-request latency should be tracked separately, the distinction between symptoms and causes, the rule that every page should be actionable, the warning about alert fatigue and the distinction between black-box and white-box monitoring.
The incident routine, the runbook template, the three examples, the alert counts and the simulator's figures are the author's illustrations and practical guidance for teaching, not data from real teams. The availability arithmetic in the simulator is exact for the target you choose.
Tags: Forward Deployed Engineer Observability Monitoring Runbooks Incident response