Lesson 5.2: Packaging, releases and rollbacks
A release that cannot be undone is a gamble. Learn to package software, number releases, ship in stages and keep a tested way back for every change.
By the end of this lesson you will be able to package a customer's software so that it installs the same way every time, number each release so that anyone can say what is running, ship changes in stages that limit the damage when something is wrong, and keep a tested way back for every release. You will learn a build-once rule, a release checklist, four ways to roll out a change and the special care that database changes need. You will see three releases, including one that went badly, and use a simulator to see how much a staged rollout limits the harm of a faulty release. This lesson follows 5.1, where you learned where the software runs. Here you learn how to put new versions there, safely, again and again. You do not need to be an infrastructure specialist, though some familiarity with the idea of versions and installs helps.
In the first weeks of a project the main question is whether the thing works. After it goes live, a second question becomes more important: can it be changed safely? Every customer system needs changes, to fix problems, to add what the first users asked for and to keep up with the world around it. A system that can be changed safely improves quickly. A system that no one dares to touch, because the last release broke something and nobody knows exactly what is running, slowly decays. The practices in this lesson are what separate the two.
Why releases go wrong
Most failed releases come from a small set of causes, and none is exotic.
The environments differ. It worked in test and failed in production, because a setting, a library version or a permission was not the same.
Nobody knows exactly what is running. A fix was applied by hand on one server and not recorded, so the next release overwrites it or behaves differently there.
Changes arrive in large batches. Dozens of changes released at once make it hard to tell which caused the problem.
There is no way back. The release is a one-way door: when it fails, the team is forced to fix it forward, under pressure, in front of the customer.
The data changed too. A change to the structure of the database was applied and cannot easily be reversed.
Each cause has a practice that addresses it, and together they make releases dull. Dull is the aim. A release that nobody notices is a release that went well.
Packaging: one artefact, built once
The first practice is to package the software into a single, self-contained, versioned unit, often called an artefact. A common form is a container image: a bundle that holds the application, the libraries it needs and the instructions for starting it. The same bundle runs on a developer's laptop, in the customer's test environment and in production.
Three rules make packaging work.
Build once, deploy the same thing everywhere. The package that you tested is the package that goes to production, byte for byte. Do not rebuild it for each environment, because a rebuild can pick up different versions of libraries and produce something subtly different from what you tested. If production is running something nobody tested, the testing proved little.
Keep settings outside the package. The package holds the code. Environment-specific values, such as addresses, credentials and limits, are supplied when it starts, as Lesson 5.1 recommended. This is what allows one artefact to run in test and in production with different settings.
Make it immutable. Once a version is built and published, never change it. If something needs to change, build a new version. This sounds fussy and prevents an entire class of confusion, where two machines both claim to run "version 1.4" while holding different contents.
For customers who cannot reach the internet, the package may be a complete bundle, carried in and installed from an internal source, with the list of every library and its version, as Lesson 5.1 described.
Numbering releases
Every release needs a name that anyone can read and that tells you something. A widely used convention is semantic versioning, which writes a version as three numbers, major, minor and patch, such as 2.4.1.
The patch number goes up for fixes that do not change how the software is used: 2.4.1 to 2.4.2.
The minor number goes up for new features that are backward compatible, so existing users and connected systems carry on working: 2.4.2 to 2.5.0.
The major number goes up for changes that break compatibility, such as a changed interface that other systems rely on: 2.5.0 to 3.0.0.
The convention also says that a released version is never altered: any change means a new version. In customer work, the value is mostly in the clarity. When someone says "the nightly job misbehaved after the update", you can ask "from which version to which?" and look up exactly what changed. Pair each version with short release notes in plain words: what changed, why, what to watch and how to go back.
You do not need to follow every detail of the convention. You do need a scheme that everyone understands, a record of which version runs where, and the discipline to make no untracked changes to a running system.
A release checklist
Releases become reliable when the routine is written down and followed every time. A useful checklist has four stages.
Before the release.
The change is reviewed, and the tests pass.
The new version is built once and stored with its number.
Release notes are written, including the plan for going back.
Approval is recorded, following the customer's change process from Lesson 5.1.
The people who will be affected know when it will happen.
During the release.
Note the time and the version being replaced.
Deploy to a test environment that matches production, and run the checks there.
Deploy to production in the way chosen, such as in stages.
Run a short set of smoke tests: a few quick checks that the main things work, such as that the service starts, can reach its data and can complete one real, harmless task.
After the release.
Watch the main signals for a set period, with someone responsible.
Confirm to the customer that it is done and working.
Record what was released, when and by whom.
If something is wrong.
Follow the rollback plan without debate, if the thresholds you set in advance are crossed.
Tell the customer plainly and promptly what happened and what is being done.
Write a short account afterwards of the cause and what will change.
The details vary by customer, but the shape is stable, and having it written means that nobody invents a plan under pressure at eleven at night.
Four ways to roll out a change
How you put a new version into use matters as much as what is in it. Four patterns are common, from the riskiest to the most careful.
All at once. Replace the old version everywhere in one step. It is the simplest, and if the new version is faulty, everyone is affected immediately. It suits small, low-risk changes and systems with very few users.
Rolling. Replace the old version a little at a time, such as one server after another. If the first ones show a problem, you stop. At any moment, some users may be on the old version and some on the new, so the two must be able to work side by side.
Blue-green. Run two complete, nearly identical production environments. One, say blue, serves the live traffic. You install the new version in the other, green, and test it there. When you are satisfied, you switch the traffic from blue to green. Blue stays intact, so if the new version fails, you switch straight back. Martin Fowler's description of the technique points out that this gives near-zero downtime and a fast way to undo a bad release, and that it also lets you test your recovery routine with every release. The cost is running two environments, and the main difficulty is the database, which both sides usually share.
Canary. Release the new version to a small share of users first, watch the results and widen it in steps if all is well. The name comes from the miner's canary: an early warning, exposed to the danger first. If the new version is faulty, only the small group is affected, and you return them to the old version. Canary releases suit systems with many users, where a small share is a meaningful test.
Choose by the cost of failure. A change that touches money, messages to customers or patient information deserves staging. A change to a screen's colour does not.
Predict first: you release a new version at 11 p.m. in the customer's change window. Within ten minutes the error rate has doubled. You applied a change to the database structure as part of the release. What do you do first, and what makes the decision hard? Decide before reading on.
The first action is to follow the rollback plan you wrote before the release, which should say, in advance, what error level triggers it and who decides. What makes the decision hard is the database change. If the change was made so that both the old and new versions of the software can use the database, then rolling back the application is safe and quick. If it removed or renamed something the old version needs, going back may break it, and you may be forced to repair forward. This is why the plan must be made before the release, and why database changes get their own rules, below.
Database changes need extra care
Code can be replaced and rolled back easily, because it holds no state of its own. Databases hold the customer's real data, and the structure of that data cannot simply be reversed by installing an older version. Many release disasters involve a database change.
The safest approach is to separate changes to the database from changes to the application, and to make each in a way that does not break the other. A common pattern is often called expand, then contract. Martin Fowler outlines the same approach, under his own wording, in his description of blue-green deployments: release a database change that supports both versions, then the new application, and only later remove support for the old one.
1. Expand. First release a database change that supports both the old and the new application. For example, add a new column, leaving the old one in place. Check that it works. You now have a safe point to return to.
2. Move. Release the new application, which uses the new column. If it fails, the old application still works, because the old column is intact.
3. Contract. After the new version has been stable for some time, release a change that removes the old column, which nothing now needs.
The cost is that one change takes three releases instead of one. The benefit is that at every step, you can go back. For changes that touch customer money or personal records, it is well worth it.
Two further habits help. Back up before a risky change, and check that the backup can actually be restored, since an untested backup is a hope. And be careful with writes during a switch: if users are saving data while you switch from one version to another, decide in advance how those writes will be handled, for example by putting the system briefly into read-only mode.
Speed of rollback is the other half of limiting harm. Every minute the faulty version is live, users are hit. The simulator below adds up how long it takes to notice, decide and roll back, and compares a team that has agreed its trigger in advance with one that debates it.
At the defaults, the fault is live for thirty minutes and hurts about 1,200 users. With the trigger agreed in advance, the decision takes a minute and the count falls to about 640.
Feature flags
A feature flag is a switch, in the software's configuration, that turns a new behaviour on or off without a new release. You release the code with the feature switched off, then switch it on for a few users, then for more. If it misbehaves, you switch it off in seconds, with no rollback of the software.
Flags are powerful for customer work, because they separate two risks: the risk of installing new code and the risk of exposing a new behaviour. You can install on a quiet day, and expose on your own schedule. They also let you turn a feature on for the customer's test group first, which suits a pilot.
They have costs. Every flag is a branch in the code that must be tested in both states, and old flags that stay forever make the code harder to understand. Remove a flag when the feature is fully released.
Worked example 1: the quiet release
The lender's reminder system is live. The engineer needs to change how messages are grouped, by moving accounts that paid last month to the easy group.
The change is small, but it affects who gets a message, so it matters. The engineer builds version 1.3.0 once, with tests, and writes release notes: "Accounts that paid within the last month move to the easy group. Risk: some accounts may change group. Rollback: redeploy 1.2.4." The change goes behind a flag, off by default.
The deploy takes four minutes in the lender's change window. The smoke test checks that the service starts, reads the nightly file and produces the expected count for one test account. The next morning, with the head of collections watching, the engineer turns the flag on for the control group's neighbouring team only. After two days of normal results, it is turned on for everyone. Nobody outside the project noticed any of it, which is the best sign that it went well.
Worked example 2: the release that broke the nightly job
The hospital group's bed screen needs a new field for the bed type. A developer on the engineer's team makes the change in one step: the database structure is changed to rename an old column, and the new application version reads the new name. Both go out together at 6 a.m. on a Sunday.
The screen works. But the nightly job that loads data from the second hospital, written weeks earlier and not part of this release, still writes to the old column name. That night it fails. On Monday morning the screen shows stale data for that hospital, and nobody has been told. The failure was found two hours after the shift began, by a charge nurse.
The postmortem finds three causes. The rename should have been done in the expand-and-contract way, so that both names worked for a while. The release did not list every system that writes to that table. And the alert for a missing nightly load, which Lesson 3.3 recommended, was not yet in place for this hospital. The fix is made forward in an hour, but the lesson is that a change touching shared data has a reach wider than the code you changed. Ask: what else reads or writes this?
Worked example 3: the canary that saved a day
A retailer's order suggestion service has about 400 stores using it. The new version changes how the forecast is combined with the stock level. The engineer decides on a canary: first five stores, chosen to include a large one, a small one and one with unusual products.
Within an hour, one of the five stores shows suggestions of zero for a handful of fast-selling items. The cause is a unit mismatch, boxes against single items, that appears only for products sold in cases. The engineer switches the five stores back, in two minutes, fixes the conversion, adds a test for case-packed products and releases again to the five stores the next day. All 400 stores then follow in three steps over a week.
Had the version gone to all 400 at once, those zero suggestions would have reached every store that ordered that morning, and the retailer would have run out of its best sellers across the chain. The canary cost a week of staging and saved a very bad Monday.
How much does staging limit the harm?
The benefit of a staged rollout can be put in numbers. If a faulty release affects a share of the users it reaches, then releasing to everyone at once exposes all of them. Releasing first to a small group exposes only that group, until you notice and go back.
The simulator below shows the difference. Set the number of users, the share in the first stage, the share of users who would be hit by the fault and the hours it takes to notice. It shows how many users are hit with an all-at-once release and with a staged one, and the reduction. It also plots the harm by the size of the first stage.
The model is simple. It assumes the fault is noticed within the hours you set and that harm scales with the number of users exposed. Real faults vary a great deal, and some are slow to appear. But the main point holds: the harm of a bad release is roughly proportional to how many people receive it before you stop, and you control that number.
Try three experiments. First, keep the defaults and read the reduction. Second, raise the first stage from 5 to 25 per cent and see how the benefit shrinks. Third, raise the share of users hit by the fault from 20 to 80 per cent, which describes a severe fault, and see why staging matters most for the worst releases.
At the defaults, a fault that hurts 20 per cent of 100,000 users harms 20,000 if released to everyone, but 1,000 if the first stage is 5 per cent, 95 per cent less. Notice that the benefit depends on the first stage being small. The cost is time: a staged rollout may take several days, which is why you reserve it for changes that matter.
Rolling back well
A rollback is only as good as its preparation. A few rules help.
Decide before you release. Write down the signals that trigger a rollback, such as an error rate above some level, and who decides. In the moment, people hesitate.
Practise it. Roll back in the test environment during rehearsals, so that you know it works and how long it takes.
Keep the old version ready. The previous package must be available, with its settings.
Know what cannot be undone. Messages sent, payments made and emails delivered cannot be recalled. For such actions, the safeguard is to stage and to check before, not to plan on reversing after.
Say when you will roll forward. Sometimes going back is more dangerous than fixing forward, for instance after a one-way database change. Know in advance which case you are in.
Tell the customer. A short, calm message: what happened, what you did, what the state is now and when you will update again.
Common mistakes
Rebuilding for each environment. Build once and deploy the same package everywhere.
Changing production by hand. Every untracked change makes the next release a surprise. If you must, record it and bring it back into the code.
Releasing in large batches. Small, frequent changes are easier to understand and to undo.
No version numbers. Without them, nobody can say what is running or what changed.
One-step database changes. Use expand, move, contract, and back up first.
No smoke tests. A few quick checks after a release catch obvious breakage in minutes.
No rollback plan. Write it before you release, and practise it.
Forgetting what else touches the data. List every system that reads or writes what you are changing.
Leaving flags behind. Remove a flag when the feature is fully released.
Key takeaways
Most failed releases come from differences between environments, untracked changes, large batches, no way back and database changes.
Package the software as one versioned, immutable artefact, build it once and deploy the same thing everywhere, with settings kept outside it.
Number releases clearly, for example with major, minor and patch numbers, and write short release notes with the rollback plan.
Roll out in a way matched to the risk: all at once, rolling, blue-green or canary. Staging limits the harm to the users reached before you stop.
Make database changes in three steps, expand, move and contract, and back up first.
Decide the rollback triggers and practise the rollback before the release. Some actions cannot be undone, so stage and check before them.
Check yourself
1. What does "build once, deploy the same everywhere" mean, and why does it matter?
The package you tested is exactly what goes to production, never rebuilt. A rebuild can pick up different library versions, so production would run something nobody tested.
2. Explain expand, move and contract for renaming a database column.
First add the new column while keeping the old one, so both versions of the software work. Then release the application that uses the new column. After it is stable, remove the old column.
3. How does a canary release limit harm?
It sends the new version to a small share of users first, watches the results and widens in steps. A fault affects only the small group, who can be returned to the old version.
4. A fault would hurt 30 per cent of the users who receive it. With 50,000 users, how many are hurt if released to all at once, and if the first stage is 10 per cent and the fault is caught there?
All at once: 50,000 × 0.30 = 15,000. Staged: 50,000 × 0.10 × 0.30 = 1,500.
5. Why decide the rollback triggers before the release?
Under pressure, people hesitate and argue. A pre-agreed trigger and decision-maker turns the choice into a routine step.
Try this
This exercise takes about forty minutes and produces a release plan for a change you know. Use a small system of your own, a customer example or an imagined one.
1. Describe a realistic change, such as adding a field or changing a rule, in one sentence.
2. Write the version numbers before and after, and say whether the change is a patch, minor or major, and why.
3. Write the release notes in four lines: what changed, why, what to watch and how to go back.
4. Choose a rollout method, and write why it fits the risk.
5. If the change touches a database, write the expand, move and contract steps.
6. Write five smoke tests, each a quick check that would show the release is broken.
7. Write the rollback triggers, the person who decides and the first three steps.
8. Use the simulator with your own numbers to estimate how much harm a staged release would avoid.
A good result is a one-page release plan with a version, release notes, a rollout choice, smoke tests and rollback triggers. It is the document that turns a nervous late-night deployment into a routine one.
Sources
[Blue-green deployment](https://martinfowler.com/bliki/BlueGreenDeployment.html), Martin Fowler. Used for the description of two near-identical environments with a switch between them, the fast rollback it allows, the use of the same mechanism to test recovery, and the advice to split database changes from application releases so that each is compatible with both versions.
[Semantic Versioning](https://semver.org/). Used for the major, minor and patch convention and the rule that a released version is never altered.
The three releases, the failure causes and the simulator's figures are the author's illustrations for teaching, not data from real teams. The release checklist and the rollback rules are practical guidance written for this lesson.
Tags: Forward Deployed Engineer Releases Rollbacks Deployment Software delivery