Lesson 3.2: APIs, authentication and webhooks

Most customer integrations run on APIs, tokens and webhooks. Learn how requests, logins, retries and event alerts work, and build them to survive failure.

By the end of this lesson you will be able to explain how two systems talk to each other through an interface, read an interface's documentation and make a request, describe how a program proves who it is, and handle the four things that go wrong in almost every real integration: expired logins, rate limits, failed requests and duplicate or out-of-order notifications. You will learn the difference between asking a system for data and being told when something happens, see three realistic integration problems solved and use a simulator to see how many retries are enough. This lesson follows 3.1, where you found which systems hold the data and how each can be reached. Here you learn how to reach the ones that offer an interface. It helps if you have seen a little code, but the ideas are explained from scratch.

If you remember one thing from this lesson, let it be this: an integration that works on the first day is not finished. Every real connection between two systems fails sometimes. The network drops, a login expires, the other side is slow, the same message arrives twice. An engineer who builds only for the case where everything works delivers something that works in the demo and breaks in the second week. An engineer who builds for failure delivers something the customer can rely on. That difference is much of what this lesson is about.

What an interface is

Two programs usually talk to each other through an agreed set of requests, each with an address and a defined reply. This agreement is called an interface, or API, short for application programming interface. A bank's loan system might offer an interface that lets another program ask for an account's balance or add a note to it. You do not need to know how the loan system works inside. You only need to follow the agreement.

Most interfaces you meet in customer work use the same web technology that browsers use. A request has four parts worth knowing.

An address. Often called an endpoint, such as https://loans.example.com/accounts/1234. It says what you are asking about. A method. The kind of request. GET asks for information. POST sends new information, such as creating a record. PUT or PATCH change an existing record, and DELETE removes one. Headers. Extra information about the request, including your login details and the format you are sending. A body. The content you are sending, usually structured text in a format called JSON, which looks like a list of labelled values.

The reply also has a few parts. The most important is a status code, a number that says what happened. The groups are easy to remember: numbers starting with 2 mean success, with 4 mean you did something wrong (or are not allowed), and with 5 mean the other side had a problem. The common ones are 200 for success, 401 for "I do not know who you are", 403 for "I know who you are but you may not do this", 404 for "not found", 429 for "too many requests" and 500 for "something broke on our side".

You will meet other styles too. Some customers use older interfaces, some use files dropped on a server and some use database connections. The ideas below apply to most of them, but the web style is the most common for new work.

Reading the documentation

The first skill is reading an interface's documentation quickly. Good documentation answers a few questions, and you should look for each.

What is the base address, and is there a separate test version? How do I prove who I am? What requests are available for the data I need, and what do they return? How are long lists divided into pages? What are the limits on how many requests I can make? What do errors look like?

Then try one request by hand before you write any code. Tools exist that let you send a request and see the reply. If you can get a single successful reply and understand each field, you have taken the biggest step. Most delays in integrations come from skipping this and writing a lot of code against a misunderstanding.

When the documentation is poor, as it often is for customers' internal systems, the fallback is to ask the system's owner for an example request and a real reply, and to test carefully in a non-production copy. Note anything the documentation does not say, such as what a blank value means, and ask the steward.

Proving who you are

A system must know who is asking, so that it can decide what to allow and keep a record. There are a few common ways a program proves its identity, from simplest to most flexible.

An API key. A long secret string that you send with each request. It is simple, and it works like a password for a program. The risks are that it does not expire by itself and that anyone who sees it can use it.

A username and password. Sent with each request. This is common in older systems.

A token from a login step. Your program first logs in with a credential, receives a short-lived token and sends the token with each later request. The token expires, so a leaked one stops working soon. This is the idea behind OAuth 2.0, the standard that many modern systems use. The standard describes four roles. The resource owner controls the data, often a person. The client is the program that wants to use it. The authorization server checks the identity and issues tokens. The resource server holds the data and accepts tokens. An access token has a limited scope and lifetime. A refresh token, where one is issued, lets the client get a new access token without asking the person again.

For programs that run by themselves, with no person present, as most integrations do, the standard provides a client credentials flow: the program authenticates with its own credentials and receives an access token. The specification says this is meant only for programs that can keep their secrets safe, and that no refresh token should be issued in this case. In practice, your program gets a new token whenever the old one expires.

The practical lessons are these.

Keep secrets out of code. Store keys and passwords in a protected place, such as the customer's secret manager or protected environment settings, never in a file that is saved with your code. Ask for the least access you need. If the interface allows read-only access to three tables, ask for that, not full access. Lesson 3.4 treats this in depth. Expect tokens to expire. Your program should notice a 401 reply, fetch a new token and retry once, not crash. Know how to rotate. Keys should be replaceable without breaking the integration. Ask how the customer rotates secrets, and make sure your program can pick up a new one.

Four things that go wrong

Almost every integration problem falls into one of four groups. Build for them from the beginning.

1. The login expires. The token you were using stops working in the middle of a job. The fix is to handle the 401 reply by getting a new token and trying again, and to avoid caching tokens longer than their life.

2. You are told to slow down. Most interfaces limit how many requests you may send in a given time. When you exceed it, you receive a 429 reply, often with a header that says how long to wait. The right response is to wait and then retry, not to retry at once, which makes it worse.

3. The request fails for a reason that may pass. A timeout, a dropped connection or a 500 reply may succeed a moment later. The standard approach is to retry a few times, waiting longer between attempts. Doubling the wait each time is common and is called exponential backoff. Adding a little randomness to each wait stops many programs from retrying at the same instant.

4. Retrying does damage. If a request creates something, such as a payment or a record, and the reply is lost, then retrying may create it twice. The cure is to make requests safe to repeat, by including a unique key with each request so that the other side can recognise and ignore a repeat. This property is called idempotency. When the interface supports it, use it. When it does not, check whether the thing already exists before creating it.

Predict first: your program is loading a day's records from a customer's system, and partway through, the system replies with 429, "too many requests". What should the program do next? Think of two things to avoid before deciding.

The right response is to pause, for as long as the reply suggests or a short time with backoff if it does not, and then continue from where it stopped. Two things to avoid are retrying immediately in a tight loop, which keeps you over the limit and may get your access blocked, and starting again from the beginning, which wastes requests and may create duplicates. A good program also remembers how far it got, so that after a failure it resumes instead of starting over, and it spreads its requests so that it stays under the limit in the first place.

Pagination and large jobs

A request for a large list rarely returns everything at once. The reply gives one page, for example the first 100 records, plus a way to ask for the next page. Your program must follow the pages until the end. Two pitfalls appear often.

Stopping at the first page. A test with ten records works, and production, with ten thousand, quietly returns only the first hundred. Always test with a realistic volume. Records that change while you read. If new records arrive while you page through, you may miss some or see some twice. Ask the system owner how to read a stable set, for example by asking for "everything changed since this time" and recording the time of your last successful run.

For large jobs, work in batches, record progress after each, and design the job so that running it twice produces the same result as running it once.

Asking versus being told

There are two ways to learn that something has changed in another system.

Polling. You ask repeatedly: "anything new?" It is simple and under your control, but it wastes requests when nothing has changed and delays your reaction until the next ask. Webhooks. The other system tells you when something happens, by sending a message to an address that you provide. A payment arrives, a record changes, and a notification lands in your program within seconds. You do not need to ask, and you react quickly.

Webhooks sound better, and often are, but they add a new job: you now run a service that must be reachable, secure and reliable. The best practices come from the people who run large webhook systems, and they are worth learning as rules.

Check that the message is genuine. Anyone can send a message to your address. Reputable systems sign each message, with a secret that only you and they know, so you can check that it came from them and was not changed on the way. Never act on an unsigned message from the internet. Signed messages usually include a timestamp, and you should reject ones that are too old, to defeat an attacker who replays a message they captured. Answer quickly. Return a success reply right away, then do the real work afterwards. If you take too long, the sender thinks the delivery failed and tries again. Expect duplicates. Senders retry when they do not get a reply, so you may receive the same event twice. Record the identifier of each event you have processed, and ignore repeats. Do not rely on order. Events can arrive in a different order from the one in which they happened. Your program must cope, for example by fetching the latest state of the record when an event arrives instead of trusting the event's content alone. Process in a queue. A burst of events can overwhelm a program that handles each one while the sender waits. Put events in a queue and work through them at a rate your system can handle. Plan for outages. Senders typically retry for some time, and some let you resend events manually. But if your service is down long enough, you will miss events. A periodic catch-up job that asks "what changed since my last good run?" covers the gaps.

Many production systems use both: webhooks for speed and a slower polling job as a safety net.

Files and other routes

Not every customer has an interface. Many systems exchange data through files: a nightly export placed on a secure file server, or sent to a mailbox. The same ideas apply, with different details. Check that the file arrived and is complete, compare the number of rows with what the sender says it sent, handle the file arriving late or twice, and never process a file that is still being written. Treat a file feed as an interface with its own failure modes, and write down the agreement: when it arrives, what it contains and what happens when it does not.

Worked example 1: the payment notifications

A lender's collections pilot needs to know when a borrower pays, so that it stops sending reminders. The payment gateway offers webhooks. The engineer builds a small service that receives them.

On the first day, it works. In week two, the gateway has an outage and then replays the events of three hours in a burst, many of them already delivered. The service, which updates the account on each event, marks several accounts as paid twice and sends confirmation messages twice to some borrowers.

The fixes follow the rules above. The service now checks the signature and the timestamp of each message, stores the event identifier and ignores repeats, answers quickly and puts the work in a queue. A catch-up job runs every hour and asks the gateway for payments since the last good run. After these changes, the next outage causes no duplicates. The engineer also adds an alert that fires if no events arrive for thirty minutes during business hours, so that a silent failure is noticed.

Worked example 2: the expired login

A retailer's integration pulls daily sales from a partner's interface using a token that lasts one hour. The engineer's first version fetches the token once at the start and uses it throughout. For small days the job takes ten minutes. During the festival season, the job takes ninety minutes, and halfway through, every request returns 401. The job fails, and nobody notices until the next morning.

The fix has two parts. The program now watches for 401, fetches a new token and retries the request once. It also writes down how many records it has loaded, so that a failed job can restart from there. The engineer adds a morning check that compares the number of records loaded with the number the partner reports, and sends a message if they differ. A single expired token used to cause a silent gap. Now it causes, at worst, a short delay and a note.

The expired-token story comes down to two numbers: how long the token lives and how long the job takes. The chart shows them for the retailer's integration.

Worked example 3: the retry storm

A logistics company's integration calls a partner's interface for the status of each of 5,000 shipments every night. The partner allows 100 requests a minute. The first version sends requests as fast as it can. After 100 requests, the partner returns 429. The program retries immediately, 50 times a second, and the partner temporarily blocks the company's key for an hour.

The fix is to respect the limit. The program spreads its requests at about 80 a minute, which finishes in roughly an hour, reads the wait time in the 429 reply when it does arrive, and uses exponential backoff with some randomness for other failures. It also uses a bulk endpoint that the partner offers, which returns forty shipments per request, so that the whole job needs 125 requests. A job that used to be blocked now finishes in two minutes. The lesson is general: read the limits and the bulk options in the documentation before you write the loop.

The retry storm came from ignoring a limit and a bulk option. Set the number of items, how many a single request can carry and the limit, and see how long the job takes.

At the defaults, 5,000 single-item requests at 80 a minute take about an hour. If one request can carry 40 items, there are 125 requests and the job takes under two minutes.

How many retries are enough?

How many times should a program retry a failed request? Too few and a brief glitch becomes a failed job. Too many and a real outage ties your program up for a long time, or you hammer a struggling system. The arithmetic helps.

If each attempt fails with some chance, and attempts are independent, the chance that every attempt fails falls fast with each retry. With waits that double each time, the total waiting time also grows fast, so you are trading reliability against delay. The simulator below shows both. Set the chance that one attempt fails, the number of retries and the length of the first wait. It shows the chance that the request never succeeds, the longest time you would wait and the average number of attempts.

The independence assumption is a simplification, since real failures often come in bursts, but it shows why three to five retries with backoff handles most brief problems, and why a long outage needs a different design: a queue and a catch-up job, not endless retries.

Try three experiments. First, keep the defaults and see how much a failure rate of 20 per cent drops with three retries. Second, raise the failure rate to 70 per cent, which looks like an outage, and see that no sensible number of retries rescues it. Third, increase the retries to eight and notice how long the worst-case wait becomes.

At the defaults, a 20 per cent failure rate with three retries means a request fails for good only 0.16 per cent of the time, the longest wait is 14 seconds and the average number of attempts is about 1.25. Three retries nearly always rescue a brief glitch at a small cost. Raise the failure rate to 70 per cent, and the same three retries still leave a 24 per cent chance of failure. This is the shape to remember: retries cure glitches and cannot cure outages.

Testing and handing over

A few habits make integrations safer to run.

Use a test environment. Ask for a sandbox, with fake data, and test failure cases there: expired tokens, slow replies, duplicate events. Log what matters. Record each request's outcome, the status code and how long it took, but never log secrets or sensitive data. Alert on silence. The most dangerous failure is one that makes no noise. Alert when expected data does not arrive. Document the agreement. Write down the interface, the limits, the schedule and what happens on failure, so that the next person can run it. Plan for change. Interfaces change. Ask how the customer announces changes, and check which version you are using.

Later modules cover monitoring and handover in more depth. For now, remember that the first version is a draft and that failures are the normal case.

Common mistakes

Coding before trying one request by hand. A single successful request teaches more than an hour of reading. Putting secrets in code. Use protected storage, and know how to rotate. Assuming tokens last. Handle 401 by getting a new token and retrying once. Retrying instantly. Back off, add randomness and respect limits and wait times. Retrying unsafe requests. Use an idempotency key, or check before you create. Testing only small data. Pagination and limits show up at production size. Trusting unsigned webhooks. Verify the signature and the timestamp. Doing the work before answering. Reply quickly, then queue the work. Assuming events arrive once and in order. Record event identifiers and fetch the latest state. No safety net. Add a catch-up job and an alert for silence.

Key takeaways

An interface is an agreed set of requests and replies. Learn to read the documentation and to send one request by hand first. Programs prove who they are with keys, passwords or short-lived tokens. Keep secrets out of code, ask for least access and expect tokens to expire. Four things go wrong in almost every integration: expired logins, rate limits, transient failures and unsafe retries. Build for each from the start. Webhooks tell you when something happens, and need signature checks, quick replies, duplicate handling, no reliance on order and a queue. Use polling or catch-up jobs as a safety net for missed events and outages. Retries cure glitches but not outages. Alert on silence, because quiet failures are the most dangerous.

Check yourself

1. What do status codes starting with 2, 4 and 5 mean? 2 means success, 4 means the request was wrong or not allowed, and 5 means the other system had a problem. 2. A job fails halfway with a 401. What is the right response? Get a new token and retry the request once, then continue from where the job stopped. Do not crash or start over. 3. Why must a webhook receiver record the identifiers of events it has processed? Senders retry when they get no reply, so the same event can arrive more than once. Recording identifiers lets you ignore repeats. 4. With a 30 per cent chance of failure per attempt and 2 retries, what is the chance the request never succeeds? 0.3 to the power of 3, which is 0.027, so 2.7 per cent. 5. Why use a catch-up job even if you have webhooks? Webhooks can be missed during an outage on either side. A periodic job that asks what changed since the last good run fills the gaps.

Try this

This exercise takes about forty minutes and gives you a first working integration. Use any free public interface that needs no payment, such as a public weather, currency or open-data service.

1. Find the documentation. Write down the base address, how to authenticate, the request you will use and any limits. 2. Send one request by hand, using a tool that lets you see the reply. Write down the status code and what each field in the reply means. 3. Write a short program, in any language, that makes the request and prints one value. Put any key in an environment setting and not in the code. 4. Add handling for failure: if the reply is 429 or 5xx, wait and retry up to three times, with the wait doubling each time. Test it by pointing at a wrong address, and see what happens. 5. If the service returns lists in pages, make your program read two pages. 6. Use the simulator above to work out the chance of never succeeding with your retry settings, assuming a 20 per cent failure rate. 7. Write four lines of notes that a colleague could use to run it: what it does, where the key is stored, the limits and what to do if it fails.

A good result is a short working program, a page of notes and a clear view of how it behaves when things go wrong. Doing this once with a harmless public service gives you the habits that matter when the same pattern is running against a customer's real systems.

Sources

[The OAuth 2.0 Authorization Framework (RFC 6749)](https://www.rfc-editor.org/rfc/rfc6749), the standards document. Used for the four roles, access and refresh tokens, and the client credentials flow, including that it is meant for programs that can keep secrets and that no refresh token should be issued. I read the first part of the document. [Receive Stripe events in your webhook endpoint](https://docs.stripe.com/webhooks), Stripe documentation. Used as an example of good practice from a large webhook provider: verifying signatures and timestamps, replying quickly with a success status, handling duplicate events by recording event identifiers, not relying on event order, processing in a queue and rolling secrets. It describes one company's system, so check the details of the system you integrate with. The three integration stories, the request outcomes and the simulator's figures are the author's illustrations for teaching, not data from real companies.

Tags: Forward Deployed Engineer APIs Authentication Webhooks Integration