Integrating third-party APIs: the failure modes that only appear in production

Calling someone else's HTTP API is easy to start and surprisingly deep to finish. The request succeeds in development, and the integration looks done. What is missing is every branch where the other service does not behave: it takes forty seconds, it returns a 500, it accepts your request and drops the connection before you learn that it did, or it starts refusing you because you sent requests too quickly.

These are the failure modes I have run into doing automation and integration work, including during an unpaid AI/ML automation role at Hwy Haul in mid-2026, and while building IntakeKit, an unpublished personal project. None of it is exotic. All of it is the kind of thing that is much cheaper to build in at the start than to retrofit after an incident.

Always set a timeout

Most HTTP clients default to no timeout or an extremely long one. That default is close to always wrong, because it converts a slow dependency into an outage of your own service: your workers block, the pool exhausts, and requests that have nothing to do with that API start failing.

import requests

# connect timeout, read timeout
resp = requests.get(url, timeout=(3.05, 10))

Two separate numbers, because they mean different things. The connect timeout covers establishing the TCP connection — if that takes more than a few seconds the host is unreachable and waiting longer will not help. The read timeout is how long you will wait for data after connecting, which depends on what the endpoint actually does.

Set these per endpoint rather than globally. A token refresh should time out in two seconds; a report-generation endpoint may legitimately need sixty. A single global value is either too tight for the slow one or too loose for the fast one.

Retry the right errors, and only the right ones

Retrying blindly is worse than not retrying. Repeating a request that failed because it was malformed just produces the same 400 four more times, and repeating a non-idempotent write can duplicate real-world effects.

A workable division:

• Retry: connection errors, timeouts, 429, 502, 503, 504. These are transient by nature.
• Do not retry: 400, 401, 403, 404, 422. The request is wrong; sending it again will not fix it. Fail loudly instead.
• Depends: a bare 500. Some services return it for transient faults and some for a permanent bug. Retry it once or twice at most, and log it distinctly so you can tell which kind you are dealing with.

Backoff needs jitter

Exponential backoff — waiting 1s, then 2s, then 4s — is standard. The part that is frequently skipped is randomisation, and skipping it creates a specific failure.

If a service briefly goes down and two hundred of your requests fail simultaneously, pure exponential backoff means all two hundred retry at exactly one second, then all two hundred at exactly two seconds. You have built a synchronised herd that hits the recovering service in waves, which is often what keeps it down.

import random, time

def call_with_retry(fn, attempts=5, base=0.5, cap=30.0):
    for attempt in range(attempts):
        try:
            return fn()
        except Transient as exc:
            if attempt == attempts - 1:
                raise
            # full jitter: sleep a random amount within the backoff window,
            # so concurrent clients spread out instead of retrying in lockstep
            window = min(cap, base * (2 ** attempt))
            time.sleep(random.uniform(0, window))

Full jitter — sleeping a uniform random time between zero and the window, rather than the window itself — spreads retries across the interval and is what actually prevents the herd. It is one line and it is the difference between helping a recovering service and hammering it.

Also honour Retry-After when the response includes it. If a service explicitly tells you when to come back, your backoff formula is a worse guess than its instruction.

Idempotency: the retry you cannot see the result of

Here is the case that motivates everything else. You POST a request that creates something. The server processes it successfully. The response is lost — connection reset, timeout, load balancer restart. Your client sees a failure and retries. Now the thing has been created twice.

Retries are only safe when the operation is idempotent: performing it twice has the same effect as performing it once. GET, PUT, and DELETE are usually idempotent by design. POST usually is not.

The standard fix is an idempotency key — a unique identifier you generate for the logical operation and send with every attempt:

import uuid

key = str(uuid.uuid4())   # generated ONCE per logical operation
resp = session.post(
    url,
    json=payload,
    headers={"Idempotency-Key": key},
    timeout=(3.05, 10),
)

The critical detail: generate the key once, outside the retry loop, and reuse it for every attempt of that same operation. A key generated inside the loop is a new key each time, which is exactly equivalent to having no key at all. This is the most common way idempotency is implemented incorrectly.

If the API you are calling does not support idempotency keys, you need your own guard: record the operation with a unique constraint in your own database before calling, and check it before retrying.

Rate limits are a design input, not an error

Treating 429 as an exception to be caught is a reactive approach that works until you have any real volume. Better to read the limit headers most APIs return and stay under the ceiling deliberately:

X-RateLimit-Limit: 1000
X-RateLimit-Remaining: 47
X-RateLimit-Reset: 1786000000

With Remaining visible you can slow down before being refused. For batch work, a token-bucket limiter on your side is more predictable than discovering the ceiling by hitting it.

Two related habits. Cache aggressively when the data does not change often — the cheapest request is the one you do not send, and honouring ETag or Last-Modified with a conditional request often returns a 304 that costs no quota. And batch when the API supports it: one request fetching a hundred records is not merely faster than a hundred requests, it consumes a hundredth of the rate limit.

Pagination, and the trap in offset pagination

Never assume the first page is all of it. A common bug is reading results from the first response and ignoring next entirely, which works perfectly in testing against a small dataset and silently truncates in production.

Prefer cursor pagination when it is offered. Offset pagination — ?page=3&per_page=50 — has a real correctness problem: if a record is inserted or deleted while you are paging, the window shifts and you can skip or duplicate records. Cursor pagination, which passes an opaque pointer to the last item seen, is stable under concurrent modification.

cursor, out = None, []
while True:
    params = {"limit": 100}
    if cursor:
        params["cursor"] = cursor
    page = get(url, params=params)
    out.extend(page["data"])
    cursor = page.get("next_cursor")
    if not cursor:
        break

Always bound the loop. A malformed response that keeps returning the same cursor turns this into an infinite loop that will happily consume your entire rate limit; a maximum page count costs one line.

Webhooks are untrusted input

When a service calls you instead, the direction reverses and so do the assumptions. A webhook endpoint is a public URL that anyone can POST to.

Verify the signature on every request, using the shared secret and a constant-time comparison — hmac.compare_digest in Python, not ==, because ordinary string comparison returns early on the first differing byte and leaks timing information. Check the timestamp too, and reject anything older than a few minutes, or a captured valid request can be replayed indefinitely.

Expect duplicates. Webhook delivery is at-least-once in essentially every implementation, so the same event will occasionally arrive twice. Store the event ID and ignore ones you have already processed.

Respond fast. Acknowledge with a 2xx immediately and do the real work asynchronously. If your handler takes twelve seconds, the sender times out, marks delivery failed, and retries — and now you are processing the same event repeatedly while the queue backs up.

Log the correlation ID

Most APIs return a request identifier, often as X-Request-Id. Log it with every call, successful or not.

The reason is practical: when something goes wrong and you contact the provider's support, "request req_8f2a... at 14:32 UTC returned a 500" gets a specific answer. "Some of our requests are failing sometimes" gets a form response. That identifier is the difference between a resolved ticket and a week of round-trips.

Log the status code, the latency, and the endpoint alongside it. Latency percentiles over time are how you notice a dependency degrading before it fails outright.

A checklist before calling an integration done

1. Every request has an explicit connect and read timeout.
2. Retries cover transient failures only, with exponential backoff and jitter.
3. Retry-After is honoured when present.
4. Non-idempotent writes carry an idempotency key generated outside the retry loop.
5. Rate-limit headers are read, and there is a plan for approaching the ceiling.
6. Pagination is followed to completion, with a bound on iterations.
7. Webhook signatures are verified in constant time, with replay and duplicate protection.
8. Correlation IDs, status codes, and latencies are logged.
9. Secrets come from the environment, never from source control.
10. There is a defined behaviour when the dependency is entirely unavailable — queue, degrade, or fail clearly, but decided rather than accidental.

None of this is difficult individually. The reason it is worth doing up front is that each of these failures is invisible in development and obvious only in production, usually at the least convenient moment. The general principle is the one that applies to parsing messy real-world input as much as to network calls: decide what happens in the bad case deliberately, because if you do not decide, the behaviour still exists — you just have not chosen it.