API Development Built for the Day It Fails

Why do integrations break in production?

Because production has conditions that development does not — network timeouts, rate limits, provider outages, and data shaped in ways the test cases never contained.

The specific failures repeat. A request times out, gets retried, and creates a duplicate record. A provider changes a field and the parser fails silently. A rate limit is hit during a busy period and requests are dropped rather than queued. A webhook arrives twice and is processed twice.

None of these are exotic, and all of them are cheap to handle at build time and expensive to diagnose afterwards. That is the whole argument for designing failure handling alongside the feature rather than after the first incident.

What is idempotency and why does it matter?

An idempotent operation produces the same result whether it runs once or five times. For anything that creates records, charges money or sends messages, it is the difference between a safe retry and a serious problem.

The scenario is ordinary: a request to create an order times out. The client does not know whether the server processed it. Retrying risks a duplicate order; not retrying risks a lost one. Without idempotency there is no correct choice.

With an idempotency key, the retry is safe — the server recognizes the request it already handled and returns the same result rather than creating a second order. Any API that creates or charges should support it, and any integration with one should use it.

Should we build an API or use a platform integration?

Use the existing integration if one covers your case. Custom API work is worth it when no off-the-shelf connector fits, when the data mapping is genuinely specific to your business, or when reliability requirements exceed what a generic tool provides.

Middleware platforms sit usefully between the two. For low-volume, non-critical flows they are often the right answer — cheaper than building and adequate for the job.

Where they struggle is volume, complex transformation, and failure visibility. When a middleware flow breaks silently at three in the morning, diagnosing it is frequently harder than debugging code you own. That trade is worth making deliberately rather than by default.

How do you handle a third-party API changing?

By assuming it will. Providers deprecate versions, change field names and adjust rate limits, usually with notice and occasionally without.

What limits the damage is defensive parsing that fails loudly rather than silently, schema validation on responses, monitoring that alerts when error rates rise, and pinning to a specific API version where the provider supports it.

The failure mode worth avoiding is the silent one. An integration that keeps running while quietly dropping a field can corrupt data for weeks before anyone notices, and reconstructing what was lost is far more expensive than the outage would have been.

What does a good API contract look like?

Explicit, versioned and boring. Every endpoint documented with its parameters, its response shape and every error it can return — including the ones that are unlikely, because those are the ones nobody handles.

The most useful thing a contract does is make the error cases first-class. Documentation that describes only the successful response leaves every consumer to discover the failure modes in production, one incident at a time.

Consistency matters more than elegance. If one endpoint returns a date as a string and another as a timestamp, every consumer writes special-case code forever. Pick a convention, apply it everywhere, and write it down.

A machine-readable specification is worth the effort. It generates documentation, validates responses in tests, and gives consumers a client without them hand-writing one. It also makes a breaking change visible at review time rather than at runtime.

How do you handle a third party going down?

By deciding in advance what your system does without them, which is a product question as much as a technical one.

Some dependencies are fatal — a payment provider being unavailable means you cannot take payment, and the honest response is a clear message rather than a spinner. Others are not: a recommendation service failing should degrade to a sensible default, not take the page down with it.

Circuit breaking is the pattern that prevents one slow dependency from consuming your capacity. After a number of failures, stop calling for a while, serve the fallback, and try again later. Without it, requests queue up waiting on something that is not coming back.

The subtler failure is a third party that is up but wrong — returning stale data, empty results or a changed field name. Validating what comes back, rather than trusting it, is what turns a silent data corruption into a logged error somebody can act on.

How should an API be versioned?

With a strategy chosen before you have consumers, because retrofitting one is considerably harder than starting with it.

The core distinction is between additive and breaking changes. Adding an optional field or a new endpoint breaks nothing and needs no version bump. Removing a field, renaming one, or changing a type breaks every consumer that relied on it, and needs a deliberate path.

Whichever mechanism you choose, the commitment matters more than the mechanism: how long an old version is supported, how consumers are told, and how the deprecation is communicated. An API with no deprecation policy is one that either never improves or breaks people without warning.

For internal APIs with a small number of known consumers, this can be lightweight. For anything public or partner-facing, it is a product commitment with real cost, and it should be priced as one.

What makes an integration debuggable?

Logs that record what was sent, what came back and how long it took — with a correlation identifier that lets you follow one request across every system it touched.

Without correlation, diagnosing a failure means comparing timestamps across several log files and hoping. With it, one identifier retrieves the whole path. It costs almost nothing to add at the start and is genuinely difficult to add later.

Alerting has to distinguish between a request failing and a pattern of failures. One timeout is normal. Twenty in a minute is an incident. An alert on every individual error trains everyone to ignore alerts, which is worse than having none.

And logs need to be readable without exposing what they should not. Request bodies containing personal data or credentials are a common accidental leak. What gets logged is a decision to make deliberately, at build time, rather than something to discover during an audit.

What is idempotency and why does it decide the design?

An idempotent operation produces the same result whether it runs once or five times. It matters because networks are unreliable and retries are automatic, which means some of your requests will arrive more than once whether or not you planned for it.

The classic failure is a payment. A request times out, the client retries, and the original request had actually succeeded — so the customer is charged twice. Nothing was defective; the operation simply was not safe to repeat.

The standard solution is an idempotency key: the client generates a unique identifier for the attempt, the server records it, and a repeat with the same key returns the original result rather than performing the action again. It is a small amount of work at build time and prevents a category of problem that is very unpleasant to discover in production.

Reads are naturally safe. Updates that set a value are usually safe. Anything that creates, increments or triggers an external side effect is not, and those are the operations that need this designed in from the start rather than added after an incident.

How do you plan for a third party changing?

By assuming they will, and by making the change visible quickly when it happens rather than discovering it through a reconciliation failure weeks later.

The first defense is validating responses instead of trusting them. If a field you depend on is missing or has changed type, that should raise an error immediately at the boundary, not propagate a null through your system until something else breaks in a way that is hard to trace back.

The second is isolation. Third-party calls belong behind a layer of your own, so a change affects one file rather than being scattered across the codebase. It also makes testing possible without hitting their service, which matters more than it sounds.

The third is simply paying attention. Deprecation notices arrive by email to whoever registered the integration, often someone who has since left. Making sure those reach a monitored address is unglamorous and prevents a genuine outage.