Speak to us
A Disastrous Documentation Change

API definitions|Business|DDD|Domain Driven Development|

A Disastrous Documentation Change

Disclaimer

The following story relates to real facts but has been heavily redacted and modified to protect all involved NDA-covered parties, as well as to simplify the story for the end reader.

The story of a storefront and its API

This story revolves around 3 main parties:

  • the end customer
  • a software company providing a shop frontend
  • a software company that provided the API backing the shop

Roave collaborated closely with the software company by providing the API backend: We helped bring the product to its “1.0” state, with considerable effort going into building an API on top of a legacy application substrate.

We won’t go into the details of the software architecture: much could have been improved, but that is beside the purpose of this article.

Once 1.0 was reached, the customer added heavy performance requirements on the product, which required all involved parties to scramble and improve.

The decision was taken to rewrite the backing API with a completely new software stack, involving different technologies, new teams, new deployments, but retaining the same API definitions as the 1.0 stack.

API 2.0

While writing a new software stack, the team switched from PHP + Symfony to Go, which resulted in many accidental API changes.

The API that was affected with the change:

definitions:
  response.Product:
    required:
      - id
      # other fields
      - price
    properties:
      # ... plenty of fields here
      price:
        type: integer
      best_effort_price:
        type: integer

This API stayed exactly the same, but due to how swaggo and the Go programming language handle structures, the actual generated API documentation changed as follows:

definitions:
  response.Product:
-    required:
-     - id
-     # other fields
-     - price
    properties:
      # ... plenty of fields here
      price:
        type: integer
      best_effort_price:
        type: integer

The actual “why this happens” is part of how Go handles default values for struct {} declarations, and is also not relevant for this tale.

With this API change, consumers lost clarity on which fields are guaranteed to be there or not.

Where did it go wrong?

The API team kept the behavior of the endpoints exactly the same as on the 1.0 implementation.

The shop frontend team started relying on the API 2.0 documentation, and confused the “lack of required declarations” with “all fields are there at all times”.

If you look closely at the API definitions, best effort_price is not a required response field, and may be absent.

The shop frontend team implemented logic that relied on best_effort_price for critical store interactions.

Things kept running, and then…

To provide more context in understanding the problem, best_effort_price is a computed field that requires background processing: for various months, that background processing has been operating smooth, fast, providing almost near-real-time data for the API.

Due to some issues with background processing, best_effort_price momentarily stopped “working” (remember it was never required) after a few months of smooth sailing.

As a result of the field not being available, the storefront-side experienced a partial outage.

The end customer escalated the problem immediately, ringing all alarm bells for all teams involved.

The issue got solved?

The API team fixed the temporary problem by recovering the workers that were responsible for computing best_effort_price, and explained how the API got mis-interpreted.

To prevent the issue from occurring again, we planned follow-up tasks to:

  • the API documentation could be adjusted to highlight the optionality of the field
  • the API could be re-designed to always provide best_effort_price

And then it failed again…

A few months passed, and the same exact failure with best_effort_price resurfaced.

The situation caused the same kind of outage, and the customer ended up blaming the API provider with the outage:

  • customer trust was eroded
  • the API provider company paid huge fines due to SLA agreements not being respected

Eventually, the situation recovered, but the economical and political damage was done.

Turns out that:

  • the API provider didn’t work on any of the raised issues, due to other business priorities
  • the shop frontend provider didn’t react to given feedback, possibly for similar reasons.

Eventually, the API team decided to re-implement the missing field so that it can never go missing again, even at the cost of performance and data imprecision.

Learnings

A lot of problems became more evident as the situation I described unfolded: many of these will be obvious to more experienced developers, but

Documentation vs real-world

The biggest learning of all here is that an API is not what’s documented, but rather the observed system behavior: it does not matter if a system is statically or dynamically typed, what matters is how consumers use it, and the assumptions they build up on top of it.

Assumptions need to be fed back in the business domain, and loose ends require adjusting continuously.

Software project rewrites

Despite each team doing their best to migrate from 1.0 to 2.0, things will go wrong somewhere. A simple non-functional requirement like the API documentation has led to major misunderstandings.

Software rewrites will, regardless of anyone’s effort, always lead to some regression. There must be understanding by all involved parties that switching technologies comes with substantial risk.

In our case, the promise of Go solving all our issues led to a lot of API-related frustration for many reasons besides what was outlined in this article.

Layers of management

The software bug we experienced was a misunderstanding. Add “our people talk to your people” to the mix, and you get a Chinese whispers chain that amplifies frustration.

The politics of this project involved software teams that weren’t able to communicate directly with each other. A frank discussion would’ve saved everyone a ton of time and might have sparked new ideas on how to fix the situation more intelligently.

Software developers are smart people: don’t gate them behind layers of management.

Because of the amount of management involved, the priority of the bugfix got diluted, and everyone was impacted in the end due to inaction.

What to do next?

The main thing that we will change on our end is how to approach problems reported by customers. Instead of expecting action on the consumer side, a solution that comes from within the software components that are in our control is to be favored.

If the problem can be solved within one software team, deferring responsibility to a consuming team is probably a less preferable choice.

Even if the change to be applied “feels wrong”. The ultimate priority is smooth operation of the system at large.

Back to all