The week Dan stopped doing library work

The pull request was opened at 14:22 on a Wednesday. The body of the change was eight lines. The lockfile diff was a hundred and ninety four lines, which is normal for that lockfile. The CI ran for nine minutes and twelve seconds and finished green. Two reviewers approved within the hour. The PR was merged at 16:48. We deployed to production at 17:11 on the Wednesday because we deploy on Wednesdays.

The PR title was chore(deps): bump @acme/http-client from 4.7.2 to 4.7.3. It was the kind of PR you do not read carefully. You read it, you skim the changelog, you check that CI is green, you approve. The bot opens twenty of these a week. Mostly they are fine. This one was not.

What you would see if you looked at the diff was this:

- "@acme/http-client": "4.7.2",
+ "@acme/http-client": "4.7.3",

What you would see if you opened the upstream changelog was this:

## 4.7.3 (2026-03-04)
### Bug fixes
- Race condition in connection pool teardown during graceful shutdown
- Memory leak when AbortController aborts during DNS resolution
- Default retry policy migrated from internal defaults to opt-in
  configuration (see migration guide)

The migration guide was a separate page. The link did not appear in the package's main README. The link did appear in the release notes if you scrolled to the bottom. The reviewers had not scrolled to the bottom. Neither had the author of the PR, who was Dan, a senior engineer who had been with us for two years and who had been doing exactly the right thing.

The migration guide said this:

Versions before 4.7.3 retried failed requests three times by default with exponential backoff. This behavior caused issues for users who expected requests to be idempotent and who were unintentionally triggering retries for non-idempotent operations. As of 4.7.3, retries default to zero. To preserve the previous behavior, set retries: 3 in your client constructor.

We had not set retries: 3 in our client constructor. We had been relying on the default since 2020.

Thursday 09:14: the first sign

The first sign was a Slack message from the platform team's on-call. It said checkout p99 latency creeping. Investigating. The on-call was Ana. Ana had been on the rotation for fourteen months and she knew how to read a Datadog dashboard. She saw p99 at 1.4 seconds, up from a baseline of 380 milliseconds. She did not know yet that p99 was the wrong metric to be looking at.

The thing that was actually happening was that error rates had gone up about six tenths of a percent. That increase was almost invisible on the dashboard because the y-axis on the error rate panel was capped at five percent and the increase pushed us from 0.04% to 0.61%. The change looked like the line had wiggled at the bottom. The retried-then-succeeded requests, which had previously been hidden inside our latency window, were now showing up as failures. The latency for the failing requests was now zero (because they failed fast instead of retrying), but the affected paths surfaced the failures to users who would normally not have seen them, and those user-facing flows had additional client-side retries, and those retries were not configured for the new default, and so on. The cascade was already happening at 09:14. We did not know it yet.

Ana paged the checkout team's primary, who was Marcus. Marcus joined the war room at 09:21. He asked Ana what had changed. Ana checked the deploy log. The deploy log showed eleven deploys since the previous night, four of them to checkout's repos, none of them to checkout's core code path. Marcus said it was probably an upstream issue. They started looking at the dependencies.

Thursday 11:08: the second sign

At 11:08, the recommendation engine started dropping events. The drop was small. The drop was also silent. The recommendation engine ate the dropped events and continued processing. The user-facing impact was that some recommendations were stale. Nobody complained about stale recommendations within the first hour. Nobody complained within the second hour either. The drop was invisible to the recommendation team for the rest of the day.

The drop was happening because the recommendation engine's enrichment pipeline made an HTTP call to a feature store. The feature store call was timing out on roughly two percent of requests. Previously those timeouts had been retried and succeeded ninety eight percent of the time. Now they failed once and the events were dropped because the enrichment pipeline had been written to assume that any failure of the feature store was a transient issue that would self-heal.

A different cascade was beginning at 11:08. We did not know that one either.

Thursday 14:30: the third sign

The audit log started missing entries at 14:30. The audit log was not under any active monitoring because the audit log was supposed to be reliable. The audit log was unreliable now. We did not have an alert on audit log gaps. We had assumed we did not need one. The audit log gaps were silently accumulating.

Two days later, when the audit log team went to do their weekly compliance report, the report did not match the totals in the production database. The audit log team opened a ticket. The ticket sat in a queue overnight before being looked at. By the time it was looked at, the cascade had been running for three full days.

Thursday 16:40: Marcus calls staff

Marcus called Priya, our staff engineer on platform, at 16:40. The conversation was on the phone, not in Slack. I am paraphrasing from memory because Priya described it to me later. Marcus said checkout was still seeing elevated latency and error rates and they could not find the cause. Priya asked what had changed. Marcus said nothing they could see. Priya asked what had been deployed in the last forty eight hours across all repos that touched checkout's request path. Marcus said he did not have that information assembled and was working on it. Priya said she would help him assemble it.

The work of assembling it took Priya and Marcus three hours that evening. They had to look at:

Repo	Deploys in window	Notes
acme/checkout	4	None touch the request path
acme/cart	2	Both are styling changes
acme/billing	1	Adds a new field, gated behind feature flag
acme/orders	3	Two are bot bumps, one is a migration
acme/auth-mw	0	No deploys this week
acme/payments	5	All feature work, behind flags
acme/promo-engine	1	Internal refactor
acme/feature-store	0	No deploys

None of the deploys appeared to touch the request path in a way that should have caused the latency spike. The deploys all looked benign. The deploys were all benign. The cause was upstream of all of them, in a library bump that had happened in a different repo.

The library bump did not appear in this table because Priya and Marcus had not thought to include the shared-library repo. The shared-library repo bot opens twenty PRs a week. Nobody thinks of it as a deploy. It is a dependency update. It propagates to the services through the next build. The next build for some services is "whenever we deploy", which for some of these services was "weeks from now". For the services that had deployed since Wednesday, the bump had taken effect at the moment of deploy. The services that had not deployed since Wednesday were still on the old behavior.

Priya did not know to look at the shared-library repo. Neither did Marcus. They went home at 21:00 with no answer.

Friday 03:14: Ana pages a second time

The pager woke Ana at 03:14 on Friday morning. The same alert as Thursday. p99 had crept higher overnight. She joined the war room. There was nobody else in it. She called the platform on-call rotation. She got Yuki, who was the secondary. Yuki was groggy. Yuki was helpful.

They looked at the dashboards for an hour. They could see the latency. They could not find the cause. Yuki asked Ana whether they should escalate. Ana said she did not know what they would escalate to. There was no incident commander. There was no clear cause. There was a slow burn that nobody had named yet.

They escalated anyway. The incident commander was Reza. Reza joined the war room at 04:20. Reza had been with us for three years. Reza had seen this kind of slow burn before. Reza asked what had been deployed in the last seventy two hours. Yuki shared the table that Priya and Marcus had built the previous evening. Reza looked at the table. Reza asked whether any shared libraries had been updated.

Yuki said he did not know. Reza said to find out.

Friday 06:55: the commit

It took Yuki two hours to find out. He had to clone the shared-libraries monorepo, run a script to identify all bumps in the last seventy two hours, cross reference with the services that had deployed in the window, and look at each bump's release notes. There were nineteen bumps in the window. He read all nineteen release notes. He found the migration guide for @acme/http-client 4.7.3 at 06:55.

He sent a message to the war room: Found it. http-client 4.7.3 changed default retry from 3 to 0. We did not set retries explicitly anywhere. This affects every service that uses http-client without explicit retry config. That is most of them.

The war room went quiet for about a minute. Then Reza said OK. We are pinning to 4.7.2 in every repo that has deployed since the bump. Yuki, lead. Marcus, help. I want it done by 09:00.

Marcus and Yuki spent the next two hours opening twenty six pull requests. Each PR pinned @acme/http-client to 4.7.2 and added a comment explaining why. They merged the PRs. They redeployed every affected service. The latency normalized by 10:30. The error rate normalized by 11:00. The audit log gaps stopped accumulating by 11:20, although the gap from Thursday and Friday morning was a permanent hole in the record.

Friday afternoon: the count

We spent Friday afternoon counting what had broken.

Service	Symptom	First seen	Resolved
checkout	Latency spike from transient retries failing	Thu 09:14	Fri 10:30
orders	Idempotency-key collisions from failed checkout retries	Thu 13:40	Fri 10:35
recommend-engine	Dropped events from feature-store calls	Thu 11:08	Fri 11:00
audit-log	Silently missing entries during outbound failures	Thu 14:30	Fri 11:20
search-reindexer	Permanent stop on transient kafka producer failure	Thu 22:10	Fri 09:00
notifications	Duplicate sends from upstream retry storm	Fri 02:15	Fri 10:35
auth-mw	Token refresh failures during transient calls	Fri 04:40	Fri 10:45

Seven services. Seven different symptoms. The user-facing impact was distributed. The financial impact was modest because we caught it inside forty eight hours. The reputational impact was worse, because two enterprise customers had noticed the checkout latency on Thursday afternoon and had asked their account managers what was going on. The account managers had not had an answer. The customers had filed support tickets. The support tickets were assigned to a team that did not know about the incident. By the time the support tickets were resolved, the customers had a chip on their shoulders that lasted six weeks.

Monday: the postmortem

The postmortem was scheduled for Monday at 14:00. We invited Dan, Marcus, Ana, Yuki, Reza, Priya, and a representative from each of the seven affected service teams. The room was full. The mood was not angry. It was tired.

Reza ran the postmortem. He walked through the timeline. He showed the table. He explained the cascade. He read the migration guide aloud. He said the cause was the default behavior change in the library and the absence of explicit retry configuration in our client constructors. He said nobody in the room had failed.

Dan asked, twenty minutes in, whether he should have caught it in review. Reza said he could not have. Reza said the changelog did not surface the migration guide on the package's main page. Reza said the diff had been eight lines. Reza said CI had passed because we did not have integration tests that exercised retry behavior across service boundaries.

I watched Dan's face when Reza said this. Dan was relieved. Dan was also not convinced. Dan was the engineer whose name was on the commit, and no amount of postmortem reframing was going to remove the fact that Dan's commit was the one we had to roll back, in twenty six pull requests, across two days, with one staff engineer running point and another helping. Dan knew this. Dan said the right things in the room. Dan said he would write a runbook update. Dan did write the runbook update.

The runbook update said that all library bumps that touched network or storage code should have an integration test added before approval. The runbook update was added to the contributing guide. The contributing guide was read by the same number of people who had read the previous version of the contributing guide, which was approximately zero per quarter.

What Dan did next

Dan did not push back. Dan did not file a complaint. Dan did the thing that engineers do when they have lost confidence, which is to stop doing the kind of work that exposes them. The kind of work was shared-library maintenance. Dan had been the de facto owner of three shared libraries. He had been doing the patient, unsexy work of keeping them current, reviewing the bot PRs, occasionally writing the migration guides for our own internal libraries. After the incident he stopped reviewing bot PRs. He stopped opening migration guides for the internal libraries. He shifted his focus to the feature work he had been doing on the side, which was lower-risk, more visible, and more aligned with the next promotion cycle.

I did not blame Dan. I would not have done anything different in his place. The system had punished Dan for a thing that Dan could not have prevented, and the system would punish him again if he kept doing the same work. The rational move was to stop. Dan was rational.

Six months after the incident, the three shared libraries Dan had owned had not been bumped past their state at the time of the incident. Two of them had accumulated CVE notifications that no other engineer wanted to pick up. The platform team had quietly added "shared library maintenance" to their roadmap for next quarter. The platform team's roadmap had eighteen items on it. Shared library maintenance was item fifteen. It was not going to happen next quarter.

The cost of the incident was not the seven services or the two days of war room or the eight hours of postmortem prep. The cost was Dan's continued absence from the work he had been doing, multiplied by the four other engineers who had been watching Dan and who had also quietly stopped picking up the shared-library work. The total cost was the rate at which our shared libraries were now drifting toward unmaintainability, which was much higher than it had been before the incident, which we would not feel until we hit a CVE we could not patch around or a forklift migration we could not afford.

The thing that would have stopped this

The thing that would have stopped this is a system that, when Dan's PR was opened, could have looked at the diff and said something like:

This bump changes the default retry behavior in @acme/http-client from 3 to 0. The library is currently used at 47 distinct call sites across 11 repositories. Of those, 4 explicitly set retries in their constructor (safe). 43 rely on the default (affected).

Of the 43 affected call sites, 18 are in user-facing request paths with downstream cascade potential. 7 of those 18 are in services that have surfaced transient errors in production logs in the last 30 days.

Recommended action: set retries: 3 explicitly at the 43 affected call sites before merging. Alternatively, hold this PR pending coordinated rollout.

The comment would have been the first thing Dan read on his own PR. The reviewers would have read it too. The decision about whether and how to land the bump would have been informed by the comment. Nobody would have approved a coordinated rollout in the next half hour. Dan would have known what to do. The merge would have either been delayed until the call sites were fixed, or fixed in the same PR.

The comment is not magic. The information in it is information our codebase already has. The 47 call sites are findable with grep. The 4 explicit configurations are findable by parsing the constructor calls. The 18 user-facing request paths are findable by following our request-id flow through the system. The 7 services with transient error history are findable in our error tracker. All of this information exists, distributed across grep, AST tools, request tracing, and error tracking. None of it is assembled in a way that a human reviewer can see at the moment they need it.

The reason it is not assembled is that the work of assembling it is not anybody's job. The work was on the platform team's roadmap for last quarter. It is on the platform team's roadmap for this quarter. It will be on the platform team's roadmap for next quarter. There will always be a more urgent item than "assemble the cross-cutting view of our dependencies and behaviors so engineers can ship library bumps with confidence". The more urgent items will produce more visible wins. The cross-cutting view will continue to not be built.

What I want for the next Dan

I want the next library bump to come with the comment. I want the comment to be the first thing the author reads. I want the comment to make the review take twenty minutes instead of two minutes, because the comment will reveal things that need to be talked about. I want the merge to happen when the things have been talked about and the explicit configurations have been added.

I want the cascade to not happen. I want the cascade to be invisible, in the sense that the cascade never starts. I want Dan to keep being the engineer who picks up the bot PRs, because the system will help him do that work well. I want Dan to know, in his bones, that the next bump is safe to land, because the system told him exactly which call sites would be affected and he handled all of them.

This is what we built blastmap to do. We did not build it because Dan's incident was rare. We built it because Dan's incident is what is happening, in some shape, right now, in your codebase, in a dependency you have not looked at in a month. The cost compounds quietly. The compound is the velocity you do not have, the libraries you do not bump, the chilling effect you do not measure but feel anyway. The way out is the comment on the PR. The comment is built from the system the model is built from. We are building it. Slowly enough that we get it right. Fast enough that the next Dan does not give up the work.