Undisclosed Client: Improving On-Call - Stagebuilder Consulting

1% Better

Systemically improving on-call to rebuild customer trust.

Reducing support resolution time from days to hours. Delivered without adding new headcount.

At a glance

The impact of on-call becoming just another week.

	BEFORE		AFTER
Time to Resolve	3.5 hours	→	90 minutes
Pages per shift	45	→	15
Recurring incident classes	23 / quarter	→	0
On-Call Sentiment	Dread	→	Just another week

The Mechanism

A repeatable operating pattern. Diagnose, prioritize, execute.

Diagnose

On-call was constant firefighting.

A reactive rotation with no internal signal for product quality.

Customer bug reports were the monitoring system.

Blueprint

Designed both technical and people-centric systems to improve response.

Bespoke monitoring systems to understand product health.

A handoff discussion focused on systemic improvement.

Build

Delivered a health dashboard with product leaders to understand actual customer experience before they notified us.

Deployed agentic tooling to safely process server logs and reduce debugging time.

Ship

Mean time to resolve fell from 3.5 hours to 90 minutes.

Core recurring incident classes dropped to zero.

Encore

Product health dashboard informs business OKRs.

Tech debt became technical investments that gets real roadmap time.

The Story

Adding more people to on-call won’t fix the systemic problem.

Problem

Customers were the monitoring system.

On-call was constant firefighting. A reactive rotation that never got ahead of the next support issue. Product quality degraded in places nobody could see, so customers reported problems before any dashboard did. Understanding an incident meant reading logs across five separate systems. Hours passed before anyone could reproduce the bug. Nothing carried forward, so the same failures kept returning.

Every support ticket pulled an engineer in. Churn climbed each quarter while tech debt lost every prioritization fight, and the rotation was something engineers dreaded.

Change

Fix the system behind the symptoms.

The expected answer was more headcount or a full rewrite that stopped all feature work. Neither choice was acceptable. The reliability work ran alongside the roadmap, so no feature slipped.

We designed two systems, one technical and one about people. A product-quality health score, built with the product owners and architects who knew which workflows customers depend on, surfaced degradation on a dashboard before a customer felt it.

Every rotation left on-call a little better. AI joined triage to summarize each incident and point at a likely cause. Structured handoffs carried context between shifts, and we gave support the tools to address common user mistakes.

RESULT

57% faster to resolve, and a rotation nobody dreads.

57% less time to resolve. MTTR fell from 3.5 hours to 90 minutes.
Recurring incident classes reduced to zero. Proper technical investments in the roadmap removed 23 recurring incidents.
Pages per shift dropped from 45 to 15. Consistent removal of noisy signals reduce context switching.
No rewrite, no new headcount. The roadmap never paused.
The score now shapes the business. Leaders use it for OKRs, and tech debt earns roadmap room as an investment in quality.

The Details

For the readers who want the mechanics. Open what you need.

Product quality drove the score. Built with product owners and engineering leads, the health score rated the workflows customers actually depend on. This data allowed the teams to setup custom monitors to discover a problem before a customer felt it. Over weeks a dashboard became a record of product health that leaders used to shape OKRs and performance metrics.

AI shortened the path to resolution. With read-only access to production logs, it summarized each incident and pointed at a likely cause, so the engineer’s first hour went to deciding what to do. An expert can tell at a glance whether the model is reasoning down the right path, which keeps a human firmly in control of the call.

One improvement per rotation made it compound. Every shift retired a noisy page, closed a live Sentry issue, downgraded a flaky alert, or left a run book behind. Small changes stacked into a quieter rotation week over week.

FEATURE WORK KEEPS MOVING

The reliability work ran alongside delivery, so the roadmap never paused.

No rewrite was needed.

No extra headcount was added.

CUSTOMER TRUST SETS THE PRIORITY

Whatever was costing trust by the minute got attention first.

Each bug was a betrayal of customer trust and risk of churn.

STOPPED

Never communicating about ongoing issues when changing shifts.

Using customers as the primary monitoring system.

Prioritizing feature work over product quality.

The product health score is still in use today. It’s used to inform business-level OKRs and determine how to prioritize technical investments alongside feature work.

The AI tooling has expanded to proactively address low health scores and give a resource to customer support for more technical customer issues.

Still in production as of 2025.

IF THIS SOUNDS LIKE YOUR TEAM

Does every support issue feel like you’re fighting a fire?

If so, the rotation is missing a system that improves both the on-call process and the software underneath it. More people and a rewrite feel like the answer, but at the cost of your budget and roadmap.

Bring us the 3 a.m. pages. We’ll make them rare.

Let’s Talk →