On PagerDuty?

PagerDuty rotation or routing missing the right person? We sort it out.

PagerDuty gets blamed for a lot of 3am phone calls and even more silent nights when someone should have been woken and nobody was. The usual cause is a rotation override from eight months ago, a routing key pointing at the wrong service, or an escalation policy with a blank layer. We find it and close it.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

pagerduty.log

still broken

What it is

PagerDuty at a glance

PagerDuty is an incident response platform founded in 2009 and taken public in 2019. It receives events from monitoring tools, decides who should be paged based on service routing and on call schedules, and runs the escalation ladder when nobody acknowledges.

The platform now covers alerting, incident command, runbook automation, customer service operations and AIOps. For most teams the heart of it is still the services, event rules, schedules and escalation policies that decide who gets woken up and in what order.

Type
Incident response and on call platform
First released
2009
License
Proprietary SaaS, per user pricing with tiered plans
Maintained by
PagerDuty, Inc.
Typically used for
On call scheduling, incident response, alert routing, escalation management, status communication

The honest version

PagerDuty pros and cons

No technology is all upside. This is where PagerDuty earns its place and where it tends to cause trouble.

Where PagerDuty is strong

  • The incumbent in on call toolingMature escalation policies, schedules with layers, integrations with most monitoring tools, and well understood operational conventions mean new hires usually know how it works already.
  • Integrations with everythingDatadog, New Relic, Sentry, AWS CloudWatch, Prometheus, GitHub, Slack and hundreds more land events natively, so wiring up a new source is a routing key away.
  • Schedules handle real on call patternsLayered schedules, overrides, follow the sun rotations and restrictions cover the complicated cases that smaller tools get wrong six months in.
  • Runbook automation included on higher plansRundeck, acquired by PagerDuty, turns the first few troubleshooting steps into automation triggered from the incident itself.
  • Good mobile appsThe iOS and Android apps actually wake people up, and the acknowledge and escalate actions work reliably even on flaky networks.
  • Analytics on response timeMTTA, MTTR, who gets paged most, and which services account for the most incidents are all reported out of the box rather than needing a separate dashboard tool.

Where PagerDuty bites

  • Per user pricing adds up fastCharging per on call user per month makes a mid sized engineering org pay more for alerting than for its monitoring stack, and the pricing has crept upward every renewal.
  • Rotation overrides lingerA one time swap set by a manager months ago keeps paging the wrong person, and nobody notices until an incident goes to voicemail.
  • Event rule editor is complexThe rules engine can suppress, deduplicate and route, but nested conditions with regex and token extraction are fiddly to reason about and rarely well documented inside the org.
  • Routing keys are easy to copy wrongA paste that lands against the wrong service produces events in a graveyard service that nobody is oncall for, and the dashboard shows green.
  • SMS and voice delivery can lagCarrier filtering, international routing and quiet hours settings occasionally delay a page, which matters more than any other category of bug this product has.
  • Service and team modelling needs ownershipServices, teams and escalation policies rot when teams reorganise, and keeping the model in sync with the actual org is a job nobody is officially assigned.

Sound familiar?

Common PagerDuty problems we fix

The PagerDuty problems that bring people to us, and what you have once each one is fixed.

override active for 14 more days

A long override is paging the wrong person

A short term swap was set up with no end date, or a cover arrangement after a role change was never cleaned up. Everyone forgets about it until a weekend incident wakes the wrong engineer.

Overrides audited and cleaned

routing key not found

Events land in a dead service

A routing key paste went to the wrong service, or the source integration references a service that was deleted. Events look sent, nothing is paged.

Routing keys mapped to real services

oncall is empty between 02:00 and 06:00

Schedule has a hole overnight

A layer only covers part of the week, or a handoff time crosses into a time range nobody is assigned. Nobody is woken because, strictly, nobody is on call.

Schedule covers every hour

escalation policy has empty layer

Escalation ladder leads nowhere

Layer two is blank, so when the primary does not acknowledge the incident escalates into nothing. Usually the result of a user leaving the company and being removed without being replaced.

Every layer has a real target

integration not received event in 7 days

An integration has gone quiet

A monitoring tool stopped forwarding events because its API key was rotated or its service was decommissioned, and nobody noticed because the absence of pages looks exactly like success.

Heartbeat monitors on each source

handoff crosses daylight saving

Daylight saving breaks the handoff

A handoff time set to a local wall clock drifts across daylight saving boundaries, so the schedule either doubles up or leaves a gap on the switching weekend.

Handoffs anchored to UTC

webhook signature verification failed

Downstream automation stops firing

A webhook signing secret rotated on PagerDuty but not on the receiver, so every incident webhook is rejected and the Slack room goes quiet.

Webhooks accepted again

incident auto-resolved after 24h

Incidents auto-resolve without acknowledgement

A service that never acknowledges pages has the auto resolve timeout doing the job, which turns real incidents into closed tickets nobody investigated.

Auto resolve set honestly or disabled

What we do

PagerDuty help, from rescue to on call strategy

Call us for one failing rotation, or let us set up the whole on call programme.

Rescue
Rotations paging the wrong person, routing keys missing events, or escalation policies that lead nowhere. We trace it and close it the same day where we can.
Rotation and schedule audit
Overrides reviewed, handoffs anchored, coverage verified against the real roster, and schedules documented so the next manager does not inherit a mystery.
Service and routing design
Services, teams, routing keys and integrations modelled against how your engineering org actually works, with runbooks attached where they help.
Event rule tuning
Deduplication, suppression and auto resolution tuned down so pages reflect real work rather than noisy dashboards, with analytics to prove it.
Integrations and automation
Datadog, Sentry, Prometheus and custom sources wired up, plus Rundeck or external runbooks launched from the incident itself.
Managed Tech
Quarterly on call reviews, override cleanup, response time analytics, and a backup pair of eyes when a late night page turns into a real incident.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running PagerDuty alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

PagerDuty questions

Something else? Call+1 737 727 7327

Why did our incident escalate past everyone without being acknowledged?

Usually one of three things: an empty escalation layer after a leaver was removed, a rotation override that was still active months after the cover ended, or an SMS and voice delivery delay on a particular carrier. We pull the audit log for the incident, walk the schedule and escalation policy, and find the specific point where the chain broke.

How do we stop old overrides from lingering?

PagerDuty supports time bound overrides with an explicit end date, but nothing stops a manager from setting an open ended one. We review every active override, enforce end dates going forward, and set up a monthly audit so cover arrangements do not outlive the reason they were created.

Why do events show as delivered but nobody is paged?

The routing key is landing on a service that has no escalation policy, or the service has one that resolves immediately, or deduplication is suppressing events that look similar. The event rule console shows the full pipeline for each event; we walk it from source to acknowledgement to find the step that silently dropped it.

Should we move off PagerDuty to something cheaper?

Sometimes. Opsgenie, incident.io and Grafana IRM all do the core job well, and the pricing difference matters once your on call roster grows. The real cost of moving is not the tool but the integrations and the tribal knowledge in the schedules. We model the migration and tell you whether the savings justify the switch.

Can you help us set up on call for the first time?

Yes. We work with the team to choose the right rotation cadence, write the first escalation policies, wire up the monitoring sources, and run a tabletop exercise before anyone is woken up for real. Starting small and growing the model is less painful than inheriting one later.

How much does PagerDuty help cost?

Estimates are no charge. Describe the rotations, the services and what is going wrong, and we come back with a price and a plan before changing anything.

Free estimate

PagerDuty missing the right people?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.