Tech Lead - Disaster Recovery and Incident Management

When a database fails at 3am, or a SaaS provider we depend on goes dark, somebody has to already know what happens next. That's not a document sitting in a drive nobody's opened since the last audit. It's a tested, current, engineering-grade plan — and a person who can run the room when it matters.

We're not hiring someone to keep a DR binder up to date. We're hiring someone to own reliability as an engineering discipline: designing the failure tests, running the incidents, and turning every near-miss into a system that's harder to break next time.

Why This Matters

Trading for Anyone, Anywhere, Anytime means our platform doesn't get to have a bad day. Millions of traders, real money, live markets, around the clock — an outage isn't an inconvenience, it's a trader who couldn't close a position. Regional outages, database failures, and SaaS disruptions aren't hypotheticals; they're a matter of when, not if.

Most companies treat disaster recovery as a compliance exercise: a runbook written once, tested rarely, and trusted blindly. We're building the opposite — DR and incident management as connected disciplines that make the platform measurably more resilient, with real failover tests, real RTO/RPO targets tied to business risk, and AI tooling that helps us model failures and correlate signals faster than a human staring at five dashboards at once.

Why Deriv

• We run live DR failover tests, not tabletop exercises that stay theoretical
• Our incident command spans engineering, infrastructure, SRE, Compliance, and Risk — in real time, not in a post-mortem doc three weeks later
• We're building toward AI-assisted failure modelling, incident signal correlation, and automated runbooks — not replacing judgment, but giving you better information faster when every minute matters

We share what we learn. Deriv<ed> is where we write about what we're building, what breaks, and what we figure out the hard way.

What You’ll Do

Own disaster recovery as an engineering discipline

• Design, schedule, and run DR exercises for business-critical applications — database failures, regional outages, SaaS provider disruptions, live failover tests
• Challenge business impact analyses directly with system and SaaS owners until RTO/RPO targets reflect actual business risk, not last year's assumptions
• Keep DR strategies, runbooks, and readiness dashboards matched to what's actually running in production — not what the architecture diagram from two years ago says

Command incidents when it counts

• Lead critical and major incidents from declaration to resolution, coordinating engineering, infrastructure, SRE, Compliance, and Risk in real time
• Make the fast calls under pressure: assess blast radius, set priority, pull in the right responders, and be the single source of truth while everyone else is asking "what's happening"
• Brief the right level of detail to the right audience — on-call engineers need different information than the C-suite, and giving everyone the same update either drowns engineers in noise or leaves leadership guessing

Turn incidents into architecture, not just tickets

• Lead post-incident reviews that find the systemic weakness, not just the symptom
• Hold teams accountable for closing findings from DR exercises, incidents, and audits — a finding that never closes is just a risk you've agreed to keep taking
• Challenge new systems before they reach production if they don't meet recovery requirements

Use AI as a working tool, not a buzzword

• Model failure scenarios and simulate downstream impact before they happen for real
• Correlate incident signals and surface patterns across past incidents instead of relying on institutional memory
• Automate runbooks and improve DR/incident reporting so the next 3am page has better information waiting for it

Who You Are

You've run live incidents, not just read about them

• 7 - 12 years across disaster recovery, business continuity, incident management, infrastructure resilience, SRE, or similar. You've commanded a live P1 or P2 and know the difference between calm authority and just staying quiet.

You know cloud infrastructure well enough to ask the right question fast

• Practical knowledge of AWS, GCP, Azure, or similar — compute, storage, databases, networking, and where each of them actually breaks. You've run live DR failover tests, not just written the plan for one.

You think in frameworks but talk in outcomes

• Practitioner-level understanding of ITIL service management, business continuity, problem management, and service design — but you translate that into decisions people can act on, not slideware.

You communicate clearly when everyone else is panicking

• Strong written and verbal communication under pressure. You can brief a war room and an executive in the same hour without either one leaving confused.

You treat resilience posture as a fact, not a feeling

• You give leadership honest, data-backed visibility into where recovery capability actually stands against business risk — including when the answer isn't good.

You're fluent with AI tools, not just aware of them

• Comfortable using AI for scenario modelling, incident triage support, pattern detection, and reporting — you know it accelerates the work, not that it replaces the judgment call.

You close the loop

• You hold teams accountable for fixing what DR exercises, post-incident reviews, and audits find — including when it's not your team's mess to clean up.

Bonus Points

• Direct experience interfacing with Compliance and Risk on regulatory classifications, audit readiness, and evidence management
• Experience building real-time DR readiness dashboards or leadership reporting on resilience posture
• Experience moving continuity processes from manual checklists toward automated or self-healing infrastructure

The Honest Reality

This is demanding work. You'll be the person paged when a database fails and the person expected to have already tested for it. You'll challenge system owners on assumptions they've held for years, and sometimes they won't like it. Post-incident reviews will surface things that make people uncomfortable, including decisions you made in the moment with incomplete information.

But you'll build the muscle that keeps a platform serving millions of traders standing when things go wrong — and you'll have the authority, in the moment that matters, to make the calls that matter.

If you want DR to be a document nobody reads, this isn't it. If you want to own reliability as an engineering discipline, it might be.