
Lead Software Engineer - Production Support & SRE
Alumni · LONDON, United Kingdom
Strong hiring record · median 8 days open
- First seen
- 16 November 2026
- Reposted
- Not since September 2026
All reasons
In its favour
- Posted today
- Still listed on the employer's own board
The CIB Rates Production Management Technology team comprises technical leaders and self-motivated engineers who are passionate about ensuring the stability, resilience, and continuous improvement of mission-critical front-office trading applications. The team operates as an engineering-led Production Management function - one that writes code, builds platforms, and applies AI to detect, diagnose, and remediate issues before they impact the business.
As a Lead Software Engineer within Rates Production Management, you will combine end-to-end production ownership of the Rates application estate with hands-on software engineering. You will design and build automation, observability, and AI-driven tooling that delivers early detection of batch, risk, and market data issues, accelerates root cause analysis, and enables safe, automated remediation. You will serve as a trusted technical authority during major incidents, production readiness reviews, and resiliency exercises, while championing Site Reliability Engineering (SRE) principles and the responsible adoption of AI across the team.
Key Responsibilities
- Serve as the technical production owner for critical Rates front-office services - batch/EOD processing, real-time pricing, risk computations, trade management, and market data pipelines
- Lead technical response and service restoration for P1/P2 incidents, coordinating across Application Development, Infrastructure, and business stakeholders to accelerate recovery
- Lead post-incident reviews (RCAs/post-mortems), identify recurring failure patterns, and partner with development teams on root cause remediation to eliminate recurrence and reduce technical debt
- Maintain strong production governance — access controls, operational procedures, runbooks, documentation, audit readiness, MEPC execution, and operational review processes
- Design, build, and maintain production-grade software (primarily Python) for operational tooling, self-healing workflows, and automated runbooks that reduce manual toil and MTTR
- Contribute code changes directly to Rates applications to fix production defects, improve performance, and harden resiliency, following the full SDLC - design, automated testing, peer review, and controlled release
- Build reusable engineering components, APIs, and integrations (job schedulers, ticketing, messaging, data stores) that scale improvements across the CIB application portfolio
- Design and deliver AI/ML-based early-detection capabilities - anomaly detection on batch timings, job volumes, risk outputs, and market data, with predictive alerting on SLA breaches before business impact
- Define and improve SLIs, SLOs, error budgets, and reliability metrics for the Rates application suite, reporting production health to senior technology and business leadership
- Design and deploy monitoring, telemetry, dashboards, and alerting (Geneos, OpenTelemetry, Grafana, or equivalent)
- Drive capacity planning, performance profiling, resiliency testing, and BCP/failover readiness across multi-region deployments
- Represent Production Management in architecture and production readiness reviews for new features, platform changes, and cloud migration
- Build and maintain strong relationships with Application Development, DevOps, Front Office business users, and GTI/Infrastructure teams
- Communicate effectively with all levels of management - from technical deep-dives to executive-level summaries of production health and automation impact
Technical Expertise (Required)
- Experience: 7+ years' experience in front-office production support, SRE, or production engineering within Investment Banking, preferably supporting Fixed Income Rates trading applications (pricing, risk, trade management, EOD/batch processing)
- Software Engineering: Formal training or certification in software engineering concepts; strong Python skills at production-code level - object-oriented design, testing, debugging complex distributed systems, and building automation tooling within a full SDLC with code review and CI/CD
- AI & Automation: Hands-on experience applying AI/ML or LLM-based tooling to operational problems (anomaly detection, triage, RCA, automated remediation, reporting) and leading effective use of AI-assisted development tools, with a strong understanding of responsible AI - data sensitivity, guardrails, and validation of outputs
- Monitoring & Observability: Extensive experience with Geneos and/or Grafana, Splunk, Dynatrace, OpenTelemetry, log aggregation, and alerting frameworks
- Databases: Strong SQL (Oracle, Postgres) and query performance tuning in high-throughput environments
- Platform Knowledge: Linux/UNIX (RHEL), shell scripting (bash/ksh), job scheduling systems, and batch orchestration
- Incident Management: Proven experience leading complex P1 incidents, managing war rooms, problem management, and producing actionable RCAs that eliminate recurrence
Technical Expertise (Preferred)
- Knowledge of Rates products (Swaps, Options, Inflation, Bonds/Futures, Repo/Securities Lending) and the front-to-back trade lifecycle
- Experience building agentic AI workflows, prompt engineering, or integrating LLMs with internal tooling (e.g. MCP servers, knowledge graphs, retrieval-augmented generation)
- Hands-on experience with public cloud (AWS), infrastructure-as-code (Terraform, CloudFormation), and containerisation (OpenShift, Kubernetes, Docker)
- Experience with time-series/statistical analysis and data libraries for building data-driven detection and dashboards
- Experience with capacity management, performance profiling, and memory analysis in Python/Java environments
General Competencies
- Proven ability to lead under pressure - managing multiple critical incidents simultaneously while maintaining clear communication
- Engineering mindset - automates repeatable work, measures outcomes, and treats operational problems as software problems
- Strategic thinker with the ability to balance immediate production needs against long-term platform and automation improvements
- Strong ownership mentality with a track record of driving initiatives from conception to measurable outcomes (e.g. reduced MTTR, fewer incidents, improved SLA adherence)
- Excellent communication skills (verbal and written) - able to translate complex technical issues into business impact language
- Curiosity and continuous learning, particularly in emerging AI technologies, applied with sound judgement on risk and control
Skills
- Python
- OpenTelemetry
- Grafana
- LLM
- Splunk
- Dynatrace
- SQL
- Oracle Database
- PostgreSQL
- Linux
- Unix
- RHEL
- Shell
- Bash
- Prompt Engineering
- Model Context Protocol
- Retrieval-Augmented Generation
- AWS
- Terraform
- AWS CloudFormation
- OpenShift
- Kubernetes
- Docker
- Java
More jobs at Alumni
All 103- Software Engineer III- ( Java Fullstack, AWS)Alumni · Hyderabad, Telangana, Indiatoday
- Senior Thunderhead/SmartDX DeveloperAlumni · Bengaluru, Karnataka, Indiatoday
- Software Engineer III- React, Micro frontend, AWS and JavaAlumni · Hyderabad, Telangana, Indiatoday
- Automation Lab AnalystAlumni · Warszawa, Mazowieckie, Polandtoday
- SRE III- NetworkAlumni · Bengaluru, Karnataka, Indiatoday
Similar roles
- Lead Software Engineer - Production Support & SREJPMC · LONDON, United Kingdomtoday
- Lead Software Engineer - Production Support & SREChase- · LONDON, United Kingdomtoday
- Senior Software EngineerVerisk Careers | Verisk · Manchester, Greater Manchester, United Kingdomtoday
- Senior Fullstack EngineerNebius · Amsterdam, Netherlands · Germany · Israel +3today
- Senior/Staff Platform Engineer - Rust (d/f/m)Jobgether · UK · EUR 82,200–93,000/yrtoday
- Senior ML EngineerJobgether · UKtoday