Big Data Principal

Senior / Lead Data Engineer

Engineering –Enterprise Data Platform

Level

Lead/Principal Data Engineer / Technical Lead– hands-on

Experience

12+ years in data engineering (Senior); 5+ years including team leadership (Lead)

Reports to

Enterprise Data Architect

Domain

Financial Services / Banking (strongly preferred)

Location

Hybrid – to be confirmed

Engagement

Permanent / Long-term Contract

About the Role

We are looking for a Senior / Lead Data Engineer to join the engineering organisation a multi-year banking transformation programme delivering capabilities across Accounts (Real-Time Ledger), Payments, Foreign Exchange and Client Lifecycle Management, underpinned by an Enterprise Data Platform built on Snowflake.

This is a hands-on build role. Working closely with the Enterprise Data Architect, you will turn the target-state data architecture into a production platform: standing up the platform foundations, engineering data acquisition and transformation, and implementing canonical data models and data products. You will act as the engineering counterpart to the Data Architect – implementing the design, feeding back on buildability, and setting the engineering standards others follow. At Lead level, you will also shape and lead the data engineering team.

Key Responsibilities

Platform foundations

• Build Snowflake environment structures – layered databases and schemas (Raw / Conformed / Consumption), environment separation (development / test / production) and naming conventions
• Implement role-based access control, service accounts, virtual warehouse design, resource monitors and cost guardrails
• Manage the platform as code – Snowflake objects, roles, grants and warehouses through infrastructure-as-code and version control
• Establish Git workflows and CI/CD pipelines for data code, with automated build, test and deployment across environments

Data acquisition and CDC

• Assess source systems and recommend fit-for-purpose acquisition patterns – full vs incremental, batch vs micro-batch vs event, and file vs API vs direct database vs message hub
• Advise on where change data capture should be applied and owned – log-based CDC at source, application-emitted change events, or warehouse-side change detection – and implement the chosen patterns
• Engineer CDC and replication from relational, document and time-series application databases, handling initial loads, deletes, schema evolution, lag and backfills
• Ingest event streams from messaging platforms into Snowflake, covering ordering, idempotency, deduplication, replay and error handling
• Configure and optimise managed ELT connectors (e.g. Fivetran), and build file- and API-based ingestion where required
• Design and build a metadata-driven ingestion framework – sources, objects, load types (full, incremental, CDC), schedules and target mappings held as configuration, with control tables for watermarks, run status, audit and reprocessing – so new feeds are onboarded through metadata rather than new code
• Make the framework reusable across ingestion channels (managed ELT, file, API, database CDC and event feeds), with generic, parameterised loading, schema-drift handling and standard error handling
• Prove patterns through technical spikes, and record decisions as Architecture Decision Records

Transformation and modelling

• Design and build dbt projects from the ground up – structure, layering, conventions, macros, packages, tests and documentation
• Run dbt both natively in Snowflake (dbt Projects on Snowflake) and on dbt Cloud, building portable, convention-led projects that move between the two with minimal rework
• Implement canonical data models – including industry-aligned models such as BIAN-based business objects – from source-to-canonical mapping specifications and governed reference data
• Build dimensional and domain-oriented data products for reporting, regulatory, analytics and AI consumers
• Engineer identity resolution, key management and cross-source reconciliation, and feed implementation findings back into the models and mappings

Quality, governance and operations

• Embed data quality as code – tests, freshness checks, reconciliations and data contract enforcement
• Implement Snowflake governance controls – tagging and classification, dynamic data masking, row access policies and PII handling aligned to GDPR – with lineage that supports BCBS 239
• Build monitoring and alerting for pipeline failures, freshness, data quality and cost, and define run-books and support processes

Orchestration and scheduling

• Design and implement orchestration and scheduling natively in Snowflake using Tasks and task graphs (DAGs) – dependencies, root and finaliser tasks, serverless vs warehouse-backed compute, schedules and event-triggered runs
• Orchestrate ingestion, dbt execution (e.g. EXECUTE DBT PROJECT), data quality checks and downstream refreshes end to end within Snowflake, with retries, failure handling, alerting and notifications
• Use Dynamic Tables where declarative, target-lag-driven pipelines are a better fit than task-based scheduling, choosing between incremental and full refresh modes and managing their cost
• Build operational visibility over task and Dynamic Table runs – run history, latency, failures and credit consumption – and keep orchestration designs portable to external orchestrators where needed

AI-assisted engineering

• Use Snowflake CoCo (formerly Cortex Code) to accelerate development, with generated code reviewed, tested and deployed through standard CI/CD
• Apply Snowflake Cortex AI capabilities – AISQL functions, Cortex Search, semantic views, Cortex Analyst and Cortex Agents – where they add value, and design data products to be AI-ready

Leadership and collaboration

• Work as the engineering counterpart to the Enterprise Data Architect, contributing to design decisions and ADRs
• Collaborate with Integration, Solution and Application teams on source extracts, event contracts and data contracts
• Define and own engineering standards, templates and reusable patterns
• Lead level: lead, mentor and help recruit data engineers, own code review, and plan and report delivery

Primary Skills (Essential)

Candidates are expected to have strong, hands-on depth in all of the following:

Skill area

What we expect

SQL

Advanced, production-grade SQL – complex joins, window functions, set-based transformation, MERGE / upsert patterns, query profiling and performance tuning

Data Warehousing

Solid grounding in warehouse principles – layered / medallion architectures, ELT, incremental processing, slowly changing dimensions, history and audit, reconciliation, partitioning and performance

Data Modelling

Dimensional modelling (Kimball – facts, dimensions, conformed dimensions); implementing canonical / enterprise data models; normalised vs denormalised trade-offs; SCD Types 1/2; working from logical models and mapping specifications

Snowflake

Hands-on platform engineering – schema and layer design, RBAC, virtual warehouses, Snowpipe, Streams & Tasks, zero-copy cloning, Time Travel, masking and row access policies, resource monitors, cost optimisation

dbt

Building dbt projects from scratch – layering, incremental models, snapshots, macros, packages, tests, docs; dbt Projects on Snowflake and dbt Cloud

Data Acquisition & CDC

Designing full, incremental and CDC-based acquisition; log-based CDC tooling (e.g. Fivetran / HVR, Debezium, Kafka Connect, Snowflake Openflow); batch, micro-batch and event-driven ingestion

Managed ELT

Fivetran or comparable – connector configuration, schema drift, history mode, re-syncs, consumption cost management

Metadata-driven Ingestion

Designing and building configuration-led ingestion frameworks – metadata / control tables, parameterised and reusable load patterns, watermarking, audit, restartability and reprocessing

Snowflake Orchestration

Snowflake Tasks and task graphs (DAGs), Streams & Tasks patterns, Dynamic Tables (target lag, incremental vs full refresh), serverless tasks, alerts and notifications, monitoring of task history and cost

Engineering Practice

Python for data engineering and automation; Git; CI/CD (e.g. GitHub Actions,); infrastructure-as-code (e.g. Terraform, schemachange); automated testing

Snowflake AI

Good working knowledge of Snowflake CoCo (formerly Cortex Code) and Snowflake Cortex AI, and a disciplined approach to AI-assisted development in a regulated environment

Secondary Skills (Desirable)

Experience in any of the following will strengthen an application:

Skill area

What we value

Cloud (AWS)

S3, IAM, KMS, PrivateLink and networking basics, secrets management

Application Databases

Sourcing safely from PostgreSQL, Oracle, MongoDB, TimescaleDB and Redis – logical replication, redo logs, change streams, impact on source performance

Event Streaming

NATS JetStream or Kafka; Snowpipe Streaming; event schema contracts and replay strategies

Advanced Snowflake

Snowpark (Python), Snowflake Horizon catalog, data sharing

Modelling Standards

BIAN, ISO 20022 or other industry reference models; awareness

Operations

Data observability tooling, FinOps for Snowflake and ELT consumption, external orchestration tools (e.g. Airflow, Dagster)

Banking Domain

Core banking / ledger, payments, FX, client onboarding / KYC or regulatory reporting data; GDPR and BCBS 239

Certifications

SnowPro Core or SnowPro Advanced

Data Engineer; dbt Analytics Engineering; AWS certification

Experience & Profile

• Senior: 10+ years in data engineering, including at least 3 years of hands-on Snowflake delivery in production. Lead: 12+ years, including leading a data engineering team or workstream
• Strong track record in data warehousing and data modelling, with experience implementing models designed by architects as well as designing their own
• Has stood up a new data platform or major platform workstream – environments, access control, automation – rather than only extending an existing one
• Has built a dbt project from the ground up, including standards, testing and CI/CD
• Has designed and built a metadata-driven ingestion framework that has been used to onboard multiple sources in production
• Strong hands-on experience scheduling and orchestrating production pipelines natively in Snowflake with Tasks / task graphs and Dynamic Tables
• Has designed and implemented data acquisition across heterogeneous sources, and can clearly explain the trade-offs between full, incremental and CDC approaches
• Comfortable reading and implementing architecture artefacts – canonical models, mapping specifications, ADRs – and offering well-reasoned implementation feedback
• Clear communicator, able to work with architects, application teams and business stakeholders

Senior vs Lead – Role Distinction

Aspect

Senior Data Engineer

Lead Data Engineer

Focus

Hands-on build of pipelines, models and platform components

Hands-on build plus technical direction of the engineering workstream

Standards

Applies and contributes to engineering standards

Defines and owns engineering standards and reusable patterns

People

Peer reviews and supports other engineers

Leads, mentors and helps recruit the team; owns code review

Planning

Estimates and delivers own work

Plans, estimates and reports delivery

Architecture

Implements the design; raises issues

Engineering counterpart to the Data Architect; co-authors ADRs