Senior Data Engineer

Role

Senior Data Engineer

Skill

• Fluent in Python programming language
• Proficient in SQL
• Understanding of OLTP concepts including normalised schema design
• Understanding of OLAP concepts and data model design
• Experience leading the designing and building data pipelines, ETL processes, and managing data infrastructure
• Experience leading workflows automation, managing task dependencies and transformations using Apache Airflow and DBT
• Experience in real time streaming pipelines like Kafka/pubsub and, Pyspark/dataflows
• A strong analytical mindset, a passion for resolving complex technical challenges, and the ability to find creative solutions to issues
• Excellent communication and interpersonal skills, with the ability to lead technical discussions, build consensus, and work effectively with cross-functional teams
• Deep understanding of cloud migration strategies, best practices, and potential pitfalls
• Experience with Google Cloud Platform, AWS or Azure including services for data storage, processing, and analytics
• Experience with cloud deployment and test automation using a CI/CD solution such as Cloud Build or Concourse
• Familiarity with consumer behaviour data (Google Analytics, Adobe Analytics or Snowplow Analytics) and/or subscribers data
• Proficient usage of AI to gain working efficiency to focus on higher impact architectural tasks

Experience and Role Responsibilities

• 5 + Senior Data Engineer
• Design, develop and maintain data pipelines that extract, load and transform data from various sources into the data warehouse
• Design and implement data models that align with business requirements and support efficient querying and analysis
• Ensure the team's orchestration pipelines are reliable, scalable and efficient whilst considering factors including data volume, velocity and variety
• Work across the data engineering teams, vendors and internal stakeholders to contribute to the continuous improvement of data products, processes and systems.
• Follow data engineering best practices including coding, DevOps, documentation and metadata standardisations
• Experience with the management and nature of audience tracking and subscribers datasets
• Take ownership and accountability for assigned work, including development and end-to-end testing
• Ability to conduct comprehensive feasibility assessments of architectural and data modelling options, systematically evaluating the trade-offs, advantages, and limitations of each proposed solution
• Ensure high standard and ongoing quality of Engineering output whilst meeting project timelines
• Document existing and future solution designs to enhance the team's collective knowledge