Data Engineer

Key Responsibilities

• Design and develop automated ELT processes ensuring a high level of data quality and reliability, using tools such as Spark, Apache Airflow, DBT, Python, Databricks, etc.
• Build and maintain batch and real-time data processing using Apache Kafka and AWS services like SNS, SQS, and Kinesis, supporting microservice architectures.
• Maintain the data lake in AWS and Databricks, building well-governed, query-oriented data models for efficient consumption.
• Implement monitoring and observability for the pipelines you work on.
• Collaborate cross-functionally, assisting other teams in identifying, collecting, and processing valuable information for diverse business use cases.
• Support the Data Science team by helping ensure stable, scalable environments for model deployment and experimentation.
• Build and maintain CICD pipelines, following best practices for automatic deployment and consistency.
• Help administer and maintain deployed infrastructure.
• Follow and contribute to engineering standards and best practices across the Data Engineering team.
• Deliver work on time, managing your own priorities in line with team commitments.

Skills and Experience Required

• 3+ years in data engineering or related data processing roles.
• Strong experience working with varied data formats and sources (JSON, CSV, Parquet, APIs, multiple DB engines).
• Hands-on experience with data modelling in Big Data platforms (e.g. Databricks, Glue, Snowflake).
• Familiarity with event-driven architectures and AWS.
• At least 2 years of hands-on experience with AWS, especially:
• Core: IAM, S3, EC2, VPC
• Data Services: EMR, Lambda, Batch, SNS, SQS, DynamoDB, Glue, Athena
• Ability to set up and maintain monitoring, alerting, and debugging solutions for your pipelines.
• Experience collaborating with Data Science teams to support production-grade pipelines.

Knowledge

• Solid understanding of computing systems (OS, memory, networks, etc.).
• Proficient in data analysis and interpretation, with a good grasp of statistics and data quality methods.
• Strong SQL and performance tuning skills.
• Good knowledge of Big Data technologies, Spark, and object storage systems.
• Proficiency in Python (and optionally Scala), with strong coding and testing practices.
• Familiarity with Docker; Kubernetes experience is a plus.
• Airflow (or similar) for orchestration.
• Good understanding of IaaC and CICD principles.
• Dashboard and storytelling experience for data visualization.

Other

• Strong communication skills.
• Proactive mindset and collaborative team spirit.
• Comfortable working within an agile team setup.