Data Engineer

What you’ll do

• Design, build and maintain data processing ETL pipelines & data products in a multi-cloud, multi-region, distributed processing context choosing the right technologies.
• Drive data investigations to deliver a resolution of technical, procedural, and operational issues.
• Solve complex business problems by utilizing disciplined development methodology, producing scalable, flexible, efficient and supportable solutions using appropriate technologies.

Skills

Data engineering patterns

Sound knowledge of the different data engineering patterns to determine what to use and when

Data architecture

Understanding of data architecture and frameworks like Data mesh, data fabric etc.

Big Data Technologies

Proficiency in big data technologies such as data lake, EMR, Glue for analysing large datasets.

Programming Languages

Proficiency in programming languages commonly used in data engineering, such as Python, Java, or Scala. with OOP expertise

Data Warehousing

Strong knowledge of data warehousing solutions preferably Amazon Snowflake.

ETL/ELT Tools

Familiarity with orchestration tools

Database Systems

Expertise in both relational and NoSQL databases.

Data Modelling

Skill in designing efficient data models, both for OLAP and OLTP systems.

Streaming Data

Knowledge of streaming data technologies.

Version Control

Experience with version control systems

Git.

Containerization and Orchestration

Understanding of containerization technologies (Docker) and container orchestration platforms (Kubernetes).

Data Security

Knowledge of data encryption, access control, and compliance with data privacy regulations.

Monitoring and Logging

Proficiency in setting up monitoring and logging solutions for data pipelines using different tools