Data Engineer - Databricks

Overview

We are looking for a seasoned Data Engineer to work with our team and clients to develop enterprise-grade data platforms, data lakes and lakehouses, services, and pipelines using Databricks, with a focus on data governance and cataloging using Unity Catalog. We are looking for more than just a Data Engineer; we are seeking a technologist with excellent communication and customer service skills and a passion for data and problem solving.

Responsibilities

• Design, develop, and support the migration of data lake and lakehouse environments using Databricks, with a focus on performance, reliability, and scalability.
• Implement and support Databricks Unity Catalog to enable centralized data governance, discovery, access control, and management across data lake and lakehouse environments.
• Assess and understand existing ETL jobs, workflows, data marts, BI tools, reports, and associated data dependencies.
• Design, develop, and optimize scalable data pipelines and data processing solutions using Databricks.
• Address technical inquiries concerning customization, integration, enterprise architecture, and the features and functionality of data products.
• Develop and support database, data warehouse, data mart, data lake, and lakehouse solutions in cloud environments such as AWS, Azure, or GCP.
• Apply hands-on expertise with Databricks, Unity Catalog, SQL, PySpark/Python, and cloud technologies to support data engineering and governance solutions.
• Support an Agile software development lifecycle.
• Contribute to the continued growth of our AI & Data Exploitation Practice.

Qualifications

• Ability to obtain and maintain a U.S. government security clearance.
• Bachelor's degree in computer science, information systems, engineering, or a related technical discipline.
• 2–4 years of industry experience developing production software and solving complex technical problems.
• 2–4 years of direct experience in Data Engineering, including experience with tools and technologies such as:
• Big data technologies: Databricks, Apache Spark, Delta Lake, or similar.
• Relational SQL: preferably T-SQL; alternatively PostgreSQL or MySQL.
• Data pipeline and workflow management tools: Databricks Workflows, Airflow, AWS Step Functions, or similar.
• Cloud services: Databricks on AWS, S3, EC2, RDS, or Azure equivalents.
• Object-oriented or scripting languages: PySpark/Python, Java, C++, Scala, or similar.
• Data lakehouse architecture and technologies such as Delta Lake or Apache Iceberg.
• Demonstrated hands-on experience implementing and working with Databricks Unity Catalog, including data cataloging, governance, access controls, and permissions.
• Experience applying data governance practices within Databricks data lake or lakehouse environments.
• Advanced SQL knowledge and experience working with relational databases, including query development and optimization, as well as familiarity with a variety of database technologies.
• Experience manipulating, processing, and extracting value from large, disparate datasets.
• Ability to assess existing data pipelines, understand their purpose and functionality, and efficiently reimplement or modernize them using Databricks.
• Experience working with structured and unstructured data.
• Experience architecting data systems, including transactional systems and data warehouses.
• Experience with the software development lifecycle (SDLC), CI/CD practices, and development, test, and production environments.
• Experience with additional data cataloging and governance tools such as Informatica EDC, Collibra, Alation, Microsoft Purview, or Amazon DataZone is a plus.
• Demonstrated commitment to data governance and data management best practices.
• Experience working in an Agile environment.
• Experience supporting project teams of developers and data scientists building web-based interfaces, dashboards, reports, analytics solutions, and machine learning models.