Senior Data Engineer

Overview

We are looking for a seasoned Senior Data Engineer to work with our team and our clients to develop enterprise grade data platforms, services, and pipelines. We are looking for more than just a "Senior Data Engineer", we need a technologist with excellent communication and customer service skills and a passion for data and problem solving.

Responsibilities

• Lead and architect the migration of data environments with a focus on performance and reliability.
• Assess and understand ETL jobs, workflows, BI tools, and reports.
• Address technical inquiries concerning customization, integration, enterprise architecture, and general features and functionality of data products.
• Design and develop database and data warehouse solutions in cloud environments leveraging platforms such as AWS, Azure, or GCP.
• Design, develop, and optimize scalable data solutions using Databricks.
• Leverage Microsoft Purview to support data cataloging, lineage, metadata management, and integration across the data environment.
• Support an Agile software development lifecycle.
• You will contribute to the growth of our Data Exploitation Practice!

Qualifications

• Ability to obtain and maintain a U.S. government security clearance.
• Bachelor's degree and 10+ years of total experience.
• 5-7 years of direct experience in Data Engineering, including experience with tools such as:
• Big data tools: Hadoop, Spark, Kafka, etc.
• Relational SQL and NoSQL databases, including Postgres and Cassandra.
• Data pipeline and workflow management tools: Azkaban, Luigi, Airflow, etc.
• Cloud services across AWS, Azure, or GCP.
• Data streaming systems: Storm, Spark-Streaming, etc.
• Search tools: Solr, Lucene, Elasticsearch.
• Programming and scripting languages: Python, Java, C++, Scala, etc.
• Experience with Microsoft Purview.
• Experience with Databricks.
• Experience with Wiz or similar cloud security platforms.
• Advanced working SQL knowledge and experience working with relational databases, query authoring and optimization (SQL), as well as working familiarity with a variety of databases.
• Experience with message queuing, stream processing, and highly scalable ‘big data’ data stores.
• Experience manipulating and processing large, disconnected structured and unstructured datasets to support analysis and extract value.
• Experience constructing complex queries to analyze results using databases or in a data processing development environment.
• Experience with data modeling tools and processes.
• Experience architecting transactional data systems and data warehouses.
• Experience aggregating results and/or compiling information for reporting from multiple datasets.
• Experience working in an Agile environment.
• Experience supporting project teams of developers and data scientists building web-based interfaces, dashboards, reports, and analytics/machine learning models.