Sr. Data Engineer

Title

Sr. Data Engineer

Duration

11/23/2026 - 12/31/2026

Work Hours

40hrs/week

Location

400 North Street, 5th Floor, Harrisburg, PA 17120

Responsibilities

• Create and maintain optimal data pipeline architecture.
• Design and implement cloud-based data solutions using AWS services such as S3, Glue, Redshift, EMR, Athena, Kinesis, and Lambda.
• Build and maintain data orchestration workflows using tools such as Apache Airflow, AWS Step Functions, or similar services.
• Assemble large, complex data sets that meet functional / non-functional business area requirements.
• Engineer analytical data environments that support advanced statistical analysis, predictive modeling, machine learning, and evidence-building activities.
• Partner with data scientists, research analysts, and policy analysts to ensure enterprise data assets support reproducible research and statistical modeling.
• Identify, design, and implement internal process improvements: e.g. automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability.
• Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and 'big data' technologies.
• Build analytics tools that utilize the data pipeline to provide actionable insights into education and workforce outcomes, operational efficiency, and other key business performance metrics.
• Develop and maintain data pipeline monitoring, alerting, and logging frameworks.
• Implement data quality checks and validation frameworks across pipelines.
• Develop dashboards, reports, analytical models, and/or other decision-support products.
• Work with stakeholders including the Executive, Data, Design, and Support teams to assist with data-related technical issues and support their data infrastructure needs.
• Ensure compliance with applicable Commonwealth policies, data governance standards, privacy requirements, security controls, and interagency data-sharing agreements.
• Create data tools for analytics and data scientist team members that assist them in building and optimizing data products needed to support ongoing operations and data-informed decision-making.
• Work with data and analytics professionals to strive for greater functionality in our data systems.

Required Qualifications

• Advanced SQL knowledge and proficiency working with relational databases, query authoring (SQL), as well as working familiarity with a variety of databases.
• Experience building and optimizing 'big data' data pipelines, architectures, and data sets.
• Advanced knowledge and proficiency designing, building, and refining dashboards, reports, analytical models, and decision-support products for executive and non-technical users.
• Experience working with student-level higher education data, including establishing linkages to other sources (e.g. employment and wage data) to measure outcomes and/or generate insights for decision-making).
• Experience performing root-cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
• Strong analytic skills related to working with unstructured datasets.
• Experience building processes supporting data transformation, data structures, metadata, dependency, and workload management.
• A successful history of manipulating, processing, and extracting value from large, disconnected datasets.
• Working knowledge of message queuing, stream processing, and highly scalable 'big data' data stores.
• Strong project management and organizational skills.
• Experience supporting and working with cross-functional teams in a dynamic environment.

Required / Desired Skills Skill

Required / Desired

Amount

of Experience

Hands-on experience with cloud-based data services such as AWS (e.g. S3, EC2, Glue, EMR, RDS, Redshift, Athena, Lambda, DynamoDB)

Required

5

Years

Proficiency in object-oriented/object function scripting languages

e.g. Python, R, Java, C++, Scala

Required

5

Years

Proficiency in relational SQL and NoSQL databases, including Oracle, MS SQL Server, Postgres, Cassandra.

Required

5

Years

Experience with version control systems such as Git and CI/CD pipelines

Required

5

Years

Knowledge of data modeling techniques including dimensional modeling, data vault, and star/snowflake schemas.

Required

5

Years

Familiarity with data cataloging and metadata management tools such as AWS Glue Data Catalog.

Required

5

Years

Understanding of data governance, data lineage, and data quality frameworks.

Required

5

Years

Knowledge of REST API design and integration patterns.

Required

5

Years

Experience with big data tools

e.g. Hadoop, Spark, Kafka.

Required

5

Years

Experience with data pipeline and workflow management tools

e.g. Azkaban, Luigi, Airflow.

Required

5

Years

Experience with data integration services solutions from vendors such as Informatica, MuleSoft, Talend, and TIBCO.

Required

5

Years

Experience with stream-processing systems

e.g. Storm, Spark-Streaming.

Required

5

Years