Airflow Architect – Big Data Systems

Airflow Architect – Big Data Systems Austin, TX

Gross $135K including $5K variable

Contracting $60/hr,

Job Summary

Need an experienced Airflow Architect to design, architect, and lead enterprise-scale data orchestration solutions for modern big data platforms. The ideal candidate will have deep expertise in Apache Airflow , Big Data technologies , cloud-native architectures , and data engineering best practices to build scalable, reliable, and high-performance data pipelines.

Key Responsibilities

• Architect and implement large-scale data orchestration frameworks using Apache Airflow.
• Design, develop, and Eligible to workimize ETL/ELT pipelines for batch and near real-time processing.
• Build scalable workflows integrating Hadoop, Spark, Kafka, Databricks, data lakes, and data warehouses.
• Define Airflow deployment strategies, DAG standards, security, monitoring, and governance.
• Implement workflow scheduling, dependency management, error handling, retries, alerting, and SLA monitoring.
• Lead cloud-based data platform implementations on AWS
• Eligible to workimize pipeline performance, scalability, reliability, and cost efficiency.
• Establish CI/CD processes and DevOps practices for Airflow environments.
• Collaborate with business stakeholders, data engineers, architects, and platform teams to translate requirements into technical solutions.
• Mentor engineering teams on Airflow architecture, data orchestration patterns, and best practices.

Required Skills

Airflow & Data Engineering

• Expert knowledge of Apache Airflow DAG design, scheduling, operators, sensors, and executors.
• Strong experience in workflow orchestration, metadata management, and pipeline monitoring.
• Hands-on expertise in ETL/ELT development and data pipeline automation.

Big Data Technologies

• Apache Spark / PySpark
• Hadoop ecosystem
• Apache Kafka
• Data Warehousing concepts
• Data Lakes and Lakehouse architectures
• NoSQL databases

Programming

• Python (advanced)
• SQL (advanced)
• Shell scripting
• Core Java (preferred)

Cloud Platforms

• AWS (EMR, Glue, S3, Redshift)

Preferred Qualifications

• 10+ years of experience in data engineering and big data platforms.
• 3+ years of architecture experience designing enterprise data solutions.
• Experience with Kubernetes, Docker, and cloud-native deployments.
• Knowledge of Data Mesh, Data Fabric, and Modern Data Platforms.
• Experience leading architecture reviews and technology strategy discussions.

Success Metrics

• Pipeline reliability and SLA adherence
• Workflow performance Eligible to workimization
• Reduced operational incidents
• Improved data quality and observability
• Scalable and reusable orchestration frameworks