Technology Lead | Big Data - Hadoop | Hadoop

Job Description

Job Title

Technology Lead | Big Data - Hadoop | Hadoop Work Location: Maryland HeightsMO63043 Contract duration: 12 Target Start Date: 16 Oct 2026 Does this position require Visa independent candidates only? Yes hybrid work set-up

Job Details

Must Have skills

AWS, Hadoop

Detailed Job Description

Key Responsibilities

• Design, build, and optimize scalable batch and real-time data pipelines using Apache Spark (Scala/Python), Core Java, and Apache Kafka to process high-volume data.
• Manage and maintain enterprise data Lakehouse architectures spanning AWS and Cloudera environments, leveraging S3, Apache Iceberg, and Hive for scalable storage and querying.
• Implement Change Data Capture (CDC) and write complex SQL to integrate data from relational databases (Oracle, SQL Server) into dimensional data warehouses.
• Automate and orchestrate pipeline workflows using Unix shell scripting and Python.
• Lead and support migration of legacy Hadoop workloads to AWS cloud-native architectures.
• Monitor, troubleshoot, and tune JVM and Spark job performance to ensure reliability and efficiency at scale.
• Ensure all data pipelines and storage practices adhere to regulatory compliance and data governance standards.
• Collaborate with cross-functional teams (data architects, analysts, platform engineers) to translate business requirements into robust data solutions.

Required Skills & Technology

Strong programming experience in Scala/Python (Spark) and Core Java Hands-on experience with Apache Kafka for streaming data pipelines Working knowledge of AWS services (S3) and Cloudera platform tools (Hive, Iceberg) Strong SQL skills, including CDC concepts, across Oracle and SQL Server Experience with dimensional data warehouse modeling Proficiency in Unix shell scripting and Python for automation Experience migrating on-prem Hadoop workloads to cloud platforms Understanding of JVM internals and Spark performance tuning techniques

Preferred Qualifications

Prior experience in the Communications/Telecom domain (CDRs, network telemetry data) Familiarity with data governance and regulatory compliance frameworks Bachelor's degree in computer science, Engineering, or a related field, or equivalent practical experience

Minimum years of experience

8-10 years

Certifications Needed

No

Top 3 responsibilities you would expect the Subcon to shoulder and execute Manage and maintain enterprise data lakehouse architectures spanning AWS and Cloudera environments, leveraging S3, Apache Iceberg, and Hive for scalable storage and querying. Implement Change Data Capture CDC and write complex SQL to integrate data from relational databases Oracle, SQL Server into dimensional data warehouses.

Automate and orchestrate pipeline workflows using Unix shell scripting and Python. Lead and support migration of legacy Hadoop workloads to AWS cloudnative architectures.

M

Interview Process (Is face to face required?) Yes

Any additional information you would like to share about the project specs/ nature of work