PySpark / Python Data Engineer

Role

PySpark / Python Data Engineer – Tech Lead role

Duration

6-12+ Months Contract

Location

California, - Remote (Some visits may require in future)

Note

Prefer candidates from PST Time Zone

Please find the job details below

Job Summary

We are seeking a seasoned professional with expertise in building data engineering and analytics solutions within AWS ecosystems. The ideal candidate should have deep experience in PySpark, Python, and end‐to‐end data pipeline development, including job orchestration, workflow design, and data mapping. The role requires the ability to translate complex business logic, stored procedures, and SQL triggers into scalable PySpark implementations. Experience with data streaming on Spark clusters and API design is highly desirable. Knowledge of Palantir Foundry is a strong plus.

Details

Time Zone

Must be able to work in PST hours.

Minimum Qualifications

MS or equivalent experience in Computer Science, MIS, or related technical fields; 10–15+ years of overall experience, with 5+ years in data engineering/ETL ecosystems using PySpark, Python, and Java.

Key Responsibilities

• Translate business requirements into technical solutions using PySpark and Python frameworks.
• Lead data engineering initiatives for complex analytics challenges.
• Plan and execute tasks, track progress, and document work following best practices.
• Identify and implement process improvements, including scalable infrastructure design and workflow automation.
• Participate in Agile/Scrum ceremonies.
• Provide technical guidance to team members across functional and technical domains.
• Build infrastructure for large‐scale data access and ensure data quality/metadata management.
• Collaborate with leadership to strengthen data‐driven decision‐making.

Required Skills

• Strong expertise in PySpark and Python.
• Experience with Pandas, APIs, and Spark Streaming.
• Solid understanding of database design fundamentals.
• Familiarity with CI/CD tools and infrastructure‐as‐code frameworks.
• Experience writing production‐grade code, including unit/integration tests and schema validations.
• Knowledge of Palantir Foundry (Ontology modeling, API configuration, Foundry Typescript) and exposure to Power BI or Tableau are significant advantages.