Python/PySpark developer

Technologies

Python, PySpark, SQL, Spark, Hadoop Ecosystem

Role Overview

A Senior Python and PySpark Developer leads the design, development, and delivery of scalable Python and PySpark-based solutions. The role ensures application quality, performance, maintainability, and efficient processing of large-scale datasets while guiding the engineering team and collaborating with stakeholders.

Key Responsibilities

• Lead design and development of Python and PySpark applications
• Develop scalable data processing pipelines using PySpark
• Design and optimize ETL/ELT solutions for large-volume datasets
• Define coding standards, best practices, and development guidelines
• Review and optimize application and Spark job performance
• Handle complex business logic, integrations, and data transformations
• Work with structured and unstructured data across multiple sources
• Implement data quality, validation, and reconciliation processes
• Collaborate with Data Architects, Data Engineers, and Business stakeholders
• Mentor junior and mid-level developers on Python, PySpark, and data engineering best practices
• Support release planning, production stability, troubleshooting, and performance tuning
• Ensure adherence to security, governance, and compliance standards

Required Skills

Python

• Expert-level Python development experience
• Strong understanding of Object-Oriented Programming (OOP), design patterns, and coding best practices
• Experience in building scalable and maintainable applications
• Strong debugging, performance tuning, and code review skills

PySpark & Big Data

• Strong hands-on experience with PySpark and Apache Spark
• Experience developing distributed data processing applications
• Good understanding of Spark architecture, executors, partitions, and cluster computing concepts
• Experience with Spark SQL, DataFrames, Window Functions, UDFs, and optimization techniques
• Knowledge of Spark performance tuning including partitioning, caching, broadcasting, and resource management
• Experience handling large-scale datasets in distributed environments
• Familiarity with data formats such as Parquet, Avro, ORC, and JSON
• Experience integrating Spark with cloud storage and relational databases through JDBC

SQL & Data Engineering

• Strong SQL development and query optimization skills
• Experience designing ETL/ELT pipelines and data transformation frameworks
• Understanding of data warehousing concepts and dimensional modeling
• Experience working with databases such as SQL Server, Oracle, PostgreSQL, DB2, or Snowflake