Mid-level Big Data Engineer

Role

Mid-level Big Data Engineer

Position Type

Full-Time Contract (40hrs/week)

Contract Duration

6 months + extendable

Work Schedule

8 hours/day (Mon-Fri)

Location

Hybrid - Wed/Thursday to be in onsite in Hyderabad, India

About the role

• Design, develop, test, and deploy scalable Big Data solutions on AWS Cloud.
• Build and maintain batch and streaming data pipelines using Scala, Python, Spark, and PySpark.
• Analyze, process, and transform large volumes of structured and unstructured data.
• Develop and support distributed data processing frameworks and data ingestion platforms.
• Integrate new data sources and tools into existing big data ecosystems.
• Optimize performance of Spark, Hadoop, EMR, and distributed computing workloads.
• Design and implement cloud-native APIs and microservices supporting data products and operational platforms.
• Collaborate with product owners, architects, QA teams, and business stakeholders throughout the software development lifecycle.
• Automate data workflows, deployments, monitoring, and operational processes.
• Evaluate emerging Big Data, AI, and cloud technologies and prototype innovative solutions.
• Ensure adherence to security, governance, reliability, and software engineering best practices.

Required Skills

• Bachelor's degree in Computer Science, Computer Engineering, Information Technology, or equivalent experience.
• 5-8 years of experience in software engineering and Big Data application development.
• Strong hands-on experience with: Scala, Python, Apache Spark / PySpark, Hadoop ecosystem (HDFS, MapReduce, Hive), Kafka or similar event-streaming technologies
• Strong AWS experience with services such as: EMR, S3, EC2, ECS/EKS, Airflow (MWAA preferred), Step Functions, API Gateway, Lambda, DynamoDB and/or Aurora/RDS
• Experience building and supporting large-scale batch and API-based data processing solutions in AWS.
• Strong Linux and shell scripting skills.
• Experience with data ingestion, transformation, data modeling, schema design, and data lifecycle management.
• Strong understanding of performance tuning for Spark, Hadoop, EMR, and distributed systems.
• Experience with Agile/Scrum development methodologies.
• Understanding of secure software development practices and cloud security fundamentals.
• Strong analytical, troubleshooting, and problem-solving skills.
• Excellent written and verbal communication skills.

Preferred Skills

• Experience with Generative AI, AI-assisted development tools (GitHub Copilot, Amazon Q, Cursor, etc.), and AI-driven automation.
• Understanding of Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), Vector Databases, and AI/ML solution integration.
• Experience building AI-enabled data platforms and data pipelines supporting ML workloads.
• Experience with DevOps and CI/CD tools such as Git, GitHub, GitLab, Bitbucket, Jenkins, Maven, Gradle, and Artifactory.
• Experience with containerization technologies such as Docker and Kubernetes.
• Knowledge of automated testing frameworks and data quality validation tools.
• Experience working in enterprise-scale cloud environments.
• AWS certification(s) and/or Databricks certification(s) are a plus.