Talend/data bricks & snowflake

Title

Talend/data bricks & snowflake

Location

Hyderabad

Exp

6-8 Yrs

Job Description

Key Responsibilities

• Design, develop, and maintain scalable and extensible data pipelines for collecting, storing, transforming, and analyzing large datasets from multiple sources.
• Build and optimize ETL/ELT pipelines using Talend, Databricks, Snowflake, Azure Data Factory, and related technologies.
• Develop and manage data ingestion and replication frameworks using Fivetran and HVR for batch and near real-time data movement.
• Design and implement scalable data lakehouse architectures using Databricks, Delta Lake, and Snowflake.
• Develop PySpark and Spark SQL solutions to process and transform large-scale datasets efficiently.
• Implement data quality frameworks, automated testing, monitoring, and reconciliation processes to ensure data integrity and reliability.
• Build and maintain API-based integrations to support enterprise-wide data consumption.
• Optimize Snowflake performance through warehouse tuning, query optimization, clustering strategies, and cost management.
• Adhere to data governance standards to ensure data quality, consistency, integrity, usability, and security.
• Implement CI/CD pipelines and DevOps best practices for automated deployment and platform reliability.
• Perform root cause analysis and troubleshoot complex data-related issues across multiple systems.
• Collaborate with business stakeholders, product teams, architects, and engineering teams to deliver high-quality data products and solutions.
• Support modernization initiatives by migrating legacy ETL workloads from Talend, Informatica, and DataStage to cloud-native platforms.

Minimum Qualifications

• 5+ years of experience in Data Engineering, Data Warehousing, and Analytics solutions.
• 5+ years of experience designing and developing ETL/ELT solutions using Talend, Informatica, DataStage, or similar tools.
• 3+ years of hands-on experience with Databricks, PySpark, and Spark SQL.
• 3+ years of experience with Snowflake, including performance tuning, data modeling, and workload optimization.
• Experience with Fivetran and/or HVR for Change Data Capture (CDC), replication, and data ingestion.
• Strong programming and scripting skills in SQL, Python, Stored Procedures, and Unix Shell Scripting.
• Experience designing and developing cloud-based data pipelines using Azure Data Factory.
• Strong understanding of dimensional modeling, data warehousing concepts, and data lake architectures.
• Experience creating test plans, test strategies, integration testing, and data validation frameworks.
• Experience implementing CI/CD and DevOps practices using GitHub, Jenkins, JFrog, and related tools.
• Knowledge of data governance, data quality, metadata management, and security best practices.
• Experience working with structured, semi-structured (JSON/XML), and unstructured datasets.
• Experience working in Agile and Scrum environments.

Preferred Qualifications

• Bachelor's degree in Computer Science, Engineering, Mathematics, Information Systems, or a related discipline.
• Experience building enterprise-scale analytics and reporting platforms using Azure Cloud services.
• Hands-on experience with Delta Lake and Lakehouse Architecture.
• Experience supporting retail, merchandising, customer analytics, or supply chain data platforms.
• Experience migrating legacy ETL solutions to Databricks and Snowflake.
• Exposure to streaming technologies such as Kafka or Azure Event Hubs.
• Knowledge of Infrastructure as Code (Terraform) and DataOps practices.
• Databricks, Snowflake, Azure Data Engineer, or related cloud certifications.
• Exposure to Power BI or other BI and visualization platforms.

Primary Screening Skills (Must-Have)

• Databricks
• Snowflake
• PySpark
• Python
• SQL
• Azure Data Factory
• Fivetran/HVR
• CI/CD and GitHub

Secondary Skills

• Talend / Informatica / DataStage
• Data Modeling
• DevOps
• API Integration
• Data Governance

Nice-to-Have Skills

• Delta Lake
• Kafka / Azure Event Hubs
• Terraform
• Power BI
• DataOps Practices