Data Engineering

Project Role

Application Support Engineer

Project Role Description

Act as software detectives, provide a dynamic service identifying and solving issues within multiple components of critical business systems.

Must have skills

Data Engineering

Good to have skills

NA

Minimum 5 year(s) of experience is required

Educational Qualification

15 years full time education

Job Role Summary

In this role, the candidate will serve as a Monitoring Data Pipeline Engineer, responsible for designing, building, and maintaining robust, scalable, and highly observable data pipelines that ensure reliable data flow across the enterprise. The position emphasizes end to end observability, including metrics, logs, traces, data quality checks, and pipeline performance monitoring—to enable timely diagnostics, proactive issue prevention, and overall operational excellence.

Job Role Responsibilities

Build and maintain monitoring dashboards for data pipelines (latency, failures, SLAs).

Design, develop, monitor, and troubleshoot Apache NiFi data flows for enterprise-scale data integration and ingestion.

Implement distributed tracing and logging across ingestion, processing, and storage layers.

Configure real time alerts using tools like Grafana, Datadog, Azure Monitor, CloudWatch, etc.

Enhance pipeline performance through profiling, optimization, and scalability improvements.

Ensure consistent metadata capture using Data Catalog tools.

Investigate pipeline failures and performance issues.

Conduct RCA and implement preventive automation.

Work closely with Data Engineers, SRE/DevOps, Platform Teams, and Product Owners.

Provide insights into pipeline health to business stakeholders.

Influence the development of platform-wide observability standards.

Job Role Requirements

• Bachelor's/master s degree in computer science or equivalent with score 6 GPA and above
• 4–8 years of experience in Data Engineering, Observability, or Platform Engineering
• Strong understanding of modern data pipelines (batch and streaming).

Experience with at least one cloud ecosystem

Azure / AWS / GCP.

Proficiency in monitoring/observability platforms

Datadog, Grafana, CloudWatch, Azure Monitor, Dynatrace, etc.

Hands-on experience with

• Streaming frameworks (Kafka, Spark Streaming, EventHub, etc.)

Mandatory hands-on experience with Apache NiFi, including

• Flow design and development, Performance tuning, Troubleshooting and production support.
• Monitoring and observability, Error handling and flow optimization

Hands-on experience with

Apache NiFi , Kafka, Spark Streaming, EventHub or equivalent streaming technologies

• Good understanding of CI/CD, Docker, Kubernetes, APIs & Microservices.
• Programming/scripting skills in Python, SQL, Bash, or similar
• Experienced in modern monitoring tools in IT Ops domain
• Proficient in tools for logging, alerting and monitoring onprem/cloud infrastructure, services and performance
• Good understanding of analysis, design, implementation, testing, monitoring - SDLC
• Strong knowledge of the version control system (e.g. GitHub)
• Good database knowledge. Understanding concept of RDBMS i.e. nested queries, records maintenance.
• Knowledge of IT Operations space and Site Reliability Engineer job role is an added advantage

Soft skills

• Excellent written and verbal communication skills
• Agile methodologies and project execution

Technical Skills

• Apache NiFi (Mandatory)
• Kafka
• Data Pipeline Monitoring & Observability
• AWS / Azure / GCP
• Python & SQL
• Grafana / Datadog / CloudWatch / Azure Monitor
• CI/CD, Docker & Kubernetes
• RCA, Alerting & Troubleshooting
• Data Engineering (Batch & Streaming)
• GitHub

Additional Information

• The candidate should have minimum 5 years of experience in Data Engineering.
• This position is based at our Bengaluru office.
• A 15 years full time education is required.