Senior Data Engineer - MS Fabric

Roles & Responsibilities

We are seeking a skilled Senior Data Engineer with expertise in Databricks to join our dynamic data team. The ideal candidate will design, build, and maintain scalable data pipelines and architectures to support our organization's data-driven initiatives and leverage Databricks to process large-scale datasets, optimize data workflows, enable advanced analytics and machine learning, and integrate Power BI for data visualization and reporting.

Key Responsibilities

• 5-8 years of professional work experience in a relevant field
• Proficient in Microsoft Fabric platform, Azure Databricks, ADF, Delta Lake, SQL Data Warehouse, Unity Catalog.
• Good Experience on Microsoft Dynamics 365Experience/ prior knowledge on semi structure data and Structured Streaming, Azure synapse, data lake, data warehouse.
• Proficient in creating Azure Data Factory pipelines for ETL/ELT processing; copy activity, custom Azure development etc.
• Good knowledge of SQL and Python for data manipulation, transformation, and analysis
• Understand business requirements to set functional specifications for reporting applications

Data Pipeline Development

Design, develop, and maintain robust, scalable data pipelines using Databricks, Apache Spark, and other cloud-based technologies.

Data Integration

Ingest, transform, and integrate data from diverse sources, including APIs, databases, streaming platforms, and third-party systems, into Databricks for analytics and reporting.

Power BI Integration

Develop and optimize data models and datasets in Databricks for use in Power BI, ensuring efficient data connections and high-quality visualizations.

Performance Optimization

Optimize data workflows, API calls, and queries on Databricks and Power BI for performance, cost-efficiency, and scalability.

Data Modeling

Build and maintain data models to support business requirements, ensuring data quality, consistency, and accessibility for analytics and reporting.

Cloud Integration

Implement data solutions on cloud platforms (e.g., AWS, Azure, GCP) integrated with Databricks, Power BI, and API ecosystems.

Security & Compliance

Implement data governance, security, and compliance best practices within Databricks, Power BI, and API environments.

Technical Skills

• Proficiency in Databricks, including Delta Lake, Spark SQL.
• Strong programming skills in Python, Scala, or Java.
• Experience with Apache Spark for big data processing.
• Knowledge of SQL for querying and transforming data.
• Proficiency in Power BI for creating data models, DAX queries, and interactive dashboards.

Preferred Qualifications

• Databricks certification (e.g., Databricks Certified Data Engineer Associate/Professional).
• Experience with real-time data processing and streaming (e.g., Kafka, Spark Streaming).

Educational qualification

BE/B.Tech or Equivalent Degree

Experience

2-5 Years

Mandatory/requires Skills

Strong hands-on expertise in Python (or Java), NLP, RegEx, SpaCy, NLTK and DevOps

Preferred Skills