Data Engineer

Job Overview

Nextlink is seeking for a Data Engineer with strong expertise in Palantir Foundry, Python, and PySpark to design, develop, and maintain scalable data engineering solutions. The ideal candidate will have hands-on experience with Foundry Ontology, data pipelines, modular applications, and end-to-end data solution implementation, along with a solid understanding of data engineering principles and distributed computing.

Key Responsibilities

1. Palantir Foundry Platform Expertise

• Develop and maintain data pipelines, Foundry Ontology, and modular applications within the Palantir Foundry platform.
• Design and implement end-to-end data solutions, from data ingestion and transformation to data modeling and consumption.
• Integrate data from multiple source systems into Palantir Foundry.
• Build and maintain scalable, reusable data workflows aligned with business requirements.
• Leverage Foundry capabilities to support data-driven applications and analytics.

2. Programming and Scripting – Python & PySpark

• Develop Python and PySpark scripts for data processing, transformation, cleansing, and automation.
• Build reusable Python functions and libraries to extend Palantir Foundry capabilities.
• Develop modular and maintainable code for ETL workflows and data transformation processes.
• Process large datasets efficiently using Python libraries such as Pandas and NumPy, along with PySpark for distributed processing.
• Optimize code for performance, scalability, and reliability.
• Use SQL for data querying, validation, and data modeling, as required.

3. Data Engineering Fundamentals

• Design, develop, and maintain ETL/ELT pipelines for data ingestion and transformation.
• Build efficient data models and optimize data processing workflows.
• Work with structured and unstructured data from multiple sources.
• Implement data quality checks, validation, and error-handling mechanisms.
• Ensure data consistency, reliability, and availability across pipelines.

4. Big Data and Distributed Systems

• Develop scalable data processing solutions using Apache Spark and PySpark.
• Apply distributed computing concepts to handle large-scale data processing requirements.
• Troubleshoot pipeline failures and resolve data processing issues.
• Perform performance tuning and optimization of Spark jobs and data workflows.
• Follow best practices for scalable and efficient data engineering.

Requirements

Essential Skills

• Hands-on experience with Palantir Foundry and its core capabilities.
• Strong programming skills in Python and PySpark.
• Experience developing and maintaining ETL/ELT pipelines.
• Knowledge of Foundry Ontology, data pipelines, and modular applications.
• Strong understanding of data transformation, data modeling, and data integration.
• Experience with Apache Spark and distributed data processing.
• Understanding of performance optimization and scalable data processing techniques.

Desirable Skills

• Familiarity with SQL for querying and data modeling.
• Experience with Pandas and NumPy for data manipulation and analysis.
• Knowledge of data quality, validation, and pipeline monitoring practices.
• Experience integrating data from multiple source systems.
• Understanding of data governance and enterprise data engineering best practices.

Keywords

Palantir Foundry, Palantir Core, PySpark, Python, Apache Spark, Data Engineering, ETL/ELT, Foundry Ontology, Data Pipelines, Modular Applications, SQL, Data Transformation, Data Modeling, Distributed Computing, Performance Optimization.

Primary Skills

Digital – PySpark, Palantir Core

Role Focus

Palantir Foundry development, Python/PySpark-based data engineering, and end-to-end data pipeline implementation.