Data Platform Tech Lead

• Design, build, and maintain large-scale batch and streaming data pipelines (ETL/ELT) across multiple products
• Develop and optimize Spark workloads for performance, cost, and reliability on large data volumes
• Build and evolve our Data Lake / Lakehouse, including data modeling, partitioning, and storage formats
• Write production-grade backend services and REST APIs that expose data platform capabilities
• Build in data quality, observability, and monitoring from the start, and own your pipelines in production end to end
• Automate and improve the data platform, including AI-assisted development and LLM-powered data workflows
• Work closely with R&D and product teams, and contribute to design and code reviews
• 6+ years of hands-on Data Engineering experience building and operating production-scale pipelines
• Proven experience with building and maintaining a large scale Data Lake
• Deep, hands-on experience with Spark (tuning, partitioning, debugging large jobs)
• Strong Python and advanced SQL
• Backend development experience: REST APIs, microservices, service design patterns
• Broad database experience across relational, analytical/columnar, NoSQL, and streaming stores
• Experience with cloud-based data platforms
• Practical experience with AI tools and agents, such as integrating or orchestrating LLM-powered workflows in production
• Strong engineering fundamentals: testing, CI/CD, code quality, and performance profiling
• Self-directed: you take a problem from design to production with little guidance and suggest improvements without being asked

Nice to have

• Hands-on Databricks experience is a major advantage
• Azure data services or EventHub
• Streaming frameworks and real-time processing (e.g., Spark Structured Streaming, Kafka, Flink)
• Table formats such as Delta Lake or Iceberg
• Background in cyber security, SaaS, or large-scale enterprise environments
• Experience with data governance, observability, or self-service platforms

#LI-NK2