DevOps Engineer - Data Platform Department (DPD)

Job Description

Department Overview

The Data Platform Department (DPD) at Rakuten Group develops and maintains a comprehensive data platform, empowering over 70 Rakuten services with solutions for data ingestion, discovery, governance, analytics, and querying. We support data-driven decision-making across one of Japan's largest data ecosystems, providing the tools and infrastructure to support key domains such as Data Lakes, Data Warehouses, and Business Intelligence.

Position

Position Details

We are seeking a DevOps Engineer to manage the end-to-end operations of our enterprise Databricks platform hosted on Azure and GCP. The role will focus on platform administration, automation, CI/CD, monitoring, security, incident management, and ensuring a secure, reliable, and highly available platform.

Key Responsibilities

• Manage day-to-day operations of Azure and GCP Databricks environments
• Automate infrastructure provisioning and platform configuration using Terraform
• Build and maintain CI/CD pipelines for Databricks deployments
• Monitor platform health, performance, availability, and costs
• Troubleshoot production issues, perform root cause analysis, and drive operational improvements
• Manage platform security, access controls, secrets, and audit logging
• Collaborate with engineering, cloud, networking, and security teams to support platform operations

Mandatory Qualifications

<Databricks Administration>

• 3+ years of hands-on experience administering Azure Databricks in production environments
• Strong knowledge of Unity Catalog, workspace administration, cluster policies, compute management, Jobs & Workflows, permissions, secret scopes, and audit logs
• Experience managing multiple Databricks workspaces across development, test, and production environments

<Cloud Platforms>

• Strong hands-on experience with Microsoft Azure and working knowledge of Google Cloud Platform (GCP)
• Experience with cloud networking concepts including VNet/VPC, Private Link/Private Endpoints, firewall rules, DNS, VPN/Peering, and secure connectivity

<Infrastructure as Code & Automation>

• Hands-on experience with Terraform for provisioning and managing cloud infrastructure
• Experience automating infrastructure deployment, configuration management, and environment standardization

<CI/CD & Source Control>

• Experience building and maintaining CI/CD pipelines using Azure DevOps, GitHub Actions, GitLab CI, or Jenkins
• Strong knowledge of Git, branching strategies, release management, and deployment automation

<Monitoring & Incident Management>

• Experience implementing monitoring, logging, and alerting using Azure Monitor, Log Analytics, GCP Cloud Monitoring, or similar tools
• Strong experience in production support, incident management, root cause analysis (RCA), and operational runbook creation

<Security & Identity>

• Experience implementing RBAC, Microsoft Entra ID (Azure AD), IAM, service principals, managed identities, secret management, and audit logging
• Good understanding of cloud security best practices and compliance requirements

<Scripting & Automation>

• Proficiency in Python and Bash (PowerShell is an advantage)
• Experience developing automation scripts for operational and deployment tasks

<DevOps Practices>

• Strong understanding of DevOps principles, Infrastructure as Code, CI/CD, GitOps, change management, and platform reliability engineering

<Nice to Have Skills>

• Experience managing multi-cloud Databricks platforms (Azure & GCP)
• Knowledge of Delta Sharing, Delta Live Tables, Databricks Asset Bundles, and Lakehouse architecture
• Experience with Docker, Kubernetes, Grafana, Prometheus, Splunk, or Datadog.
• Experience with cloud cost optimization (FinOps)
• Azure, GCP, Databricks, or Terraform certifications

Preferred Qualifications

• Strong troubleshooting and analytical skills
• Experience supporting mission-critical production platforms
• Excellent communication and documentation skills
• Automation-first mindset with a focus on continuous improvement

#engineer #infrastructureenginee #technologyplatformdiv