Senior Kafka Engineer / Administrator

Position Title

Senior Kafka Engineer / Administrator

Location

St. Louis, MO (Onsite)

Duration

Long Term Contact

Job description

We are migrating a portfolio of Apache Kafka clusters from on-premises platforms (Cloudera and Confluent) to AWS MSK, managed through a CI/CD pipeline built on Terraform and GitLab. We are looking for a senior, hands-on Kafka engineer to own day-to-day Kafka administration across both environments, drive the migration, and coordinate closely with producer and consumer application teams to ensure a smooth, low-risk cutover. This is a senior individual-contributor role for someone who has run Kafka at scale in production, is comfortable in a hybrid on-prem/cloud world, and can act as the technical point of contact between platform and application teams.

Basic Qualifications

• 5 years in data/platform/infrastructure engineering
• 4 years operating Apache Kafka in production

Required Skills

• Hands-on administration experience with Cloudera and Confluent Platform
• Production experience with AWS MSK (provisioning, configuration, IAM auth, networking)
• Proven experience migrating Kafka workloads to the cloud with minimal disruption
• Strong Terraform and GitLab CI/CD experience for infrastructure automation
• Hands-on Datadog experience for monitoring, dashboards, and alerting
• Deep understanding of Kafka internals: partitions, replication, ISR, offsets, consumer groups, and delivery semantics
• Solid AWS fundamentals: VPC, security groups, IAM, KMS, CloudWatch, PrivateLink
• Strong Linux, networking, and scripting skills (Bash, Python)
• Excellent communication and coordination skills to work across producer/consumer teams

Preferred Qualifications

• Experience in a hybrid on-prem/cloud environment
• Experience with GitOps practices

Role and responsibilities

• Administer and operate Kafka clusters across Cloudera, Confluent Platform, and AWS MSK
• Manage topics, partitions, replication, retention, quotas, ACLs, and consumer groups
• Tune brokers, producers, and consumers for throughput, latency, and reliability
• Handle patching, upgrades, capacity planning, and cluster health/performance troubleshooting
• Configure and maintain security: TLS/SSL, SASL, mTLS, IAM (for MSK), RBAC/ACLs, and encryption at rest and in transit
• Plan and execute migration of clusters and workloads from on-prem (Cloudera/Confluent) to AWS MSK
• Design cutover strategies with minimal downtime and no data loss
• Build and maintain Kafka/MSK infrastructure using Terraform
• Manage automated deployments and configuration through GitLab CI/CD pipelines
• Build and maintain monitoring, alerting, and dashboards in Datadog for Kafka/MSK
• Serve as the primary technical liaison between the Kafka platform team and producer/consumer application teams