Senior Machine Learning Systems Engineer, LLM Inference Performance on AWS Trainium (26-01636) -

Role name

Senior Machine Learning Systems Engineer, LLM Inference Performance on AWS Trainium

Work Location

Canada (Remote)

Start date

Immediate availability.

Background check MANDATORY

Request ID

1HTA3I

Role overview

We are seeking a hands-on machine learning systems engineer to optimize production-grade large language model inference across current and next-generation AWS Trainium platforms, including Trainium2 and Trainium3. The work focuses on decode latency, throughput under representative load, price-performance, long-context and multi-chip scaling, and reproducible benchmarking against leading GPU platforms.

As part of a small, senior engineering team, you will own substantial workstreams end to end, from model-level optimization through measurement, analysis, and documentation. You will communicate technical findings and business implications clearly to customer engineers and leadership.

This is a hands-on, customer-facing role focused on delivering measurable production outcomes.

Key responsibilities

• Lead decode optimization. Improve decode latency, throughput, and cost on AWS Trainium platforms, validating correctness at every step. Initial priorities include speculative decoding and native multi-token prediction (MTP) for Qwen-family models in vLLM on AWS Neuron.
• Benchmark and assess performance rigorously. Design and run matched, reproducible tests against leading GPU platforms, controlling prompts, concurrency, cache state, and warm-up. Deliver latency-throughput Pareto frontiers and price-performance comparisons aligned with customer service-level objectives.
• Lead cross-generation optimization. Benchmark and migrate workloads across Trainium generations, including Trainium2 and Trainium3. Scale serving on multi-chip Trainium systems for long-context workloads, including contexts of 100K tokens or more, and apply architecture-specific optimizations.
• Extend model coverage. Onboard additional architectures, including hybrid state-space and mixture-of-experts models, reusing and generalizing existing implementations where practical.
• Enable customer ownership. Create runbooks, implementation guides, and benchmark documentation, and support hands-on workshops so customer engineers can operate, reproduce, and tune the stack independently.

What success looks like

• Decode latency and serving cost improve measurably while maintaining model quality.
• Performance results are reproducible, correctly scoped, and defensible across hardware platforms.
• Final benchmarks reflect representative customer workloads, not synthetic peak numbers alone.
• Customer engineers can reproduce the environment, run the benchmarks, and continue optimization independently.

Basic qualifications

• 5+ years of experience in machine learning or systems engineering, including 2+ years focused on production model inference or serving.
• Strong proficiency in Python and PyTorch, with the ability to read and modify model implementations involving attention, mixture-of-experts routing, and KV-cache management.
• Hands-on experience with an LLM serving framework such as vLLM, TensorRT-LLM, SGLang, or Triton Inference Server.
• Deep understanding of LLM inference performance, including prefill and decode, continuous batching, and tensor and expert parallelism, and their impact on latency, throughput, and cost.
• Experience profiling and optimizing accelerator workloads at the framework, compiler, or kernel level, and designing rigorous performance experiments.
• Strong written and verbal communication skills, with the ability to explain engineering trade-offs to technical audiences and business impact to senior stakeholders.

Preferred qualifications

• Experience with AWS Trainium or Inferentia and the AWS Neuron SDK, including NxD Inference, vLLM on Neuron, or Neuron Kernel Interface (NKI), or with migrating workloads across accelerator generations.
• Experience implementing or evaluating speculative decoding, such as draft-model approaches, Medusa, EAGLE, or MTP.
• Kernel development experience in NKI, Triton, or CUDA.
• Experience with hybrid or state-space architectures such as Mamba, large mixture-of-experts models, or FP8 / INT8 quantization with model quality and accuracy validation.
• Experience in a customer-facing engineering, consulting, or technical enablement role.