EXAMPLE PROJECT BRIEF

LLM inference on Amazon EKS

Run a customer-facing model endpoint with GPU capacity, release controls and visibility into the cost of serving each request.

Illustrative delivery scope

The workload

A product team is preparing a model-backed API for customer traffic. Model startup is slow, demand changes through the day and a shared GPU pool makes it difficult to attribute spend.

The API needs a tested relationship between traffic, latency and GPU capacity before the team can plan a wider product launch.

PROPOSED ENGINEERING WORK

Delivery scope

Serving environment

Define the model/runtime memory requirements, request path and GPU node pools. Configure container images, health/readiness checks, artifact access and workload permissions.

Capacity and release behavior

Exercise model loading, representative traffic, scale-up and a failed rollout. Review batching and queue behavior with the ML engineers, then set capacity and rollback rules for the endpoint.

Operating evidence

Connect endpoint latency/errors, queue depth, model version, GPU memory/activity and allocated spend. Document the service owners and the investigation path for a slow request.

Illustrative workflow · scoped to your requirements
  1. Versioned model & image
  2. Amazon EKS / GPU pool
  3. Inference API
  4. Latency & cost telemetry

Project deliverables

  • Infrastructure and workload configuration
  • Load-test and model-startup findings
  • Serving dashboards and workload cost baseline
  • Release/rollback procedure and incident runbook

How to assess the result

Customer response time

Record latency and failed requests across an agreed traffic profile, including a capacity change.

Serving economics

Attribute capacity and spend to the workload and compare cost per completed request against the baseline.

Release recovery

Exercise a failed release and return the endpoint to the accepted version.

Targets are set from the workload baseline and the product’s requirements during scoping.

CLOUD & PLATFORM ENGINEERING

Tell us about your infrastructure.

Tell us what is happening and what you want to improve. The introductory scoping call is free. Any technical review follows a proposal confirming scope, deliverables and fee.

Request an Infrastructure ReviewAn introductory conversation is free. Scope and fees for any assessment or delivery are confirmed in a proposal.