Initializing Techhub.cafe

midAWS Machine Learning

Compare real-time, serverless, asynchronous and batch inference on SageMaker

Updated Aug 1, 2026

Short answer

Real-time gives consistent low latency at always-on cost; serverless scales to zero with cold starts; asynchronous queues long-running or large-payload requests; batch transform scores datasets offline with no persistent endpoint.

Deep explanation

Choosing an inference option is a latency/cost/payload trade-off, and interviewers use it to test whether you reason about requirements rather than defaulting to real-time.

Real-time endpoints — persistent instances, single-digit to low-double-digit millisecond latency, autoscaling on invocations or latency. Constraints: 60-second request timeout, ~6 MB payload. Cost accrues 24/7. Use for anything user-facing and synchronous.

Serverless inference — no instance management, scales to zero, billed per millisecond of compute plus data processed. Cold starts on the first request after idle (typically seconds, worse for large models). No GPU support. Ideal for spiky, low-volume or intermittent traffic where idle cost dominates.

Asynchronous inference — requests go on a queue, results land in S3 with an SNS notification. Supports payloads up to 1 GB and long processing times, autoscales to zero when the queue empties. Use for large documents, video, or heavy generative work.

Batch transform — an ephemeral cluster scores an entire S3 dataset and shuts down. No endpoint exists afterwards. Cheapest per prediction by a wide margin. Use for scheduled scoring of whole populations.

Python
# Serverless — no instance type at all
from sagemaker.serverless import ServerlessInferenceConfig
model.deploy(serverless_inference_config=ServerlessInferenceConfig(
    memory_size_in_mb=4096, max_concurrency=20))


# Asynchronous — big payloads, long jobs, scales to zero
from sagemaker.async_inference import AsyncInferenceConfig
model.deploy(
    initial_instance_count=1, instance_type="ml.g5.xlarge",
    async_inference_config=AsyncInferenceConfig(
        output_path="s3://my-bucket/async-out/",
        notification_config={"SuccessTopic": success_topic_arn}),
)


# Batch — score 50M rows, then the cluster disappears
transformer = model.transformer(
    instance_count=10, instance_type="ml.m5.xlarge",
    strategy="MultiRecord", max_payload=6,
    output_path="s3://my-bucket/scores/",
)
transformer.transform("s3://my-bucket/to-score/", content_type="text/csv", split_type="Line")

The reasoning to voice in an interview: does a human wait for this answer? If no, batch or async, and cost drops by an order of magnitude. If yes, is traffic steady enough to justify always-on instances, or spiky enough that serverless cold starts are an acceptable trade?

Real-world example

A document platform runs all four. A real-time endpoint classifies document type on upload (users wait). Asynchronous inference runs a large extraction model over 200-page PDFs taking minutes each. Serverless serves a rarely-used legacy classifier that would otherwise cost $200/month idle. And a nightly batch transform re-scores the entire archive when a new model ships — 40 million documents on a transient fleet for about the price of two days of a real-time endpoint.

Common mistakes

  • - Defaulting to real-time for everything, then wondering why inference dominates the bill.
  • - Choosing serverless for latency-sensitive traffic without load-testing cold starts.
  • - Trying to push a 200 MB payload or a 5-minute inference through a real-time endpoint and hitting the limits.
  • - Running batch transform with `instance_count=1` on a huge dataset when it parallelises almost linearly.
  • - Forgetting that async and serverless scale to zero, so the first request after idle pays a startup penalty.

Follow-up questions

  • How do you mitigate serverless cold starts?
  • When is batch transform preferable to a scheduled endpoint call loop?
  • How does autoscaling work on a real-time endpoint?

More AWS Machine Learning interview questions

View all →