Initializing Techhub.cafe

midAWS Machine Learning

What is SageMaker Feature Store and what is the difference between the online and offline store?

Updated Aug 1, 2026

Short answer

Feature Store is a managed repository for curated features. The online store serves single records at millisecond latency for real-time inference; the offline store keeps the full history in S3 for training — and sharing one definition between them is what prevents training/serving skew.

Deep explanation

Training/serving skew is one of the most damaging and hardest-to-detect failures in production ML: the feature is computed one way in the training pipeline and subtly differently in the serving path, so the model sees inputs at inference it never saw in training. Feature Store attacks this by making both paths read the same definitions and, ideally, the same ingested values.

Offline store — append-only history in S3 (Iceberg or Glue-catalogued), queryable via Athena. This is what you build training sets from.

Online store — a low-latency key-value store returning the latest record per identifier, for inference.

A feature group can write to both. You ingest once and both paths are populated.

The subtle capability that separates a good answer: point-in-time correct joins. Training on the current value of a feature leaks information the model would not have had at prediction time. The offline store keeps EventTime on every record, so you can reconstruct the value as of the moment of each label.

Python
from sagemaker.feature_store.feature_group import FeatureGroup
import pandas as pd, time


fg = FeatureGroup(name="customer-metrics", sagemaker_session=session)
fg.load_feature_definitions(data_frame=features_df)


fg.create(
    s3_uri="s3://my-bucket/feature-store",
    record_identifier_name="customer_id",
    event_time_feature_name="event_time",     # mandatory: enables PIT correctness
    role_arn=role,
    enable_online_store=True,                 # both stores from one definition
)


features_df["event_time"] = time.time()
fg.ingest(data_frame=features_df, max_workers=8, wait=True)

Reading at inference:

Python
rt = boto3.client("sagemaker-featurestore-runtime")
rec = rt.get_record(
    FeatureGroupName="customer-metrics",
    RecordIdentifierValueAsString="cust-88213",
)
features = {f["FeatureName"]: f["ValueAsString"] for f in rec["Record"]}

And a point-in-time correct training query against the offline store:

SQL
SELECT l.customer_id, l.label, f.orders_30d, f.avg_basket
FROM labels l
JOIN customer_metrics f
  ON f.customer_id = l.customer_id
 AND f.event_time <= l.label_time          -- never look into the future
 AND f.event_time = (
       SELECT MAX(f2.event_time) FROM customer_metrics f2
       WHERE f2.customer_id = l.customer_id AND f2.event_time <= l.label_time)

Real-world example

A subscription business had a churn model scoring far worse in production than in backtests. The cause was leakage: the training join used each customer's current support-ticket count, which for churned customers included tickets filed during cancellation. Moving to Feature Store with point-in-time joins dropped offline AUC from 0.91 to 0.83 — and raised production AUC to match, because 0.91 had never been real.

Common mistakes

  • - Omitting or fudging `EventTime`, which makes point-in-time correctness impossible and invites leakage.
  • - Enabling the online store for features only ever used in training, paying for low-latency storage no one reads.
  • - Assuming ingestion is instantly queryable offline — there is a lag before records land in S3 and appear to Athena.
  • - Putting extremely high-cardinality or rapidly-changing features online without considering cost and write throughput.
  • - Treating Feature Store as a data lake. It is a curated feature layer
  • raw data belongs in S3.

Follow-up questions

  • What exactly is training/serving skew and how does Feature Store reduce it?
  • How do you handle features that must be computed at request time?
  • What is the cost profile of Feature Store?

More AWS Machine Learning interview questions

View all →