AWS Emulator Passes 56.5% of Its Fidelity Tests and Ships Releases 90 Minutes Apart. Pin One Before CI Trusts It.
A fidelity score is a pass rate over AWS's test suite, not over your API calls, and only behaviour with a passing test is protected from change.
On 11 November AWS published AWS Emulator, a local program that answers AWS API calls on your laptop or in CI without an account and without a bill. It covers 438 API models across more than 200 services, the code is dedicated to the public domain under CC0, and AWS says autonomous agents wrote all of it. The team states it has not written or reviewed any of the code.
Most of the coverage has gone to that last sentence. The part that affects your pipeline is quieter. The launch release passed 56.5% of its own published tests, and the version number has moved from 1.0.322 to 1.0.489 in six days. A team that wires this into CI this week has to answer two questions before the first green build means anything: what does a fidelity score count, and which release did we test against?
What AWS Emulator is, and what AWS says it is not
The emulator is a single Rust binary serving HTTPS on 127.0.0.1:9297. You point the AWS CLI or an SDK at it with environment variables and run your code unchanged. State lives in memory and disappears when the process stops. An AWS write-up of how it was built describes three groups of agents: one writes test scenarios from the public API models and verifies them against real AWS, one replays those tests against the emulator and files the failures, and one fixes a single issue per agent behind a merge gate.
The documentation is blunt about scope. The emulator is for building and testing, not for production and not for security testing. Its security page says a request AWS would deny might succeed locally, and one AWS would allow might fail. It behaves as one account, with no calls across accounts or Regions. The repository takes no pull requests. You file an issue, and an agent picks it up.
A fidelity score counts tests, not your calls
AWS defines the terms in its fidelity documentation. A fidelity test is a short SDK program that runs a workflow against real AWS and asserts the result, the resulting state and, for invalid calls, the exact error. The emulator passes only if it matches AWS on every call in that workflow. A fidelity score is the number of a service's tests that pass out of the total for that service, such as 40 of 50.
Three properties of that definition matter more than the headline number.
First, the denominator is the test suite, not your application. A service at 68% is not a service where 68% of your requests behave correctly. If your code calls five DynamoDB operations and all five sit in the passing set, your effective fidelity is far higher than the score. If one of them sits in the failing set, the score flatters you.
Second, the pass bar is strict in a way that drags scores down for reasons you may not care about. One wrong error message fails the whole workflow. Step-level results show the gap: an independent reading of the launch scorecard counted 68,007 individual steps with 73.2% passing, against 56.5% of whole tests.
Third, AWS says the emulator matches most closely on control plane APIs, the ones that create, describe and delete resources. Data plane fidelity, and support for CloudFormation, Terraform and the CDK, varies by service. Application tests mostly exercise the data plane. A service can score well because bucket creation matches while the object behaviour your code depends on is still in the backlog.
The figures below come from i-SCOOP's analysis of the scorecard for release 1.0.322, which put the total at 24,243 of 42,937 tests. Across 342 services with at least ten tests, the median pass rate was 61.3%, 40 services reached 90% or more, and 124 sat below 50%.
| Service | Tests | Pass rate |
|---|---|---|
| EventBridge | 967 | 71.5% |
| KMS | 505 | 71.3% |
| Step Functions | 982 | 70.0% |
| SNS | 635 | 69.3% |
| DynamoDB | 1,694 | 68.1% |
| SQS | 1,523 | 67.8% |
| S3 | 2,814 | 62.7% |
| Lambda | 3,006 | 51.9% |
| EC2 | 2,470 | 38.8% |
| CloudFormation | 4,206 | 18.9% |
Treat any second-hand table, including this one, with care. MarkTechPost's account of the same launch gives KMS as 291 of 310 tests, or 93.9%, and a total of 42,857 tests. We could not reconcile the two sets of figures from the public pages. AWS's own instruction explains why this happens: compare scores only when they use the same version of the fidelity tests. The suite is versioned separately from the emulator, tests are added and retired, and 40 of 50 can become 45 of 60 in a release where nothing got worse.
Passing tests do not regress. Everything else can change in a patch.
The most useful sentence in the documentation is on the versioning page. Behaviour covered by a passing fidelity test does not regress. Behaviour that no fidelity test covers can be changed by any release to match AWS more closely, and if your tests relied on the earlier response, they can start to fail.
That splits your test suite into two populations. Assertions that sit on top of a passing fidelity test are stable. Assertions on anything else are a bet on today's behaviour, and fidelity fixes ship as patch versions, the smallest increment in the scheme. Major and minor numbers protect how you start and configure the binary. They promise nothing about responses.
AWS's advice follows from this: write tests that expect what AWS does, not what the emulator does today. That sounds obvious until an agent writes the test. A coding agent iterating against a local endpoint will assert on whatever comes back, including a response the emulator has wrong. The test goes green, gets committed, and breaks on a later patch that fixes the emulator. The failure then looks like an upstream regression when it is a correct fix exposing a wrong assertion.
Releases arrive faster than most teams upgrade
When we read the releases page on 17 November, it listed seven releases that day between 01:05 and 10:15, roughly one every 90 minutes, ending at 1.0.489. Release 1.0.385 was dated the evening of 14 November. That is 104 version numbers in under three days, each one a batch of fixes titled by service.
Two policies collide at that rate. AWS recommends the latest version and publishes security fixes only there. Older releases get nothing, and a security fix may include a breaking change inside a major version. The same page tells you to install a specific version if new releases change your results.
Following "latest" means two CI runs an hour apart can test against different emulators, so a red build no longer tells you whether your commit or the emulator changed. Pinning forever means running a build that will never be patched and that drifts further from AWS each day. Neither default is safe. The workable answer is a pinned version with a scheduled bump, handled like any other dependency update: a pull request that changes one number, runs the suite, and shows the diff in results before anyone merges it.
Where a green local run tells you nothing
Some gaps are documented and some were found by people poking at it. A field test published on DEV ran 40 workflow checks across nine services on release 1.0.322 and all 40 passed. The same author ran 32 permission cases, and 27 matched what AWS documentation says should happen. All five mismatches allowed a request AWS denies, including a write that should have been blocked by a permissions boundary and another that ignored a session policy. Their expected results came from documentation, not from runs on real AWS, so read it as a strong hint, not a measurement.
The list of things a local pass cannot vouch for is longer than permissions:
- IAM policies, permissions boundaries, bucket policies and KMS key policies. AWS tells you not to test these on the emulator at all.
- Anything across accounts or Regions. Fidelity tests are verified in us-west-2, with global services in us-east-1.
- Operations the emulator has not built yet. These return HTTP 501 with the code NotImplemented, which is honest but easy for a retry wrapper to swallow.
- Scale. The DEV test hit a 64 MiB ceiling on a bucket and 1,000 messages on a queue.
- Throttling, service quotas, latency and outages. A matching response body says nothing about how AWS behaves under load.
A CI setup for AWS Emulator that survives the release rate
The setup below assumes a Debian-based runner. The package URL pattern comes from the versioning page, and the version shown is an example, not a recommendation.
#!/usr/bin/env bash
set -euo pipefail
# 1. Pin. Bump this in its own pull request, on a schedule.
EMULATOR_VERSION="1.0.489"
BASE="https://downloads.aws.com/aws-emulator/packages/aws-emulator/LTS"
curl -fsSLO "${BASE}/aws-emulator_${EMULATOR_VERSION}-1_amd64.deb"
sudo dpkg -i "aws-emulator_${EMULATOR_VERSION}-1_amd64.deb"
# 2. Record what ran, next to the test results.
aws-emulator -v | tee emulator-version.txt
# 3. No real credentials in this shell. A misrouted call must fail.
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN AWS_PROFILE
# 4. Telemetry is on by default. Decide, and write the decision down.
export EMULATOR_TELEMETRY=off
aws-emulator &
# 5. Refuse to run tests unless the endpoint is set explicitly.
# See "Pointing your AWS SDK at AWS Emulator" for the variables
# and the generated credentials and certificate.
: "${AWS_ENDPOINT_URL:?must point at the emulator}"The script handles the mechanical part. Four habits around it do the rest.
- Inventory before you trust. List the operations your code calls and look them up in manifest.jsonl in the evaluation suite repository. AWS ships an agent skill that does this for a repository and reports which parts you can rely on. An operation with no fidelity test is unverified, not broken, and it is the part that can change under you.
- Make 501 a hard failure. Check that no retry or fallback path converts NotImplemented into a skipped step or an empty result.
- Source assertions from AWS. For any behaviour not covered by a passing fidelity test, take the expected value from a run against a real account or from the service documentation, and say so in the test. Review agent-written tests for assertions copied from emulator output.
- Keep a small real-AWS stage. Permissions, cross-account paths, infrastructure-as-code applies and the handful of data plane behaviours your product depends on belong there. It can run nightly or before release. It cannot be zero.
If you run the fidelity tests yourself with the published Python runner, store the test-suite version alongside the emulator version. Two scores with different suite versions are not comparable, and that includes your own from last month.
What to watch next
Local AWS testing is not new. DynamoDB Local dates from 2013, and LocalStack, Moto and Floci have covered this ground for years. InfoQ's report notes the emulator is not a drop-in replacement for those tools on permission testing or infrastructure-as-code pipelines. What is new is a vendor publishing the failing tests alongside the passing ones, and a codebase of about 16 million lines of Rust that changes many times a day with no human commit.
The numbers worth tracking are not the overall percentage. Watch whether data plane scores for the services you use close the gap with control plane scores, whether the release rate settles into something a dependency bot can follow, and whether AWS starts backporting security fixes. Until then, the emulator is a fast first filter with a published list of what it gets wrong, and the pin in your CI config is what turns that list into something you can reason about.
Frequently asked questions
Related reading
Slack Classic Apps Deprecation Was Dated 16 November. Slack Paused It, and Migrating Still Changes Your Bot.
Slack's own changelog still says classic apps stop working on 16 November 2026. A notice on the same page says the change is paused. Here is which statement governs, and what migrating alters in your bot.
Postgres 14 Reaches End of Life on 12 November. On RDS, the Date That Bills You Is 1 March 2027.
The Postgres community stops patching version 14 on 12 November 2026. Amazon RDS keeps it in standard support until 28 February 2027, then charges. Here is how to read the gap.
.NET 8 and .NET 9 Lose Support on the Same Day, 10 November. Your Inventory Is the Hard Part.
Microsoft ends support for .NET 8 and .NET 9 on 10 November 2026. Changing TargetFramework is easy; finding the Dockerfiles, CI images, hosts and packages that still pin the old runtime is the real work.