Anonymous health checks on 78 registry MCP servers: 51.3% complete the full call sequence

A dated, reproducible health probe of 78 registry-listed MCP servers.
Pennyforge Studio · 2026-10-05 · second dated point 2026-10-06 · repro + raw data

The Model Context Protocol ecosystem is growing fast: thousands of servers are listed in
public registries, SDKs are downloaded tens of millions of times a month, and the protocol
just went through (2026-07-28) its most substantial revision since launch. But almost nobody
has measured what actually happens when a client tries to use one of these servers without
an account, a token, or prior knowledge.

We did, on a bounded cohort, with everything logged.

What we did

• Cohort (n=78): the endpoints from one public MCP registry's alphabetical a–b slice that answered an initialize call on 2026-10-02. (Of 186 listed in that slice, 78 answered initialize; 87 gave HTTP 401, the rest errored or timed out — the "listed" population is already smaller than the registry suggests.)
• Happy path per server: initialize → tools/list → one safe tools/call: the first tool whose declared inputSchema has no required properties and whose name is read-style (get/list/search/…), called with empty arguments. 12-second caps, 8 parallel workers, ~20 minutes wall clock, $0.
• Version probe (2026-10-05, same cohort): a second initialize requesting the 2026-07-28 protocol version, to see which spec revision each server actually speaks.
• Re-probe (2026-10-06, same cohort): the version probe re-run, plus the 2026-07-28-revision wire-form test and the deprecation-exposure probes (server/discover, sampling/createMessage, roots/list, logging/setLevel) — the second dated point in §6½.
• All probes anonymous (no accounts, no keys, one declared User-Agent).

What we found

1. 40 of 78 (51.3%) complete the full anonymous happy path

HEALTHY 40 · CALL-ERR 19 · LIST-ONLY 15 · TOOLS-LIST-ERR 2 · INIT-ERR 2.

"Listed in a registry" is not "works anonymously." A little under half of the servers
that answer at all let an anonymous client complete a real tool call.

2. Tiered auth is the silent failure mode (19.2% gated somewhere in the pipeline)

13 servers let anonymous clients initialize and tools/list — then require
a bearer token, API key, personal link, OAuth sign-in, or an account at the
tools/call stage. Two more gate even at tools/list. In total 15/78 (19.2%)
are gated somewhere in the pipeline, and the gate is invisible until you try to call.

The credential types we observed: 6× bearer-token, 2× api-key, 1× OAuth sign-in,
1× personal-link, 3× account-signup (one operator, three endpoints), plus two generic
401s that don't say what credential they want. One dated price point from the errors:
a "free" probe key at $19 / 30 days. One server was running on its operator's own
expired API key — it 401s everyone, including its author.

3. Schema honesty: declared required: [] ≠ callable with {}

6 servers (7.7%) declare a tool with no required properties, then demand arguments in
the error text ("property_coverage requires address OR both latitude and longitude",
"need is required when research_id is absent"…). A careful client-side analysis found
5 more endpoints where the "other" failures were the same shape — input the schema
didn't declare as required. If your agent framework trusts inputSchema.required,
it fails systematically against these.

4. Registry hygiene: 2/78 are broken listings

• One registry entry ships an unexpanded URL template: https://mcp.biel.ai/sse?project_slug={project_slug}&domain={domain} → Cloudflare 530 "Origin DNS error" for everyone, including the registry's own users.
• One serves its HTML marketing landing page at /mcp with HTTP 200 to an initialize request.

5. Latency is not the problem

init p50 453 ms, tool-call p50 468 ms, p90 1.8 s. The servers that work, work fast.
The ones that fail, fail at auth — not at speed.

6. Spec-version adoption: the ecosystem is de facto stateless, de jure behind

Requesting the 2026-07-28 revision (finalized three months ago), servers answer with the
newest revision they support:

Three months after the stateless revision shipped, only 9% of answering servers have
migrated to it — while 88.5% (69/78) never issue the Mcp-Session-Id header at all,
even though the revisions they declare still describe sessions. The spec makes session IDs
optional (MAY), so nobody is violating anything — but the ecosystem has already
converged on stateless operation without declaring the revision that makes it official.
(One operator had all three of its endpoints on 2026-07-28; one server declared
2026-07-28 while still requiring a bearer token at call time.)

Note on method: our 2026-10-02 probe requested the 2025-06-18 version, so by negotiation
semantics it under-reports newer versions (a server answering "2025-06-18" may support
more). The 2026-07-28 request reveals each server's true maximum. The 10-02 probe did
catch genuinely ancient servers (2× 2024-11-05, 5× 2025-03-26) that still report the same
floor on 10-05.

6½ Two days later (2026-10-06) — the second dated point

We re-ran the version probe on the same 78 endpoints two days ahead of the originally
planned follow-up, adding two measurements the first run lacked: (i) a second
initialize in the 2026-07-28 revision's own wire form — version declared per-request
in params._meta plus the MCP-Protocol-Version header, with params.protocolVersion
absent — and (ii) probes for the capabilities the 2026-07-28 revision deprecates or
standardizes.

Adoption is flat, the cohort is churning. 7 of 71 answering servers (9.8%) negotiate
2026-07-28 — the same 7 as on 10-05, no migrations in either direction. But 5 endpoints
(6.4% of the cohort) simply went down in 48 hours: DNS/TLS dead, no error page. A
registry-listed MCP endpoint's two-day survival rate, as of this writing: 93.6%.

One spec, two wire forms — and the two disagree. The 2026-07-28 revision moved
version declaration into the per-request _meta field (plus the
MCP-Protocol-Version header on HTTP). But the dominant TypeScript SDK's request schema
still requires params.protocolVersion as a string. Measured per server: 34 of 71
answering servers (47.9%) reject the new revision's own canonical wire form — most
with the exact validation error params.protocolVersion: expected string — while 37
(52.1%) accept it. At this date, being spec-conformant to the new revision and
maximally compatible with the installed base are mutually exclusive.

Of the 7 servers actually running the 2026-07-28 revision, 4 accept the canonical form
and negotiate it in full; the other 3 accept the form but negotiate down (two to
2025-06-18, one to 2024-11-05) — their _meta handling silently caps the negotiated
revision below what their handshake answers. All 7 still accept a plain 2025-11-25
handshake, so old clients keep working; it is the new form that is not yet
interoperable.

Deprecation is declared, not felt. Of the 71 answering servers, 19 (26.8%) still
expose sampling/createMessage, 18 (25.4%) still expose roots/list, and 14 (19.7%)
still expose logging/setLevel — all three deprecated in the 2026-07-28 revision
(SEP-2577) — while only 20 (28.2%) expose the new revision's mandatory server/discover.
Anonymous completion is rarer still: 0 servers complete a sampling round-trip, 0 complete
roots/list, 1 accepts logging/setLevel anonymously, and exactly 1 completes
server/discover anonymously. The deprecated surface is still the working surface.

The old transport is effectively extinct. 72 of 77 reachable endpoints (93.5%)
expose Streamable HTTP-style paths; the one remaining /sse-style URL is the broken
unexpanded-template listing from above. The deprecated HTTP+SSE transport is dead in
this cohort — silently, with no deprecation notice anywhere.

7. The error surface has no standard

The same underlying condition — "you need credentials" — surfaced in at least six
different shapes across the cohort: HTTP 401 with a JSON body (8×), plain-text error
inside a successful HTTP 200 tool call (9×, of which 7× as JSON is_error content, 2×
inside SSE), JSON-RPC error objects with custom codes (-32001/-32002/-32602, 4×),
a Cloudflare 530 page, and an HTML page. An agent framework that wants to recover
gracefully ("ask the user for a token") has to parse all of these. There is no
MCP-level error vocabulary yet.

What already exists (and what this is not)

Per-server security scanners for MCP do exist as of this date: Cisco AI Defense's
mcp-scanner (Apache-2.0, PyPI, active — YARA + LLM + their inspect API, plus
dependency-CVE and "production readiness" static analysis), Snyk's agent-scan
(covers MCP servers), Tencent's AI-Infra-Guard (red-teaming platform with an MCP scan),
and smaller OSS efforts. Those tools scan a server you are about to install.
What did not exist, to our knowledge, before this post: a dated, cohort-level,
publicly reproducible health table of the registry-listed servers themselves —
"as of 2026-10-05, of the 78 answering endpoints in this cohort, 51.3% complete the
anonymous happy path, and spec-version adoption is 52.6% / 9.0% / …" — with the probe
script, raw responses, and a second dated point (2026-10-06) included below. The security
scanner answers "is this server malicious?"; this answers "does the ecosystem
work, and is it migrating to the new spec?" The two are different artifacts, and the
second one is a recurring measurement, not a one-off scan.

What a registry could do about this (the top 3)

1. Record the credential type behind each gate and label the entry "requires bearer token / API key / OAuth" instead of letting it look "broken".
2. Expand or drop unexpanded URL templates before listing; content-type-check the initialize response so HTML pages and CF 530s never become live entries.
3. Distinguish missing-input errors from credential gates — the two are different fix owners (server author vs. registry policy) and currently look identical to the client.

Limitations (read before quoting numbers)

• One registry's alphabetical a–b slice, n=78, single day, single call per server, one "safe" tool per server. The direction (about half complete the happy path; tiered auth is the top silent failure) is robust; the exact percentages are cohort-specific.
• The cohort excludes servers that were down on 2026-10-02 (87× 401 + 15 other errors in the 186 listed) — so "51.3% of answering servers" is the honest denominator, not "51.3% of listed servers".
• "Safe call" = no required schema args + read-style name + empty arguments. A stricter client would fail more; a smarter one (filling optional args) would pass more.
• Anonymous probes only; servers that are fine for a paying user count as gated here.
• We probed public endpoints at low frequency with a declared UA; treat as a courtesy benchmark, not a load test.
• The §6½ wire-form result measures one concrete encoding of the 2026-07-28 declaration (params._meta + MCP-Protocol-Version header, no params.protocolVersion). Servers may accept other conformant encodings we did not send; "rejects the canonical form" is the honest phrasing, "rejects the revision" is not. The 2026-10-06 re-probe used a legacy-handshake initialize for the adoption table, so the two dates are directly comparable on revision adoption but not on the wire-form column.

Reproduction

• Harness: .tmp-sweep/r165/mcp-health-harness.py (happy path, no deps, stdlib only), .tmp-sweep/r165/mcp-version-reprobe.py (10-05 version probe), and .tmp-sweep/r165/mcp-reprobe-1008.py (10-06 re-probe: version adoption + wire-form test + deprecation-exposure probes, stdlib only)
• Raw data: data/artifacts/mcp-health-harness-2026-10-05/ (results.json, version-reprobe-2026-10-05.json, reprobe-2026-10-06.json, report.md, reprobe-report-2026-10-06.md, per-server-table.md)
• Cohort source: the 2026-10-02 registry-slice matrix (186 endpoints)
• Run cost: $0 (public endpoints; ~300 requests on 10-05, ~600 on 10-06, ~25 + ~1 min wall clock)

Pennyforge is a small one-person studio. We have no commercial interest in any server
probed; all endpoints were discovered through the public registry listing.