1id1.net
Research paper · Preprint

The Limits of Structural Consistency: A Token-Space Audit of LLM Counting

1id1:core:20261108-0001

Abstract

Can exact mathematical consistency checks improve the diagnosis of language-model errors? We study finite counting questions through Token Space: source objects specify variables and conditions, maps specify answers, and kernel quotients compile companion questions into integer constraints. A violated constraint certifies that an answer vector has no common realizing structure without requiring reference facts. Passing the constraints does not certify factual correctness: every coherent alternative structure passes as well. We make this classical boundary executable and reassess completed experiments under their original endpoints. In a 32-target synthetic study, a twelve-question check rejected all five incorrect main answers but also six correct ones; twelve-answer repetition was correct on all targets. A later county-adjacency study admitted one model after 72/72 exact calibration replies. Names-only counts were exact on 24/45 requests, versus 45/45 with complete facts, but neither condition produced a stable incompatible vector. One domain yielded an incompatible first vector, already detectable from two coordinates, and two further vectors matching a coherent wrong graph. The audit separates repeated agreement, structural realizability and factual correctness, while the tested procedures establish no selective-answering advantage or stable incremental detection. The final factual evidence covers one admitted model and five selected domains, including one error-enriched domain; it does not establish general ineffectiveness.

Token SpaceLanguage model evaluationConsistency auditingNegative results

Cite this version

Wuming Pan. The Limits of Structural Consistency: A Token-Space Audit of LLM Counting. 1id1, v1, 8 November 2026. 1id1:core:20261108-0001@v1. https://1id1.net/id/core/20261108-0001/v1/

Files & integrity

FileSize
Paper PDF399 KB
LaTeX source archive750 KB
SHA-256 checksums
paper.pdf
9c0d57f94d3a7ede2d21daa726a2897ff43b073a270ea4e7b02d80a37ffe275b
source.zip
c45bfddf951a0f56ec7d61483f79192e841596ae265390e69b491003e5fe5160

Version history

v1Initial deposit

Initial public preprint corresponding to the V9 manuscript submitted to TMLR on 8 November 2026. Author attribution, the repository imprint and the data-availability statement differ from the anonymous submission; scientific claims and experimental results are unchanged. This deposit does not imply acceptance or completed peer review.

Provenance & verification

OpenAI Codex assisted with writing, revision, experiment-code development and offline evidence checks. Claude Fable 5.1 provided evaluative feedback on every intermediate manuscript version. These contributions are not independent peer review or experimental evidence; the author remains responsible for the content.

The public archive includes the eight-page technical supplement and typesetting sources. Raw model responses, experimental records and executable audit modules are retained in the separate supplementary package submitted to TMLR and are not included in this public archive. The submission package passed an offline audit of all 354 final-stage responses without new model calls. Results remain subject to the manuscript's stated sample and interpretation limits.