A security test that fails one run in four is worse than no test
Decided
tamper with an encryption envelope by flipping bits in the decoded bytes, never by editing a base64url character. Two tests in the AES-256-GCM suite — "refuses an envelope whose ciphertext has been altered" and "refuses an envelope whose authentication tag has been altered" — mutated the envelope by replacing the final base64url character, "A" for anything else and "B" for an "A". That is not a mutation. base64 packs six bits per character, so when a payload's byte length is not a multiple of three, the final character carries two or four low-order bits that decode to nothing. Both fields land in exactly that case: the test's ciphertext is 22 bytes and a GCM tag is always 16, and 22 mod 3 = 16 mod 3 = 1, which is the four-dead-bit case. Whenever the original final character and its replacement agreed on their two significant bits, the "altered" envelope decoded to byte-for-byte the original, open() correctly succeeded, and the assertion failed with Missing expected exception.
Affects: backend/test/integrations.test.ts Supersedes nothing. Fixes a defect introduced with D-012's test suite.
Two tests in the AES-256-GCM suite — "refuses an envelope whose ciphertext has been altered" and "refuses an envelope whose authentication tag has been altered" — mutated the envelope by replacing the final base64url character, "A" for anything else and "B" for an "A". That is not a mutation. base64 packs six bits per character, so when a payload's byte length is not a multiple of three, the final character carries two or four low-order bits that decode to nothing. Both fields land in exactly that case: the test's ciphertext is 22 bytes and a GCM tag is always 16, and 22 mod 3 = 16 mod 3 = 1, which is the four-dead-bit case. Whenever the original final character and its replacement agreed on their two significant bits, the "altered" envelope decoded to byte-for-byte the original, open() correctly succeeded, and the assertion failed with Missing expected exception.
Measured over 20,000 trials: 25.0% of runs produced a no-op mutation. The byte-level flip now used produced 0 in the same 20,000.
Why this is worth an entry rather than a one-line fix. The failure mode is not the flake, it is what a flake does to a team. This test is the one that asserts a database attacker cannot alter a stored OAuth refresh token and have it decrypt to something a downstream service will send to QuickBooks as a bearer token. If it reddens a quarter of the time for no reason anyone can reproduce, the rational response — re-run CI — is also the response that would ignore it on the day the property genuinely broke. An unreliable assertion about a security property is strictly worse than no assertion, because it consumes the attention that would otherwise notice its absence.
The alternative considered and rejected: deleting the two tests as low-value, on the grounds that GCM's authenticity guarantee is the library's to uphold rather than ours. Rejected because what these tests actually pin is *ours*: that open() propagates the failure instead of catching it and returning a partial or empty string, which is a mistake this codebase could make in a future refactor and the library could not stop.
This would be wrong if the envelope format changed such that field lengths became multiples of three, at which point the old character-flip would have been correct all along. That is not a reason to go back — a test whose correctness depends on a payload's length modulo three is a test waiting to break again.