tls: "decryption failed or bad record mac" on an intact stream when a zlib stream backpressures an HTTPS response through an HTTP CONNECT proxy

[yx64]

Version

v24.21.0; Also reproduces on v20.20.2 - see the runtime matrix under Additional information.

Platform

Windows: Microsoft Windows NT 10.0.26200.0, x64
Linux:   Ubuntu-24.04 WSL2 guest on the same host, x64
---
Windows: Microsoft Windows NT 10.0.19045.0, x64

Subsystem

tls - the failure is in the TLS record read path. zlib is the trigger (the backpressure shape) and http is the layer that masks the error, but neither is where the fault is.

What steps will reproduce the bug?

An HTTPS response is piped through a node:zlib stream that applies backpressure to the
response, with the connection going through an HTTP CONNECT proxy:

const res = await get(url);                  // HTTPS through a CONNECT tunnel
await pipeline(
  res,
  countBytes,                                // plain Transform
  zlib.createZstdDecompress(),               // or createGzip() - both reproduce
  countBytes2,
  slowSink,                                  // anything that lags the consumer
  fileOrNullSink
);

Requirements, each with a control that turns it off:

• The body must be large enough to reach steady state (~80 MB observed; smaller was not
  characterised).
• The zlib stream must be directly downstream of res. Moving a lagging plain Transform
  in front of the zlib stream does not reproduce.
• The consumer downstream of zlib must lag enough to backpressure the response. Packet capture
  shows repeated TCP Zero Window segments on the client's receive side. Without the artificial
  lag the failure still occurs, but intermittently; the lag only raises the trigger rate to about
  100%.
• The relay in front of the client must be an HTTP CONNECT proxy; see "How often does it
  reproduce".

Self-contained script (Node standard library only, no third-party dependencies). It needs Node

= 22.15 for zlib.createZstdDecompress; substitute createGzip() on older versions:

// repro.mjs
//
//   node repro.mjs --url <large HTTPS artifact, ~80 MB or more> --proxy <http://host:port>
//   node repro.mjs --url <same> --proxy <same> --decode none      # control: no decompressor
//   node repro.mjs --url <same> --proxy <same> --drain-first      # control: no backpressure
import http from 'node:http';
import https from 'node:https';
import tls from 'node:tls';
import zlib from 'node:zlib';
import { Readable, Transform, Writable } from 'node:stream';
import { pipeline } from 'node:stream/promises';
import { parseArgs } from 'node:util';

const { values } = parseArgs({ options: {
  url: { type: 'string' },
  proxy: { type: 'string' },
  decode: { type: 'string', default: 'zstd' },
  lag: { type: 'string', default: '100' },
  'drain-first': { type: 'boolean', default: false }
} });

const target = new URL(values.url);
const proxy = new URL(values.proxy);

async function connect() {
  const req = http.request({
    host: proxy.hostname,
    port: Number(proxy.port || 80),
    method: 'CONNECT',
    path: `${target.hostname}:${target.port || 443}`
  });
  req.end();
  return new Promise((resolve, reject) => {
    req.once('connect', (r, socket) =>
      r.statusCode === 200 ? resolve(socket) : reject(new Error(`CONNECT ${r.statusCode}`)));
    req.once('error', reject);
  });
}

const tunnel = await connect();
const res = await new Promise((resolve, reject) => {
  const req = https.request({
    host: target.hostname,
    port: Number(target.port || 443),
    path: target.pathname + target.search,
    createConnection: () => tls.connect({ socket: tunnel, servername: target.hostname })
  }, resolve);
  req.once('error', reject);
  req.end();
});

let wire = 0;
const count = new Transform({ transform(c, _e, cb) { wire += c.length; cb(null, c); } });

// Consumer lag: hold the callback for a timer tick every 1 MB. This is what makes
// the response backpressure.
const lagMs = Number(values.lag);
let acc = 0;
const lag = new Transform({
  highWaterMark: 4194304,
  transform(c, _e, cb) {
    acc += c.length;
    if (acc < 1048576) return cb(null, c);
    acc = 0;
    setTimeout(() => cb(null, c), lagMs);
  }
});

const sink = new Writable({ highWaterMark: 4194304, write(_c, _e, cb) { cb(); } });

try {
  if (values['drain-first']) {
    // Control: consume the response with no backpressure, decompress afterwards.
    const chunks = [];
    await new Promise((resolve, reject) => {
      res.on('data', (c) => { wire += c.length; chunks.push(c); });
      res.once('end', resolve);
      res.once('error', reject);
    });
    await pipeline(Readable.from(chunks), zlib.createZstdDecompress(), lag, sink);
  } else {
    const stages = values.decode === 'zstd' ? [zlib.createZstdDecompress()] : [];
    await pipeline(res, count, ...stages, lag, sink);
  }
  console.log(`OK wire=${wire}`);
} catch (err) {
  console.log(`FAIL code=${err.code} msg=${JSON.stringify(err.message)} wire=${wire}`);
  console.log(`res.complete=${res.complete} res.aborted=${res.aborted}` +
    ` res.readableEnded=${res.readableEnded} res.destroyed=${res.destroyed}`);
}

Run each configuration several times, and run a passing configuration in the same minute: the
failure is intermittent, so a single run proves nothing in either direction.

How often does it reproduce? Is there a required condition?

With the consumer lag in place, essentially every run fails - the transfer stops at
99.76% - 99.96% of the declared body. Without the lag it fails intermittently, which is why
every measurement in this report was paired with a same-minute positive control.

Three conditions must hold at once:

1. A node:zlib stream directly downstream of the response.
2. Backpressure from the consumer reaching the response through that zlib stream.
3. A relay that re-chunks the stream with a particular write-size distribution - see
   Additional information.

Condition 3 is the one that is not reproduced without a real relay. The reproduction below
still needs one, and this issue does not claim any specific proxy is required - only that a
third-party relay is currently the only thing that triggers it dependably on this path. A fully
local rig (self-signed TLS origin on loopback) never reproduces.

What is the expected behavior? Why is that the expected behavior?

A complete, correctly transmitted TLS stream must decrypt regardless of how the application
applies backpressure. Backpressure is an application-level property and cannot change the bytes
on the wire - and in this case the bytes were proven intact offline (see Additional
information). Failing to decrypt them is a defect in the reading side.

Secondarily, the HTTP client should not translate a TLS failure into aborted /
ECONNRESET: that rewrites the root cause into a network error and is why applications
downstream blame the proxy or the network.

What do you see instead?

ERR_SSL_DECRYPTION_FAILED_OR_BAD_RECORD_MAC
error:0A000119:SSL routines:tls_get_more_records:decryption failed or bad record mac

followed by a fatal bad_record_mac alert being sent to the peer.

Event order, from listeners attached to every object in the chain:

21.153 tlsSocket.error  ERR_SSL_DECRYPTION_FAILED_OR_BAD_RECORD_MAC
21.153 rawSocket.close
21.153 tlsSocket.close
21.154 res.aborted
21.154 res.error        ECONNRESET: aborted
21.155 (pipeline tears down every downstream stage)

The TLS error is first; everything after it is a consequence.

What the application sees, because Node's HTTP client observes the socket closing while
res.complete === false and replaces the error with connResetException('aborted'):

code=ECONNRESET  msg="aborted"
res.complete=false  res.aborted=true  res.readableEnded=false  res.destroyed=true

Additional information

The ciphertext on the wire is intact

The inner TLS stream (the CONNECT tunnel on loopback) was captured in full, and the session keys
were exported from the failing run with --tls-keylog. Decrypting the capture offline with
those keys: 8,491 application-data records, 80,603,843 bytes, zero decryption failures. The
same bytes, processed in-process by the running Node client at the same moment, produced a MAC
failure.

Commands used:

# capture the inner TLS stream, full packets
tshark -i <loopback interface> -f "tcp port <proxy port>" -w <capture> -a duration:70

# export session keys from the failing run
node --tls-keylog=<keylog> repro.mjs ...

# any record that failed to decrypt would appear in expert info
tshark -r <capture> -o tls.keylog_file:<keylog> -d tcp.port==<proxy port>,http \
       -Y "tcp.stream==<n> && _ws.expert" -T fields -e frame.number -e _ws.expert.message

# and the decrypted application-data volume must match expectations
tshark -r <capture> -o tls.keylog_file:<keylog> -d tcp.port==<proxy port>,http \
       -Y "tcp.stream==<n> && tls.record.content_type==23" -T fields -e tls.record.length

--export-objects http,<dir> is not usable here: it silently exports nothing when the response
body is large.

Platform / TLS-library / runtime matrix

Same proxy, same origin, same artifact, comparable consumer lag. Every row was run with a
same-minute positive control, because the failure is intermittent. The percentage is where the
transfer stopped, not a failure rate.

Runtime	TLS library	Event library	Platform	Result (stop point, % of declared body)
Node 24.21.0	OpenSSL 3.5.8 (bundled)	libuv 1.52.1	Windows 11 x64	FAIL 99.963%
Node 24.21.0	OpenSSL 3.5.8 (bundled)	libuv 1.52.1	Linux (WSL2)	FAIL 99.760%
Node 20.20.2	OpenSSL 3.0.19 (bundled)	libuv 1.46.0	Linux (WSL2)	FAIL 99.963%
Python 3.12.3	OpenSSL 3.0.13 (system)	-	Linux (WSL2)	OK, correct hash

The two Node failures start with the same error text but go through different OpenSSL internals:

Node 24 / 3.5.8    tls_get_more_records   tls_common.c
Node 20 / 3.0.19   ssl3_get_record        ssl3_record.c

So: not platform-specific (the Windows overlapped-I/O read path is out), and not an OpenSSL
version regression (3.0.19 and 3.5.8 both fail, via different code). The difference that matters
is the runtime - same OpenSSL major, Python completes and Node does not.

Mechanism hypothesis (narrowed, not proven)

Python lets OpenSSL read the socket itself; Node pushes TCP chunks into a memory BIO and lets
OpenSSL read from there. If Node reuses, truncates or misaligns the buffer it feeds to that BIO
across a backpressure pause/resume, the result is exactly bad record mac. This fits every known
condition: needs backpressure (for the pause/resume), needs a relay (so records straddle read
boundaries), platform-independent, OpenSSL-version-independent.

Caveat: Python's read shape (blocking read plus sleeps) is not identical to Node's stream
backpressure, so its pass shows the failure is not universal to TLS clients under backpressure -
it does not prove that no other read shape can drive OpenSSL into this.

The relay's write-size sequence

Failing and passing paths were compared by recording the size of every downstream write the relay
made:

	failing relay	passing relay (ssh -L)
writes	1461	1295
smallest write	616	2608
writes <= 16 KB	14.0%	2.4%
writes <= 4 KB	6.6%	0.4%
common small sizes	2824 / 5648 / 8472 (multiples of 1412)	rare
inter-arrival p50 / p95	0.152 / 16.95 ms	0.175 / 12.56 ms

Inter-arrival timing is nearly identical; the measured difference is concentrated in write
sizes. The failing relay is a local HTTP CONNECT proxy (mihomo) forwarding to a remote egress;
the passing one is an ssh -L tunnel to the same egress host.

Replaying the recorded sequence through a purpose-written CONNECT proxy on the passing path does
reproduce the failure - but only sometimes, so write sizes are a relevant characteristic, not
a sufficient one:

Synthesised	Result
Recorded write-size sequence	2 failures out of 7 attempts
Write sizes plus recorded inter-arrival gaps	0 out of 4
Either of the above plus unbounded read-ahead	0 out of 2
Local TLS origin forced to emit small (2824-byte) records	0 out of 1
Uniform chunking at 16384 / 4096 / 1412 bytes, with TCP_NODELAY	0 out of 3

The recorded median inter-arrival gap is 0.152 ms. JavaScript timers cannot reproduce
sub-millisecond arrival structure, so anything below 1 ms degrades to setImmediate. That is a
tooling ceiling here, not a matter of trying more patterns; going further would need
packet-level shaping or pcap replay.

What these attempts do establish is that no data tampering is involved - a relay that merely
re-chunks a correct byte stream can trigger it at least sometimes.

The stop point

The failure always lands very close to the end, and repeatedly on exactly the same offset -
including across two different upstream egress servers in different regions running
different proxy-server implementations:

bytes delivered into the pipeline: 80,428,448
bytes emitted by the decompressor: 222,453,760

Across all runs the stop point takes a small number of discrete values in the range
99.76% - 99.96% of the body rather than a continuum. That suggests fixed client-side buffer
boundaries rather than anything network-timing-dependent, but the value is not constant, so it
should not be read as a single magic offset.

Ruled out

Each item has a controlled experiment behind it.

Hypothesis	Verdict	Control
The proxy corrupts bytes	No	Offline decryption of the captured ciphertext with --tls-keylog keys: all records decrypt, no failures. Same trigger conditions over plain HTTP at comparable throughput: 104,857,600 bytes, SHA-256 identical to baseline, while a same-minute HTTPS positive control failed. Inserting a purpose-written proxy in front of it does not neutralise the failure.
That specific proxy is strictly required	Unresolved	Replaying its recorded write-size sequence through a purpose-written proxy on a path it is not part of reproduced the failure 2/7. Enough to show no tampering is needed; not enough to call it unnecessary.
Windows-specific I/O read path	No	Same build reproduces on Linux.
OpenSSL version regression	No	OpenSSL 3.0.19 and 3.5.8 both reproduce, through different internal code paths.
TLS records being small	No	A local origin forced to emit 2824-byte records does not reproduce.
Data expansion by decompression	No	createGzip() on already-compressed input (in ~= out) reproduces.
Slow consumer alone	No	A plain Transform with the same total lag and the same backpressure completes.
Per-chunk async deferral	No	A Transform deferring every chunk via setImmediate, same total lag, completes.
libuv threadpool contention	No	UV_THREADPOOL_SIZE of 1 and of 32 both reproduce.
Backpressure on res	Required	Draining the response fully into memory first, then decompressing with the same lag downstream, completes with a correct hash.
A zlib stream in the chain	Required	Same lag and same backpressure; the only difference is whether that stage is zlib.
Ordering	Matters	Putting the lagging plain Transform before the zlib stream does not reproduce. Backpressure must reach res through the zlib stream.
Going through the proxy	Required	The identical configuration over an ssh -L tunnel to the same egress host completes with a correct hash.

Workaround

Do not place a streaming decompressor directly downstream of the HTTP response. Drain the
response without backpressure, then decompress. Verified on the same path with the same
decompressor and the same total lag: correct output hash.

Not yet tested

• macOS
• Bare-metal Linux (the Linux result above is from WSL2; a positive there is still decisive, but
  the kernel is a vendor fork behind a virtual NIC)
• A non-OpenSSL runtime such as Go, which would tell whether any TLS implementation can be driven
  into this by the same relay
• HTTP/2 (the failing path is HTTP/1.1 inside the tunnel)
• Which relay characteristic is actually required (see the synthetic-relay table above)