Skip to content

Fixes against real-world usage: Bounded literal reads, LOGIN tag isolation - #141

Open
Pzixel wants to merge 2 commits into
chatmail:mainfrom
Pzixel:outreach/0.11.3-patches
Open

Fixes against real-world usage: Bounded literal reads, LOGIN tag isolation#141
Pzixel wants to merge 2 commits into
chatmail:mainfrom
Pzixel:outreach/0.11.3-patches

Conversation

@Pzixel

@Pzixel Pzixel commented Sep 2, 2026

Copy link
Copy Markdown

Two changes that turned out to be required in real-world use against a large fleet of mailboxes behind an IMAP proxy, some of them on misbehaving servers.

  • Bounded response buffers and bounded literal reads. ImapStream parses a response only once the whole server-declared literal is in memory, so one FETCH BODY[] of a huge message, or a server that announces a bogus literal size, grows the buffer without limit and stalls the connection. Running about a thousand concurrent sessions, that was both a memory hazard and a way for a single mailbox to block a worker. Client::new_with_max_response_size caps the parse buffer; read_response_with_literal_prefix reads one response while keeping only a bounded prefix of an oversized literal, reports the declared size, and closes the response stream so the caller can record the message as truncated and move on. LiteralAwareResponse / LiteralPrefix carry the outcome; MockStream::with_max_read_size exists to test partial reads.
  • LOGIN completion matched by its own tag. login treated the first tagged status it saw as its result, whatever the tag. Some servers (and proxies) deliver a stale tagged NO for an earlier command first; the client then reported an authentication failure for credentials that were fine, and the mailbox was marked as failing auth. Only the completion whose tag equals the LOGIN request's tag is considered now; others are ignored.

Carried over from the email-stats vendored copy of 0.11.3:

- ImapStream::new_with_max_response_size and
  read_response_with_literal_prefix: read one response while retaining only
  a bounded prefix of a server-declared literal larger than the limit; a
  capped outcome closes the response stream. New LiteralAwareResponse and
  LiteralPrefix types carry the outcome.
- LOGIN completion matching only considers the tag of the LOGIN request;
  completions for other tags are ignored (test added).
@Pzixel
Pzixel force-pushed the outreach/0.11.3-patches branch from 2da4626 to 1487607 Compare September 2, 2026 12:57
@Pzixel Pzixel changed the title Fixes against real-world usage: Bounded literal reads, LOGIN tag isolation, public internals Fixes against real-world usage: Bounded literal reads, LOGIN tag isolation Sep 2, 2026
Collapse the nested `if let` in `login` and in the literal-prefix
decode path into let-chains, and reformat the affected block.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@link2xt

link2xt commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Some servers (and proxies) deliver a stale tagged NO for an earlier command first

Are you using LOGOUT command, reusing the same connection to LOGIN and then receive responses for requests that were pipelined earlier? async-imap does not really support pipelining, could it be that you have started some command and then dropped the future reading from a stream and this is why you did not read this response? Otherwise I don't see how you can receive a response with mismatching tag.

If it is a proxy or server bug and it somehow sends responses with tags that were never requested or duplicates them, could you report this to https://github.com/modern-email/defects/issues with all the details such as the version of the proxy/server that is doing this?

I have extracted this fix into #142 for easier review since it is basically a one-line change.

@Pzixel

Pzixel commented Sep 2, 2026

Copy link
Copy Markdown
Author

Answering directly: no pipelining, no LOGOUT + reuse, and no dropped read future can reach LOGIN. Every session is a fresh TCP+TLS connect, greeting, LOGIN, and any timeout or error poisons the session (we drop the transport, never reuse it), so a command whose completion we didn't read can't leave a leftover tag for a later one. LOGIN is the first tagged command on that socket anyway.

The source was a third-party IMAP proxy we don't control, which we've since blocked outright. I don't have a wire trace saved

Still worth fixing on this side, since if *tag == id was already there, check_status_ok just sat outside it, so login judged a response it had already decided wasn't its own.

#142 is fine, please land it. Note that here is another change in this PR for bounded literal reads: at ~1000 concurrent sessions doing UID FETCH ... BODY.PEEK[] the stream buffers the entire declared literal before parsing, so one huge message blows the worker's memory budget. I had servers (again outside our our control) reporting 500mb or even gigabyte messages that would be just 10kb in reality.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants