BIP-39: say what separates two words

[fametrano]

The BIP never says it. The rule exists — the Japanese wordlist page requires
the ideographic space U+3000, added in #130 — but it is not in the BIP text,
and the note there holds a caveat: ASCII and ideographic spaces are
equivalent "as long as your code never shows the user an ASCII space
separated phrase or tries to split the phrase input by the user".

The reference implementation splits the input, on U+0020, and so cannot read
back the Japanese sentences it writes: to_entropy raises on all 24
Japanese vectors of its own vectors.json. The fix of the reference
implementation is open there: trezor/python-mnemonic#145

This adds three sentences to "From mnemonic to seed": the separator, the
Japanese exception, and what it means for software that splits a sentence
before normalizing it. No test vector changes, and nothing that is correct
today becomes incorrect.

Found downstream in btclib: btclib-org/btclib#258.

🤖 Generated with Claude Code

[fametrano]

If this lands and there is interest, I am happy to write the follow-up on the
other half of the question: what a reader should do with whitespace that is
not the separator — a leading or trailing space, the line break of a paper
backup wrapped over two lines, a doubled space from a copy-paste.

The BIP mandates NFKD and says nothing more, so implementations differ and the
failure mode is quiet: to_seed hashes whatever it is given, so a sentence
that a checksum check would reject still derives a seed — a valid-looking
wallet that is not the user's. Two answers look defensible: refuse anything
but the canonical single-space form, or collapse any run of whitespace after
NFKD. No test vector changes under either; btclib, which I maintain, collapses.

I left it out of this PR because it is new normative text rather than a
clarification of what the wordlist page already requires, and it should not
hold up the separator sentence. Glad to open it separately if the authors want
it, or to drop it if the view is that a Deployed BIP should not grow new
requirements.

[murchandamus]

cc authors: @prusnak, @ebfull, @slush0, @voisine

[prusnak]

On several occasions I raised my will that we should drop non-English wordlists from BIP39, but I was always blocked by editors.

Since I am not able to edit my own standard, I don't really care about issues related to non-English use.

[fametrano]

prusnak I do understand your position, and I agree with you about English only.

That said, I do care about fixing inconsistencies, and I would love it if your "I don't really care" could become an ACK.

Also, the possible follow-up — what to do with whitespace that is not the separator, a leading or trailing space and so on — concerns English too, not just Japanese.

[fametrano]

@murchandamus a month has passed since the authors were cc'd. One of them, @prusnak, has answered: not an objection, but not an ACK either, and scoped to non-English use; the other three have not replied. Is a non-objection from an author enough to move a clarification like this forward under the "Pending acceptance" label, or does it need an explicit ACK? Happy to wait longer if that is the process, I just want to know what the thread is waiting on.

[optout21]

A few shallow thoughts on this:

• I'm not in favor of any change to this BIP related to non-English languages, as the industry consensus seems to be that the spec is sub-optimal and it's not worth supporting.

• In a comment you mention trailing white space. I think this is an important point, as any white space issue (leading/trailing white space, duplicate separators, non-space white space) can have significant effect -- a different wallet derived depending on the white space handling!

• As generally BIP's are not versioned, I find it dangerous modifying in a way which can make a compliant implementation non-compliant, or, if an implementation is changed to be compliant and that results in a different wallet for the same input. I think such changes should be addressed only a follow-up BIP.

• What I can imagine to be changed in this BIP is to document the deficiency, and add a clarification that the current form of the BIP does not specify the handling of the mentioned white space issues. A clarification only, not a change to the mandated rules.

Nonetheless, if the industry standard is white space normalization (before hashing), then this could be reflected in the spec, but this is non-trivial to assess.

[fametrano]

We agree on the principle, and I want to say that first. A BIP that is already widely implemented should not be changed in a way that makes a conforming implementation non-conforming, and much less in a way that derives a different wallet from the same input. If this PR did either, I would withdraw it. I think it does neither, and the PR description did not make that clear enough.

No input derives a different wallet. The seed is PBKDF2 over the NFKD of the sentence, and NFKD maps U+3000 to U+0020, so the two ways of writing a Japanese sentence already agree today. Taking the standard all-zero-entropy Japanese vector:

あいこくしん ×11, then あおぞら     (12-word all-zero-entropy Japanese vector, U+3000 separators)

seed, U+3000 separators: 646f1a38134c556e948e6daef213609a62915ef5…
seed, U+0020 separators: 646f1a38134c556e948e6daef213609a62915ef5…

Both forms also pass the checksum. This PR does not touch derivation at all.

What the paragraph is about is splitting a sentence into words, which is what an implementation does to look the words up and to verify the checksum. Split on U+0020 before normalizing and that sentence is 1 word; split after normalizing and it is 12. So the text can change whether an implementation accepts a sentence, never which wallet it derives.

And it is not a new rule. bip-0039-wordlists.md, linked from this BIP, already states it in bold:

Developers implementing phrase generation or checksum verification must separate words using ideographic spaces / accommodate users inputting ideographic spaces.

and then explains that after normalization "dealing with ASCII or Ideographic space is the same". The body of BIP-39 never mentions the separator at all. So the paragraph lifts an existing "must" out of a linked file and into the text implementers actually read.

On your first point, the paragraph does not change anything about non-English wordlists either — it repeats the rule the wordlists file already sets.

So the question I would put back to you: with the above, does this fall within the clarification you say in your fourth point you could accept — the deficiency documented, no mandated rule changed? If not, I would rather fix the wording than argue the point: what would you need it to say?

[optout21]

You are correct, the change in its current form does not affect wallet derivation. I was carried away by the mention of trailing spaces, but that's not addressed in the current PR. My comments were general, but not valid here, my apologies!

The proposed clarification is being added to the "From mnemonic to seed" section, but I don't think it belongs there. The section describes the NFKD Unicode normalization before PBKDF2.
Also, I don't think the spec document itself is the right place for a rule specific to single language, the bip-0039/bip-0039-wordlists.md seems better suited for that.

Looking at the "Japanese" section of bip-0039/bip-0039-wordlists.md, it mentions about separators, the ideographic space, and how it is converted to ASCII space through NFKD. The description there (although not as clearly written as possible) implies the rules for separation, verification, and generation. What extra detail is being added by the proposed change? I see at most the simple rule that words are joined using a space character, the rest is all present already or implied.

I think a better place to mention the space separator would be after the sentence:
"Finally, we convert these numbers into words and use the joined words as a mnemonic sentence."
"Words are joined using a single space character (for non-English language exceptions see [[bip-0039/bip-0039-wordlists.md]] Wordlists (Special Considerations))."

[fametrano]

Thank you for the detailed follow-up — the placement note is fair, and I've moved the paragraph. The seed section covers NFKD before PBKDF2; this is about how a sentence is written and read, so it now sits right after the sentence you named.

On content, I'd keep one clause beyond your version, and the reason is in the file you linked. It does address splitting, but only as a precondition:

as long as your code never shows the user an ASCII space separated phrase or tries to split the phrase input by the user, dealing with ASCII or Ideographic space is the same

Verifying the checksum requires recovering the individual words, and in practice that means splitting the sentence — so that precondition does not hold for an implementation that checks a mnemonic it was given. What is left unsaid is the order splitting happens in. Split a Japanese sentence on U+0020 before normalizing and the whole sentence reads as one token; normalize first and it splits into the individual words, since NFKD maps U+3000 to U+0020. That order is the detail I think is missing rather than implied.

What I've pushed is your sentence plus that clarification:

Words are joined using a single space character (U+0020), except in Japanese, where the ideographic space (U+3000) is used; see [[bip-0039/bip-0039-wordlists.md|Wordlists (Special Considerations)]]. A sentence is split back into words after NFKD normalization, which maps U+3000 to U+0020; splitting on U+0020 before normalization reads a Japanese sentence as a single word.

Two sentences, in the section you chose. If you'd rather the second lived in the wordlists file instead of the BIP, I'm happy to move it — the order is what matters to me, not where it's written.

[optout21]

I think it has improved, thanks! I still have some comments:

• The first sentence is fine with me, although I would prefer to leave out the concrete details here, but I can accept the current form as well.
• The second sentence is about verification, that doesn't belong to the Generation section. Unfortunately, the document doesn't have a Verification section (that could be improved...), the closest is this paragraph in the "From mnemonic to seed" section:
  "Although using a mnemonic [...] issue a warning if it is invalid."
  I would propose moving the second sentence after this paragraph.
• In general, details are good in a spec, but redundancy is not, as it violates the single source of truth principle (and it also makes future updates riskier). I feel the current form still repeats too much. I would formulate something like
  "When splitting a mnemonic string into words for validation, the rules regarding word separators laid out above (in the "Generating" section) should be observed."

[fametrano]

Both moves make sense; I've pushed them.

The separator rule now appears once, in "Generating the mnemonic", in the form you proposed. The second sentence has moved where you suggested, right after the paragraph that requires a checksum to be computed — that paragraph is what makes it concrete, since computing the checksum is what forces the sentence to be split. I also dropped the restatement you flagged: it no longer repeats how words are joined, only what the split has to cope with.

What I kept, rather than a pointer back to "Generating", is that requirement. The rules laid out there are about joining, and bip-0039-wordlists.md states the verification side in terms of the separator as well — implementers "must separate words using ideographic spaces / accommodate users inputting ideographic spaces". Read as a rule for splitting, that isn't enough in either direction:

                         split on U+3000 | split on U+0020 | NFKD then split
sentence joined U+3000         12        |        1        |       12
sentence joined U+0020          1        |       12        |       12

So the requirement is that the split accept either separator, and the way to get it wrong is splitting on U+0020 without normalizing first — which is what the reference implementation's to_entropy() does (mnemonic.py:155, against to_mnemonic() joining with U+3000 at :210). The sentence now reads:

Computing that checksum requires splitting the sentence into words, which must accept either separator: NFKD normalization maps U+3000 to U+0020, so splitting after normalization is one way; splitting on U+0020 before normalization reads an ideographic-space-separated sentence as a single word.

I deliberately avoided prescribing an order: Electrum splits on whitespace first and normalizes each word afterwards, which is equally correct.