Fix BETWEEN with leading-dot decimals being mis-lexed (issue601)

[AmadNaseem]

Fixes #601.

sqlparse mis-lexes BETWEEN/AND when the range bounds are decimals written without a leading zero:

>>> import sqlparse
>>> sqlparse.parse("select * from t where l_discount between .03 and .06")[0].flatten()
# before: 'between' and 'and' come out as Token.Name (not Keyword),
#         corrupting the grouping into Identifiers like 'l_discount between'

The same query with 0.03 / 0.06 parses fine, which points at the leading dot.

Cause: the qualified-name rule in SQL_REGEX (sqlparse/keywords.py) is [A-ZÀ-Ü]\w*(?=\s*\.) — it matches a bareword followed by a ., intended for schema.name. But it also fires on between in between .03, mistaking the decimal point for a member-access period, so the keyword is swallowed as a name.

Fix: narrow the lookahead to [A-ZÀ-Ü]\w*(?=\s*\.(?!\d)), so the . only counts as member access when it is not followed by a digit. This is safe because SQL identifiers cannot start with a digit, so name.<digit> is never valid member access. Qualified names still work: schema.table, db.schema.table, t.col, t.*, a.b.c.d all verified unchanged. .03/.06 were already matched by the float rule; they just weren't reached before because the name rule consumed the keyword.

Added a regression test in tests/test_regressions.py and a CHANGELOG entry. Full test suite passes (488 passed).

[Copilot]

Pull request overview

Fixes a lexer edge case where a keyword immediately preceding a leading-dot float (e.g. BETWEEN .03) could be mis-tokenized as a Name due to the qualified-identifier regex, which then breaks downstream grouping.

Changes:

• Tighten the qualified-name lookahead in SQL_REGEX to ignore . when it begins a numeric literal (.(\d...)).
• Add a regression test for between .03 and .06 tokenization (issue601).
• Document the fix in CHANGELOG.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
sqlparse/keywords.py	Refines the qualified-name regex to avoid consuming keywords before leading-dot floats.
tests/test_regressions.py	Adds a regression test ensuring between/and remain Keyword tokens with leading-dot floats.
CHANGELOG	Adds a bug-fix entry describing the corrected lexing behavior (issue601).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.