Skip to content

[3.0][Testing] Case insensitive comparisons (part 2 of 2) — guard the convention in the unit suite - #9597

Open
albertlast wants to merge 7 commits into
SimpleMachines:release-3.0from
albertlast:3.0/tests-ci-token-guard
Open

[3.0][Testing] Case insensitive comparisons (part 2 of 2) — guard the convention in the unit suite#9597
albertlast wants to merge 7 commits into
SimpleMachines:release-3.0from
albertlast:3.0/tests-ci-token-guard

Conversation

@albertlast

@albertlast albertlast commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator

Description

Depends on #9596. It is branched from it, so until that one merges the diff here shows
its commit too. The commits belonging to this pull request are the second and third.

Part 1 gives a comparison a way to say whether it folds case. This is what notices when
one does not say. What makes it worth a test rather than a review habit is the failure
mode: a case sensitive comparison returns fewer rows instead of erroring, so nothing
reaches smf_log_errors, nothing fails in CI, and the first person to notice is a member
whose search came back empty. #9592 and #9593 both sat in the tree for years.

There are two ways to get this wrong, so there are two scans.

1. Folding neither side. The comparison relies on the collation, which folds on MySQL
and does not on PostgreSQL. Counted per file against a baseline:

public const BASELINE = [
    'Sources/Actions/Admin/Bans.php' => 3,
    'Sources/Actions/Admin/Subscriptions.php' => 1,
    'Sources/Actions/Profile/Summary.php' => 2,
    'Sources/Search/SearchApi.php' => 3,
    'Sources/Security.php' => 2,
];

All eleven are real, not noise. Three are #9592, three are #9593, and the two in
Security.php are folded by the ban_like identifier when the query runs rather than in
the query text, so they stay listed with a note saying why.

This scan reads LIKE only. member_name = {string:name} and $member_name = $string
are the same line to a scanner, and an UPDATE ... SET clause looks like both; guessing
would produce a baseline of about fifty entries that are mostly PHP assignments, which is
a test people learn to ignore.

2. Folding only the column. This is the worse one, and it is why the second scan
exists. A folded column can never equal an unfolded value, so instead of matching too much
the comparison matches nothing at all on PostgreSQL — including the row it was looking
for. It also hides, because the query says {column_ci:} and reads as handled. The first scan
cannot find these by construction: it treats {column_ci:} as proof a decision was made.

public const UNFOLDED_VALUES = [
    'Sources/Actions/Admin/Members.php' => 1,
    'Sources/Actions/AutoSuggest.php' => 2,
    'Sources/Actions/Register2.php' => 2,
    'Sources/Actions/RequestMembers.php' => 1,
    'Sources/PersonalMessage/PM.php' => 1,
    'Sources/PersonalMessage/Search.php' => 2,
    'Sources/Profile.php' => 1,
];

These are correct only when the caller folds the value in PHP first, which is a claim
about code elsewhere, so the list names which callers do and which do not. Five do.
Five do not, and are faults: Register2.php and Profile.php are #9594, and
PersonalMessage/Search.php is the same fault in the search for a personal message by its
author, which is not yet filed.

This scan reads equality as well as LIKE, which the first cannot. A line carrying
{column_ci:} is SQL, so an = on it is a comparison rather than an assignment — which is how
#9594 comes into range.

Sources/User.php left this list when #9596 gained {array_string_ci:}, which folds every
value in an IN list, so that comparison folds both sides now.

Confirmed to fail when it should. Adding 'WHERE real_name LIKE {string:x}' to
Sources/Actions/Groups.php:

The set of case sensitive comparisons on user text has changed.
...
     'Sources/Actions/Admin/Subscriptions.php' => 1,
+    'Sources/Actions/Groups.php' => 1,
     'Sources/Actions/Profile/Summary.php' => 2,

A third test asserts both scans find something, so that emptying either one does not leave
the tests above passing against an empty baseline.

Neither scan asserts anything about behaviour, so this is not a substitute for the fixes.
It stops the lists growing while they are worked through, and lowering a number is the
point of them rather than a chore attached to them.

One case is out of reach of both: Sources/User.php:3798 builds
email_address ' . $comparison . ' where $comparison holds LIKE or = from an
earlier line, so neither the column nor the operator is on a line a scanner can pair up.
That comparison folds its values and not its column, the mirror of the second list.

Reading every file under Sources/ costs about 60ms on a normal filesystem (1606 files).
It is slower over a Docker bind mount on Windows, which is a local artifact.

Issues References (Fixes|Related|Closes)

  1. Related: [3.0] Case insensitive comparisons (part 1 of 2) — a {ci:} type for the query language #9596 — the {column_ci:} type this guards the use of
  2. Related: [3.0]: Email bans do not match members with uppercase in their address on PostgreSQL #9592
  3. Related: [3.0]: Searching by user name is case-sensitive on PostgreSQL #9593
  4. Related: [3.0]: Two accounts can register on the same email address on PostgreSQL #9594
  5. Related: Enable an automated AI reviewer on pull requests (CodeRabbit is free for public repos) #9526 — the same gap this covers mechanically, where an AI reviewer would cover it by judgement

🤖 Generated with Claude Code

Whether a string comparison folds case is decided by the database engine:
MySQL folds it in the column's collation, PostgreSQL compares exactly. Callers
handled that themselves by reading Db::$db->case_sensitive and wrapping the
column in LOWER(), which put the decision at every call site and left it out
wherever somebody did not think to add it.

{ci:column} moves it into the query string, where the substitution layer
already lives. It expands to the bare column on MySQL and to LOWER(column) on
PostgreSQL. {ci_string:key} is the matching value type, for the places that
were folding the value in SQL rather than in PHP.

The column is named inline rather than through $db_values, so that a
comparison shows in the query text which column it folds. Only a column name,
optionally qualified by a table alias, is accepted.

Memberlist keeps its own LOWER() loop, because it folds expressions such as
COALESCE(group_name, '') as well as plain columns, and those are not what
{ci:} accepts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: albertlast <mathiaspapealbert@hotmail.com>
The array_ modifier is already a prefix on the types that take a list, so a ci
prefix would leave the list form as {array_ci_string} or {ci_array_string}.
As a suffix it composes with what is there: {string_ci} beside {string}, and
{array_string_ci} beside {array_string}.

{array_string_ci} folds each value in the list the way {string_ci} folds one,
which lets User::addQueryCustomizationsForLoadType() hand the names over as
they came instead of folding them itself.

That last one is a behaviour change, and the only one in this branch. Folding
the names in PHP with strtolower() left them compared against a column folded
by SQL LOWER(), and the two disagree outside ASCII: a member named ÄNNA gives
'änna' on the column and 'Änna' from strtolower(), which never match on
PostgreSQL. Both sides now fold the same way.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: albertlast <mathiaspapealbert@hotmail.com>
albertlast and others added 5 commits September 3, 2026 16:04
The token says what it takes. {ci:} left a reader to work out that the thing
inside it was a column name rather than a value key, which is the opposite way
round from every other type in the query language.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: albertlast <mathiaspapealbert@hotmail.com>
A LIKE against text a person typed has to say whether it folds case, because
the engines disagree about it. Written as a bare column it says nothing, and
the query then matches on MySQL and not on PostgreSQL. That failure returns
fewer rows instead of erroring, so it reaches neither the error log nor CI.

This counts, per file, the comparisons on the columns holding names, email
addresses and hostnames that do not fold case in the query text, and holds the
count against a baseline. A new one fails the suite and names the file; fixing
an old one means lowering its number in the same commit.

The scan covers LIKE only. `member_name = {string:name}` and
`$member_name = $string` are the same line to a scanner, and the SET clause of
an UPDATE looks like both, so equality is left out rather than guessed at.

The two comparisons in Security.php are folded by the 'ban_like' identifier at
the point the query runs rather than in the query text, so they stay in the
list with a note saying why.

Reading every file under Sources costs about 60ms on a normal filesystem.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: albertlast <mathiaspapealbert@hotmail.com>
Folding one side is worse than folding neither. A folded column can never equal
an unfolded value, so rather than matching too much on PostgreSQL the comparison
matches nothing at all, including the row it was looking for. It is also the
harder one to see, because the query says {ci:} and reads as handled.

The first scan cannot find these, by construction: it treats {ci:} as proof that
a decision was made. So a second scan lists the comparisons that fold a column
while comparing it against a value the query leaves alone. Those are correct
only when the caller folded the value in PHP, which is a claim about code
elsewhere, so the list names which callers do and which do not.

This one reads equality as well as LIKE. A line carrying {ci:} is SQL, so an =
on it is a comparison rather than a PHP assignment, which is what put equality
out of reach of the first scan.

Five of the eleven entries are known faults: Register2.php and Profile.php are
SimpleMachines#9594, and PersonalMessage/Search.php is the same fault in the search for a
personal message by its author.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: albertlast <mathiaspapealbert@hotmail.com>
{ci_string} became {string_ci} and gained {array_string_ci}, so the scan that
decides whether a comparison declares its case handling has to recognise both.

User.php leaves the second list as a result. It hands its names to
{array_string_ci:}, which folds every value in the list, so the comparison no
longer folds one side only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: albertlast <mathiaspapealbert@hotmail.com>
{ci:} became {column_ci:}, so the scan that decides whether a comparison
declares its case handling has to look for the new spelling. Getting this wrong
fails loudly rather than quietly: every converted call site would stop looking
folded at once, and the baseline would grow by all of them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: albertlast <mathiaspapealbert@hotmail.com>
@albertlast
albertlast force-pushed the 3.0/tests-ci-token-guard branch from 0df2a14 to bd84584 Compare September 3, 2026 14:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant