[3.0][Testing] Check that an upgraded database matches a fresh install - #9531
[3.0][Testing] Check that an upgraded database matches a fresh install#9531albertlast wants to merge 5 commits into
Conversation
|
Checked the reading against Three indexes were missing entirely. An index on an expression stores Three out of 113 on a stock install, and silently: the tool reported no difference where an upgrade had failed to create any of them. The compatibility functions were not read. The two readings agree object for object now:
All Verified by breaking a database on purpose — dropping I checked MySQL the same way, against Worth saying why the tool reads the catalogue rather than diffing two dumps, since that is the obvious alternative. A dump is ordered and formatted for restoring, not for comparing — |
|
Did the same against Two things were being read too shallowly, though. A generated column was compared without its expression. `modified_time` bigint unsigned GENERATED ALWAYS AS (coalesce(json_unquote(json_extract(`edit_history`,'$[0][0]')),0)) STORED
`modified_name` varchar(255) GENERATED ALWAYS AS (coalesce(json_unquote(json_extract(`edit_history`,'$[0][6]')),'')) STORED
`modified_reason` varchar(255) GENERATED ALWAYS AS (coalesce(json_unquote(json_extract(`edit_history`,'$[0][7]')),'')) STOREDA wrong path in one of those produces a column of exactly the right type holding quietly the wrong value, which is about the worst thing this could fail to notice. Comparing
Checked by breaking things on purpose again — moving one JSON path and putting one table back to So both engines have now been checked against their own dump tool, and both found something. That is the more useful outcome than agreement would have been. |
The installer tells you to delete it and cannot do it itself: the ?delete link it offers is a GET, and command line arguments only ever reach $_POST, so nothing on the CLI path ever gets there. Leaving it behind is not cosmetic. Settings.php redirects every request back into the installer while the file exists, so the forum the script just built is unreachable, and SMF puts a "MAJOR SECURITY RISK: you have not removed install.php" box on every page it shows an administrator - which also lands in front of anything else a test or a person is trying to read on that page. Deleting it is safe for a reinstall because install_one() calls reset.sh first, and reset.sh clears Settings.php and then blocks until the entrypoint has staged a fresh copy. Adds a check in front of the two installer passes to say so out loud when it has not: without one, php reports "Could not open input file: install.php", which reads like a broken script rather than a stack that was never made installable. Signed-off-by: albertlast <mathiaspapealbert@hotmail.com>
Two forums side by side, each with its own administrator, and a password
chosen at install time is a combination that ends in hand written SQL
sooner or later - which is a poor way to answer a question as ordinary as
"is this the password?".
user.sh answers it. list shows the accounts, check says whether SMF would
accept a password and exits 0 or 1 so it can be used in a conditional,
and reset sets a new one. --engine reads the settings use-engine.sh saved
for the other engine, so the forum that is not currently live can be
looked at without switching to it and back.
Two details that stop it being a thin wrapper around an UPDATE:
- The hashing goes through Security::hashPassword() rather than being
written here, so what lands in the table is by construction what
Login2 reads back out. A script that hashes passwords its own way is
a script that eventually disagrees with the forum.
- reset clears passwd_flood too. SMF locks an account out for a while
after enough wrong guesses, and a new password behind a live lockout
behaves exactly like a password that did not take.
check also points out an account that is not activated, which fails to
log in with an entirely correct password.
The password is passed to the container through the environment rather
than in the argument list, which anything able to read the process table
can see. Also completes the file list in the README, which still only
described the image and had none of the scripts in it.
Signed-off-by: albertlast <mathiaspapealbert@hotmail.com>
The installer builds the schema from Sources/Db/Schema/v3_0/ in one go. The upgrader arrives at the same place through a hundred-odd migrations applied to whatever 2.1 left behind. They are meant to converge, and nothing we have checks that they do -- so a column left at the wrong type, an index that was never created, or a primary key quietly dropped goes unnoticed until it is a bug report from someone whose forum upgraded years ago. .docker/schema-tool.php reads the shape of a database -- tables, columns, indexes, sequences, the names in settings -- and compares two of those readings. It talks to the engine directly rather than through SMF, because the database worth looking at is frequently one SMF would refuse to run on. .docker/compare-upgrade.sh drives it: empty, load a 2.1 dump, upgrade, read, reinstall from scratch, read again, report. --baseline takes any 2.1 dump, so this works against a real forum as well as against the synthetic baseline from the 2.1 environment. Two kinds of difference are reported but do not decide the exit code, since a real forum always has some: what is in settings, and the order the columns of a table sit in. And if the upgrade does not reach the end, it stops there and says so rather than comparing anyway -- a half-upgraded database differs from a fresh install in hundreds of places, every one of them the honest consequence of the migrations that never ran. Signed-off-by: albertlast <mathiaspapealbert@hotmail.com>
Checking the reading against pg_dump --schema-only of the same database found two gaps, both of them the kind of thing this is supposed to notice. Three indexes were missing entirely. An index on an expression stores 0 in indkey and has no pg_attribute row, so joining to that table dropped those keys, and an index every one of whose keys is an expression disappeared with them -- idx_member_name_low, idx_real_name_low and idx_birthdate2 on a stock install. pg_get_indexdef() per key renders a plain column and an expression alike, and needs no join at all. The compatibility functions were not read. find_in_set(), instr(), from_unixtime(), the group_concat aggregate and the rest are created at install; a query naming one of them fails outright where it is absent, which makes a missing one a worse problem than a missing index, not a lesser one. The two readings now agree object for object: 72 tables, 113 indexes, 69 primary keys, 41 sequences, 19 functions. All pg_dump still reports that this does not are the public schema and the comment on it. Signed-off-by: albertlast <mathiaspapealbert@hotmail.com>
Same check as the last commit, the other way round: against mysqldump --no-data --routines --triggers --events. Column for column and index for index the two agree, 538 and 179, and SMF creates no routines, triggers or events on MySQL, so there is no equivalent of the missing functions. Two things were being read too shallowly, though. EXTRA says that a column is generated and never what from. smf_messages has three STORED columns read out of the edit_history JSON -- modified_time, modified_name and modified_reason -- and a wrong path in one of them gives a column of the right type holding quietly the wrong value. Comparing GENERATION_EXPRESSION is what tells $[0][7] from $[0][9]. ROW_FORMAT was not read at all. It is not decoration on this schema: COMPACT caps an index key at 767 bytes where DYNAMIC allows 3072, so a table left behind in the older format is one where half of SMF's indexes cannot be created at their full width -- and index width is already the second largest group of differences an upgrade produces. AUTO_INCREMENT is still deliberately not read. It measures how much a database has been used, not what shape it is. Signed-off-by: albertlast <mathiaspapealbert@hotmail.com>
a28f7cc to
9ef9720
Compare
2 participants
Description
The installer builds the schema from
Sources/Db/Schema/v3_0/in one go. The upgrader arrives at the same place through a hundred-odd migrations applied to whatever 2.1 left behind. They are meant to converge, and nothing we have checks that they do — so a column left at the wrong type, an index that was never created, or a primary key quietly dropped goes unnoticed until it turns up as a bug report from someone whose forum upgraded two years ago.Two files:
.docker/schema-tool.phpreads the shape of a database — tables, columns, indexes, sequences, the variable names insettings— and compares two of those readings. It talks to the engine directly rather than through SMF, since the database worth looking at is frequently one SMF would refuse to run on..docker/compare-upgrade.shdrives it: empty the database, load a 2.1 dump, upgrade it, read the schema, reinstall from scratch, read that too, report the difference.--baselinetakes any SQL dump of a 2.1 database, so this works against a real forum as well as against the synthetic baseline from #9330.Two kinds of difference are reported but do not decide the exit code, because a real forum always has some: what is in
settings, and the order the columns of a table sit in. And if the upgrade does not reach the end, the script stops there and says so rather than comparing anyway — a half-upgraded database differs from a fresh install in hundreds of places, every one of them the honest consequence of the migrations that never ran, and none of them worth reading. That check is worth more than it sounds: the first run of this produced 315 differences, all of them because the upgrade had stopped a third of the way through.The tool underneath is usable on its own against any two SMF databases on the same engine — two forums you have, or one forum before and after something you are testing.
What it says today
On MySQL, against the 2.1 baseline from #9330, with the
DropTimeOffsetdefects from #9521 patched so the upgrade can finish: 55 differences in the schema, in four groups.Text columns are one size larger than they should be. Around 35 of the 55.
ALTER TABLE … CONVERT TO CHARACTER SET utf8mb4promotes atextcolumn tomediumtextto preserve its byte capacity, so an upgraded forum ends up withmessages.bodyasmediumtextwhere a fresh install hastext, andmail_queue.bodyaslongtextagainstmediumtext. Wider rather than narrower, so nothing breaks, but it is not the schema the code was written against.Index prefix lengths disagree, in both directions.
Indexes 2.1 had that 3.0 does not define are never dropped, so an upgraded forum carries
log_spider_hits.idx_id_spider,log_spider_hits.idx_log_timeand a uniquelog_subscribed.id_subscribethat a fresh install has not got.log_packagesends up with the same column indexed twice, asfilename(191)andidx_filename(15).mentionskeeps defaults the fresh schema drops —id_member,id_mentionedandtimeareDEFAULT 0after an upgrade and have no default at all in a fresh install.Outside the schema, four settings a fresh install writes are missing after an upgrade —
cpu_count,forum_uuid,mostOnlineUpdated,robots_txt_search— and the 2.1-era ones the upgrade means to delete are all still there, which is #9530.On PostgreSQL the script stops before it can compare anything:
AlertsObsoleteusesUPDATE … JOIN, which is MySQL-only. That is #9519, and #9524 is the fix.None of the above is addressed here. This adds the thing that finds it.
Notes for review
Depends on #9344 for
.docker/install-forum.sh, and reads better after #9330, which is where the baseline it was written against comes from. Nothing outside.docker/and one.gitignoreentry changes, so it cannot affect a running forum.Issues References (Fixes|Related|Closes)
Related to #9330, #9344, #9519, #9521, #9530