Refactor Postgres migrator

[nadav-govari]

Description

Advisory locks reading: https://flaviodelgrosso.com/blog/postgresql-advisory-locks

Issue 1

Today, the Postgres migrator begins a transaction, and runs all migrations inside their own transaction block, such that if any one fails, the whole thing gets unwound.

This is undesirable for two reasons:

1. The migrations build on each other; migration 3 failing does not mean theres a problem with migration 2
2. Concurrent DDL cannot be run inside of a transaction block

This refactors the migrator to run each migration directly, without a wrapping block. sqlx already applies migrations transactionally for us.

Issue 2

Migrations must be successfully applied on startup for a metapod to mark itself ready on a deployment. Sqlx takes out a Postgres advisory lock, which is a Postgres feature that allows for utilizing Postgres' atomicity to lock on custom resources, which you define with a key of (int, int). It then runs the migrations until completion.

This is a problem for any migration that takes more than the readiness timeout, which is currently 5 minutes. The readiness probe will kill the metastore pod mid applying migrations, and will start crashlooping. Additionally, other metastore pods will queue up to acquire this same lock so that they can too apply migrations.

The fix for this is as follows:

1. Migrations that are absolutely required for functionality stay where they are, with diesel guard inspecting all migrations in CI and preventing migrations that acquire exclusive locks on a table from being merged.
2. Migrations that can be applied concurrently, such as index creation, are moved to a second migrations directory introduced in this PR, postgres_deferred, and applied async post-deployment. To prevent the same racing behavior described above, Postgres advisory locks are again reused. A pod will attempt to acquire the advisory lock; if it succeeds, it is the migrations leader and will attempt to apply the migrations. If it doesn't, then it will assume another pod has the lock, and continue on with its day. At all points, live traffic is still served.

Finally, ignore_missing on migrations is marked true. This will allow us forwards and backwards compatibility with migrations and binaries, as of this change. It will be up to us to ensure that the code also respects this.

How was this PR tested?

Postgres integ tests, local cluster tests incoming.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a1b638bbbd

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 50775793dd

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".