Skip to content

Harden Elasticsearch tests against container startup flakes - #1445

Merged
danielgerlag merged 1 commit into
masterfrom
cursor/harden-elasticsearch-tests-78c8
Sep 8, 2026
Merged

Harden Elasticsearch tests against container startup flakes#1445
danielgerlag merged 1 commit into
masterfrom
cursor/harden-elasticsearch-tests-78c8

Conversation

@danielgerlag

Copy link
Copy Markdown
Owner

Describe the change

Elasticsearch-Tests has been failing on master and open PRs (including #1443 / #1444) while every other provider job stays green. The failure is not an assertion flake in the indexer: all 7 tests die in ElasticsearchDockerSetup.InitializeAsync with:

Initialize sequence timed-out
GET /_cluster/health?wait_for_status=green
Connection refused (127.0.0.1:<mapped-port>)

Squadron starts docker.elastic.co/elasticsearch/elasticsearch:7.17.29 and then polls cluster health green through NEST. The mapped port never accepts connections within the 5-minute timeout (single-node bootstrap / mmap / geoip download / NEST product-check during startup). Last time this job was green on this repo was around October 2025; recent master runs fail the same way every time.

Describe your implementation or design

Test-only hardening. No production Elasticsearch indexer behavior change.

  1. Replace Squadron with Testcontainers ContainerBuilder (same approach as Azure/Oracle tests).
  2. Pin ES 7.17.29 (matches NEST 7.x used by the provider).
  3. Single-node env that actually starts on GHA ubuntu-24.04: security/ML/geoip off, node.store.allow_mmap=false, 512MB heap.
  4. Wait for HTTP 200 on /_cluster/health (yellow is fine) instead of NEST wait_for_status=green.
  5. Unique index name per test instance; refresh the index instead of Thread.Sleep(1000).
  6. Raise vm.max_map_count in the Elasticsearch CI job.

Tests

Existing SearchIndexTests (7 facts) still cover search/filter. They now prove documents are searchable after an explicit refresh on an isolated index.

This environment has no Docker daemon, so the tests were compiled (net8.0) but not executed here. CI Elasticsearch-Tests is the real verification.

Breaking change

No.

Additional context

Failing test names (all same fixture timeout):

  • should_search_on_reference
  • should_search_on_custom_data
  • should_filter_on_custom_data
  • should_filter_on_alt_custom_data_with_conflicting_names
  • should_filter_on_reference
  • should_filter_on_status
  • should_filter_on_date_range

The refresh-race / shared-index hypothesis was not the current CI failure. Unique indexes + refresh are still included so those races cannot return once the container starts.

Open in Web Open in Cursor 

Replace Squadron's wait-for-green NEST health check with Testcontainers
HTTP readiness, pin ES 7.17.29, and give the single-node container
mmap-safe / no-geoip settings. Use a unique index per test and refresh
before search instead of a fixed sleep.

Co-authored-by: Daniel Gerlag <danielgerlag@users.noreply.github.com>
@danielgerlag
danielgerlag marked this pull request as ready for review September 7, 2026 22:55
@danielgerlag
danielgerlag merged commit 9f6f699 into master Sep 8, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants