Move JSON ingestion into elasticgraph-json_ingestion via an indexer extension#1302
Open
jwils wants to merge 1 commit into
Open
Move JSON ingestion into elasticgraph-json_ingestion via an indexer extension#1302jwils wants to merge 1 commit into
jwils wants to merge 1 commit into
Conversation
jwils
requested review from
BrianSigafoos-SQ,
bsorbo,
ellisandrews-toast,
jwondrusch,
marcdaniels-toast,
myronmarston and
rossroberts-toast
as code owners
July 7, 2026 21:21
This was referenced Jul 7, 2026
jwils
force-pushed
the
joshuaw/json-ingestion-adapter
branch
2 times, most recently
from
July 7, 2026 21:47
bd53a07 to
bcc1c1a
Compare
jwils
force-pushed
the
joshuaw/ingestion-adapter-seam
branch
from
July 7, 2026 21:47
1f731a7 to
6e17704
Compare
jwils
force-pushed
the
joshuaw/json-ingestion-adapter
branch
from
July 8, 2026 02:10
bcc1c1a to
4724804
Compare
jwils
force-pushed
the
joshuaw/ingestion-adapter-seam
branch
from
July 8, 2026 02:10
6e17704 to
a9d37cc
Compare
jwils
added a commit
that referenced
this pull request
Jul 8, 2026
## Why To support ingestion in any data format while keeping format-specific logic in its own gem (`elasticgraph-json_ingestion`, the upcoming `elasticgraph-proto_ingestion`), the indexer needs an extension mechanism analogous to the GraphQL one: schema definition extensions register runtime extensions in the schema artifacts, and the runtime component applies them at boot. This PR adds the runtime metadata storage for that. ## What - Add `indexer_extension_modules` to `RuntimeMetadata::Schema`, mirroring `graphql_extension_modules` - Add a `RuntimeMetadata::IndexerExtension` wrapper (twin of `GraphQLExtension`) - Nothing registers or consumes these yet, and empty lists are pruned from dumped YAML, so no schema artifacts change ## Verification - `script/lint`, `script/type_check` - `script/run_gem_specs elasticgraph-schema_artifacts` (100% coverage) - `script/run_gem_specs elasticgraph-schema_definition` - `bundle exec rake schema_artifacts:check` (no artifact changes) ## Stack Current PR is marked with `->`. - -> [#1298 Add indexer_extension_modules to runtime metadata](#1298) - [#1299 Add register_indexer_extension schema definition API](#1299) - [#1300 Apply indexer extension modules when the Indexer boots](#1300) - [#1301 Extract an ingestion adapter seam inside elasticgraph-indexer](#1301) - [#1302 Move JSON ingestion into elasticgraph-json_ingestion via an indexer extension](#1302)
jwils
added a commit
that referenced
this pull request
Jul 8, 2026
## Why Ingestion format gems need a way to ensure their indexer-side behavior is active whenever a schema is defined with their schema definition extension--the same guarantee `register_graphql_extension` provides for GraphQL extensions (e.g. how `elasticgraph-apollo` registers its engine extension). ## What - Add `register_indexer_extension` to the schema definition API, mirroring `register_graphql_extension` - Registered extensions are dumped into the `indexer_extension_modules` runtime metadata added in #1298 ## Verification - `script/lint`, `script/type_check` - `script/run_gem_specs elasticgraph-schema_definition` (100% coverage) ## Stack Current PR is marked with `->`. - [#1298 Add indexer_extension_modules to runtime metadata](#1298) - -> [#1299 Add register_indexer_extension schema definition API](#1299) - [#1300 Apply indexer extension modules when the Indexer boots](#1300) - [#1301 Extract an ingestion adapter seam inside elasticgraph-indexer](#1301) - [#1302 Move JSON ingestion into elasticgraph-json_ingestion via an indexer extension](#1302)
jwils
force-pushed
the
joshuaw/json-ingestion-adapter
branch
from
July 8, 2026 16:15
4724804 to
f1b5472
Compare
jwils
force-pushed
the
joshuaw/ingestion-adapter-seam
branch
from
July 8, 2026 16:15
a9d37cc to
73bb297
Compare
jwils
added a commit
that referenced
this pull request
Jul 8, 2026
## Why With runtime metadata storage (#1298) and the registration API (#1299) in place, `elasticgraph-indexer` needs to actually apply registered extensions--introducing the indexer extension concept, mirroring how `elasticgraph-graphql` applies GraphQL extension modules from both settings YAML and runtime metadata. ## What - `Indexer#initialize` extends configured extension modules onto the instance: first from the new `indexer.extension_modules` setting (mirroring `graphql.extension_modules`), then from the runtime metadata's `indexer_extension_modules` - Regenerated config schema artifacts for the new setting - Nothing registers an indexer extension yet, so this is a no-op for all existing schemas ## Verification - `script/lint`, `script/type_check` - `script/run_gem_specs elasticgraph-indexer` (100% coverage) - `script/run_gem_specs elasticgraph-local` ## Stack Current PR is marked with `->`. - [#1298 Add indexer_extension_modules to runtime metadata](#1298) - [#1299 Add register_indexer_extension schema definition API](#1299) - -> [#1300 Apply indexer extension modules when the Indexer boots](#1300) - [#1301 Extract an ingestion adapter seam inside elasticgraph-indexer](#1301) - [#1302 Move JSON ingestion into elasticgraph-json_ingestion via an indexer extension](#1302)
…xtension The JSON ingestion adapter and JSON-schema-based record preparer factory move from elasticgraph-indexer to elasticgraph-json_ingestion, and the JSON ingestion schema definition extension registers an indexer extension in runtime metadata that makes the adapter available at indexing time. Any schema defined with JSON ingestion support automatically gets JSON event ingestion--no settings needed--and elasticgraph-indexer no longer contains (or refers to) any JSON-specific ingestion logic. The be_a_valid_elastic_graph_event matcher moves to elastic_graph/json_ingestion/spec_support/event_matcher, and suites that build indexers from the shared test schema artifacts gain a development dependency on elasticgraph-json_ingestion (whose indexer extension those artifacts now reference).
jwils
force-pushed
the
joshuaw/ingestion-adapter-seam
branch
from
July 8, 2026 21:25
73bb297 to
9b5aa5c
Compare
jwils
force-pushed
the
joshuaw/json-ingestion-adapter
branch
from
July 8, 2026 21:25
f1b5472 to
292fbd2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
JSON parsing and validation has never conceptually belonged to
elasticgraph-indexer--it should live inelasticgraph-json_ingestionalongside the JSON schema generation, with the indexer staying ingestion-format-neutral soelasticgraph-proto_ingestion(and others) can plug in the same way.What
elasticgraph-json_ingestion(JSONIngestion::IngestionAdapter,JSONIngestion::RecordPreparerFactory)JSONIngestion::SchemaDefinition::APIExtensionregisters aJSONIngestion::IndexerExtensionviaregister_indexer_extension, which contributes the adapter toIndexer#ingestion_adapters. Zero configuration: any schema defined with JSON ingestion support automatically gets JSON event ingestion at indexing time, activated through the schema artifacts' runtime metadataelasticgraph-indexerno longer contains or references any JSON-specific ingestion logic;Indexer#record_preparer_factoryis removed, andoperation_factoryraises a clearConfigErrorwhen a schema has no ingestion extensionsbe_a_valid_elastic_graph_eventmatcher moves toelastic_graph/json_ingestion/spec_support/event_matcher(breaking require-path change)elasticgraph-json_ingestionRemaining follow-up
Indexer::TestSupport::Convertersstill builds JSON-envelope events (json_schema_version); moving it requires touchingelasticgraph-local's runtime fake-data indexing, so it's deferred to a follow-up PR.Verification
script/quick_build(5160 examples, 0 failures, plus site validation with doctests)run_each_gem_specCI mode), all green with 100% coverageStack
Current PR is marked with
->.