Skip to content

Map DateTime to google.protobuf.Timestamp and document temporal string formats#1306

Open
jwils wants to merge 2 commits into
joshuaw/proto-field-numbersfrom
joshuaw/proto-timestamp
Open

Map DateTime to google.protobuf.Timestamp and document temporal string formats#1306
jwils wants to merge 2 commits into
joshuaw/proto-field-numbersfrom
joshuaw/proto-timestamp

Conversation

@jwils

@jwils jwils commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Why

Mapping DateTime to string loses the type safety proto offers — a string can be malformed in ways a google.protobuf.Timestamp cannot, and proto consumers get language-native timestamp types. For the remaining string-typed temporal scalars, the proto type is much wider than the ElasticGraph type, so the expected format deserves documentation at the point of use (per review discussion).

What

  • Map DateTime to google.protobuf.Timestamp, importing google/protobuf/timestamp.proto automatically. A Timestamp is a UTC instant, so a publisher's original UTC offset is not preserved; t.protobuf type: "string" remains available as an override.
  • t.protobuf gains two options usable by any scalar:
    • import: maps a scalar to an externally defined proto type, emitting the needed import statement
    • comment: documents the expected format on each generated field
  • The built-in string-typed temporal scalars (Date, LocalTime, TimeZone) use comment: to document their formats (e.g. // ISO 8601 date, e.g. "2024-11-25"). Values are still validated at ingestion time, exactly as with JSON ingestion.
  • Format comments compose with the existing source name: comments and propagate to the innermost nested-list wrapper field.

Risk Assessment

Low — only affects the unreleased elasticgraph-proto_ingestion extension from #1080.

References

Update — 2026-07-10

Nested-list wrapper comment propagation was removed after #1080 changed to reject lists of lists. Format comments continue to apply to supported scalar and single-list fields.

@jwils
jwils force-pushed the joshuaw/proto-field-numbers branch from f62a859 to 90e2af5 Compare July 9, 2026 17:29
@jwils
jwils force-pushed the joshuaw/proto-timestamp branch from 5a2e67d to 1537869 Compare July 9, 2026 17:29
@jwils
jwils force-pushed the joshuaw/proto-field-numbers branch from 90e2af5 to 616dbfd Compare July 9, 2026 18:13
@jwils
jwils force-pushed the joshuaw/proto-timestamp branch 2 times, most recently from 721fe97 to d516c31 Compare July 10, 2026 15:44
@jwils
jwils force-pushed the joshuaw/proto-field-numbers branch from 616dbfd to 74011e5 Compare July 10, 2026 15:44
@jwils
jwils force-pushed the joshuaw/proto-timestamp branch from d516c31 to 9e7369b Compare July 10, 2026 15:52
@jwils
jwils force-pushed the joshuaw/proto-field-numbers branch from 74011e5 to 075e749 Compare July 11, 2026 13:34
@jwils
jwils force-pushed the joshuaw/proto-timestamp branch from 9e7369b to 9e2b508 Compare July 11, 2026 13:36
@jwils
jwils force-pushed the joshuaw/proto-field-numbers branch from 075e749 to 8d6e03f Compare July 11, 2026 13:53
@jwils
jwils force-pushed the joshuaw/proto-timestamp branch from 9e2b508 to 06e1531 Compare July 11, 2026 13:54
@jwils
jwils force-pushed the joshuaw/proto-field-numbers branch from 8d6e03f to 0a476c1 Compare July 11, 2026 13:59
@jwils
jwils force-pushed the joshuaw/proto-timestamp branch from 06e1531 to ee20f41 Compare July 11, 2026 14:00
@jwils
jwils force-pushed the joshuaw/proto-field-numbers branch from 5f292f5 to 7a57c87 Compare July 15, 2026 23:52
@jwils
jwils force-pushed the joshuaw/proto-timestamp branch from 508c016 to 4a6aa37 Compare July 15, 2026 23:52
@jwils
jwils force-pushed the joshuaw/proto-field-numbers branch from 7a57c87 to 26fe805 Compare July 16, 2026 14:21
@jwils
jwils force-pushed the joshuaw/proto-timestamp branch from 4a6aa37 to 1aa384b Compare July 16, 2026 14:21
jwils added 2 commits July 17, 2026 09:58
…g formats

`DateTime` fields now use the well-known `google.protobuf.Timestamp` type
(with `google/protobuf/timestamp.proto` imported automatically) instead of
`string`. A Timestamp cannot be malformed the way a string can, and proto
consumers get language-native timestamp types. Note that a Timestamp is a
UTC instant, so a publisher's original UTC offset is not preserved;
`t.protobuf type: "string"` remains available to override.

To support this, `t.protobuf` gains two options usable by any scalar:

- `import:` maps a scalar to an externally defined proto type, emitting the
  needed `import` statement in `schema.proto`.
- `comment:` documents the expected format on each generated field, used by
  the built-in `string`-typed temporal scalars (`Date`, `LocalTime`,
  `TimeZone`) whose proto type is wider than the ElasticGraph type (e.g.
  `// ISO 8601 date`). Values are still validated at ingestion time, just
  as with JSON ingestion.
@jwils
jwils force-pushed the joshuaw/proto-field-numbers branch from 26fe805 to 8803f5e Compare July 17, 2026 15:05
@jwils
jwils force-pushed the joshuaw/proto-timestamp branch from 1aa384b to b94fd0e Compare July 17, 2026 15:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant