fix(cluster): accept charon's null/omitempty cluster-file shapes by varex83agent · Pull Request #581 · NethermindEth/pluto

varex83agent · 2026-08-03T11:19:23Z

Problem

pluto's versioned cluster-definition and distributed-validator structs are missing serde defaults for several fields that charon legitimately emits as JSON null or omits entirely. As a result, serde rejects cluster files that charon writes.

Concretely, charon (v1.7.1 defaults to writing cluster-file v1.10, and also reads v1.8/v1.9) can produce:

"operators": null and "validators": null — Go marshals a nil slice as null, and these fields have no omitempty, so the key is always present but may be null.
an absent timestamp and absent name — both omitempty, so the key is dropped when empty.
an absent public_shares and absent partial_deposit_data on distributed validators — both omitempty.

#[serde(default)] alone handles absent keys but not an explicit null, so the null-marshaled slices additionally need DefaultOnNull.

Fix

operators, validators (definition v1.8 / v1.9 / v1.10): #[serde(default)] + #[serde_as(as = "DefaultOnNull")] — accepts both null and an absent key, deserializing to an empty vec. Mirrors the existing deposit_amounts handling.
timestamp, name (v1.8 / v1.9 / v1.10): #[serde(default)]. (v1.10 already had it for name; v1.8/v1.9 did not.)
public_shares, partial_deposit_data (DistValidatorV1x8orLater): #[serde(default)].

All three definition versions are fixed because charon reads v1.8/v1.9/v1.10 (cluster/version.go: supportedVersions).

Notes

The existing Definition deserializer enforces num_validators == validators.len(), so validators: null is only valid alongside num_validators: 0 — which is exactly what charon emits. The test reflects this.
Serialization is unchanged: DefaultOnNull's SerializeAs forwards to the normal serializer, so non-empty slices still serialize as arrays.

Tests

definition_accepts_null_slices_and_absent_omitempty_fields drives the v1.8/v1.9/v1.10 fixtures with operators/validators set to null and name/timestamp removed, asserting they parse to empty/default.
dist_validator_v1x8_accepts_absent_omitempty_fields strips public_shares and partial_deposit_data from a real lock fixture's validator and asserts it parses.

cargo test -p pluto-cluster (98 tests), clippy, and fmt all pass.

🤖 Generated with Claude Code

pluto's versioned cluster-definition and distributed-validator structs lacked serde defaults for several fields that charon legitimately emits as JSON `null` or omits entirely, so deserialization rejected files charon writes: - `operators` and `validators` — charon has no `omitempty` on these, so a nil slice marshals as `null`. Added `#[serde(default)]` plus `DefaultOnNull` so both `null` and an absent key deserialize to an empty vec (charon v1.7.1 defaults to writing v1.10 but also reads v1.8/v1.9, so all three definition structs are fixed). - `timestamp` and `name` — `omitempty`, so absent when empty. Added `#[serde(default)]` (v1.10 already had it for `name`; v1.8/v1.9 did not). - `public_shares` and `partial_deposit_data` on `DistValidatorV1x8orLater` — `omitempty`, so absent when empty. Added `#[serde(default)]`. Root cause is a missing `#[serde(default)]` (and `DefaultOnNull` for the null-marshaled slices) on fields charon can omit or null out. Tests drive each definition version and the distributed validator through the null/absent shapes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

emlautarom1

The underlying problem is not fully solved:

Checked the corpus at crates/cluster/src/examples/ using this branch:

                              main             this PR
cluster-definition-000..006   PARSES (all 7)   PARSES (all 7)
cluster-lock-000              FAILS -> invalid type: null, expected a string
cluster-lock-001              FAILS -> invalid type: null, expected a string
cluster-lock-002              FAILS -> missing field `name`
cluster-lock-003              FAILS -> invalid value: string "", expected hex bytes convertible to target type

All seven definitions passed before this PR too, so the corpus result is unchanged end to end. Charon's own TestExamples round-trips and signature-verifies every one of these files.

The default-written v1.10 shape also still fails, with this branch applied:

baseline (unmodified)                            PARSES
distributed_validators: null                     FAILS -> invalid type: null, expected a sequence
builder_registration.message.fee_recipient = ""  FAILS -> invalid value: string ""...
builder_registration.message.pubkey = ""         FAILS -> ...
builder_registration.signature = ""              FAILS -> ...
deposit_data[0].pubkey = ""                      FAILS -> ...

This is due to the following:

lock.rs isn't touched. LockV1x8orLater.distributed_validators has no default, and Go marshals the nil slice as null with no omitempty, so charon's own empty-cluster lock is rejected. Same fix as the rest of this PR.
A second root cause the PR doesn't cover: [u8; N] hex fields reject "" while Vec<u8> hex fields accept it, and charon's to0xHex(nil) returns exactly "". That accounts for cluster-lock-003 and all four builder-registration/deposit failures above.
The pre-v1.8 structs are in scope in practice. The description says charon reads v1.8/v1.9/v1.10, but supportedVersions lists v1_0 through v1_10. cluster-lock-000/001 (v1.1.0, config_signature: null) and -002 (v1.2.0, absent name) are real files that fail today.

I suggest to mirror Charon's TestExamples criteria: parse every file in crates/cluster/src/examples/ raw, locks included (only definitions are processed today).

Feel free to fold the changes onto this PR or create follow ups.

Extends the versioned-parser fixes to the cases the review flagged: - lock.rs: distributed_validators on every Lock version now accepts charon's nil-marshaled `null` slice (default + DefaultOnNull), so an empty-cluster lock parses. - Pre-v1.8 definitions (v1.0-v1.7) get the same treatment as v1.8+ for name/timestamp/fee_recipient_address/withdrawal_address (omitempty) and operators/validators (null-marshaled slices). - OperatorV1x1/V1x2orLater config_signature/enr_signature accept `null` (v1.0/v1.1 marshal `[]byte` directly; ethHex tolerates null on read). - HexBytes now deserializes `""` to a zero-filled `[u8; N]`, matching charon's `to0xHex(nil) == ""` for fixed SSZ vectors (Bytes48/32/...). The blanket TryFrom impl is split into concrete Vec<u8> / [u8; N] impls. Adds a corpus test mirroring charon's TestExamples that parses every file in crates/cluster/src/examples/, locks included, and drops the manual empty-hex rewrite in parse_example_lock now that the parser handles it natively. Co-Authored-By: Bohdan Ohorodnii <35969035+varex83@users.noreply.github.com>

varex83agent · 2026-08-10T12:07:58Z

Thanks @emlautarom1 — addressed all four points; the whole examples/ corpus (locks included) now parses.

lock.rs untouched → fixed. distributed_validators on every Lock* version (v1x0or1 … v1x8orLater) now has #[serde(default)] + DefaultOnNull, so charon's nil-marshaled null slice (empty-cluster lock) parses.
[u8; N] hex fields reject "" → fixed. HexBytes now deserializes "" to a zero-filled [u8; N], matching charon's to0xHex(nil) == "" (an SSZ Bytes48/Bytes32/… of a nil slice is exactly N zero bytes). The blanket TryFrom<Vec<u8>> impl is split into concrete Vec<u8> / [u8; N] impls; Vec<u8> still decodes "" to an empty vec. This covers cluster-lock-003 and the builder-registration/deposit-data empty-hex cases.
Pre-v1.8 in scope → fixed. name/timestamp/fee_recipient_address/withdrawal_address (charon omitempty) get #[serde(default)], operators/validators get DefaultOnNull, and OperatorV1x1/OperatorV1x2orLater config_signature/enr_signature accept null. That fixes cluster-lock-000/001 (v1.1.0, config_signature: null) and -002 (v1.2.0, absent name).
Mirrored TestExamples. New parses_every_example_file test reads every file in crates/cluster/src/examples/ and parses definitions as Definition and locks as Lock. Also dropped the manual empty-hex rewrite in parse_example_lock now that the parser handles "" natively.

cargo test -p pluto-cluster (all example files parse), clippy, and fmt pass.

emlautarom1

Before merging, ensure:

Merge the latest changes on main
On the parses_every_example_file, after each parse run verify_hashes and verify_signatures to match the Charon test.

emlautarom1 · 2026-08-24T19:06:05Z

    fn parse_example_lock(json: &str) -> Lock {
-        let mut value: serde_json::Value = serde_json::from_str(json).unwrap();
-
-        if let Some(validators) = value
-            .get_mut("distributed_validators")
-            .and_then(serde_json::Value::as_array_mut)
-        {
-            for validator in validators {
-                let Some(deposit_data) = validator
-                    .get_mut("deposit_data")
-                    .and_then(serde_json::Value::as_object_mut)
-                else {
-                    continue;
-                };
-
-                for (field, len_bytes) in [
-                    ("pubkey", 48usize),
-                    ("withdrawal_credentials", 32usize),
-                    ("signature", 96usize),
-                ] {
-                    if deposit_data
-                        .get(field)
-                        .and_then(serde_json::Value::as_str)
-                        .is_some_and(str::is_empty)
-                    {
-                        let zeros = "00".repeat(len_bytes);
-                        deposit_data[field] = serde_json::Value::String(format!("0x{zeros}"));
-                    }
-                }
-            }
-        }
-
-        serde_json::from_value(value).unwrap()
+        // Empty hex strings (charon's `to0xHex(nil)`) for fixed-size deposit
+        // fields are handled by the `HexBytes` deserializer, so parse directly.
+        serde_json::from_str(json).unwrap()
    }


nit: inline this function

…aults

Mirror charon's `TestExamples`: after parsing each example cluster file, run `verify_hashes` and `verify_signatures` (noop eth1 client, matching charon's nil). Inline the now-trivial `parse_example_lock` helper. Co-Authored-By: Bohdan Ohorodnii <35969035+varex83@users.noreply.github.com>

varex83agent · 2026-08-25T14:10:55Z

Addressed the pre-merge points:

Merged latest main into the branch.
parses_every_example_file now mirrors charon's TestExamples — after parsing each example file (definitions and locks), it runs verify_hashes() and verify_signatures(). The signature check uses a noop EthClient (EthClient::new("")), the equivalent of charon's VerifySignatures(nil) — none of the fixtures carry ERC-1271 smart-contract signatures.
Inlined the now-trivial parse_example_lock helper (nit).

cargo test -p pluto-cluster (104 tests), clippy, and fmt all pass.

varex83agent and others added 2 commits August 3, 2026 13:19

Merge branch 'main' into fix/cluster-serde-defaults

2779e88

emlautarom1 requested changes Aug 3, 2026

View reviewed changes

varex83 requested a review from emlautarom1 August 11, 2026 17:21

emlautarom1 approved these changes Aug 24, 2026

View reviewed changes

varex83agent and others added 2 commits August 25, 2026 16:06

Merge remote-tracking branch 'origin/main' into fix/cluster-serde-def…

a97d89e

…aults

varex83 merged commit a974b22 into main Aug 25, 2026
16 checks passed

varex83 deleted the fix/cluster-serde-defaults branch August 25, 2026 15:24

Navigation Menu

Uh oh!

fix(cluster): accept charon's null/omitempty cluster-file shapes - #581

fix(cluster): accept charon's null/omitempty cluster-file shapes#581
varex83 merged 5 commits into
mainfrom
fix/cluster-serde-defaults

varex83agent commented Aug 3, 2026

Uh oh!

emlautarom1 left a comment •

edited

Loading

Uh oh!

varex83agent commented Aug 10, 2026

Uh oh!

emlautarom1 left a comment

Uh oh!

emlautarom1 Aug 24, 2026

Uh oh!

varex83agent commented Aug 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

varex83agent commented Aug 3, 2026

Problem

Fix

Notes

Tests

Uh oh!

emlautarom1 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

varex83agent commented Aug 10, 2026

Uh oh!

emlautarom1 left a comment

Choose a reason for hiding this comment

Uh oh!

emlautarom1 Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

varex83agent commented Aug 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

emlautarom1 left a comment •

edited

Loading