Skip to content

HDDS-16354. Make the dfsrw read/write ratio configurable - #11219

Open
ermahesh wants to merge 4 commits into
apache:masterfrom
ermahesh:HDDS-16354
Open

ermahesh wants to merge 4 commits into
apache:masterfrom
ermahesh:HDDS-16354

Conversation

@ermahesh

@ermahesh ermahesh commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Follow-up from HDDS-14524.

The dfsrw (dfs-read-write-validator) freon workload issues exactly one read
after every write, so its read/write mix is pinned at 1:1. Real workloads are
rarely balanced, and the benchmark is more useful when the mix can be tuned
toward read-heavy or write-heavy.

This PR adds a --read-write-ratio option that sets how many reads each write is
followed by:

--read-write-ratio effect
1.0 (default) one read per write, the workload as it is today
4 read-heavy: each write is read back four times
0.25 write-heavy: one read back every fourth write

The existing per-path CRC32 validation and stale-read detection are unaffected. A
read still picks a random file from the history of the thread that wrote it and
compares against the checksum of the most recent write of that path, so both
corruption and an overwritten path returning older bytes are still detected.

Why a ratio rather than a --read-threads / --write-threads split

A read-only thread would have no write history of its own, so it would have to
read paths owned by other threads. Those paths are overwritten concurrently,
which is precisely the situation the stale-read check treats as a failure.
Splitting threads would therefore have meant weakening the validation, so every
read is kept inside the thread that wrote the file.

Fractional ratios

A ratio that is not a whole number is accumulated per thread rather than rounded
on each write:

private int readsDue(double readsPerWrite) {
  readCredit += readsPerWrite;
  int reads = (int) readCredit;
  readCredit -= reads;
  return reads;
}

Rounding per write would collapse every ratio below 0.5 to zero reads, leaving a
"validator" that never validates. Carrying the remainder makes 0.25 read back
every fourth write and 1.5 alternate between one and two reads. At the default
of 1.0 the credit lands on exactly 1.0 every time, so the default path is
unchanged.

Notes

  • -n keeps its meaning: one task is one write, and the number of files written
    for a given -n does not change with the ratio.
  • The file-write and file-read-validate timers are unchanged, so the two
    sides of the mix stay separately measurable.
  • A ratio that is not a positive finite number is rejected at startup, alongside
    the existing --size, --buffer and --max-files-per-thread checks.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16354

How was this patch tested?

New unit test TestHadoopFsReadWriteRatio in hadoop-ozone/freon.

The existing integration test TestHadoopFsReadWriteValidator is unchanged and
continues to cover the default behaviour against a real cluster.

The dfsrw workload paired one read with every write, pinning the mix at
1:1.  Real workloads are rarely balanced, so --read-write-ratio now sets
how many reads a write is followed by: 4 reads back each write four
times, 0.25 reads back every fourth write, and the default of 1 leaves
the workload as it was.

Reads stay inside the thread that wrote the file, so a read still
validates the CRC32 of the latest write of the path it reads, and an
overwritten path returning older bytes is still detected.  A ratio that
is not a whole number is accumulated per thread rather than rounded on
every write, otherwise every ratio below 0.5 would read back nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@chihsuan chihsuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking this up! @ermahesh I left a few comments.

@chihsuan

Copy link
Copy Markdown
Contributor

cc @yandrey321 if you’d like to take a look. This follow-up implements the configurable write/read ratio you suggested here #10651 (comment). Thanks!

…ctly

- Rename --read-write-ratio to --reads-per-write, and describe the reads as
  validation reads that each pick a file the thread wrote at random, rather
  than as read-backs of the write just made.
- Carry the ratio in whole millionths of a read instead of accumulating it as
  a double, which drifted: adding 0.1 ten times gives 0.9999999999999999, so
  the read the tenth write was due slipped to an eleventh. The option is now
  a BigDecimal, which also removes the NaN and infinity checks.
- Parameterize the corruption point of everyReadOfATaskValidatesContent over
  the first, second and third read, so it no longer covers only the last.
- Extend FlippingInputStream from FSInputStream, which supplies the
  positioned reads and drops three forwarding methods.

Adds carriesFractionalRatioWithoutDrift, which fails on the old accumulator.
Review feedback: express the read/write mix as a percentage and draw
per operation instead of issuing a fixed number of reads after every
write.

Every operation of the run now independently draws whether it writes a
file or reads one back, so -n counts operations of both kinds and the
split holds on average rather than exactly. This removes the fixed-point
carry machinery that was needed to spread a fractional reads-per-write
ratio across writes.

Two invariants the ratio model got for free need holding explicitly: a
thread whose history is still empty has nothing to read and writes
instead, and the file a write goes to is now named after the write
sequence in the thread's own marker rather than the task counter, which
no longer counts that thread's writes. That keeps a thread within
--max-files-per-thread however its reads and writes happen to fall.
… change

The file a write goes to is named after the task counter again, as it was
before --read-percent. Deriving it from the thread's own write sequence
broke the path recycling the framework provides: taskLoop passes
counter % testNo, so a time-based run cycles through -n ids and writes
over its paths, while the write sequence only wraps after
--max-files-per-thread. testValidateOverwrittenPaths saw 317 paths where
it expects at most 8, and the tool stopped exercising overwritten reads
in --duration runs, which is the point of that mode.

The memory bound the write sequence was meant to protect holds either
way, since the id is taken modulo --max-files-per-thread.

testWriteReadValidate expected -n files. -n now counts operations of both
kinds, so it takes the number of writes from the metrics and checks that
reads and writes together account for every operation.

testPathsWrapAtMaxFilesPerThread now runs at --read-percent 0. It asserts
an exact file count, and with reads drawn in, whether every path of the
cycle got written at all would have been left to chance.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants