feat: warn when hash/range ingest publishes oversized segments

[davidalexander3986]

Fixes #19573.

Description

Hash and range native batch ingest can publish segments much larger than maxRowsPerSegment / targetRowsPerSegment when a partition key is hot. Operators had no signal for that besides noticing huge segments later.
This PR counts published segments whose row count exceeds maxRowsPerSegment × 2, then warns, emits a metric, and records the count on the completion report.

Detect oversized segments at supervisor publish

After a successful publish, ParallelIndexSupervisorTask runs the check only for HASH or RANGE when getMaxRowsPerSegment() is non-null (so hashed + numShards with no max is skipped). Ratio is 2.0. For range, resolved max is already target + target/2, so the warn is effectively 3× the user’s targetRowsPerSegment.
If the count is > 0:

• WARN on the supervisor task
• metric ingest/segments/oversized
  Live reports omit the field. Completion reports include oversizedSegments (Long, omitted when null). Sequential IndexTask (maxNumConcurrentSubTasks: 1) does not run this check.

Tests

• IndexTaskUtils.getOversizedSegments unit cases (threshold, null totalRows, mixed)
• Multi-phase hashed and range skewed ingest expecting oversizedSegments = 1; even range expects 0; hashed + numShards omits the field
• Task report serde round-trips a non-null oversizedSegments
• Metric listed in metrics.md, logging emitter allow-list, Prometheus, and StatsD catalogs

Implementation decision

In the cited issue it was recomended to set this logic in the PartialHashSegmentGenerateTask hooks and to use Appenderator.getRowCount(SegmentIdWithShardSpec) to get the row count. I ended up not doing this because I realized in the partial tasks we don't yet have the final segments. The final segments are only available after the final merge, right after publishing to S3. So, I instead went with wiring this through in the PartialSegmentMergetask instead.

Side effect

Those jobs merge and push DataSegments without going through an appenderator, so totalRows was never set. The supervisor still emits ingest/rows/published via IndexTaskUtils.getTotalRowCount(), which skips nulls, so the metric was always 0. After this change the metric should emit correctly for range and hash index_parallel jobs.

Release note

You can now have visibility into when your range and hash partitioning settings are resulting in creating overly large segment files (hot partitions) in a time chunk. Native batch hash and range ingest now warns and emits a count of segments ingest/segments/oversized for all published segments with more than 2× maxRowsPerSegment rows (for range, that is 3× targetRowsPerSegment). The count is also on the task completion report as oversizedSegments.

---

Key changed/added classes in this PR

• IndexTaskUtils
• ParallelIndexSupervisorTask
• PartialSegmentMergeTask
• IngestionStatsAndErrors
• AbstractBatchIndexTask

---

This PR has:

• been self-reviewed.
• added documentation for new or modified features or behaviors.
• a release note entry in the PR description.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• added integration tests.
• been tested in a test Druid cluster.

Verification

Ran the cluster locally, and submitted an ingestion spec with hashed partitioning for wikipedia, using channel as a partition key:

...
"tuningConfig": {
      "type": "index_parallel",
      "maxRowsInMemory": 25000,
      "forceGuaranteedRollup": true,
      "maxNumConcurrentSubTasks": 2,
      "partitionsSpec": {
        "type": "hashed",
        "maxRowsPerSegment": 2000,
        "partitionDimensions": ["channel"]
      }
    }
  }
...

since channel is often wikipedia#en for many rows, this will cause a hot partition for the same time chunk. Confirmed in report that oversizedSegments is set:

and the warn log: