Skip to content

feat(api): harden the bulk upload CSV contract (#2362)#2366

Merged
jh-RLI merged 1 commit into
developfrom
feature-2362-bulk-upload-csv-contract
Jul 17, 2026
Merged

feat(api): harden the bulk upload CSV contract (#2362)#2366
jh-RLI merged 1 commit into
developfrom
feature-2362-bulk-upload-csv-contract

Conversation

@jh-RLI

@jh-RLI jh-RLI commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Summary of the discussion

This PR stacks ontop of #2364 wait until its merged and rebase this PR.

Part of #2362 (Slice 3 — CSV contract hardening). Implements the explicit
CSV contract on top of the bulk upload tracer bullet:

  • Header preflight, before the body streams: duplicate column names,
    names not in the table, and missing required columns (NOT NULL without a
    default) are each rejected with a 400 naming the offending columns —
    a wrong column mapping costs milliseconds, not gigabytes.
  • UTF-8 BOM stripped from the header, so Excel exports work unmodified.
  • Empty field = NULL, always (FORCE_NULL on all uploaded columns),
    whether the field is quoted or not. Deliberate deviation from COPY's
    native CSV rule (quoted "" = empty string): many common writers quote
    every field and would otherwise have their NULLs silently stored as empty
    strings. Do not "fix" this back to Postgres defaults — it would silently
    corrupt NULL handling for clients.
  • Sanitized, located errors: failures return the database's data-level
    message plus the CSV line number (header-adjusted) and column — e.g.
    invalid input syntax for type bigint: "boom" (CSV line 3, column id)
    and never raw SQL, server context dumps, or internal paths. When the
    server provides no diagnostics (connection lost mid-COPY), the message
    stays generic instead of echoing socket paths.

Known minor limitation (accepted): Postgres reports COPY records; a quoted
field containing embedded newlines earlier in the file shifts the reported
line relative to physical file lines.

Tests: module grows to 20 HTTP-seam tests — one per contract rule
(duplicates, missing-required incl. serial-default interplay, BOM, column
order freedom, quoted/unquoted NULL, line+column error shape with
no-internals assertions). Full suite green. Changelog entry included.

Workflow checklist

Automation

Closes #

Part of #2362

PR-Assignee

Reviewer

  • 🐙 Follow the
    Reviewer Guidelines
  • 🐙 Provided feedback and show sufficient appreciation for the work done

Slice 3 of #2362, on top of the bulk upload tracer bullet:

- Header preflight before the body streams: reject duplicate column
  names, names not in the table, and missing required columns
  (NOT NULL without default), each with a 400 naming the offenders.
- Strip a UTF-8 BOM from the header (Excel exports).
- FORCE_NULL on all uploaded columns: an empty field is NULL whether
  quoted or not - a deliberate deviation from COPY's native CSV rule,
  because many writers quote every field and would silently store
  empty strings instead of NULLs.
- Sanitized failure responses: the database's data-level message plus
  the CSV line number and column (header-adjusted), never raw SQL,
  server context dumps, or internal paths - including the no-diagnostics
  fallback (lost connection), which stays generic.

Test module grows to 20 HTTP-seam tests covering each contract rule.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jh-RLI
jh-RLI force-pushed the feature-2362-bulk-upload-csv-contract branch from 204a99b to d1fb848 Compare July 17, 2026 15:13
@jh-RLI
jh-RLI merged commit 0057cf3 into develop Jul 17, 2026
4 of 5 checks passed
@jh-RLI
jh-RLI deleted the feature-2362-bulk-upload-csv-contract branch July 17, 2026 15:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant