Skip to content

Add a safe compatibility path for opaque FUSE imports - #905

Draft
dktrekkie wants to merge 2 commits into
Listenarrs:canaryfrom
dktrekkie:feature/836-fix-unraid-fuse-identity
Draft

Add a safe compatibility path for opaque FUSE imports#905
dktrekkie wants to merge 2 commits into
Listenarrs:canaryfrom
dktrekkie:feature/836-fix-unraid-fuse-identity

Conversation

@dktrekkie

Copy link
Copy Markdown

Summary

This is a compact proof of concept for #836. It addresses two independent
filesystem boundaries exposed by an unattended download import on an Unraid
user share:

  1. Linux can return an opaque file handle without proving that the backing
    filesystem provides the persistence guarantees required for durable object
    identity.
  2. A translated download source outside the configured library root can require
    explicit case semantics when read-only automatic detection is inconclusive.

The implementation deliberately adds generic policy and carrier seams instead
of special-casing Unraid, shfs, a download-client vendor, or a mount path. It
makes unattended imports safe and functional through Listenarr's existing
compatibility copy-and-retain path.

This is proposed as an interim compatibility path, not as the final hardlink
solution for Unraid. Its purpose is to unblock affected imports without granting
destructive authority from identity evidence that was reproduced as unstable.
It gives the project room to design verified hardlink publication for weak
storage separately. Until then, affected HardlinkCopy imports complete as
full copies and retain their sources, with the time and storage cost described
below.

Reproduced failure sequence

The download client completed successfully, and Listenarr could enumerate and
probe every source file. Import then failed before publication because the
library root no longer matched its persisted physical identity. Confirming the
root only helped temporarily: the opaque fuse.shfs handle changed after the
same directory was looked up again, so Listenarr treated the unchanged path as
a different directory.

Making FUSE identity fail closed allowed the root to enter Listenarr's existing
limited-storage contract, but exposed a second boundary. The translated local
download source is outside the library root and still resolved case semantics
with Auto. On fuse.shfs, the flags ioctl was unsupported and the read-only
existing-entry probe was inconclusive, so import stopped before publication.

With the final PoC image, the library root and translated source were configured
explicitly as Sensitive. The same unattended workflow then completed: four
release artifacts were compatibility-published, the audio file was registered,
the follow-up scan completed, and all source files were retained.

Why this design

The initial working prototype inferred case behavior from the exact
fuse.shfs mount subtype. That solved the reproduced system but coupled generic
filesystem code to one Unraid implementation. It was removed.

The final design instead separates two questions:

  • whether opaque filesystem-object evidence is trusted for durable identity;
  • which case semantics describe a specific download source endpoint.

Both decisions are named and independently replaceable. The identity decision
is enforced once at the existing Linux generation-candidate funnel, and the
source setting is parsed once per processing job. No per-file probing loop,
trust cache, vendor detection, or downloader-specific branch is introduced.

Changes

Filesystem-object identity trust

  • Add a named Auto / Trusted / Untrusted trust policy.
  • Apply the policy once at the existing Linux generation-candidate funnel, so
    every consumer gets the same result without call-site plumbing.
  • Under Auto, keep opaque handles on known non-FUSE filesystems, but do not
    probe or persist them on FUSE or when filesystem type is unavailable.
  • Preserve an independently available inode generation, including generation
    zero, and retain the target branch's existing FID/fallback behavior.
  • Share one uncached fstatfs helper with the existing filesystem-semantics
    code. No trust decision is cached.

Download-source case semantics

  • Add an optional named-only download-client setting,
    sourceCaseSensitivityMode, defaulting to Auto for existing clients.
  • Parse it once per processing job and carry it through the import boundary to
    resolution of the translated local source path.
  • Fall back to whole-set Auto when archive extraction or staging means a file
    set no longer has one trustworthy source endpoint.

Hardlink behavior and trade-off

Listenarr already contains native Linux hardlink support, and this PoC does not
modify that implementation. The existing publication planner only selects its
durable hardlink path when source object identity is durable and the destination
authorizes filesystem mutation.

The configured action is named HardlinkCopy, and Listenarr already treats copy
as its fallback when a hardlink cannot be created. This PoC therefore uses an
existing product fallback rather than inventing a new publication behavior.
However, the distinction still matters: on the affected weak FUSE path the
fallback becomes systematic, so the user's performance and storage goal is not
met even though the import itself succeeds.

Under this PoC, Auto does not grant that authority from an opaque FUSE handle.
The planner therefore selects AdditiveCopyRetainSource, even when the requested
completed-file action is HardlinkCopy. Compatibility publication creates a
new file, copies and flushes all bytes, verifies the destination, registers it,
and retains the source.

The end-to-end validation source and destination were exposed through one
container mount and were physically located on the same backing ZFS pool, so
cross-device layout was not the reason for the fallback. A post-import check
showed different inodes with link count one: the successful result was a full
copy, not a hardlink.

The fallback is not represented internally as a successful hardlink. Each
import result preserves RequestedAction=HardlinkCopy and
EffectiveAction=Copy, together with durable_identity_unavailable and a
compatibility-publication message. Those details are recorded in history,
although the current UI may not make the distinction prominent.

This is a deliberate safety trade-off, not a claim that Unraid cannot support
hardlinks. It means imports take copy time and consume duplicate space while the
source remains present. The unresolved product work is to support verified
hardlink publication on weak storage without treating link success as proof
that later destructive cleanup is safe. That likely requires additional
journaling, verification, policy, and UI work, but not necessarily a rewrite of
the native hardlink implementation. That extension is intentionally outside
this PoC.

Intentional PoC boundary

This PR demonstrates the architecture and provides a usable mitigation for the
reproduced import without claiming to finish the product design. It can be
reviewed either as a small compatibility change that is safe to ship while the
hardlink path is designed, or as a concrete reference implementation for that
later work. It does not add UI, API schema, migration, automatic Unraid
detection, downloader-specific behavior, or a final choice of where the
identity-trust setting should live.

For validation, sourceCaseSensitivityMode=Sensitive was added manually to the
existing download-client settings JSON; no database column was introduced. The
current UI does not carry this unknown property and can remove it when a client
is edited. Likewise, the identity policy currently uses Auto at the seam and
does not expose a product-level carrier. These are explicit PoC limitations,
not finished configuration UX.

Maintainer decisions intentionally left open are:

  • where an identity-trust carrier belongs, if it should be configurable;
  • how source endpoint semantics should be exposed or derived;
  • whether weak storage should support verified hardlink publication separately
    from destructive move/source-cleanup authority.

Testing

  • dotnet format listenarr.slnx --no-restore --verify-no-changes: passed.
  • Full Windows backend run: 3,039 passed and 211 platform tests skipped; the
    same four unrelated baseline failures remained.
  • Affected native Linux groups: 167 passed, one Windows-only test skipped, and
    the same four established baseline failures remained.
  • The identity commit alone compiled and ran its focused Windows groups with
    48 passed, 15 platform skips, and zero failures.
  • Read-only native probes showed that the affected FUSE share and an XFS
    control expose the same raw handle shape. Production policy rejected the
    FUSE handle while retaining usable identity on XFS and ZFS controls.
  • Explicit source and library-root semantics both resolved as authoritative
    and case-sensitive on the affected system.
  • A production image built successfully, completed fresh-database migrations,
    and returned HTTP 200 in an isolated smoke container.
  • One fresh unattended import through SABnzbd completed with zero retries. This
    validates the generic path with the client used to reproduce FUSE mounts pass the durable-identity probe: name_to_handle_at returns an ephemeral nodeid, so the root folder is Healthy with mutation until the inode is evicted #836; the code
    contains no SABnzbd-specific branch. Four release artifacts were
    compatibility-published, the audio file was registered and scanned, and all
    four source files were retained. Destructive filesystem authority remained
    disabled.

Notes

  • The production validation used Listenarr's existing weak-storage
    copy-and-retain path; no source-deletion authority was enabled.
  • The investigation, design reviews, and implementation were AI-assisted, as
    already disclosed in the issue discussion.

Classify filesystem-object handle trust once at the Linux generation-candidate funnel. Auto mode keeps known non-FUSE behavior and inode-generation evidence, while refusing opaque handles on FUSE or when filesystem type is unavailable.
Read an optional named client setting once per processing job and carry it through the import boundary. Existing clients default to Auto, while mixed archive or staging inputs conservatively fall back to whole-set Auto.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant