Support datasets mixing .ims and .ome.zarr image formats - #196
Merged
Conversation
snakebids requires every component to reduce to a single path template, and the image extension was baked into the spim template - so a dataset containing both Imaris files and OME-Zarr stores failed at dag-building with "Multiple path templates for one component". SPIMpack works around this by symlinking every image under a hardcoded .ims name, which makes zarr-backed stores invisible to pybids and silently drops those subjects from the workflow. Generate the spim component with `extension` as a wildcard so one template covers both formats, resolve each scan's concrete image path in rules through a spim_input() lookup function, and re-expose the component with the extension stripped from its wildcards/zip_lists so it cannot leak into derived output templates. Verified on a uniform all-.ims dataset (job table identical to baseline, 6078 jobs) and on a mixed 24x.ims/7x.ome.zarr dataset (all 31 subjects planned, per-subject inputs resolve to the correct format, no extension appears in output paths). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012437t8NSZLaZAwoAWK6bNT
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the Snakemake/Snakebids input handling so a single spim BIDS component can represent datasets that mix SPIM image formats (e.g., .ims alongside .ome.zarr) without failing DAG construction. It does so by making extension a wildcard for discovery, then resolving each job’s concrete input path via a lookup function while preventing extension from propagating into downstream output templates.
Changes:
- Adds an
extensionwildcard to thespimSnakebids input component and introducesspim_input()to resolve the concrete per-scan input path. - Re-exposes
inputs["spim"]withextensionremoved from its wildcards/zip lists to avoid leakingextensioninto derived output templates. - Updates rules to use
spim_input(and updates metadata prepopulation to iterate over the raw component).
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| spimquant/workflow/Snakefile | Introduces spim_raw, builds a wildcard→path lookup, defines spim_input(), and re-wraps the spim component without extension for downstream expansion. |
| spimquant/workflow/rules/common.smk | Updates stain discovery to iterate over the raw SPIM inputs (spim_raw.expand()). |
| spimquant/workflow/rules/vessels.smk | Switches raw SPIM rule inputs to spim_input. |
| spimquant/workflow/rules/templatereg.smk | Switches raw SPIM references (inputs/params) to spim_input. |
| spimquant/workflow/rules/segmentation.smk | Switches raw SPIM inputs to spim_input. |
| spimquant/workflow/rules/qc.smk | Switches raw SPIM inputs to spim_input. |
| spimquant/workflow/rules/plaques.smk | Switches raw SPIM inputs to spim_input. |
| spimquant/workflow/rules/patches.smk | Switches raw SPIM inputs to spim_input. |
| spimquant/workflow/rules/import.smk | Switches raw SPIM inputs to spim_input. |
| spimquant/workflow/rules/counts.smk | Switches raw SPIM inputs to spim_input. |
| spimquant/config/snakebids.yml | Adds extension to the spim wildcards list so a single template covers multiple formats. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Member
|
this looks good to me, will merge in after I do a quick check myself (had a review suggestion implemented to give an error if it finds duplicates, e.g. ome.zarr and ims for a subject) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
snakebids requires every component to reduce to a single path template, and the image extension was baked into the spim template - so a dataset containing both Imaris files and OME-Zarr stores failed at dag-building.
Generate the spim component with
extensionas a wildcard so one template covers both formats, resolve each scan's concrete image path in rules through a spim_input() lookup function, and re-expose the component with the extension stripped from its wildcards/zip_lists so it cannot leak into derived output templates.