Fix (b)run_fastsurfer on macOS, add python fs_time replacement, and tests for Mac in CI - #864
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Subject parsing drops unterminated final lines and passes quoted or escaped paths with their syntax characters intact.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Improves macOS Bash 3.2 compatibility and adds CI regression coverage.
Changes:
- Replaces Bash 4/GNU-only shell constructs.
- Restores overwrite warnings on macOS.
- Adds static and functional shell compatibility tests.
File summaries
| File | Description |
|---|---|
brun_fastsurfer.sh |
Reworks subject parsing and job tracking. |
run_fastsurfer.sh |
Enables overwrite detection under Bash 3.2. |
test/shell/test_bash_compat.py |
Adds compatibility tests. |
.github/workflows/unittest.yaml |
Runs shell tests on Linux and macOS. |
Review details
Suppressed comments (1)
brun_fastsurfer.sh:293
- The stdin variant has the same final-line regression: if a producer closes stdin without a trailing newline,
readfillssubject_linebut the loop body is skipped, so the final subject is never processed. Preserve a nonempty unterminated line asmapfiledid.
while IFS= read -r subject_line ; do subjects+=("$subject_line") ; done \
< <(sed "$SED_CLEANUP_SUBJECTS")
- Files reviewed: 4/4 changed files
- Comments generated: 3
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
There was a problem hiding this comment.
🟡 Changes recommended
The unresolved moderate timing-wrapper and concurrency-test issues must be addressed before approval.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 9/9 changed files
- Comments generated: 5
- Review effort level: Balanced
There was a problem hiding this comment.
🟡 Changes recommended
Five unresolved moderate issues affect parsing correctness, safe execution, lint coverage, and functional test reliability.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
brun_fastsurfer.sh:562
- Double-quoted backslashes do not always escape the next character in the shell: before an ordinary character they remain literal. This branch removes every backslash, so a valid subject value such as
"a\ b"is passed asa binstead ofa\ b, contrary to the documented shell-style parsing. Only consume the backslash before$, backtick,", or\; otherwise append it literally.
if [[ "${rest:0:1}" == "\\" ]] && [[ -n "${rest:1:1}" ]]
then out="$out${rest:1:1}" ; rest="${rest:2}"
- Files reviewed: 9/9 changed files
- Comments generated: 4
- Review effort level: Balanced
There was a problem hiding this comment.
🔵 Needs a closer look
Two moderate test-coverage gaps must be addressed before approval.
Review details
Suppressed comments (4)
Previously missed (2) — in code that hasn't changed since the last review.
test/shell/test_bash4_lint.py:62
- These patterns only match a standalone
-A/-n, so valid combined or preceding flags such asdeclare -rA valuesandlocal -r -n refbypass the Bash-4 guard. Match option groups containing the relevant flag so the lint covers all declaration spellings.
test/shell/test_brun_bash32.py:87 - The compatibility tests all expect
brun_fastsurfer.shto succeed, but this helper never checks its exit status. A regression that starts every stub correctly and then exits nonzero would therefore pass all the log-based assertions. Assert success here so every caller verifies the CLI contract as well as its side effects.
brun_fastsurfer.sh:68
- This rule is quoting-dependent:
unquotepreserves backslashes inside single quotes and before non-special characters inside double quotes, so doubling a literal backslash there produces two backslashes. Please qualify that doubling is required outside quotes.
backslash likewise has to be written \\\\.
recon_surf/fs_time:125
- This documented sample omits the literal
Lthatformat_load()prefixes to every load sample, so it is not an example of the line the rewritten command actually emits. Update the public output contract to match the implementation and the load-line test.
The 3 numbers are the system load averages for the past 1, 5, and 15 minutes.
- Files reviewed: 10/10 changed files
- Comments generated: 0 new
- Review effort level: Balanced
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved lint failures, signal handling, and test coverage gaps must be addressed before approval.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (7)
Previously missed (2) — in code that hasn't changed since the last review.
recon_surf/fs_time:284
- A Ctrl-C delivered to the foreground process group interrupts this
wait4withKeyboardInterrupt, so the wrapper emits a Python traceback and exits before writing the timing record or applying the child’s 128+signal status mapping. The self-signalling child test does not exercise this because only the child receives SIGTERM. Handle wrapper signals while still reaping the child and producing the resource line (and forward signals when only the wrapper was targeted).
test/shell/test_bash4_lint.py:112 - Skipping a missing hard-coded script makes an accidental rename or stale
SHIPPED_SCRIPTSentry pass while silently removing that script from the compatibility scan. Treat this as a test failure so the coverage list cannot shrink unnoticed.
recon_surf/fs_time:136
UP037is enabled for this newly linted executable, so this unnecessary quoted annotation is reported by Ruff.
def shorten_command(cmd: list[str]) -> "tuple[str, int]":
recon_surf/fs_time:166
- Because
UP037is selected, Ruff reports this quoted annotation now thatfs_timeis included in the lint scope.
def make_line(key: str, command: str, nargs: int, extra: str = "", when: "datetime | None" = None) -> str:
recon_surf/fs_time:179
- This non-forward quoted union violates the enabled Ruff
UP037rule and will fail the code-style job.
def field(name: str, value: "int | str") -> str:
test/shell/test_fs_time.py:40
- Ruff's enabled
UP037rule rejects this quoted built-in annotation.
def parse_fields(line: str) -> "dict[str, str]":
test/shell/test_fs_time.py:175
- This quoted union is also reported by the enabled Ruff
UP037rule.
def test_fstime_load_default(value: "str | None", load_expected: bool):
- Files reviewed: 10/10 changed files
- Comments generated: 3
- Review effort level: Balanced
mapfile is bash 4+, so --subject_list, subjects on stdin and --parallel <n> were refused on macOS, and run_fastsurfer skipped its overwrite warning there. Read the lines instead. Also drop two GNU-only dependencies in the same path: cut --output-delimiter, which silently emptied the t1 path, and an expr BRE using \| and \+, which BSD expr rejects so no subject parameters parsed at all.
A static lint for bash-4+ constructs on every PR, plus functional tests for the paths that were refused, run by a macos job because that is the only place /bin/bash is 3.2.
Both figures cover only the hypothalamus module, and one of them is the last timing line a run prints, so a seg-only run appeared to have taken as long as this single module.
fs_time wrapped /usr/bin/time -f, and -f is a GNU extension, so on macOS the probe in functions.sh failed and per-command timings were silently dropped for every run. Take the numbers from os.wait4 instead, which needs no external tool and covers both platforms. Verified against GNU time on linux: identical field layout, and maxrss identical for a 500 MB command.
The lint only reads files, so it does not need a runner with the full dependency set; the functional tests need a real bash 3.2 and stay on macOS.
Three faults in the subject-list path. read dropped a final line with no newline after it, where mapfile had kept it. The cleanup used GNU's \s, which BSD sed reads as the letter s, so it truncated any line ending in one and left trailing whitespace and blank lines in place. And the tokenizer passed on the source text of each token, so a quoted or escaped path arrived at --t1 with its quotes still attached and matched no file, which ruled out the paths with spaces that are common on macOS. The lint now finds recon_surf scripts by shebang as well as suffix, which also covers the sourced functions.sh.
The stamp was read after wait4, and the extractor adds the elapsed seconds to it, so every recorded interval was shifted forward by a full command duration. Also restore the child's default SIGPIPE, which python ignores and posix_spawn inherited, count an empty FSTIME_LOAD as on, and put the load averages in FSLOADPOST.
Inside double quotes a backslash only escapes $ ` " and \, so "a\ b" keeps its backslash; unquote stripped it unconditionally. fs_time now opens -o up front and exits 125 like /usr/bin/time, instead of spending the command's runtime and then throwing a traceback. Its tests move to a platform-independent file that runs on ubuntu as well, which is what covers the linux side of the ru_maxrss and unmeasured-field branches.
The -A and -n rules only matched a flag adjacent to the keyword, so 'declare -rA' and 'local -r -n ref' passed. A rule that stops matching looks exactly like a clean repo, hence the table of spellings it now checks in both directions. Also assert brun's exit status in the test helper, and correct two doc claims: a backslash needs doubling only outside quotes, and the load line prints an L prefix.
Ctrl-C raised KeyboardInterrupt out of wait4, so the command that was interrupted lost its timing line and left a traceback in the log. A no-op SIGINT handler lets the wait restart instead; catching and retrying does not work, because wait4 can return and only then raise, leaving nothing to reap. Also cover the two-stage --parallel_seg/--parallel_surf schedulers, and fail rather than skip when a listed script is missing, since a rename would otherwise shrink the scan unnoticed.
Makes
brun_fastsurfer.shusable on macOS, with a smaller fix torun_fastsurfer.shand new teststo keep both that way. Also port fs_time to python to be portable across OSs, enabling timing in macOS.
Apple ships bash 3.2.57 as
/bin/bash(frozen at the last GPLv2 release), and every shippedscript selects that interpreter through its own
#!/bin/bash, whatever the user's login shell is.Three separate GNU/bash-4 dependencies meant
--subject_list, subjects on stdin and--parallel <n>could not work there. The
run_fastsurfer.shchange is one guard removal, restoring the"Potentially Overwriting" warning that macOS users never saw.
Subject lists are also parsed correctly on macOS now: paths containing spaces can be quoted or escaped as in the shell, trailing whitespace and blank lines are stripped rather than a trailing s , and a list file without a final newline keeps its last subject.
Found while packaging FastSurfer for macOS #859 ; none of it is macOS-packaging specific.
1.
mapfileis bash 4+, so three features were refused outrightbrun_fastsurfer.shguarded its fourmapfilecalls withfail_bash_version_lt4, which exited with"requires at minimum bash version 4" for
--subject_list, subjects via stdin, and--parallel <n>.On macOS that guard always fires, so those three features were simply unavailable.
Replaced with
while IFS= read -r, which bash 3.2 has, so the features work rather than beingrefused. The two duplicated job-counting sites collapse into one
get_running_jobshelper, andfail_bash_version_lt4and its five call sites are gone.run_fastsurfer.shhad the same guard around its "Potentially Overwriting" warning, meaning macOSusers never saw it when a subject directory already contained files. That block is now unconditional.
2.
cut --output-delimiteris GNU-only, and silently dropped the t1 pathWith the refusal lifted, the next failure appeared:
cut -d= -f2-1000 --output-delimiter="="inprocess_by_token. BSDcutrejects the option, soimage_parameterscame out empty and theimage path in every
<subject_id>=<path>spec was lost.The option was redundant even on GNU, since
cut -falready joins with the input delimiter, which iswhy
stools.shnever had it. Removed.3.
exprwith a GNU BRE parsed nothing at allThe subject-spec tokenizer used
expr "$s" : "$regex"with a BRE containing\|and\+, both GNUextensions. BSD
exprreturns an empty string for every input, including a bare word, so nosubject parameters parsed on macOS at all.
Now matched with bash's own
=~(an ERE), which drops theexprdependency entirely. The pattern isbuilt from single-quoted pieces, because a single quote cannot appear inside a single-quoted string,
and it is verified against escaped spaces, both quote styles, and escapes nested inside double quotes.
Tests
test/shell/test_bash_compat.py, 19 tests:mapfile,readarray,declare -A,declare -n,coproc,wait -n,&>>,${x^^}/${x,,}) andreporting the file, line and a replacement. Platform independent, so it runs on every PR via
the existing ubuntu matrix (
tests: ["image", "config", "shell"]). Covers the pipeline entrypoints, all five
recon_surf/*.sh, the macOS build, the install-time scripts, and the.sh.templatefiles rendered into the installed package.new
macos-shell-testjob onmacos-15. That job also fails deliberately if/bin/bashstopsbeing 3.x, so the guard cannot rot unnoticed the way it did before.
The lint alone would not have caught issues 2 and 3: both are GNU-only options to external tools,
and only executing the code on a BSD userland finds those. That is what the mac job buys, and the
same class of risk (
sed -i,date -d,stat -c,grep -P) remains open elsewhere.Verification
brun_fastsurfer.shon both interpreters, invoked explicitly so its own shebang could not substitute3.2, with a stub via
--run_fastsurferso nothing is actually processed:--parallel 2, 4 subjects--parallel max, 4 subjectsAsserting the peak is the point: an always-empty
running_jobswould still start all four, socounting subjects would not test the job accounting. End to end the stub receives
--t1 <path> --sd <dir> --device auto --viewagg_device auto --sid subjX, with per-subject flags suchas
--seg_onlypassed through. The lint was confirmed to fail on an injectedmapfilerather thanbeing decorative.
run_fastsurfer.sh --seg_onlywas run on a real subject on macOS (bash 3.2), against a copy of anexisting output directory, and produced the warning that used to be skipped there:
WARNING: Found 27 files in subject directory $SUBJECTS_DIR/OAS1_0011_MR1:
Potentially Overwriting: ./surf/callosum.surf ./surf/callosum.thickness.w ...
That path is not in CI, since it needs real input data; the lint covers it against regressions.
The Linux skip path for the three functional tests is correct by construction but will first be
confirmed by CI.
Also in this PR: fs_time no longer needs GNU time
recon_surf/fs_timewrapped/usr/bin/time -f, a GNU extension, sofunctions.shprobed it, foundit broken and silently disabled per-command timings on every macOS run. It now measures with
os.wait4(), so there is no external tool to be missing. Verified against real GNUtimeon Linux(identical field layout,
maxrssidentical for a 500 MB command) and on macOS by a full seg-onlyrun, which produced 8 timed commands where it previously produced none.
The rewrite also fixes two things the bash version got wrong: @#@FSLOADPOST now carries the load averages its own help documents, and an empty FSTIME_LOAD counts as "on" rather than "off".
Two known limits, both documented in the file:
Mcannot read below the launcher's own footprint onLinux, which only affects commands too small for the figure to be interesting, and macOS reports no
block IO or swap counts, so
I,OandWread.there.Log output clearer
For example,
HypVINNused to report its own runtime as "whole pipeline"; that is fixed here.