Skip to content

Fix (b)run_fastsurfer on macOS, add python fs_time replacement, and tests for Mac in CI - #864

Merged
m-reuter merged 11 commits into
Deep-MI:devfrom
m-reuter:mac-bash
Sep 3, 2026
Merged

Fix (b)run_fastsurfer on macOS, add python fs_time replacement, and tests for Mac in CI#864
m-reuter merged 11 commits into
Deep-MI:devfrom
m-reuter:mac-bash

Conversation

@m-reuter

@m-reuter m-reuter commented Sep 2, 2026

Copy link
Copy Markdown
Member

Makes brun_fastsurfer.sh usable on macOS, with a smaller fix to run_fastsurfer.sh and new tests
to keep both that way. Also port fs_time to python to be portable across OSs, enabling timing in macOS.

Apple ships bash 3.2.57 as /bin/bash (frozen at the last GPLv2 release), and every shipped
script selects that interpreter through its own #!/bin/bash, whatever the user's login shell is.
Three separate GNU/bash-4 dependencies meant --subject_list, subjects on stdin and --parallel <n>
could not work there. The run_fastsurfer.sh change is one guard removal, restoring the
"Potentially Overwriting" warning that macOS users never saw.
Subject lists are also parsed correctly on macOS now: paths containing spaces can be quoted or escaped as in the shell, trailing whitespace and blank lines are stripped rather than a trailing  s , and a list file without a final newline keeps its last subject.

Found while packaging FastSurfer for macOS #859 ; none of it is macOS-packaging specific.

1. mapfile is bash 4+, so three features were refused outright

brun_fastsurfer.sh guarded its four mapfile calls with fail_bash_version_lt4, which exited with
"requires at minimum bash version 4" for --subject_list, subjects via stdin, and --parallel <n>.
On macOS that guard always fires, so those three features were simply unavailable.

Replaced with while IFS= read -r, which bash 3.2 has, so the features work rather than being
refused. The two duplicated job-counting sites collapse into one get_running_jobs helper, and
fail_bash_version_lt4 and its five call sites are gone.

run_fastsurfer.sh had the same guard around its "Potentially Overwriting" warning, meaning macOS
users never saw it when a subject directory already contained files. That block is now unconditional.

2. cut --output-delimiter is GNU-only, and silently dropped the t1 path

With the refusal lifted, the next failure appeared: cut -d= -f2-1000 --output-delimiter="=" in
process_by_token. BSD cut rejects the option, so image_parameters came out empty and the
image path in every <subject_id>=<path> spec was lost.

The option was redundant even on GNU, since cut -f already joins with the input delimiter, which is
why stools.sh never had it. Removed.

3. expr with a GNU BRE parsed nothing at all

The subject-spec tokenizer used expr "$s" : "$regex" with a BRE containing \| and \+, both GNU
extensions. BSD expr returns an empty string for every input, including a bare word, so no
subject parameters parsed on macOS at all.

Now matched with bash's own =~ (an ERE), which drops the expr dependency entirely. The pattern is
built from single-quoted pieces, because a single quote cannot appear inside a single-quoted string,
and it is verified against escaped spaces, both quote styles, and escapes nested inside double quotes.

Tests

test/shell/test_bash_compat.py, 19 tests:

  • 16 static lint tests, one per script, flagging constructs bash 3.2 lacks (mapfile,
    readarray, declare -A, declare -n, coproc, wait -n, &>>, ${x^^}/${x,,}) and
    reporting the file, line and a replacement. Platform independent, so it runs on every PR via
    the existing ubuntu matrix (tests: ["image", "config", "shell"]). Covers the pipeline entry
    points, all five recon_surf/*.sh, the macOS build, the install-time scripts, and the
    .sh.template files rendered into the installed package.
  • 3 functional tests for the paths that were refused, which need a real bash 3.2 and so run in a
    new macos-shell-test job on macos-15. That job also fails deliberately if /bin/bash stops
    being 3.x, so the guard cannot rot unnoticed the way it did before.

The lint alone would not have caught issues 2 and 3: both are GNU-only options to external tools,
and only executing the code on a BSD userland finds those. That is what the mac job buys, and the
same class of risk (sed -i, date -d, stat -c, grep -P) remains open elsewhere.

Verification

brun_fastsurfer.sh on both interpreters, invoked explicitly so its own shebang could not substitute
3.2, with a stub via --run_fastsurfer so nothing is actually processed:

bash 3.2 bash 5.3
--parallel 2, 4 subjects 4 started, peak concurrency 2 identical
--parallel max, 4 subjects 4 started, peak 4 identical

Asserting the peak is the point: an always-empty running_jobs would still start all four, so
counting subjects would not test the job accounting. End to end the stub receives
--t1 <path> --sd <dir> --device auto --viewagg_device auto --sid subjX, with per-subject flags such
as --seg_only passed through. The lint was confirmed to fail on an injected mapfile rather than
being decorative.

run_fastsurfer.sh --seg_only was run on a real subject on macOS (bash 3.2), against a copy of an
existing output directory, and produced the warning that used to be skipped there:

WARNING: Found 27 files in subject directory $SUBJECTS_DIR/OAS1_0011_MR1:
Potentially Overwriting: ./surf/callosum.surf ./surf/callosum.thickness.w ...
That path is not in CI, since it needs real input data; the lint covers it against regressions.
The Linux skip path for the three functional tests is correct by construction but will first be
confirmed by CI.

Also in this PR: fs_time no longer needs GNU time

recon_surf/fs_time wrapped /usr/bin/time -f, a GNU extension, so functions.sh probed it, found
it broken and silently disabled per-command timings on every macOS run. It now measures with
os.wait4(), so there is no external tool to be missing. Verified against real GNU time on Linux
(identical field layout, maxrss identical for a 500 MB command) and on macOS by a full seg-only
run, which produced 8 timed commands where it previously produced none.

The rewrite also fixes two things the bash version got wrong:  @#@FSLOADPOST  now carries the load averages its own help documents, and an empty  FSTIME_LOAD  counts as "on" rather than "off".

Two known limits, both documented in the file: M cannot read below the launcher's own footprint on
Linux, which only affects commands too small for the figure to be interesting, and macOS reports no
block IO or swap counts, so I, O and W read . there.

Log output clearer

For example, HypVINN used to report its own runtime as "whole pipeline"; that is fixed here.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Subject parsing drops unterminated final lines and passes quoted or escaped paths with their syntax characters intact.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Improves macOS Bash 3.2 compatibility and adds CI regression coverage.

Changes:

  • Replaces Bash 4/GNU-only shell constructs.
  • Restores overwrite warnings on macOS.
  • Adds static and functional shell compatibility tests.
File summaries
File Description
brun_fastsurfer.sh Reworks subject parsing and job tracking.
run_fastsurfer.sh Enables overwrite detection under Bash 3.2.
test/shell/test_bash_compat.py Adds compatibility tests.
.github/workflows/unittest.yaml Runs shell tests on Linux and macOS.
Review details

Suppressed comments (1)

brun_fastsurfer.sh:293

  • The stdin variant has the same final-line regression: if a producer closes stdin without a trailing newline, read fills subject_line but the loop body is skipped, so the final subject is never processed. Preserve a nonempty unterminated line as mapfile did.
  while IFS= read -r subject_line ; do subjects+=("$subject_line") ; done \
    < <(sed "$SED_CLEANUP_SUBJECTS")
  • Files reviewed: 4/4 changed files
  • Comments generated: 3
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread brun_fastsurfer.sh Outdated
Comment thread brun_fastsurfer.sh
Comment thread test/shell/test_bash_compat.py Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The unresolved moderate timing-wrapper and concurrency-test issues must be addressed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 9/9 changed files
  • Comments generated: 5
  • Review effort level: Balanced

Comment thread recon_surf/fs_time Outdated
Comment thread recon_surf/fs_time Outdated
Comment thread recon_surf/fs_time Outdated
Comment thread recon_surf/fs_time Outdated
Comment thread test/shell/test_brun_bash32.py Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Five unresolved moderate issues affect parsing correctness, safe execution, lint coverage, and functional test reliability.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

brun_fastsurfer.sh:562

  • Double-quoted backslashes do not always escape the next character in the shell: before an ordinary character they remain literal. This branch removes every backslash, so a valid subject value such as "a\ b" is passed as a b instead of a\ b, contrary to the documented shell-style parsing. Only consume the backslash before $, backtick, ", or \; otherwise append it literally.
          if [[ "${rest:0:1}" == "\\" ]] && [[ -n "${rest:1:1}" ]]
          then out="$out${rest:1:1}" ; rest="${rest:2}"
  • Files reviewed: 9/9 changed files
  • Comments generated: 4
  • Review effort level: Balanced

Comment thread recon_surf/fs_time Outdated
Comment thread test/shell/test_bash4_lint.py Outdated
Comment thread test/shell/test_brun_bash32.py Outdated
Comment thread test/shell/test_brun_bash32.py Outdated
@m-reuter
m-reuter requested a balanced review from Copilot September 2, 2026 21:27
@m-reuter m-reuter changed the title Fix brun_fastsurfer and run_fastsurfer on macOS, and test that in CI Fix (b)run_fastsurfer on macOS, add python fs_time replacement, and tests for Mac in CI Sep 2, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Two moderate test-coverage gaps must be addressed before approval.

Review details

Suppressed comments (4)

Previously missed (2) — in code that hasn't changed since the last review.

test/shell/test_bash4_lint.py:62

  • These patterns only match a standalone -A/-n, so valid combined or preceding flags such as declare -rA values and local -r -n ref bypass the Bash-4 guard. Match option groups containing the relevant flag so the lint covers all declaration spellings.
    test/shell/test_brun_bash32.py:87
  • The compatibility tests all expect brun_fastsurfer.sh to succeed, but this helper never checks its exit status. A regression that starts every stub correctly and then exits nonzero would therefore pass all the log-based assertions. Assert success here so every caller verifies the CLI contract as well as its side effects.

brun_fastsurfer.sh:68

  • This rule is quoting-dependent: unquote preserves backslashes inside single quotes and before non-special characters inside double quotes, so doubling a literal backslash there produces two backslashes. Please qualify that doubling is required outside quotes.
backslash likewise has to be written \\\\.

recon_surf/fs_time:125

  • This documented sample omits the literal L that format_load() prefixes to every load sample, so it is not an example of the line the rewritten command actually emits. Update the public output contract to match the implementation and the load-line test.
The 3 numbers are the system load averages for the past 1, 5, and 15 minutes.
  • Files reviewed: 10/10 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved lint failures, signal handling, and test coverage gaps must be addressed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (7)

Previously missed (2) — in code that hasn't changed since the last review.

recon_surf/fs_time:284

  • A Ctrl-C delivered to the foreground process group interrupts this wait4 with KeyboardInterrupt, so the wrapper emits a Python traceback and exits before writing the timing record or applying the child’s 128+signal status mapping. The self-signalling child test does not exercise this because only the child receives SIGTERM. Handle wrapper signals while still reaping the child and producing the resource line (and forward signals when only the wrapper was targeted).
    test/shell/test_bash4_lint.py:112
  • Skipping a missing hard-coded script makes an accidental rename or stale SHIPPED_SCRIPTS entry pass while silently removing that script from the compatibility scan. Treat this as a test failure so the coverage list cannot shrink unnoticed.

recon_surf/fs_time:136

  • UP037 is enabled for this newly linted executable, so this unnecessary quoted annotation is reported by Ruff.
def shorten_command(cmd: list[str]) -> "tuple[str, int]":

recon_surf/fs_time:166

  • Because UP037 is selected, Ruff reports this quoted annotation now that fs_time is included in the lint scope.
def make_line(key: str, command: str, nargs: int, extra: str = "", when: "datetime | None" = None) -> str:

recon_surf/fs_time:179

  • This non-forward quoted union violates the enabled Ruff UP037 rule and will fail the code-style job.
def field(name: str, value: "int | str") -> str:

test/shell/test_fs_time.py:40

  • Ruff's enabled UP037 rule rejects this quoted built-in annotation.
def parse_fields(line: str) -> "dict[str, str]":

test/shell/test_fs_time.py:175

  • This quoted union is also reported by the enabled Ruff UP037 rule.
def test_fstime_load_default(value: "str | None", load_expected: bool):
  • Files reviewed: 10/10 changed files
  • Comments generated: 3
  • Review effort level: Balanced

Comment thread recon_surf/fs_time Outdated
Comment thread test/shell/test_fs_time.py Outdated
Comment thread test/shell/test_brun_bash32.py
mapfile is bash 4+, so --subject_list, subjects on stdin and --parallel <n>
were refused on macOS, and run_fastsurfer skipped its overwrite warning there.
Read the lines instead. Also drop two GNU-only dependencies in the same path:
cut --output-delimiter, which silently emptied the t1 path, and an expr BRE using
\| and \+, which BSD expr rejects so no subject parameters parsed at all.
A static lint for bash-4+ constructs on every PR, plus functional tests for
the paths that were refused, run by a macos job because that is the only place
/bin/bash is 3.2.
Both figures cover only the hypothalamus module, and one of them is the last
timing line a run prints, so a seg-only run appeared to have taken as long as
this single module.
fs_time wrapped /usr/bin/time -f, and -f is a GNU extension, so on macOS the
probe in functions.sh failed and per-command timings were silently dropped for
every run. Take the numbers from os.wait4 instead, which needs no external tool
and covers both platforms. Verified against GNU time on linux: identical field
layout, and maxrss identical for a 500 MB command.
The lint only reads files, so it does not need a runner with the full
dependency set; the functional tests need a real bash 3.2 and stay on macOS.
Three faults in the subject-list path. read dropped a final line with no newline
after it, where mapfile had kept it. The cleanup used GNU's \s, which BSD sed reads as
the letter s, so it truncated any line ending in one and left trailing whitespace and
blank lines in place. And the tokenizer passed on the source text of each token, so a
quoted or escaped path arrived at --t1 with its quotes still attached and matched no
file, which ruled out the paths with spaces that are common on macOS. The lint now finds
recon_surf scripts by shebang as well as suffix, which also covers the sourced
functions.sh.
The stamp was read after wait4, and the extractor adds the elapsed seconds to it, so
every recorded interval was shifted forward by a full command duration. Also restore the
child's default SIGPIPE, which python ignores and posix_spawn inherited, count an empty
FSTIME_LOAD as on, and put the load averages in FSLOADPOST.
Inside double quotes a backslash only escapes $ ` " and \, so "a\ b" keeps its
backslash; unquote stripped it unconditionally. fs_time now opens -o up front and exits
125 like /usr/bin/time, instead of spending the command's runtime and then throwing a
traceback. Its tests move to a platform-independent file that runs on ubuntu as well,
which is what covers the linux side of the ru_maxrss and unmeasured-field branches.
The -A and -n rules only matched a flag adjacent to the keyword, so 'declare -rA'
and 'local -r -n ref' passed. A rule that stops matching looks exactly like a clean
repo, hence the table of spellings it now checks in both directions. Also assert
brun's exit status in the test helper, and correct two doc claims: a backslash needs
doubling only outside quotes, and the load line prints an L prefix.
Ctrl-C raised KeyboardInterrupt out of wait4, so the command that was interrupted lost
its timing line and left a traceback in the log. A no-op SIGINT handler lets the wait
restart instead; catching and retrying does not work, because wait4 can return and only
then raise, leaving nothing to reap. Also cover the two-stage --parallel_seg/--parallel_surf
schedulers, and fail rather than skip when a listed script is missing, since a rename would
otherwise shrink the scan unnoticed.
@m-reuter
m-reuter merged commit 27180ee into Deep-MI:dev Sep 3, 2026
5 checks passed
@m-reuter
m-reuter deleted the mac-bash branch September 3, 2026 21:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants