From f0960b77bb3954c80033bf3e2057346955f30ef4 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 19 Sep 2026 16:21:05 +0000 Subject: [PATCH 01/10] Add a browser-based manual aligner, cortex.align.webgl_manual Port of the mayavi aligner to the WebGL viewer: the functional reference volume stays on its own voxel grid, the pial and white matter surfaces are moved into its space by rotations and translations only, and they are cut off at the displayed slices so that their outline shows on each slice. - cortex/webgl/aligner.py: tornado server (reference mosaic, CTM pack, page, save endpoint), the world frame (voxel grid in mm permuted to RAS) and the JSAligner handle with tagged calls and frame waits - resources/js/aligner.js, aligner.html, resources/css/aligner.css: coronal, axial, sagittal and 3D views, cursor and slice navigation, mesh translation and rotation by mouse and keyboard with undo, colormap with range, brightness, contrast, gamma and flip, mesh color and opacity, a mode painting the volume onto the surface, saving - shaderlib.js: aligner_volume and aligner_mesh shader builders - menu.js: color picker controls - cortex.align.webgl_manual, [webgl_aligner] config defaults, docs and tests Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_0115CeZYMxr1wvRSL2V8meKZ --- AGENTS.md | 1 + cortex/align.py | 58 ++ cortex/defaults.cfg | 9 + cortex/tests/test_webgl_aligner.py | 330 +++++++ cortex/webgl/aligner.html | 64 ++ cortex/webgl/aligner.py | 472 +++++++++ cortex/webgl/resources/css/aligner.css | 105 ++ cortex/webgl/resources/js/aligner.js | 1250 ++++++++++++++++++++++++ cortex/webgl/resources/js/menu.js | 3 +- cortex/webgl/resources/js/shaderlib.js | 151 +++ docs/align.rst | 57 +- 11 files changed, 2490 insertions(+), 10 deletions(-) create mode 100644 cortex/tests/test_webgl_aligner.py create mode 100644 cortex/webgl/aligner.html create mode 100644 cortex/webgl/aligner.py create mode 100644 cortex/webgl/resources/css/aligner.css create mode 100644 cortex/webgl/resources/js/aligner.js diff --git a/AGENTS.md b/AGENTS.md index 38b4c8f61..bff22d942 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -67,6 +67,7 @@ Flow: dataviews → JSON + PNG mosaics (`data.py:Package`) + compressed CTM surf - All GLSL lives in `resources/js/shaderlib.js` as arrays of string lines assembled per-configuration — there are no `.glsl` files. - CTM packs reorder vertices: `cortex.utils.get_ctmmap` / `get_ctm2webgl_map` translate between CTM/WebGL ordering and the original surface ordering. Indexing viewer data with original vertex indices without remapping is a classic bug. - `cortex/export/headless.py` (`headless_viewer`) runs the viewer in headless Chromium via Playwright for screenshots/tests. +- `cortex/webgl/aligner.py` + `resources/js/aligner.js` (`aligner.html`, `resources/css/aligner.css`) is the browser-based manual aligner behind `cortex.align.webgl_manual`. It reuses the CTM pack, the mosaic PNG loader (`dataset.VolumeData`) and the `Shaders.aligner_*` builders in `shaderlib.js`, and works in a world frame that is the reference volume's voxel grid in mm permuted to RAS (`reference_frame`); the edited transform is the pycortex `coord` transform. - Some JS resources use CRLF line endings (e.g. `dataset.js`); keep the existing endings when editing. - `setup.py` explicitly enumerates `cortex.webgl` `package_data` patterns — new resource subdirectories must be added there or they won't ship in wheels. diff --git a/cortex/align.py b/cortex/align.py index 1940221fd..6b902f39e 100644 --- a/cortex/align.py +++ b/cortex/align.py @@ -100,6 +100,64 @@ def view_callback(aligner): return m +def webgl_manual(subject: str, xfmname: str, reference: Optional[str] = None, **kwargs): + """Open the browser-based aligner for manually aligning a functional volume + to the cortical surface of `subject`. + + This is the WebGL port of the mayavi aligner (``mayavi_manual``). The + functional reference volume stays on its own voxel grid, so its slices + are shown without resampling, and the pial and white matter surfaces are + moved into its space; in each slice view the surfaces are cut off at the + displayed slice, so that their outline can be compared with the anatomy + in the image. Only rotations and translations are possible. + + The page shows the coronal, axial and sagittal slices and a 3D view of + the three slices. In a slice view, a left drag moves the cursor, which + sets the slices of the other views and is the pivot of rotations; the + wheel (or ``[`` and ``]``) changes the slice, ctrl + wheel zooms, and a + middle (or shift + left) drag pans. A right drag or the arrow keys + translate the surfaces in the plane of the view under the mouse, ctrl + + right drag or ``q`` / ``e`` rotate them about the cursor; shift makes + the keyboard steps ten times smaller and ctrl + z undoes. The panel on + the right holds the Save button, the view mode (surface outlines on the + slices, or the volume painted on the surface, also toggled with ``m``), + the colormap with its range, brightness, contrast, gamma and flip, the + color and opacity of the surfaces, the slices and the keyboard steps. + + When Save is pressed the transform is stored into the pycortex database + as `xfmname`, as a 'coord' transform. A new transform requires + `reference`, which is copied into the database; an existing transform + is loaded together with its stored reference, and refuses to open for + editing while masks are cached for it (pass ``view_only=True`` to + inspect it). + + Parameters + ---------- + subject : str + Subject identifier. + xfmname : str + Name of the transform to create or modify. + reference : str, optional + Path to a nibabel-readable functional volume, required for a new + transform. Must be None for an existing transform. + kwargs : dict + Passed to :func:`cortex.webgl.aligner.show` (``view_only``, ``cmap``, + ``mesh_color``, ``mesh_opacity``, ``open_browser``, ``port``, ...). + + Returns + ------- + handle : cortex.webgl.aligner.JSAligner or cortex.webgl.serve.WebApp + A handle to the running aligner: ``handle.get_xfm()`` returns the + current transform, ``handle.save()`` saves it. When the aligner is + started with ``open_browser=False`` the tornado server is returned + instead; its ``get_client()`` returns the handle once a browser has + connected. + """ + from .webgl import aligner + + return aligner.show(subject, xfmname, reference=reference, **kwargs) + + def fs_manual(subject, xfmname, **kwargs): """Legacy name for cortex.align.manual. Please use that function, and see the help there.""" warnings.warn(("Deprecated name - function has been renamed cortex.align.manual" diff --git a/cortex/defaults.cfg b/cortex/defaults.cfg index fb72de6d4..a41215027 100644 --- a/cortex/defaults.cfg +++ b/cortex/defaults.cfg @@ -47,6 +47,15 @@ outline_rep = wireframe opacity = 0 colormap = gray +[webgl_aligner] +# Initial settings of the browser-based aligner (cortex.align.webgl_manual): +# the colormap of the reference volume (a 1D pycortex colormap), the color of +# the surface outlines (a matplotlib color) and the opacity of the whole +# surfaces in the 3D view (0 shows only their outlines on the slices). +colormap = gray +mesh_color = white +mesh_opacity = 0 + [paths_default] stroke = white fill = none diff --git a/cortex/tests/test_webgl_aligner.py b/cortex/tests/test_webgl_aligner.py new file mode 100644 index 000000000..825c9c09b --- /dev/null +++ b/cortex/tests/test_webgl_aligner.py @@ -0,0 +1,330 @@ +"""Tests for the browser-based manual aligner (cortex.webgl.aligner). + +The pure-python tests cover the world frame the aligner works in and the +tornado endpoints of its server. The browser test drives the aligner in +headless Chromium and is skipped without playwright. +""" + +import base64 +import io +import json +import time +import urllib.parse +import urllib.request + +import numpy as np +import pytest + +import cortex +from cortex import align, database +from cortex.webgl import aligner +from cortex.tests.testing_utils import has_playwright + +subj = "S1" +xfmname = "fullhead" + + +def _reference(): + return database.db.get_xfm(subj, xfmname).reference_nifti + + +def _open(url, data=None, timeout=30): + """Fetch `url`; posting `data` (a dict) as a form when given.""" + body = None if data is None else urllib.parse.urlencode(data).encode() + with urllib.request.urlopen(url, data=body, timeout=timeout) as resp: + return resp.read() + + +class _SaveRecorder: + """Stands in for db.save_xfm, so tests never write into the filestore.""" + + def __init__(self): + self.calls = [] + + def __call__(self, subject, name, xfm, xfmtype="magnet", reference=None): + self.calls.append(dict(subject=subject, name=name, xfm=np.asarray(xfm, dtype=float), + xfmtype=xfmtype, reference=reference)) + + def wait(self, count=1, timeout=20): + deadline = time.monotonic() + timeout + while len(self.calls) < count and time.monotonic() < deadline: + time.sleep(0.1) + return len(self.calls) >= count + + +@pytest.fixture +def recorder(monkeypatch): + rec = _SaveRecorder() + monkeypatch.setattr(database.db, "save_xfm", rec) + return rec + + +@pytest.fixture +def server(request): + """A running aligner server for the bundled transform, stopped at teardown.""" + kwargs = dict(open_browser=False, display_url=False) + kwargs.update(getattr(request, "param", {})) + srv = aligner.show(subj, xfmname, **kwargs) + yield srv + srv.stop() + + +# --------------------------------------------------------------------------- +# World frame and reference loading +# --------------------------------------------------------------------------- + + +def test_reference_frame_is_scaled_signed_permutation(): + """Each voxel axis maps to one world axis, scaled by its voxel size.""" + import nibabel + + nii = _reference() + world = aligner.reference_frame(nii) + zooms = np.asarray(nii.header.get_zooms()[:3]) + + assert world.shape == (4, 4) + assert np.allclose(world[3], [0, 0, 0, 1]) + assert np.allclose(world[:3, 3], 0) + linear = world[:3, :3] + # one nonzero entry per row and per column, of the voxel size + assert np.array_equal((linear != 0).sum(axis=0), [1, 1, 1]) + assert np.array_equal((linear != 0).sum(axis=1), [1, 1, 1]) + assert np.allclose(np.abs(linear).sum(axis=0), zooms) + + # the bundled reference is stored L, P, S: x and y flip, z does not + assert nibabel.aff2axcodes(nii.affine) == ("L", "P", "S") + assert world[0, 0] < 0 and world[1, 1] < 0 and world[2, 2] > 0 + + +def test_reference_frame_follows_axis_permutation(): + """A reference stored in a different axis order gets its axes permuted + so that world x, y, z point right, anterior and superior.""" + import nibabel + + # voxel axes are (superior, right, anterior) with 2, 3 and 4 mm voxels + affine = np.array([[0, 3.0, 0, 0], + [0, 0, 4.0, 0], + [2.0, 0, 0, 0], + [0, 0, 0, 1]]) + nii = nibabel.Nifti1Image(np.zeros((5, 6, 7), dtype=np.float32), affine) + world = aligner.reference_frame(nii) + expected = np.array([[0, 3.0, 0, 0], + [0, 0, 4.0, 0], + [2.0, 0, 0, 0], + [0, 0, 0, 1]]) + assert np.allclose(world, expected) + + +def test_load_reference_takes_first_volume_and_drops_nans(): + import nibabel + + data = np.random.RandomState(0).rand(4, 5, 6, 3) + data[0, 0, 0, 0] = np.nan + nii = nibabel.Nifti1Image(data, np.eye(4)) + epi = aligner.load_reference(nii) + assert epi.shape == (4, 5, 6) + assert epi.dtype == np.float32 + assert epi[0, 0, 0] == 0 + assert np.allclose(epi[1:], data[1:, :, :, 0]) + + +# --------------------------------------------------------------------------- +# Argument checks +# --------------------------------------------------------------------------- + + +def test_new_transform_requires_reference(): + with pytest.raises(ValueError, match="does not exist"): + aligner.show(subj, "aligner_test_missing_xfm", open_browser=False, display_url=False) + + +def test_existing_transform_refuses_new_reference(): + with pytest.raises(ValueError, match="Refusing to overwrite"): + aligner.show(subj, xfmname, reference=_reference().get_filename(), + open_browser=False, display_url=False) + + +def test_transform_with_masks_requires_view_only(monkeypatch): + import types + + monkeypatch.setattr(aligner, "glob", types.SimpleNamespace(glob=lambda pattern: ["mask_thick.nii.gz"])) + with pytest.raises(ValueError, match="cached masks"): + aligner.show(subj, xfmname, open_browser=False, display_url=False) + + +def test_align_entry_point_forwards(monkeypatch): + seen = {} + + def fake_show(subject, name, reference=None, **kwargs): + seen.update(subject=subject, name=name, reference=reference, kwargs=kwargs) + return "handle" + + monkeypatch.setattr(aligner, "show", fake_show) + assert align.webgl_manual(subj, xfmname, view_only=True) == "handle" + assert seen == dict(subject=subj, name=xfmname, reference=None, kwargs=dict(view_only=True)) + + +# --------------------------------------------------------------------------- +# Server endpoints (no browser) +# --------------------------------------------------------------------------- + + +def test_page_carries_config(server): + from PIL import Image + + base = "http://localhost:%d" % server.port + html = _open(base + "/aligner.html").decode() + assert "aligner.Aligner" in html + start = html.index('viewer = figure.add(aligner.Aligner, "main", true, ') + len( + 'viewer = figure.add(aligner.Aligner, "main", true, ') + end = html.index(");", start) + config = json.loads(html[start:end]) + + xfm = database.db.get_xfm(subj, xfmname) + nii = xfm.reference_nifti + assert config["subject"] == subj + assert config["xfmname"] == xfmname + assert config["view_only"] is False + assert np.allclose(config["xfm"], xfm.xfm) + assert np.allclose(config["world"], aligner.reference_frame(nii)) + assert config["volume"]["shape"] == list(nii.shape[::-1]) + assert config["cmap"] == "gray" + assert config["mesh_color"] == "#ffffff" + assert config["vmin"] < config["vmax"] + + # the reference is served as the float mosaic the viewer expects + png = _open(base + "/data/reference.png") + image = Image.open(io.BytesIO(png)) + nwide, ntall = config["volume"]["mosaic"] + assert image.size == (nwide * (nii.shape[0] + 1) + 1, ntall * (nii.shape[1] + 1) + 1) + + # the surfaces come from the viewer's CTM pack + ctm = json.loads(_open(base + "/ctm/%s/" % subj).decode()) + assert len(ctm["offsets"]) == 2 + assert len(_open(base + "/ctm/%s/%s" % (subj, ctm["data"]))) > 0 + + +def test_save_endpoint_stores_coord_transform(server, recorder): + base = "http://localhost:%d" % server.port + xfm = np.arange(16, dtype=float).reshape(4, 4) + resp = json.loads(_open(base + "/save", dict(xfm=json.dumps(xfm.tolist()))).decode()) + assert resp["status"] == "ok" + assert len(recorder.calls) == 1 + call = recorder.calls[0] + assert call["subject"] == subj + assert call["name"] == xfmname + assert call["xfmtype"] == "coord" + assert np.allclose(call["xfm"], xfm) + + resp = json.loads(_open(base + "/save", dict(xfm=json.dumps([1, 2, 3]))).decode()) + assert resp["status"] == "error" + assert len(recorder.calls) == 1 + + +@pytest.mark.parametrize("server", [dict(view_only=True)], indirect=True) +def test_view_only_never_saves(server, recorder): + base = "http://localhost:%d" % server.port + resp = json.loads(_open(base + "/save", dict(xfm=json.dumps(np.eye(4).tolist()))).decode()) + assert resp["status"] == "error" + assert "view only" in resp["message"] + assert recorder.calls == [] + assert '"view_only": true' in _open(base + "/").decode() + + +# --------------------------------------------------------------------------- +# Headless browser +# --------------------------------------------------------------------------- + + +def _quadrants(png): + from PIL import Image + + rgb = np.asarray(Image.open(io.BytesIO(png)).convert("RGB")).astype(np.uint32) + h, w = rgb.shape[:2] + packed = (rgb[..., 0] << 16) | (rgb[..., 1] << 8) | rgb[..., 2] + return [packed[:h // 2, :w // 2], packed[h // 2:, :w // 2], + packed[:h // 2, w // 2:], packed[h // 2:, w // 2:]] + + +def _translation(vector): + mat = np.eye(4) + mat[:3, 3] = vector + return mat + + +@pytest.mark.skipif(not has_playwright, reason="playwright and chromium are required") +@pytest.mark.timeout(400) +def test_aligner_in_headless_browser(recorder): + """The aligner loads, edits the transform in world millimeters and saves it.""" + from cortex.export.headless import _PlaywrightThread, _wait_for_viewer_loaded, filter_webgl_failures + + server = aligner.show(subj, xfmname, open_browser=False, display_url=False) + server.disconnect_on_close = False + pw = _PlaywrightThread() + handle = None + try: + pw.start("http://localhost:%d/aligner.html" % server.port, timeout=120) + handle = server.get_client() + object.__setattr__(handle, "server", server) + _wait_for_viewer_loaded(handle, timeout=240) + # software rendering is slow: the first frame follows the load + assert handle.wait_for_frame(timeout=120) >= 1 + + nii = _reference() + world = aligner.reference_frame(nii) + coord0 = np.asarray(database.db.get_xfm(subj, xfmname).xfm) + assert np.allclose(handle.get_xfm(), coord0, atol=1e-3) + + # a translation in world millimeters is applied ahead of the transform + handle.translate([2.0, -3.0, 1.5]) + coord1 = handle.get_xfm() + expected = np.linalg.inv(world) @ _translation([2.0, -3.0, 1.5]) @ world @ coord0 + assert np.allclose(coord1, expected, atol=1e-3) + + # a rotation about the cursor keeps the cursor fixed + cursor_voxel = np.asarray(handle._call("getCursor"), dtype=float) + cursor_world = (world @ np.append(cursor_voxel, 1))[:3] + handle.rotate([0, 0, 1], 10) + coord2 = handle.get_xfm() + anat = np.linalg.inv(world @ coord2) @ np.append(cursor_world, 1) + assert np.allclose((world @ coord1 @ anat)[:3], cursor_world, atol=1e-2) + assert not np.allclose(coord2, coord1, atol=1e-4) + + handle.undo() + assert np.allclose(handle.get_xfm(), coord1, atol=1e-3) + + # every view drew something, and the two view modes differ + # (snapshot waits for the frame that shows the last change) + outline = handle.snapshot() + for quadrant in _quadrants(outline): + assert len(np.unique(quadrant)) > 10 + handle.set_control("view", aligner.MODES["projected"]) + projected = handle.snapshot() + for quadrant in _quadrants(projected): + assert len(np.unique(quadrant)) > 10 + assert outline != projected + + # a colormap change and a new mesh color reach the shaders + handle.set_control("view", aligner.MODES["outline"]) + handle.set_control("image.colormap", "hot") + handle.set_control("mesh.color", "#ff0000") + recolored = handle.snapshot() + assert recolored != outline + assert handle.get_control("mesh.color") == "#ff0000" + assert handle.get_control("image.colormap") == "hot" + assert handle.get_control("view") == aligner.MODES["outline"] + + # saving stores the current transform as a coord transform + handle.save() + assert recorder.wait(1), "the save request never reached the server" + call = recorder.calls[0] + assert call["subject"] == subj and call["name"] == xfmname + assert call["xfmtype"] == "coord" + assert np.allclose(call["xfm"], coord1, atol=1e-3) + + errors = pw.browser_errors + assert not [e for e in errors if "[pageerror]" in e], errors + assert not filter_webgl_failures(errors), errors + finally: + pw.shutdown() + server.stop() diff --git a/cortex/webgl/aligner.html b/cortex/webgl/aligner.html new file mode 100644 index 000000000..315150e3b --- /dev/null +++ b/cortex/webgl/aligner.html @@ -0,0 +1,64 @@ +{% autoescape None %} +{% extends template.html %} +{% block javascripts %} + + +{% end %} +{% block jsinit %} + var viewer, figure, sock; + var viewopts = {}; +{% end %} +{% block onload %} + figure = new jsplot.W2Figure(); + viewer = figure.add(aligner.Aligner, "main", true, {{config}}); + {% if python_interface %} + sock = new Websock(); + {% end %} +{% end %} +{% block extrahtml %} + + +{% end %} diff --git a/cortex/webgl/aligner.py b/cortex/webgl/aligner.py new file mode 100644 index 000000000..88217fbf3 --- /dev/null +++ b/cortex/webgl/aligner.py @@ -0,0 +1,472 @@ +"""Browser-based manual aligner. + +Moves the anatomical surfaces (pial and white matter) in the space of a +functional reference volume, the way the mayavi aligner did. The volume +stays on its own voxel grid, so its slices are displayed without +resampling, and the surfaces are cut off at the displayed slices, which +draws their outline on the anatomy in the image. A second view mode paints +the volume onto the surfaces instead. Rendering happens in the browser +through the WebGL viewer's machinery (``cortex/webgl/resources/js/aligner.js``); +the transform is served, edited and saved through a tornado server in this +process, like ``cortex.webgl.show``. + +The entry point for users is :func:`cortex.align.webgl_manual`. +""" +import base64 +import glob +import json +import mimetypes +import os +import queue +import time +import uuid +import warnings +import webbrowser +from typing import Any, Optional, Union, cast + +import numpy as np +import numpy.typing as npt +from tornado import web + +from .. import options, utils, volume +from ..database import db +from . import serve +from .data import _pack_png +from .FallbackLoader import FallbackLoader +from .serve import P +from .view import colormaps, domain_name + +#: Name under which the reference volume is served to the page +REFERENCE_NAME = "reference" + +#: The two view modes of the page, as its `view` control names them +MODES = dict(outline="mesh + slices", projected="data on surface") + + +def reference_frame(nii) -> npt.NDArray[np.float64]: + """The voxel-to-world matrix the aligner works in for a reference image. + + The world frame keeps the voxel grid of the reference image axis aligned, + scaled to millimeters by the voxel sizes, and permuted and flipped so that + its x, y and z axes point to the subject's right, anterior and superior. + Slices of the reference image are therefore drawn without resampling, and + the surfaces are moved by rigid transforms expressed in millimeters. + + Parameters + ---------- + nii : nibabel.Nifti1Image + The reference image. + + Returns + ------- + world : (4, 4) ndarray + Affine mapping voxel indices of `nii` to world coordinates. + """ + import nibabel + + zooms = np.asarray(nii.header.get_zooms()[:3], dtype=float) + ornt = nibabel.io_orientation(nii.affine) + world = np.zeros((4, 4)) + world[3, 3] = 1.0 + for voxel_axis, (world_axis, direction) in enumerate(ornt): + world[int(world_axis), voxel_axis] = float(direction) * zooms[voxel_axis] + return world + + +def load_reference(nii) -> npt.NDArray[np.float32]: + """The reference data as a 3D float32 array in (x, y, z) voxel order. + + A 4D image contributes its first volume; NaNs are replaced by zeros. + """ + data = np.asarray(nii.get_fdata()) + while data.ndim > 3: + data = data[..., 0] + if data.ndim != 3: + raise ValueError("The reference image must have three dimensions, got shape %s" % (data.shape,)) + return np.nan_to_num(data).astype(np.float32) + + +def _color_hex(color: str) -> str: + from matplotlib.colors import to_hex + + return to_hex(color) + + +class JSAligner(serve.JSProxy[P]): + """Handle to an aligner running in the browser. + + Besides the generic attribute access of :class:`cortex.webgl.serve.JSProxy`, + this exposes the transform being edited and the controls of the page. + Its methods tag their requests, so that replies delayed by a busy page + (while it parses the surfaces, or draws a slow frame) are matched to the + right request; the generic attribute access does not have this protection. + """ + + #: Seconds to wait for the tagged reply of a call + call_timeout = 120.0 + + def _call(self, name: str, *args: Any) -> Any: + token = uuid.uuid4().hex + resp = self.send(method="run", params=["window.viewer.call", [token, name, list(args)]]) + # plain attribute lookups on a JSProxy go to the page; read the + # bookkeeping kept on the python object from its dict instead + server = vars(self).get("server") + if len(resp) == 0: + return None + deadline = time.monotonic() + self.call_timeout + reply = resp[0] + while True: + if isinstance(reply, dict) and reply.get("token") == token: + # a redraw pending at the time of the reply lands in the next + # frame; wait_for_frame waits for it + if reply.get("scheduled", False): + target = int(cast(float, reply.get("frames", 0))) + 1 + object.__setattr__(self, "_frame_target", max(target, vars(self).get("_frame_target", 1))) + if "error" in reply: + raise RuntimeError("%s: %s" % (name, reply["error"])) + return reply.get("value") + if server is None: + # without the server's queue, stale replies cannot be skipped + if reply is None: + return None + raise RuntimeError("Unexpected reply to %s: %r" % (name, reply)) + if time.monotonic() > deadline: + raise TimeoutError("No reply to %s within %.0f s" % (name, self.call_timeout)) + try: + reply = json.loads(server.response.get(timeout=1)) + except queue.Empty: + reply = None + + def wait_for_frame(self, timeout: float = 120.0) -> int: + """Block until the page has drawn the effect of the last call. + + Returns the number of frames drawn so far. Drawing happens in the + page's own animation frames, which can be slow without a GPU, so a + snapshot taken right after a change may otherwise show the previous + state. + """ + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + # the reply to this call also records a redraw still pending + frames = self._call("getFrames") + target = max(vars(self).get("_frame_target", 1), 1) + if frames is not None and int(frames) >= target: + return int(frames) + time.sleep(0.2) + raise TimeoutError("The aligner did not draw a new frame within %.0f s" % timeout) + + def get_control(self, name: str) -> Any: + """The value of a control of the page by its dotted path, such as + ``"image.vmin"`` or ``"mesh.color"``.""" + return self._call("getControl", name) + + def set_control(self, name: str, value: Any) -> None: + """Set a control of the page by its dotted path, such as + ``"image.colormap"`` or ``"view"``.""" + self._call("setControl", name, value) + + def get_xfm(self) -> npt.NDArray[np.float64]: + """The current transform, as a (4, 4) pycortex 'coord' matrix + (anatomical coordinates to voxel indices of the reference).""" + return np.asarray(self._call("getXfm"), dtype=float) + + def set_xfm(self, xfm: npt.ArrayLike) -> None: + """Replace the current transform with a (4, 4) 'coord' matrix.""" + matrix = np.asarray(xfm, dtype=float) + if matrix.shape != (4, 4): + raise ValueError("The transform must be a 4x4 matrix") + self._call("setXfm", matrix.tolist()) + + def translate(self, vector: npt.ArrayLike) -> None: + """Move the surfaces by `vector`, in millimeters along the world axes + (right, anterior, superior).""" + self._call("translate", [float(v) for v in np.asarray(vector).ravel()]) + + def rotate(self, axis: npt.ArrayLike, angle: float) -> None: + """Rotate the surfaces by `angle` degrees about the world `axis` + through the cursor.""" + self._call("rotate", [float(v) for v in np.asarray(axis).ravel()], float(angle)) + + def undo(self) -> None: + """Undo the last change to the transform.""" + self._call("undo") + + def save(self) -> Any: + """Save the current transform into the database, like the Save button.""" + return self._call("save") + + def snapshot(self, filename: Optional[str] = None) -> bytes: + """The current rendering of the four views as PNG bytes, also written + to `filename` when given. Waits for the frame showing the last change + first.""" + self.wait_for_frame() + data_url = self._call("snapshot") + png = base64.b64decode(data_url.split(",", 1)[1]) + if filename is not None: + with open(filename, "wb") as fp: + fp.write(png) + return png + + +def show( + subject: str, + xfmname: str, + reference: Optional[str] = None, + view_only: bool = False, + cmap: Optional[str] = None, + mesh_color: Optional[str] = None, + mesh_opacity: Optional[float] = None, + open_browser: Optional[bool] = None, + autoclose: Optional[bool] = None, + port: Optional[int] = None, + recache: bool = False, + types: tuple[str, ...] = ("inflated",), + title: Optional[str] = None, + display_url: bool = True, + template: str = "aligner.html", +) -> Union[JSAligner, serve.WebApp]: + """Open the browser-based aligner for a transform of `subject`. + + The functional reference volume is shown on its own voxel grid in three + slice views and a 3D view, with the pial and white matter surfaces of + `subject` cut to the displayed slices. Rotating and translating the + surfaces edits the transform; the Save button writes it into the + database as `xfmname`. See :func:`cortex.align.webgl_manual` for the + controls. + + Parameters + ---------- + subject : str + Subject identifier. + xfmname : str + Name of the transform to create or modify. + reference : str, optional + Path to a nibabel-readable functional volume, required when `xfmname` + does not exist yet. For an existing transform, leave it None: the + stored reference is loaded, and the transform is used as the starting + point. + view_only : bool, optional + Open the aligner without the possibility to save. Required to open a + transform whose masks have been cached, since such a transform cannot + be changed anymore. + cmap : str, optional + Initial colormap for the reference volume, one of the 1D pycortex + colormaps. Defaults to the `colormap` option of the `webgl_aligner` + section of the config file. + mesh_color : str, optional + Initial color of the surface outlines, as a matplotlib color. Defaults + to the `mesh_color` config option. + mesh_opacity : float, optional + Initial opacity of the whole surfaces in the 3D view (0 shows only + their outlines on the slices). Defaults to the `mesh_opacity` config + option. + open_browser : bool, optional + Open the aligner in the default browser. Defaults to the + `open_browser` option of the `webshow` config section. + autoclose : bool, optional + Stop the server when the last browser window disconnects. Defaults to + the `autoclose` option of the `webshow` config section. + port : int, optional + Port of the server; a free port is picked when None. + recache : bool, optional + Regenerate the cached surface (CTM) files. + types : tuple of str, optional + Surface types included in the CTM pack, to share the cache with the + viewer. Default ("inflated",). + title : str, optional + Title of the browser window. + display_url : bool, optional + When `open_browser` is False, display an IPython link to the aligner. + template : str, optional + Name of the tornado template of the page. Default 'aligner.html'. + + Returns + ------- + handle : JSAligner or WebApp + With `open_browser`, a handle to the running aligner (its ``server`` + attribute is the tornado server); otherwise the server itself, whose + ``get_client()`` returns the handle once a browser has connected. + """ + import nibabel + + close_on_disconnect: bool = ( + options.config.get("webshow", "autoclose", fallback="true") == "true" + if autoclose is None + else autoclose + ) + if open_browser is None: + open_browser = options.config.get("webshow", "open_browser", fallback="true") == "true" + + # The transform to start from: the stored one, or the header alignment + # of a new reference (anatomical and functional scanner spaces coincide) + try: + dbxfm = db.get_xfm(subject, xfmname, xfmtype="coord") + except IOError: + dbxfm = None + + if dbxfm is not None: + if reference is not None: + raise ValueError( + "Refusing to overwrite the reference of the existing transform %s; " + "pass reference=None to load the stored reference" % xfmname + ) + masks = glob.glob(db.get_paths(subject)["masks"].format(xfmname=xfmname, type="*")) + if len(masks) > 0 and not view_only: + raise ValueError( + "Refusing to modify transform %s because it has cached masks (%s). " + "Delete the masks to modify the transform, or pass view_only=True " + "to inspect it." % (xfmname, ", ".join(sorted(masks))) + ) + nii = dbxfm.reference_nifti + reference = nii.get_filename() + coord = np.asarray(dbxfm.xfm, dtype=float) + else: + if reference is None or not os.path.exists(reference): + raise ValueError("Reference image file (%s) does not exist" % reference) + nii = cast("nibabel.Nifti1Image", nibabel.load(reference)) + coord = np.linalg.inv(nii.affine) + + epi = load_reference(nii) + world = reference_frame(nii) + percentiles = np.percentile(epi, [1, 99]) + vmin, vmax = float(percentiles[0]), float(percentiles[1]) + if vmin == vmax: + vmin, vmax = float(epi.min()), float(epi.max()) + + # The reference is served as the float mosaic PNG the viewer uses + mosaic, mosaic_shape = volume.mosaic(epi.T, show=False) + png = _pack_png(np.ascontiguousarray(mosaic, dtype=np.float32)) + + # The surfaces come from the CTM pack the viewer uses: base positions + # are the pial surface and the `wm` attribute is the white matter + ctmfile = utils.get_ctmpack(subject, types, method="mg2", level=9, recache=recache) + + cmap_names = [name for name, _ in cast(list[tuple[str, str]], colormaps)] + if cmap is None: + cmap = options.config.get("webgl_aligner", "colormap", fallback="gray") + if cmap not in cmap_names: + warnings.warn("Colormap %s is not available, using gray" % cmap) + cmap = "gray" + if mesh_color is None: + mesh_color = options.config.get("webgl_aligner", "mesh_color", fallback="white") + if mesh_opacity is None: + mesh_opacity = float(options.config.get("webgl_aligner", "mesh_opacity", fallback="0")) + if title is None: + title = "Aligner: %s %s" % (subject, xfmname) + + config: dict[str, Any] = dict( + subject=subject, + xfmname=xfmname, + ctm="ctm/%s/" % subject, + view_only=view_only, + volume=dict( + name=REFERENCE_NAME, + subject=subject, + raw=False, + min=float(epi.min()), + max=float(epi.max()), + mosaic=[int(v) for v in mosaic_shape], + shape=[int(v) for v in epi.T.shape], + ), + images={REFERENCE_NAME: ["data/%s.png" % REFERENCE_NAME]}, + world=world.tolist(), + xfm=coord.tolist(), + cmap=cmap, + vmin=vmin, + vmax=vmax, + mesh_color=_color_hex(mesh_color), + mesh_opacity=mesh_opacity, + ) + + html = FallbackLoader([os.path.split(os.path.abspath(template))[0], serve.cwd]).load(template) + + class CTMHandler(web.RequestHandler): + def get(self, path: str): + subj, path = path.split("/") + if subj != subject: + raise web.HTTPError(404) + if path == "": + self.set_header("Content-Type", "application/json") + with open(ctmfile) as fp: + self.write(fp.read()) + else: + fpath = os.path.join(os.path.split(ctmfile)[0], path) + mtype = mimetypes.guess_type(fpath)[0] + self.set_header("Content-Type", mtype if mtype is not None else "application/octet-stream") + with open(fpath, "rb") as fp: + self.write(fp.read()) + + class DataHandler(web.RequestHandler): + def get(self, path: str): + self.set_header("Content-Type", "image/png") + self.write(png) + + class AlignerHandler(web.RequestHandler): + def get(self): + self.set_header("Content-Type", "text/html") + self.write( + html.generate( + config=json.dumps(config), + colormaps=colormaps, + default_cmap=cmap, + python_interface=True, + leapmotion=False, + title=title, + ) + ) + + class SaveHandler(web.RequestHandler): + def post(self): + self.set_header("Content-Type", "application/json") + if view_only: + self.write(json.dumps(dict(status="error", message="view only: the transform is not saved"))) + return + try: + xfm = np.asarray(json.loads(self.get_argument("xfm")), dtype=float) + if xfm.shape != (4, 4): + raise ValueError("expected a 4x4 matrix, got shape %s" % (xfm.shape,)) + db.save_xfm(subject, xfmname, xfm, xfmtype="coord", reference=reference) + except Exception as exc: + self.write(json.dumps(dict(status="error", message="not saved: %s" % exc))) + return + print("saved xfm %s for %s" % (xfmname, subject)) + self.write(json.dumps(dict(status="ok", message="saved transform %s for %s" % (xfmname, subject)))) + + class WebApp(serve.WebApp): + disconnect_on_close = close_on_disconnect + + def get_client(self): + self.connect.wait() + self.connect.clear() + return JSAligner(self.send, "window.viewer") + + def get_local_client(self): + return JSAligner(self.srvsend, "window.viewer") + + server = WebApp( + [ + (r"/ctm/(.*)", CTMHandler), + (r"/data/(.*)", DataHandler), + (r"/save", SaveHandler), + (r"/aligner.html", AlignerHandler), + (r"/", AlignerHandler), + ], + 0 if port is None else port, + ) + server.start() + print("Started aligner server on port %d" % server.port) + url = "http://%s%s:%d/aligner.html" % (serve.hostname, domain_name, server.port) + if open_browser: + webbrowser.open(url) + client = server.get_client() + # bypass JSProxy.__setattr__, which would query the page + object.__setattr__(client, "server", server) + return client + elif display_url: + try: + from IPython.display import HTML, display + + display(HTML('Open aligner: {0}'.format(url))) + except Exception: + print("Open the aligner at %s" % url) + return server diff --git a/cortex/webgl/resources/css/aligner.css b/cortex/webgl/resources/css/aligner.css new file mode 100644 index 000000000..0c4167fc2 --- /dev/null +++ b/cortex/webgl/resources/css/aligner.css @@ -0,0 +1,105 @@ +/************************************************* + * Manual aligner: a 2x2 grid of views on one canvas, + * with the controls in a panel on the right + *************************************************/ +.jsplot_axes { + position:relative; + width:100%; + height:100%; +} +#aligner { + position:relative; + width:100%; + height:100%; + overflow:hidden; + background:#000; +} +#aligner-canvas { + position:absolute; + left:0; + top:0; + z-index:2; + display:block; +} +/* Transparent overlays, one per view, that receive the mouse events */ +.aligner-view { + position:absolute; + width:50%; + height:50%; + z-index:3; + box-sizing:border-box; + border:1px solid #444; + cursor:crosshair; +} +#view-y { left:0; top:0; } +#view-z { left:0; top:50%; } +#view-x { left:50%; top:0; } +#view-3d { left:50%; top:50%; cursor:default; } +.aligner-label { + position:absolute; + left:8px; + top:6px; + color:#ddd; + font-size:10pt; + text-shadow:0px 1px 2px #000; + pointer-events:none; +} +#aligner-load { + left:50%; + top:50%; + right:auto; + width:200px; + margin:-20px 0px 0px -120px; +} +#aligner-colormaps { + display:none; +} + +/* Control panel */ +#aligner-panel { + height:100%; + overflow-y:auto; + background:#1a1a1a; + color:#ccc; + font-size:9pt; + text-align:left; +} +#aligner-panel #figure_ui { + position:static; + width:100% !important; + max-height:none; + overflow:visible; +} +#aligner-viewonly { + display:none; + padding:6px 8px; + color:#fc9; +} +#aligner-status { + min-height:1.5em; + padding:6px 8px; + color:#8fd; +} +#aligner-status.error { + color:#f88; +} +#aligner-legend { + padding:4px 8px 12px 8px; + line-height:1.4; +} +#aligner-legend h4 { + margin:10px 0px 2px 0px; + color:#eee; + font-size:9pt; +} +#aligner-legend table { + width:100%; +} +#aligner-legend td { + padding:1px 4px; + vertical-align:top; +} +#aligner-legend td.key { + white-space:nowrap; + color:#fc9; +} diff --git a/cortex/webgl/resources/js/aligner.js b/cortex/webgl/resources/js/aligner.js new file mode 100644 index 000000000..45cbddc88 --- /dev/null +++ b/cortex/webgl/resources/js/aligner.js @@ -0,0 +1,1250 @@ +//Manual aligner: moves the anatomical surfaces in the space of a functional +//reference volume, the way the old mayavi aligner did. The volume stays on +//its own voxel grid, so its slices are displayed without resampling, and the +//pial and white matter surfaces are cut off at the displayed slices, which +//draws their outline on top of the anatomy in the image. A second view mode +//paints the volume onto the surface instead, for judging the alignment from +//the pattern the data makes on the cortex. +// +//The scene lives in the "world" frame handed over by cortex/webgl/aligner.py: +//the voxel grid of the reference volume in millimeters, with the axes +//permuted and flipped so that x, y and z point right, anterior and superior. +//The transform being edited is the pycortex "coord" transform (anatomical +//millimeters to voxel indices); in the scene it appears as the world matrix +//of the surface object, world <- anatomical = config.world * coord. +var aligner = (function(module) { + + //One slice view per world axis. `look` is the viewing direction and `up` + //the screen-up direction. The coronal and axial views follow the + //radiological convention (the subject's right on the left of the screen, + //as in cortex.volume.mosaic); the sagittal view is seen from the right. + var VIEWS = { + x: {axis: 0, title: "sagittal", look: [-1, 0, 0], up: [0, 0, 1]}, + y: {axis: 1, title: "coronal", look: [ 0,-1, 0], up: [0, 0, 1]}, + z: {axis: 2, title: "axial", look: [ 0, 0, 1], up: [0, 1, 0]}, + }; + + //Placement of the four views in the 2x2 grid, matching the mayavi aligner + var LAYOUT = [ + {name: "y", col: 0, row: 0}, + {name: "z", col: 0, row: 1}, + {name: "x", col: 1, row: 0}, + {name: "3d", col: 1, row: 1}, + ]; + + var MODES = {outline: "mesh + slices", projected: "data on surface"}; + module.MODES = MODES; + + //Slice setters, one per world axis, so that the slice controls in the + //menu and the views stay in sync + var SLICE_SETTERS = ["setSagittal", "setCoronal", "setAxial"]; + + var UNDO_LIMIT = 200; + //Wheel movement (in normalized pixels) that steps one slice + var WHEEL_STEP = 30; + + //Nested 4x4 array in row-major order to a THREE.Matrix4, and back + module.matrixFromRows = function(rows) { + var m = new THREE.Matrix4(); + m.set(rows[0][0], rows[0][1], rows[0][2], rows[0][3], + rows[1][0], rows[1][1], rows[1][2], rows[1][3], + rows[2][0], rows[2][1], rows[2][2], rows[2][3], + rows[3][0], rows[3][1], rows[3][2], rows[3][3]); + return m; + }; + module.matrixToRows = function(m) { + var e = m.elements, rows = []; + for (var i = 0; i < 4; i++) { + var row = []; + for (var j = 0; j < 4; j++) + row.push(e[i + 4 * j]); + rows.push(row); + } + return rows; + }; + + //Wheel deltas are in pixels, lines or pages depending on the browser and + //the device; normalize them all to pixels (as in movement.js) + function wheelDelta(event) { + var delta = event.deltaY; + if (event.deltaMode == 1) + delta *= 18; + else if (event.deltaMode == 2) + delta *= 180; + return delta; + } + + //Resolves once every image has decoded, so that the colormap textures + //and the list of 1D colormaps can be built from them + function whenImagesLoaded(images) { + var waiting = []; + for (var i = 0; i < images.length; i++) { + var img = images[i]; + if (img.complete && img.naturalHeight > 0) + continue; + var deferred = $.Deferred(); + img.addEventListener("load", deferred.resolve); + img.addEventListener("error", deferred.resolve); + waiting.push(deferred); + } + return $.when.apply($, waiting); + } + + //Unique triangle edges of an indexed BufferGeometry, as a line index + //buffer with the same chunking (offsets) as the triangle index, so that + //the mesh can be drawn as a wireframe. three.js r69 draws wireframes of + //indexed BufferGeometries by reinterpreting the triangle index as line + //pairs, which does not give the triangle edges. + module.buildEdges = function(geometry) { + var index = geometry.attributes.index.array; + var offsets = geometry.offsets; + var edges = new Uint16Array(index.length * 2); + var edgeOffsets = []; + var total = 0; + for (var j = 0; j < offsets.length; j++) { + var start = offsets[j].start, count = offsets[j].count; + var seen = new Set(); + var chunkStart = total; + var add = function(a, b) { + var key = a < b ? a * 65536 + b : b * 65536 + a; + if (seen.has(key)) + return; + seen.add(key); + edges[total++] = a; + edges[total++] = b; + }; + for (var i = start, il = start + count; i < il; i += 3) { + add(index[i], index[i+1]); + add(index[i+1], index[i+2]); + add(index[i+2], index[i]); + } + edgeOffsets.push({start: chunkStart, count: total - chunkStart, index: offsets[j].index}); + } + var edgeGeom = new THREE.BufferGeometry(); + edgeGeom.addAttribute("index", new THREE.BufferAttribute(edges.subarray(0, total), 2)); + edgeGeom.addAttribute("position", geometry.attributes.position); + edgeGeom.addAttribute("wm", geometry.attributes.wm); + edgeGeom.offsets = edgeOffsets; + return edgeGeom; + }; + + module.Aligner = function(figure, config) { + jsplot.Axes.call(this, figure); + this.config = config; + this.loaded = $.Deferred(); + this.ready = false; + this._ready = {volume: false, mesh: false}; + this._scheduled = false; + this._draw = this.draw.bind(this); + this.nframes = 0; + + $(this.object).html($("#aligner_html").html()); + this.canvas = $(this.object).find("#aligner-canvas"); + + //Frames: voxel -> world, and anatomical -> world for the surfaces + this.world = module.matrixFromRows(config.world); + this.worldInv = new THREE.Matrix4().getInverse(this.world); + this.xfm = new THREE.Matrix4().multiplyMatrices(this.world, module.matrixFromRows(config.xfm)); + this._undo = []; + + //Voxel dimensions in voxel axis order, and for each world axis the + //voxel axis it is drawn from and the voxel size along it + var shape = config.volume.shape; + this.dims = [shape[2], shape[1], shape[0]]; + this.vax = [0, 0, 0]; + this.vscale = [1, 1, 1]; + var e = this.world.elements; + for (var a = 0; a < 3; a++) { + var best = -1; + for (var j = 0; j < 3; j++) { + var v = Math.abs(e[a + 4 * j]); + if (v > best) { + best = v; + this.vax[a] = j; + } + } + this.vscale[a] = best; + } + //The cursor (voxel coordinates) picks the displayed slices and is the + //pivot for rotations + this.cursor = [(this.dims[0] - 1) / 2, (this.dims[1] - 1) / 2, (this.dims[2] - 1) / 2]; + this.planeCoord = [0, 0, 0]; + + this._mode = MODES.outline; + this._cmapName = config.cmap; + this._showSurf = [true, true]; + this._translateStep = 1; + this._rotateStep = 1; + this.hasWM = true; + this.hoverView = null; + this._drag = null; + this._wheelAcc = 0; + + //Renderer and scene. Objects are drawn in scene order (planes, then + //the surfaces, then the crosshairs): the slice views draw their plane + //without a depth test so that the surface outline always shows on top + this.renderer = new THREE.WebGLRenderer({ + canvas: this.canvas[0], + antialias: true, + preserveDrawingBuffer: true, + alpha: false, + }); + this.renderer.setClearColor(new THREE.Color(0x000000), 1); + this.renderer.sortObjects = false; + + this.scene = new THREE.Scene(); + this.light = new THREE.DirectionalLight(0xffffff, 1.0); + this.scene.add(this.light); + this.scene.add(this.light.target); + + this.planeGroup = new THREE.Object3D(); + this.brain = new THREE.Object3D(); + this.brain.matrixAutoUpdate = false; + this.brain.matrix.copy(this.xfm); + this.brain.matrixWorldNeedsUpdate = true; + this.crossGroup = new THREE.Object3D(); + this.scene.add(this.planeGroup); + this.scene.add(this.brain); + this.scene.add(this.crossGroup); + + this.camera3d = new THREE.PerspectiveCamera(45, 1, 1, 4000); + this.camera3d.up.set(0, 0, 1); + this.controls = new jsplot.LandscapeControls(); + this.controls.addEventListener("change", this.schedule.bind(this)); + + this.views = {}; + this.viewlist = []; + for (var i = 0; i < LAYOUT.length; i++) { + var name = LAYOUT[i].name; + var div = $(this.object).find("#view-" + name)[0]; + var view = { + name: name, + div: div, + label: $(div).find(".aligner-label")[0], + col: LAYOUT[i].col, + row: LAYOUT[i].row, + rect: {left: 0, top: 0, width: 1, height: 1}, + is2d: name != "3d", + }; + if (view.is2d) { + var spec = VIEWS[name]; + view.axis = spec.axis; + view.title = spec.title; + view.look = new THREE.Vector3().fromArray(spec.look); + view.up = new THREE.Vector3().fromArray(spec.up); + view.right = new THREE.Vector3().crossVectors(view.look, view.up); + view.camera = new THREE.OrthographicCamera(-1, 1, 1, -1, 1, 4000); + view.center = new THREE.Vector3(); + view.height = 200; + view.fitted = false; + } else { + view.camera = this.camera3d; + } + this.views[name] = view; + this.viewlist.push(view); + this._bindView(view); + } + window.addEventListener("mousemove", this._onMouseMove.bind(this), false); + window.addEventListener("mouseup", this._onMouseUp.bind(this), false); + window.addEventListener("keydown", this._onKeyDown.bind(this), true); + + //Uniforms shared by the planes and the painted surface + this.volUniforms = { + data: {type:'tv', value:[null, null, null, null]}, + mosaic: {type:'v2v', value:[new THREE.Vector2(1, 1), new THREE.Vector2(1, 1)]}, + dshape: {type:'v2v', value:[new THREE.Vector2(1, 1), new THREE.Vector2(1, 1)]}, + nslices: {type:'f', value:1}, + volxfm: {type:'m4', value:this.worldInv}, + colormap: {type:'t', value:null}, + vmin: {type:'f', value:config.vmin}, + vmax: {type:'f', value:config.vmax}, + brightness: {type:'f', value:0}, + contrast: {type:'f', value:1}, + gamma: {type:'f', value:1}, + flip: {type:'i', value:0}, + outside: {type:'v3', value:new THREE.Vector3(.25, .25, .25)}, + }; + //Uniforms shared by the surface outlines + this.meshUniforms = { + color: {type:'c', value:new THREE.Color(config.mesh_color)}, + opacity: {type:'f', value:config.mesh_opacity}, + slabLo: {type:'v3', value:new THREE.Vector3()}, + slabHi: {type:'v3', value:new THREE.Vector3()}, + slabMask: {type:'v3', value:new THREE.Vector3(1, 1, 1)}, + }; + this.planeMaterials = { + flat: this._makeVolumeMaterial({lights: false, depthTest: false}), + solid: this._makeVolumeMaterial({lights: false, depthTest: true}), + }; + this.projectedMaterial = this._makeVolumeMaterial({lights: true, depthmix: true}); + this.outlineMaterials = [this._makeMeshMaterial(0, true), this._makeMeshMaterial(1, true)]; + this.surfaceMaterials = [this._makeMeshMaterial(0, false), this._makeMeshMaterial(1, false)]; + + this.planes2d = []; + this.planes3d = []; + this.planeGeoms = []; + this.crosshairs = []; + this.hemis = []; + + //Control panel on the right, holding the menu, the save status and + //the list of controls + this.ui = new jsplot.Menu(); + this.ui.addEventListener("update", this.schedule.bind(this)); + this._setupPanel(figure); + + //Colormap textures, and the menu once the colormap images are ready + //(the 1D colormaps are told apart from the 2D ones by their height) + this.colormaps = {}; + var images = $(this.object).find(".cmap img").toArray(); + whenImagesLoaded(images).done(function() { + for (var i = 0; i < images.length; i++) { + var img = images[i]; + var tex = new THREE.Texture(img); + tex.minFilter = THREE.LinearFilter; + tex.magFilter = THREE.LinearFilter; + tex.flipY = true; + tex.needsUpdate = true; + this.colormaps[img.parentNode.id] = tex; + if (typeof(colormaps) !== "undefined") + colormaps[img.parentNode.id] = tex; + } + this._buildUI(); + }.bind(this)); + + //The reference volume, through the viewer's mosaic loader + this.volume = new dataset.VolumeData(config.volume, config.images); + this.volume.loaded.done(this._volumeReady.bind(this)); + + //The surfaces, from the CTM pack the viewer uses: the base positions + //are the pial surface and the `wm` attribute the white matter + var loader = new THREE.CTMLoader(false); + loader.loadParts(config.ctm, function(geometries, materials, json) { + this._meshReady(geometries); + }.bind(this), {useWorker: true}); + }; + module.Aligner.prototype = Object.create(jsplot.Axes.prototype); + module.Aligner.prototype.constructor = module.Aligner; + + module.Aligner.prototype._setupPanel = function(figure) { + var wrapper = document.createElement("div"); + wrapper.innerHTML = $("#aligner_panel_html").html(); + var panel = wrapper.querySelector("#aligner-panel"); + $(panel).find("#aligner-controls")[0].appendChild(figure.ui_element); + if (this.config.view_only) + $(panel).find("#aligner-viewonly").show(); + this.statusElement = $(panel).find("#aligner-status"); + figure.gui.open(); + try { + figure.gui.width = 284; + } catch (e) {} + figure.w2obj.show("right", true); + figure.setSize("right", 300); + figure.w2obj.content("right", panel); + }; + + module.Aligner.prototype.showStatus = function(message, error) { + this.statusElement.text(message); + this.statusElement.toggleClass("error", !!error); + }; + + module.Aligner.prototype._makeVolumeMaterial = function(opts) { + var shaders = Shaders.aligner_volume({ + sampler: "nearest", + lights: opts.lights, + depthmix: opts.depthmix, + }); + var uniforms = THREE.UniformsUtils.merge([ + THREE.UniformsLib["lights"], + { + diffuse: {type:'v3', value:new THREE.Vector3(.7, .7, .7)}, + specular: {type:'v3', value:new THREE.Vector3(0, 0, 0)}, + emissive: {type:'v3', value:new THREE.Vector3(.35, .35, .35)}, + shininess: {type:'f', value:1}, + specularStrength: {type:'f', value:0}, + depth: {type:'f', value:0.5}, + } + ]); + for (var name in this.volUniforms) + uniforms[name] = this.volUniforms[name]; + var depth = opts.depthTest !== false; + return new THREE.ShaderMaterial({ + vertexShader: shaders.vertex, + fragmentShader: shaders.fragment, + uniforms: uniforms, + attributes: shaders.attrs, + lights: !!opts.lights, + side: opts.depthmix ? THREE.FrontSide : THREE.DoubleSide, + depthTest: depth, + depthWrite: depth, + }); + }; + + module.Aligner.prototype._makeMeshMaterial = function(depth, clip) { + var shaders = Shaders.aligner_mesh({}); + var uniforms = { + color: this.meshUniforms.color, + slabLo: this.meshUniforms.slabLo, + slabHi: this.meshUniforms.slabHi, + slabMask: this.meshUniforms.slabMask, + depth: {type:'f', value:depth}, + doClip: {type:'i', value:clip ? 1 : 0}, + opacity: clip ? {type:'f', value:1} : this.meshUniforms.opacity, + }; + return new THREE.ShaderMaterial({ + vertexShader: shaders.vertex, + fragmentShader: shaders.fragment, + uniforms: uniforms, + attributes: shaders.attrs, + transparent: !clip, + depthWrite: clip, + side: THREE.DoubleSide, + }); + }; + + //------------------------------------------------------------------------- + // Loading + //------------------------------------------------------------------------- + module.Aligner.prototype._volumeReady = function() { + var vol = this.volume; + this.volUniforms.data.value[0] = vol.textures[0]; + for (var i = 0; i < 2; i++) { + this.volUniforms.mosaic.value[i].set(vol.mosaic[0], vol.mosaic[1]); + this.volUniforms.dshape.value[i].set(vol.shape[0], vol.shape[1]); + } + this.volUniforms.nslices.value = vol.numslices; + + //World bounding box of the volume + var min = new THREE.Vector3(Infinity, Infinity, Infinity); + var max = new THREE.Vector3(-Infinity, -Infinity, -Infinity); + for (var c = 0; c < 8; c++) { + var corner = new THREE.Vector3( + (c & 1) ? this.dims[0] - 0.5 : -0.5, + (c & 2) ? this.dims[1] - 0.5 : -0.5, + (c & 4) ? this.dims[2] - 0.5 : -0.5).applyMatrix4(this.world); + min.min(corner); + max.max(corner); + } + this.bbox = {min: min, max: max}; + this.bboxCenter = min.clone().add(max).multiplyScalar(0.5); + + this._buildPlanes(); + this._updateSliceGeometry(); + + this.controls.setTarget(this.bboxCenter.toArray()); + this.controls.setRadius(1.1 * min.distanceTo(max)); + + this._ready.volume = true; + this._checkReady(); + }; + + module.Aligner.prototype._buildPlanes = function() { + var cross_material = new THREE.LineBasicMaterial({color: 0x44ccff, depthTest: false, depthWrite: false}); + var makeQuad = function() { + var geom = new THREE.Geometry(); + for (var i = 0; i < 4; i++) + geom.vertices.push(new THREE.Vector3()); + geom.faces.push(new THREE.Face3(0, 1, 2), new THREE.Face3(2, 1, 3)); + geom.dynamic = true; + return geom; + }; + for (var a = 0; a < 3; a++) { + //One quad for the slice views (drawn without depth test) and one + //for the 3D view (drawn with it), so that each mesh owns its + //geometry buffers + var geoms = [makeQuad(), makeQuad()]; + this.planeGeoms.push(geoms); + + var flat = new THREE.Mesh(geoms[0], this.planeMaterials.flat); + flat.frustumCulled = false; + this.planes2d.push(flat); + this.planeGroup.add(flat); + + var solid = new THREE.Mesh(geoms[1], this.planeMaterials.solid); + solid.frustumCulled = false; + this.planes3d.push(solid); + this.planeGroup.add(solid); + + var cross = new THREE.Geometry(); + for (var i = 0; i < 4; i++) + cross.vertices.push(new THREE.Vector3()); + cross.dynamic = true; + var lines = new THREE.Line(cross, cross_material, THREE.LinePieces); + lines.frustumCulled = false; + this.crosshairs.push(lines); + this.crossGroup.add(lines); + } + }; + + module.Aligner.prototype._meshReady = function(geometries) { + for (var i = 0; i < geometries.length; i++) { + var geom = geometries[i]; + if (geom.attributes.wm === undefined) { + //Only a fiducial surface: draw it once, as both surfaces + this.hasWM = false; + geom.addAttribute("wm", geom.attributes.position); + } + geom.addAttribute("wmnorm", mriview.computeNormal(geom.attributes.wm, geom.attributes.index, geom.offsets)); + var edges = module.buildEdges(geom); + + var hemi = {geometry: geom, edges: edges, outlines: [], surfaces: []}; + for (var s = 0; s < 2; s++) { + var line = new THREE.Line(edges, this.outlineMaterials[s], THREE.LinePieces); + line.frustumCulled = false; + hemi.outlines.push(line); + this.brain.add(line); + + var surf = new THREE.Mesh(geom, this.surfaceMaterials[s]); + surf.frustumCulled = false; + hemi.surfaces.push(surf); + this.brain.add(surf); + } + hemi.projected = new THREE.Mesh(geom, this.projectedMaterial); + hemi.projected.frustumCulled = false; + this.brain.add(hemi.projected); + this.hemis.push(hemi); + } + this._ready.mesh = true; + this._checkReady(); + }; + + module.Aligner.prototype._checkReady = function() { + if (!this._ready.volume || !this._ready.mesh) + return; + $("#dataload").hide(); + $(this.object).find("#aligner-load").hide(); + this.ready = true; + this.resize(); + this.schedule(); + this.loaded.resolve(); + }; + + module.Aligner.prototype._buildUI = function() { + var names = []; + for (var name in this.colormaps) { + if (this.colormaps[name].image.height == 1) + names.push(name); + } + names.sort(); + if (names.indexOf(this._cmapName) < 0) + this._cmapName = names.indexOf("gray") >= 0 ? "gray" : names[0]; + this.volUniforms.colormap.value = this.colormaps[this._cmapName]; + + var top = {}; + if (!this.config.view_only) + top.save = {action: this.save.bind(this)}; + top.undo = {action: this.undo.bind(this)}; + top.view = {action: [this, "setMode", [MODES.outline, MODES.projected]]}; + this.ui.add(top); + + var vol = this.config.volume; + this.ui.addFolder("image", false).add({ + colormap: {action: [this, "setColormap", names]}, + flip: {action: [this, "setFlip"]}, + vmin: {action: [this, "setVmin", vol.min, vol.max]}, + vmax: {action: [this, "setVmax", vol.min, vol.max]}, + brightness: {action: [this, "setBrightness", -1, 1, 0.01]}, + contrast: {action: [this, "setContrast", 0, 4, 0.01]}, + gamma: {action: [this, "setGamma", 0.1, 4, 0.01]}, + }); + this.ui.addFolder("mesh", false).add({ + color: {action: [this, "setMeshColor"], color: true}, + opacity: {action: [this, "setMeshOpacity", 0, 1, 0.01]}, + pial: {action: [this, "setShowPial"]}, + white: {action: [this, "setShowWhite"]}, + depth: {action: [this, "setDepth", 0, 1, 0.01]}, + }); + + var slices = {}; + var titles = ["sagittal", "coronal", "axial"]; + for (var a = 0; a < 3; a++) + slices[titles[a]] = {action: [this, SLICE_SETTERS[a], 0, this.dims[this.vax[a]] - 1, 1]}; + this.ui.addFolder("slices", false).add(slices); + + this.ui.addFolder("steps", true).add({ + "translate (mm)": {action: [this, "setTranslateStep", 0.05, 10, 0.05]}, + "rotate (deg)": {action: [this, "setRotateStep", 0.05, 10, 0.05]}, + }); + this.schedule(); + }; + + //Updates the value shown by a menu control without running its action + module.Aligner.prototype._syncControl = function(folder, name, value) { + var menu = this.ui[folder]; + if (menu === undefined || menu._controls[name] === undefined) + return; + menu[name] = value; + menu._controls[name].updateDisplay(); + }; + + //------------------------------------------------------------------------- + // Slices and cursor + //------------------------------------------------------------------------- + module.Aligner.prototype.getSlice = function(axis) { + var va = this.vax[axis]; + return Math.min(Math.max(Math.round(this.cursor[va]), 0), this.dims[va] - 1); + }; + module.Aligner.prototype._setSlice = function(axis, slice) { + var va = this.vax[axis]; + this.cursor[va] = Math.min(Math.max(Math.round(slice), 0), this.dims[va] - 1); + this._cursorChanged(); + }; + module.Aligner.prototype.setSagittal = function(slice) { + if (slice === undefined) + return this.getSlice(0); + this._setSlice(0, slice); + }; + module.Aligner.prototype.setCoronal = function(slice) { + if (slice === undefined) + return this.getSlice(1); + this._setSlice(1, slice); + }; + module.Aligner.prototype.setAxial = function(slice) { + if (slice === undefined) + return this.getSlice(2); + this._setSlice(2, slice); + }; + + //The cursor in voxel coordinates of the reference volume + module.Aligner.prototype.getCursor = function() { + return this.cursor.slice(); + }; + module.Aligner.prototype.setCursor = function(voxel) { + for (var i = 0; i < 3; i++) + this.cursor[i] = Math.min(Math.max(voxel[i], -0.5), this.dims[i] - 0.5); + this._cursorChanged(); + }; + //The cursor in world coordinates + module.Aligner.prototype.cursorWorld = function() { + return new THREE.Vector3().fromArray(this.cursor).applyMatrix4(this.world); + }; + + module.Aligner.prototype._cursorChanged = function() { + var titles = ["sagittal", "coronal", "axial"]; + for (var a = 0; a < 3; a++) + this._syncControl("slices", titles[a], this.getSlice(a)); + if (this._ready.volume) + this._updateSliceGeometry(); + this.schedule(); + }; + + //Moves the slice planes to the cursor's slices, the slabs that cut the + //surfaces to half a voxel around them, and the crosshairs to the cursor + module.Aligner.prototype._updateSliceGeometry = function() { + var e = this.world.elements; + var cw = this.cursorWorld(); + for (var a = 0; a < 3; a++) { + var va = this.vax[a]; + var slice = this.getSlice(a); + var b = (a + 1) % 3, c = (a + 2) % 3; + var vb = this.vax[b], vc = this.vax[c]; + + var lo = new THREE.Vector3(Infinity, Infinity, Infinity); + var hi = new THREE.Vector3(-Infinity, -Infinity, -Infinity); + for (var i = 0; i < 4; i++) { + var voxel = new THREE.Vector3(); + voxel.setComponent(va, slice); + voxel.setComponent(vb, (i & 1) ? this.dims[vb] - 0.5 : -0.5); + voxel.setComponent(vc, (i & 2) ? this.dims[vc] - 0.5 : -0.5); + voxel.applyMatrix4(this.world); + lo.min(voxel); + hi.max(voxel); + for (var g = 0; g < 2; g++) + this.planeGeoms[a][g].vertices[i].copy(voxel); + } + for (var g = 0; g < 2; g++) { + var geom = this.planeGeoms[a][g]; + geom.verticesNeedUpdate = true; + geom.computeFaceNormals(); + geom.computeVertexNormals(); + geom.normalsNeedUpdate = true; + } + + var coord = e[a + 4 * va] * slice + e[a + 12]; + this.planeCoord[a] = coord; + this.meshUniforms.slabLo.value.setComponent(a, coord - 0.5 * this.vscale[a]); + this.meshUniforms.slabHi.value.setComponent(a, coord + 0.5 * this.vscale[a]); + + //Crosshair: one line along each in-plane axis through the cursor + var cross = this.crosshairs[a].geometry; + for (var i = 0; i < 4; i++) { + cross.vertices[i].copy(cw); + cross.vertices[i].setComponent(a, coord); + } + cross.vertices[0].setComponent(b, lo.getComponent(b)); + cross.vertices[1].setComponent(b, hi.getComponent(b)); + cross.vertices[2].setComponent(c, lo.getComponent(c)); + cross.vertices[3].setComponent(c, hi.getComponent(c)); + cross.verticesNeedUpdate = true; + + var view = this.views["xyz"[a]]; + view.label.textContent = view.title + " " + (slice + 1) + " / " + this.dims[va]; + } + }; + + //------------------------------------------------------------------------- + // The transform + //------------------------------------------------------------------------- + //The pycortex coord transform (anatomical mm -> voxel indices) as a nested + //4x4 array, and its setter + module.Aligner.prototype.getXfm = function() { + var coord = new THREE.Matrix4().multiplyMatrices(this.worldInv, this.xfm); + return module.matrixToRows(coord); + }; + module.Aligner.prototype.setXfm = function(rows) { + this.pushUndo(); + this.xfm.multiplyMatrices(this.world, module.matrixFromRows(rows)); + this._xfmChanged(); + }; + module.Aligner.prototype.pushUndo = function() { + this._undo.push(this.xfm.clone()); + if (this._undo.length > UNDO_LIMIT) + this._undo.shift(); + }; + module.Aligner.prototype.undo = function() { + if (this._undo.length == 0) { + this.showStatus("nothing to undo"); + return; + } + this.xfm.copy(this._undo.pop()); + this._xfmChanged(); + }; + module.Aligner.prototype._xfmChanged = function() { + this.brain.matrix.copy(this.xfm); + this.brain.matrixWorldNeedsUpdate = true; + this.dispatchEvent({type: "xfm"}); + this.schedule(); + }; + + //Translates the surfaces by a world vector (mm) + module.Aligner.prototype.translate = function(vector) { + this.pushUndo(); + this._translate(new THREE.Vector3().fromArray(vector)); + }; + module.Aligner.prototype._translate = function(vector) { + var trans = new THREE.Matrix4().makeTranslation(vector.x, vector.y, vector.z); + this.xfm.multiplyMatrices(trans, this.xfm); + this._xfmChanged(); + }; + //Rotates the surfaces by `angle` degrees about a world axis through the + //cursor (or through `pivot`, a world point) + module.Aligner.prototype.rotate = function(axis, angle, pivot) { + this.pushUndo(); + this._rotate(new THREE.Vector3().fromArray(axis), angle * Math.PI / 180, + pivot === undefined ? this.cursorWorld() : new THREE.Vector3().fromArray(pivot)); + }; + module.Aligner.prototype._rotate = function(axis, radians, pivot) { + var rot = new THREE.Matrix4().makeRotationAxis(axis.clone().normalize(), radians); + var toPivot = new THREE.Matrix4().makeTranslation(pivot.x, pivot.y, pivot.z); + var fromPivot = new THREE.Matrix4().makeTranslation(-pivot.x, -pivot.y, -pivot.z); + var xfm = new THREE.Matrix4().multiplyMatrices(fromPivot, this.xfm); + xfm.multiplyMatrices(rot, xfm); + this.xfm.multiplyMatrices(toPivot, xfm); + this._xfmChanged(); + }; + + module.Aligner.prototype.save = function() { + if (this.config.view_only) { + this.showStatus("view only: the transform is not saved", true); + return "view only"; + } + this.showStatus("saving..."); + $.ajax({ + type: "POST", + url: "save", + data: {xfm: JSON.stringify(this.getXfm())}, + dataType: "json", + }).done(function(resp) { + this.showStatus(resp.message, resp.status != "ok"); + }.bind(this)).fail(function() { + this.showStatus("saving failed: no answer from the server", true); + }.bind(this)); + return "saving"; + }; + + //The canvas as drawn last, as a PNG data url. Drawing is left to the + //scheduled animation frames, so that a slow (software) render cannot hold + //up the reply to the python side; getFrames tells when a frame landed. + module.Aligner.prototype.snapshot = function() { + return this.renderer.domElement.toDataURL("image/png"); + }; + //Number of frames drawn so far + module.Aligner.prototype.getFrames = function() { + return this.nframes; + }; + //Runs a method for the python side and tags the result with the token + //of the request. The websocket protocol pairs replies with requests by + //their order and gives up on a reply after two seconds, so replies held + //up by a busy page (parsing the surfaces, a slow frame) would otherwise + //be taken for those of later requests. + //The reply also carries the number of frames drawn so far and whether a + //redraw is pending, so that the python side can wait for the frame that + //shows the effect of the call before taking a snapshot. + module.Aligner.prototype.call = function(token, name, args) { + var reply = {token: token, frames: this.nframes}; + if (!(this[name] instanceof Function)) { + reply.error = "no method " + name; + } else { + try { + reply.value = this[name].apply(this, args); + } catch (e) { + reply.error = e.message; + } + } + reply.scheduled = this._scheduled; + return reply; + }; + //A menu control by its dotted path, for instance image.vmin or mesh.color + module.Aligner.prototype.getControl = function(name) { + return this.ui.get(name); + }; + module.Aligner.prototype.setControl = function(name, value) { + this.ui.set(name, value); + }; + + //------------------------------------------------------------------------- + // Display settings + //------------------------------------------------------------------------- + module.Aligner.prototype.setMode = function(mode) { + if (mode === undefined) + return this._mode; + this._mode = mode; + this.schedule(); + }; + module.Aligner.prototype.toggleMode = function() { + this.setMode(this._mode == MODES.outline ? MODES.projected : MODES.outline); + }; + module.Aligner.prototype.setColormap = function(name) { + if (name === undefined) + return this._cmapName; + if (this.colormaps[name] === undefined) + return; + this._cmapName = name; + this.volUniforms.colormap.value = this.colormaps[name]; + this.schedule(); + }; + module.Aligner.prototype.setFlip = function(flip) { + if (flip === undefined) + return this.volUniforms.flip.value == 1; + this.volUniforms.flip.value = flip ? 1 : 0; + this.schedule(); + }; + module.Aligner.prototype.setVmin = function(value) { + if (value === undefined) + return this.volUniforms.vmin.value; + this.volUniforms.vmin.value = value; + this.schedule(); + }; + module.Aligner.prototype.setVmax = function(value) { + if (value === undefined) + return this.volUniforms.vmax.value; + this.volUniforms.vmax.value = value; + this.schedule(); + }; + module.Aligner.prototype.setBrightness = function(value) { + if (value === undefined) + return this.volUniforms.brightness.value; + this.volUniforms.brightness.value = value; + this.schedule(); + }; + module.Aligner.prototype.setContrast = function(value) { + if (value === undefined) + return this.volUniforms.contrast.value; + this.volUniforms.contrast.value = value; + this.schedule(); + }; + module.Aligner.prototype.setGamma = function(value) { + if (value === undefined) + return this.volUniforms.gamma.value; + this.volUniforms.gamma.value = value; + this.schedule(); + }; + module.Aligner.prototype.setMeshColor = function(color) { + if (color === undefined) + return "#" + this.meshUniforms.color.value.getHexString(); + this.meshUniforms.color.value.set(color); + this.schedule(); + }; + //Opacity of the whole surfaces in the 3D view (0 hides them, leaving + //the outlines on the slices) + module.Aligner.prototype.setMeshOpacity = function(value) { + if (value === undefined) + return this.meshUniforms.opacity.value; + this.meshUniforms.opacity.value = value; + this.schedule(); + }; + module.Aligner.prototype.setShowPial = function(show) { + if (show === undefined) + return this._showSurf[0]; + this._showSurf[0] = show; + this.schedule(); + }; + module.Aligner.prototype.setShowWhite = function(show) { + if (show === undefined) + return this._showSurf[1]; + this._showSurf[1] = show; + this.schedule(); + }; + //Cortical depth the data is painted at, from pial (0) to white matter (1) + module.Aligner.prototype.setDepth = function(value) { + if (value === undefined) + return this.projectedMaterial.uniforms.depth.value; + this.projectedMaterial.uniforms.depth.value = value; + this.schedule(); + }; + module.Aligner.prototype.setTranslateStep = function(value) { + if (value === undefined) + return this._translateStep; + this._translateStep = value; + }; + module.Aligner.prototype.setRotateStep = function(value) { + if (value === undefined) + return this._rotateStep; + this._rotateStep = value; + }; + + //------------------------------------------------------------------------- + // Drawing + //------------------------------------------------------------------------- + module.Aligner.prototype.resize = function() { + var w = $(this.object).width(), h = $(this.object).height(); + if (!w || !h) + return; + this.width = w; + this.height = h; + this.renderer.setSize(w, h); + for (var i = 0; i < this.viewlist.length; i++) { + var view = this.viewlist[i]; + view.rect = { + left: Math.floor(view.col * w / 2), + top: Math.floor(view.row * h / 2), + width: Math.floor(w / 2), + height: Math.floor(h / 2), + }; + } + this.schedule(); + }; + + module.Aligner.prototype.schedule = function() { + if (!this._scheduled) { + this._scheduled = true; + requestAnimationFrame(this._draw); + } + }; + + module.Aligner.prototype.draw = function() { + this._scheduled = false; + if (!this.ready || !this.width || !this.height) + return; + this.controls.update(this.camera3d); + this.renderer.enableScissorTest(true); + for (var i = 0; i < this.viewlist.length; i++) { + var view = this.viewlist[i], r = view.rect; + if (r.width < 1 || r.height < 1) + continue; + var bottom = this.height - r.top - r.height; + this.renderer.setViewport(r.left, bottom, r.width, r.height); + this.renderer.setScissor(r.left, bottom, r.width, r.height); + this._prepareView(view); + if (view.is2d && this._mode == MODES.outline) { + //Two passes: the slice first, then the outlines and the + //crosshair on top of it whatever their depth. Drawing them in + //one pass would leave the order to three.js, which draws its + //list of opaque objects back to front through the scene. + this._showLayer(view, "plane"); + this.renderer.render(this.scene, view.camera); + this._showLayer(view, "lines"); + this.renderer.autoClear = false; + this.renderer.render(this.scene, view.camera); + this.renderer.autoClear = true; + } else { + this._showLayer(view, "all"); + this.renderer.render(this.scene, view.camera); + } + } + this.renderer.enableScissorTest(false); + this.nframes++; + this.dispatchEvent({type: "draw"}); + }; + + //Sets the visibility of the objects for a view: a slice view shows its + //plane, the outline of the surfaces cut to that slice and the crosshair; + //the 3D view shows all three planes, the outlines on all of them and, + //when opaque enough, the whole surfaces. In the painted mode every view + //shows the surfaces colored by the volume instead. `layer` restricts the + //visible objects to the "plane" or the "lines" of a slice view. + module.Aligner.prototype._showLayer = function(view, layer) { + var outline = this._mode == MODES.outline; + var plane = layer != "lines", lines = layer != "plane"; + for (var a = 0; a < 3; a++) { + this.planes2d[a].visible = plane && outline && view.is2d && view.axis == a; + this.planes3d[a].visible = plane && outline && !view.is2d; + this.crosshairs[a].visible = lines && outline && view.is2d && view.axis == a; + } + var surfaces = outline && !view.is2d && this.meshUniforms.opacity.value > 0; + for (var i = 0; i < this.hemis.length; i++) { + var hemi = this.hemis[i]; + for (var s = 0; s < 2; s++) { + var shown = this._showSurf[s] && (s == 0 || this.hasWM); + hemi.outlines[s].visible = lines && outline && shown; + hemi.surfaces[s].visible = lines && surfaces && shown; + } + hemi.projected.visible = lines && !outline; + } + }; + + //Sets the slabs that cut the surfaces, the camera and the light of a view + module.Aligner.prototype._prepareView = function(view) { + var mask = this.meshUniforms.slabMask.value; + if (view.is2d) + mask.set(view.axis == 0 ? 1 : 0, view.axis == 1 ? 1 : 0, view.axis == 2 ? 1 : 0); + else + mask.set(1, 1, 1); + + if (view.is2d) { + if (!view.fitted) + this._fitView(view); + this._updateOrthoCamera(view); + this.light.position.copy(view.camera.position); + this.light.target.position.copy(view.center); + } else { + this.camera3d.aspect = view.rect.width / view.rect.height; + this.camera3d.updateProjectionMatrix(); + this.light.position.copy(this.camera3d.position); + this.light.target.position.copy(this.controls.target); + } + }; + + //Frames the whole volume in a slice view + module.Aligner.prototype._fitView = function(view) { + var size = this.bbox.max.clone().sub(this.bbox.min); + var upAxis = this._majorAxis(view.up), rightAxis = this._majorAxis(view.right); + var aspect = view.rect.width / view.rect.height; + view.height = 1.1 * Math.max(size.getComponent(upAxis), size.getComponent(rightAxis) / aspect); + view.center.copy(this.bboxCenter); + view.fitted = true; + }; + module.Aligner.prototype._majorAxis = function(vector) { + var arr = vector.toArray(), best = 0; + for (var i = 1; i < 3; i++) + if (Math.abs(arr[i]) > Math.abs(arr[best])) + best = i; + return best; + }; + + module.Aligner.prototype._updateOrthoCamera = function(view) { + var cam = view.camera; + var aspect = view.rect.width / view.rect.height; + var h = view.height, w = h * aspect; + cam.left = -w / 2; + cam.right = w / 2; + cam.top = h / 2; + cam.bottom = -h / 2; + cam.updateProjectionMatrix(); + cam.up.copy(view.up); + cam.position.copy(view.center).sub(view.look.clone().multiplyScalar(2000)); + cam.lookAt(view.center); + cam.updateMatrixWorld(); + }; + + //------------------------------------------------------------------------- + // Interaction + //------------------------------------------------------------------------- + module.Aligner.prototype._bindView = function(view) { + view.div.addEventListener("mouseenter", function() { + this.hoverView = view; + }.bind(this), false); + view.div.addEventListener("contextmenu", function(event) { + event.preventDefault(); + }, false); + if (!view.is2d) { + this.controls.bind(view.div); + return; + } + view.div.addEventListener("mousedown", this._onMouseDown.bind(this, view), false); + view.div.addEventListener("wheel", this._onWheel.bind(this, view), {passive: false}); + }; + + module.Aligner.prototype._localPos = function(view, event) { + var r = view.div.getBoundingClientRect(); + return {x: event.clientX - r.left, y: event.clientY - r.top}; + }; + + //The world point under the mouse in a slice view, on the slice plane + module.Aligner.prototype._mouseWorld = function(view, pos) { + this._updateOrthoCamera(view); + var nx = (pos.x / view.rect.width) * 2 - 1; + var ny = 1 - (pos.y / view.rect.height) * 2; + var point = new THREE.Vector3(nx, ny, 0).unproject(view.camera); + point.setComponent(view.axis, this.planeCoord[view.axis]); + return point; + }; + module.Aligner.prototype._worldToScreen = function(view, point) { + this._updateOrthoCamera(view); + var ndc = point.clone().project(view.camera); + return {x: (ndc.x + 1) / 2 * view.rect.width, y: (1 - ndc.y) / 2 * view.rect.height}; + }; + //Millimeters per screen pixel in a slice view + module.Aligner.prototype._mmPerPixel = function(view) { + return view.height / view.rect.height; + }; + + //Moves the cursor, in the plane of a slice view, to the mouse position + module.Aligner.prototype._setCursorFromMouse = function(view, pos) { + var voxel = this._mouseWorld(view, pos).applyMatrix4(this.worldInv); + for (var a = 0; a < 3; a++) { + if (a == view.axis) + continue; + var va = this.vax[a]; + this.cursor[va] = Math.min(Math.max(voxel.getComponent(va), -0.5), this.dims[va] - 0.5); + } + this._cursorChanged(); + }; + + module.Aligner.prototype._onMouseDown = function(view, event) { + event.preventDefault(); + this.hoverView = view; + var pos = this._localPos(view, event); + var state; + if (event.button === 0 && event.shiftKey) { + state = "pan"; + } else if (event.button === 0) { + state = "cursor"; + this._setCursorFromMouse(view, pos); + } else if (event.button === 1) { + state = "pan"; + } else if (event.button === 2) { + state = (event.ctrlKey || event.altKey || event.metaKey) ? "rotate" : "translate"; + this.pushUndo(); + } else { + return; + } + this._drag = {view: view, state: state, last: pos}; + if (state == "rotate") { + this._drag.pivot = this._worldToScreen(view, this.cursorWorld()); + this._drag.angle = Math.atan2(-(pos.y - this._drag.pivot.y), pos.x - this._drag.pivot.x); + } + }; + + module.Aligner.prototype._onMouseMove = function(event) { + var drag = this._drag; + if (drag === null) + return; + event.preventDefault(); + var view = drag.view; + var pos = this._localPos(view, event); + var dx = pos.x - drag.last.x, dy = pos.y - drag.last.y; + var scale = this._mmPerPixel(view); + if (drag.state == "cursor") { + this._setCursorFromMouse(view, pos); + } else if (drag.state == "pan") { + view.center.sub(view.right.clone().multiplyScalar(dx * scale)); + view.center.add(view.up.clone().multiplyScalar(dy * scale)); + } else if (drag.state == "translate") { + var move = view.right.clone().multiplyScalar(dx * scale); + move.sub(view.up.clone().multiplyScalar(dy * scale)); + this._translate(move); + } else if (drag.state == "rotate") { + var angle = Math.atan2(-(pos.y - drag.pivot.y), pos.x - drag.pivot.x); + var delta = angle - drag.angle; + if (delta > Math.PI) + delta -= 2 * Math.PI; + else if (delta < -Math.PI) + delta += 2 * Math.PI; + drag.angle = angle; + //counterclockwise on the screen is a positive rotation about the + //axis pointing at the viewer + this._rotate(view.look.clone().negate(), delta, this.cursorWorld()); + } + drag.last = pos; + this.schedule(); + }; + + module.Aligner.prototype._onMouseUp = function(event) { + this._drag = null; + }; + + //Wheel: steps through the slices; with ctrl (or a trackpad pinch) zooms + //the view about the mouse position + module.Aligner.prototype._onWheel = function(view, event) { + event.preventDefault(); + var delta = wheelDelta(event); + if (event.ctrlKey || event.metaKey) { + var pos = this._localPos(view, event); + var before = this._mouseWorld(view, pos); + var factor = Math.exp(Math.min(Math.max(delta, -100), 100) * 0.005); + view.height *= factor; + //keep the point under the mouse where it is + view.center.sub(before).multiplyScalar(factor).add(before); + } else { + this._wheelAcc += delta; + while (Math.abs(this._wheelAcc) >= WHEEL_STEP) { + var step = this._wheelAcc > 0 ? 1 : -1; + this._setSlice(view.axis, this.getSlice(view.axis) + step); + this._wheelAcc -= step * WHEEL_STEP; + } + } + this.schedule(); + }; + + module.Aligner.prototype._onKeyDown = function(event) { + var tag = event.target.tagName; + if (tag == "INPUT" || tag == "TEXTAREA" || tag == "SELECT") + return; + var key = event.key; + if ((event.ctrlKey || event.metaKey) && (key == "z" || key == "Z")) { + this.undo(); + event.preventDefault(); + return; + } + if (event.ctrlKey || event.metaKey || event.altKey) + return; + if (key == "m" || key == "M") { + this.toggleMode(); + event.preventDefault(); + return; + } + + var view = this.hoverView; + if (view === null || !view.is2d) + return; + var fine = event.shiftKey ? 0.1 : 1; + var tstep = this._translateStep * fine; + var rstep = this._rotateStep * fine * Math.PI / 180; + var toViewer = view.look.clone().negate(); + var handled = true; + switch (key) { + case "ArrowLeft": + this.translate(view.right.clone().multiplyScalar(-tstep).toArray()); + break; + case "ArrowRight": + this.translate(view.right.clone().multiplyScalar(tstep).toArray()); + break; + case "ArrowUp": + this.translate(view.up.clone().multiplyScalar(tstep).toArray()); + break; + case "ArrowDown": + this.translate(view.up.clone().multiplyScalar(-tstep).toArray()); + break; + case "q": case "Q": + this.pushUndo(); + this._rotate(toViewer, rstep, this.cursorWorld()); + break; + case "e": case "E": + this.pushUndo(); + this._rotate(toViewer, -rstep, this.cursorWorld()); + break; + case "[": case "{": + this._setSlice(view.axis, this.getSlice(view.axis) - 1); + break; + case "]": case "}": + this._setSlice(view.axis, this.getSlice(view.axis) + 1); + break; + default: + handled = false; + } + if (handled) + event.preventDefault(); + }; + + return module; +}(aligner || {})); diff --git a/cortex/webgl/resources/js/menu.js b/cortex/webgl/resources/js/menu.js index 4c4538d9b..137a704f8 100644 --- a/cortex/webgl/resources/js/menu.js +++ b/cortex/webgl/resources/js/menu.js @@ -127,7 +127,8 @@ var jsplot = (function (module) { for (var i = 2; i < desc.action.length; i++) newargs.push(desc.action[i]); - var ctrl = gui.add.apply(gui, newargs); + //a color picker, for a method that gets and sets a css color string + var ctrl = desc.color ? gui.addColor(this, name) : gui.add.apply(gui, newargs); ctrl.onChange(function(name) { parent[method](this[name]); this.dispatchEvent({type:"update"}); diff --git a/cortex/webgl/resources/js/shaderlib.js b/cortex/webgl/resources/js/shaderlib.js index dc4deb82e..753f35b46 100644 --- a/cortex/webgl/resources/js/shaderlib.js +++ b/cortex/webgl/resources/js/shaderlib.js @@ -911,6 +911,157 @@ var Shaderlib = (function() { return {vertex:header+vertShade, fragment:header+fragShade, attrs:attributes}; }, + aligner_volume: function(opts) { + //Colors each fragment with the functional volume sampled at the + //fragment's world position. The aligner uses it for its slice + //planes (unlit) and for painting the volume onto the surface (lit). + //The world frame is the functional voxel grid in millimeters, so + //volxfm is a plain scaling and the planes show unresampled voxels. + //The lookup adds brightness, contrast, gamma and a flip to the + //vmin/vmax range of the colormap. + //sampler: nearest or trilinear + //lights: whether to apply phong lighting (false for the planes) + //depthmix: mix the position between the pial (position) and white + // matter (wm) surfaces with the depth uniform + var sampler = opts.sampler || "nearest"; + var header = ""; + if (opts.lights !== undefined && !opts.lights) + header += "#define NOLIGHTS\n"; + if (opts.depthmix) + header += "#define DEPTHMIX\n"; + + var vertShade = [ + "#ifndef NOLIGHTS", + THREE.ShaderChunk[ "lights_phong_pars_vertex" ], + "#endif", + "uniform mat4 volxfm;", + "#ifdef DEPTHMIX", + "uniform float depth;", + "attribute vec4 wm;", + "attribute vec3 wmnorm;", + "#endif", + + "varying vec3 vViewPosition;", + "varying vec3 vNormal;", + "varying vec3 vPos;", + + "void main() {", + "vec3 mpos = position;", + "vec3 mnorm = normal;", + "#ifdef DEPTHMIX", + "mpos = mix(position, wm.xyz, depth);", + "mnorm = mix(normal, wmnorm, depth);", + "#endif", + "vec4 world = modelMatrix * vec4(mpos, 1.0);", + "vec4 mvPosition = viewMatrix * world;", + "vViewPosition = -mvPosition.xyz;", + "vNormal = normalMatrix * mnorm;", + "vPos = (volxfm * world).xyz;", + "gl_Position = projectionMatrix * mvPosition;", + "}", + ].join("\n"); + + var fragShade = [ + "uniform sampler2D colormap;", + "uniform float vmin;", + "uniform float vmax;", + "uniform float brightness;", + "uniform float contrast;", + "uniform float gamma;", + "uniform int flip;", + "uniform vec2 mosaic[2];", + "uniform vec2 dshape[2];", + "uniform float nslices;", + "uniform sampler2D data[4];", + "uniform vec3 outside;", + + "varying vec3 vPos;", + "#ifndef NOLIGHTS", + THREE.ShaderChunk[ "lights_phong_pars_fragment" ], + "#endif", + + utils.standard_frag_vars, + utils.samplers, + + "void main() {", + //Fragments outside the volume, and the padding between the + //mosaic tiles (NaN), get a flat color instead of data + "vec3 lo = vec3(-0.5);", + "vec3 hi = vec3(dshape[0].x, dshape[0].y, nslices) - 0.5;", + "bool inside = all(greaterThanEqual(vPos, lo)) && all(lessThanEqual(vPos, hi));", + "float value = "+sampler+"_x(data[0], vPos).r;", + "bool valid = inside && (value <= 0. || 0. < value);", + + "float norm = (value - vmin) / (vmax - vmin);", + "norm = clamp(norm * contrast + brightness, 0., 1.);", + "norm = pow(norm, gamma);", + "if (flip == 1) norm = 1. - norm;", + "vec4 vColor = texture2D(colormap, vec2(norm, 0.));", + + "gl_FragColor = valid ? vec4(vColor.rgb, 1.) : vec4(outside, 1.);", + "#ifndef NOLIGHTS", + THREE.ShaderChunk[ "lights_phong_fragment" ], + "#endif", + "}" + ].join("\n"); + + var attributes = {}; + if (opts.depthmix) { + attributes.wm = { type: 'v4', value: null }; + attributes.wmnorm = { type: 'v3', value: null }; + } + + return {vertex:header+vertShade, fragment:header+fragShade, attrs:attributes}; + }, + + aligner_mesh: function(opts) { + //Flat colored surface for the aligner. The depth uniform picks the + //surface between pial (0) and white matter (1). With doClip set, + //only fragments within the slabs (one per world axis, enabled by + //slabMask) survive, which draws the intersection of the surface + //with the displayed slices; drawn as lines this gives the outline + //of the cortex on each slice. + var vertShade = [ + "uniform float depth;", + "attribute vec4 wm;", + "varying vec3 vWorld;", + + "void main() {", + "vec3 mpos = mix(position, wm.xyz, depth);", + "vec4 world = modelMatrix * vec4(mpos, 1.0);", + "vWorld = world.xyz;", + "gl_Position = projectionMatrix * viewMatrix * world;", + "}", + ].join("\n"); + + var fragShade = [ + "uniform vec3 color;", + "uniform float opacity;", + "uniform int doClip;", + "uniform vec3 slabLo;", + "uniform vec3 slabHi;", + "uniform vec3 slabMask;", + "varying vec3 vWorld;", + + "void main() {", + "if (doClip == 1) {", + "bvec3 inside = bvec3(", + "slabMask.x > .5 && vWorld.x >= slabLo.x && vWorld.x <= slabHi.x,", + "slabMask.y > .5 && vWorld.y >= slabLo.y && vWorld.y <= slabHi.y,", + "slabMask.z > .5 && vWorld.z >= slabLo.z && vWorld.z <= slabHi.z);", + "if (!any(inside)) discard;", + "}", + "gl_FragColor = vec4(color, opacity);", + "}" + ].join("\n"); + + var attributes = { + wm: { type: 'v4', value: null }, + }; + + return {vertex:vertShade, fragment:fragShade, attrs:attributes}; + }, + cmap_quad: function() { //Colormaps the full-screen quad, used for stage 2 of volume integration var vertShade = [ diff --git a/docs/align.rst b/docs/align.rst index 06b4ad9bb..37d8a27bd 100644 --- a/docs/align.rst +++ b/docs/align.rst @@ -50,19 +50,58 @@ However, in practice, the search range is too big to be practically useful, and Manual Alignment ---------------- -.. note:: - Currently the manual aligner only works on Ubuntu 14.04. The manual - aligner uses Mayavi, which doesn't seem to be working in later versions of - Ubuntu. As an alternative to ``cortex.align.manual``, you can use - ``cortex.align.fs_manual``, which uses FreeSurfer's Freeview instead of Mayavi. - Unfortunately, the automatic alignment only gets you like 95% of the way to a good alignment. To do the final 5%, you need to manually fix it up. -Pycortex offers a GUI aligner, built using Mayavi. +Pycortex offers a GUI aligner that runs in the browser, built on the WebGL viewer. +The older aligners, built with Mayavi (``cortex.align.mayavi_manual``) and with FreeSurfer's Freeview (``cortex.align.manual``), are described further below. + +Aligning in the browser +~~~~~~~~~~~~~~~~~~~~~~~ + +To start the browser-based aligner for a new transform, pass the reference image +:: + cortex.align.webgl_manual('S1', 'example-transform', reference='./ref-image.nii.gz') + +To adjust an existing transform, leave the reference out +:: + cortex.align.webgl_manual('S1', 'example-transform') + +Note: if you are fixing a transform you had previously used for things, you will need to delete the mask files in the transform's folder. +To look at such a transform without saving, pass ``view_only=True``. + +The page shows the coronal, axial and sagittal slices of the reference image, and a 3D view of the three slices. +The reference image is drawn on its own voxel grid, so its voxels appear as they are, without resampling, and the pial and white matter surfaces are moved into its space. +In each slice view the surfaces are cut off at the displayed slice, so what you see is their outline on the slice. +You move the surfaces until this outline follows the anatomy in the image. + +* In a slice view, a left drag moves the cursor. The cursor sets the slices shown in the other views and is the pivot of rotations. The wheel or ``[`` and ``]`` change the slice, ctrl + wheel zooms, and a middle (or shift + left) drag pans. +* A right drag or the arrow keys translate the surfaces in the plane of the view under the mouse. A ctrl + right drag or ``q`` and ``e`` rotate them about the cursor, in the plane of the view under the mouse. Holding shift makes the keyboard steps ten times smaller, and ctrl + z undoes. Only rotations and translations are possible; the transform cannot stretch the brain. +* In the 3D view, a left drag rotates, a right drag pans and the wheel zooms. + +The panel on the right holds the controls. +``view`` switches between the surface outlines on the slices and the reference image painted onto the surface (``m`` toggles it too), which shows how the pattern of the data falls on the cortex. +``image`` sets the colormap, its range (``vmin`` and ``vmax``), ``brightness``, ``contrast`` and ``gamma``, and flips the colormap. +``mesh`` sets the color of the surfaces, their ``opacity`` in the 3D view (0 shows only the outlines), which of the two surfaces are shown, and the cortical ``depth`` the data is painted at. +``slices`` selects the slices, and ``steps`` sets the keyboard steps. + +To save the alignment, click ``save``. +The transform is stored into the database at once, and the window can then be closed. +The function returns a handle to the running aligner: ``handle.get_xfm()`` returns the current transform as a 4x4 matrix, and ``handle.save()`` saves it. + +The initial colormap, color of the surfaces and opacity are set in the ``[webgl_aligner]`` section of the config file. + +Mayavi aligner +~~~~~~~~~~~~~~ + +.. note:: + The Mayavi aligner only works on Ubuntu 14.04. It uses Mayavi, which + doesn't seem to be working in later versions of Ubuntu. Use + ``cortex.align.webgl_manual`` instead, or ``cortex.align.manual``, + which uses FreeSurfer's Freeview. -To start the manual aligner, call +To start the Mayavi aligner, call :: - cortex.align.manual('S1', 'example-transform') + cortex.align.mayavi_manual('S1', 'example-transform') Note: if you are fixing a transform you had previous used for things, you will need to delete the mask files in the transform's folder. You will see a window like this pop up: From e4d119832dde869c3e5c9e4fde1e3f44781e6a3b Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 19 Sep 2026 16:33:55 +0000 Subject: [PATCH 02/10] Delete stale masks when the aligner saves an edited alignment The browser aligner refused to open a transform that had cached masks. It now opens, and deletes those masks when it saves: they were cut out of the reference volume through the alignment being replaced, nothing else invalidates them, and db.save_xfm refuses to write over a transform that still has them. - cortex/webgl/aligner.py: cached_masks and clear_masks, called from the save handler before db.save_xfm; the save response names what it deleted, and the page config lists the masks - the page warns, on opening a transform that has masks, that saving deletes them and that data masked with them has to be masked again - view_only stays an explicit choice rather than something masks force Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_0115CeZYMxr1wvRSL2V8meKZ --- cortex/align.py | 13 ++- cortex/tests/test_webgl_aligner.py | 130 +++++++++++++++++++++++-- cortex/webgl/aligner.html | 3 +- cortex/webgl/aligner.py | 87 ++++++++++++++--- cortex/webgl/resources/css/aligner.css | 2 +- cortex/webgl/resources/js/aligner.js | 10 ++ docs/align.rst | 8 +- 7 files changed, 222 insertions(+), 31 deletions(-) diff --git a/cortex/align.py b/cortex/align.py index 6b902f39e..aab47be5a 100644 --- a/cortex/align.py +++ b/cortex/align.py @@ -126,10 +126,15 @@ def webgl_manual(subject: str, xfmname: str, reference: Optional[str] = None, ** When Save is pressed the transform is stored into the pycortex database as `xfmname`, as a 'coord' transform. A new transform requires - `reference`, which is copied into the database; an existing transform - is loaded together with its stored reference, and refuses to open for - editing while masks are cached for it (pass ``view_only=True`` to - inspect it). + `reference`, which is copied into the database; an existing transform is + loaded together with its stored reference. + + Saving also deletes the masks cached for `xfmname`, since they were cut + out of the reference volume through the alignment being replaced. The + page warns about this when it opens a transform that has masks, and the + save message names the ones it deleted. Data already masked with them + has to be masked again from the volumes. Pass ``view_only=True`` to + inspect an alignment without saving. Parameters ---------- diff --git a/cortex/tests/test_webgl_aligner.py b/cortex/tests/test_webgl_aligner.py index 825c9c09b..85140a6fe 100644 --- a/cortex/tests/test_webgl_aligner.py +++ b/cortex/tests/test_webgl_aligner.py @@ -8,6 +8,7 @@ import base64 import io import json +import os import time import urllib.parse import urllib.request @@ -59,6 +60,34 @@ def recorder(monkeypatch): return rec +@pytest.fixture +def stale_masks(tmp_path, monkeypatch): + """Put two cached masks for the transform in `tmp_path`. + + Only the mask paths are redirected; every other path stays as it is, so + the bundled filestore is neither read for masks nor written to. + """ + real_get_paths = database.db.get_paths + + def get_paths(subject): + paths = dict(real_get_paths(subject)) + paths["masks"] = str(tmp_path / "mask_{type}.nii.gz") + return paths + + monkeypatch.setattr(database.db, "get_paths", get_paths) + names = ["thick", "thin"] + for name in names: + (tmp_path / ("mask_%s.nii.gz" % name)).write_bytes(b"a stale mask") + return tmp_path, names + + +def _page_config(html): + """The config object the page is generated with.""" + marker = 'viewer = figure.add(aligner.Aligner, "main", true, ' + start = html.index(marker) + len(marker) + return json.loads(html[start:html.index(");", start)]) + + @pytest.fixture def server(request): """A running aligner server for the bundled transform, stopped at teardown.""" @@ -144,12 +173,16 @@ def test_existing_transform_refuses_new_reference(): open_browser=False, display_url=False) -def test_transform_with_masks_requires_view_only(monkeypatch): - import types - - monkeypatch.setattr(aligner, "glob", types.SimpleNamespace(glob=lambda pattern: ["mask_thick.nii.gz"])) - with pytest.raises(ValueError, match="cached masks"): - aligner.show(subj, xfmname, open_browser=False, display_url=False) +def test_aligner_opens_with_cached_masks(stale_masks): + """Cached masks no longer keep a transform from being edited; the page + is told about them so it can warn that saving deletes them.""" + srv = aligner.show(subj, xfmname, open_browser=False, display_url=False) + try: + config = _page_config(_open("http://localhost:%d/" % srv.port).decode()) + finally: + srv.stop() + assert config["view_only"] is False + assert config["masks"] == ["mask_thick.nii.gz", "mask_thin.nii.gz"] def test_align_entry_point_forwards(monkeypatch): @@ -164,6 +197,85 @@ def fake_show(subject, name, reference=None, **kwargs): assert seen == dict(subject=subj, name=xfmname, reference=None, kwargs=dict(view_only=True)) +# --------------------------------------------------------------------------- +# Stale masks +# --------------------------------------------------------------------------- + + +def test_cached_masks_are_listed_and_cleared(stale_masks): + tmp_path, names = stale_masks + paths = aligner.cached_masks(subj, xfmname) + assert [os.path.basename(p) for p in paths] == ["mask_thick.nii.gz", "mask_thin.nii.gz"] + # clear_masks reports the names db.get_mask takes, not the filenames + assert aligner.clear_masks(subj, xfmname) == names + assert aligner.cached_masks(subj, xfmname) == [] + assert sorted(tmp_path.glob("mask_*")) == [] + + +def test_clearing_a_transform_without_masks_does_nothing(): + # also guards against a test leaving masks in the bundled filestore + assert aligner.cached_masks(subj, xfmname) == [] + assert aligner.clear_masks(subj, xfmname) == [] + + +def test_save_deletes_the_stale_masks_first(stale_masks): + """Saving an edited alignment deletes the masks cut with the old one. + + They have to be gone before the transform is written: db.save_xfm + refuses to write over a transform that still has masks. + """ + tmp_path, names = stale_masks + seen = {} + + def save_xfm(subject, name, xfm, xfmtype="magnet", reference=None): + seen["masks"] = sorted(p.name for p in tmp_path.glob("mask_*")) + seen["xfm"] = np.asarray(xfm, dtype=float) + seen["xfmtype"] = xfmtype + + srv = aligner.show(subj, xfmname, open_browser=False, display_url=False) + xfm = np.arange(16, dtype=float).reshape(4, 4) + try: + with pytest.MonkeyPatch.context() as patch: + patch.setattr(database.db, "save_xfm", save_xfm) + resp = json.loads(_open("http://localhost:%d/save" % srv.port, + dict(xfm=json.dumps(xfm.tolist()))).decode()) + finally: + srv.stop() + + assert resp["status"] == "ok" + assert resp["masks_deleted"] == names + assert "deleted 2 stale masks (thick, thin)" in resp["message"] + assert seen["masks"] == [], "the masks were still there when the transform was written" + assert seen["xfmtype"] == "coord" + assert np.allclose(seen["xfm"], xfm) + assert sorted(tmp_path.glob("mask_*")) == [] + + +def test_a_refused_save_keeps_the_masks(stale_masks): + """A save that does not go through leaves the masks alone.""" + tmp_path, names = stale_masks + srv = aligner.show(subj, xfmname, open_browser=False, display_url=False) + try: + resp = json.loads(_open("http://localhost:%d/save" % srv.port, + dict(xfm=json.dumps([1, 2, 3]))).decode()) + finally: + srv.stop() + assert resp["status"] == "error" + assert [p.name for p in sorted(tmp_path.glob("mask_*"))] == ["mask_thick.nii.gz", "mask_thin.nii.gz"] + + +def test_view_only_keeps_the_masks(stale_masks): + tmp_path, names = stale_masks + srv = aligner.show(subj, xfmname, view_only=True, open_browser=False, display_url=False) + try: + resp = json.loads(_open("http://localhost:%d/save" % srv.port, + dict(xfm=json.dumps(np.eye(4).tolist()))).decode()) + finally: + srv.stop() + assert resp["status"] == "error" + assert [p.name for p in sorted(tmp_path.glob("mask_*"))] == ["mask_thick.nii.gz", "mask_thin.nii.gz"] + + # --------------------------------------------------------------------------- # Server endpoints (no browser) # --------------------------------------------------------------------------- @@ -175,16 +287,14 @@ def test_page_carries_config(server): base = "http://localhost:%d" % server.port html = _open(base + "/aligner.html").decode() assert "aligner.Aligner" in html - start = html.index('viewer = figure.add(aligner.Aligner, "main", true, ') + len( - 'viewer = figure.add(aligner.Aligner, "main", true, ') - end = html.index(");", start) - config = json.loads(html[start:end]) + config = _page_config(html) xfm = database.db.get_xfm(subj, xfmname) nii = xfm.reference_nifti assert config["subject"] == subj assert config["xfmname"] == xfmname assert config["view_only"] is False + assert config["masks"] == [] assert np.allclose(config["xfm"], xfm.xfm) assert np.allclose(config["world"], aligner.reference_frame(nii)) assert config["volume"]["shape"] == list(nii.shape[::-1]) diff --git a/cortex/webgl/aligner.html b/cortex/webgl/aligner.html index 315150e3b..a37c8c989 100644 --- a/cortex/webgl/aligner.html +++ b/cortex/webgl/aligner.html @@ -34,7 +34,8 @@ {% end %} +{% block css %} + +{% end %} {% block jsinit %} var viewer, figure, sock; var viewopts = {}; @@ -47,7 +49,7 @@

Slice views

Surfaces

- + @@ -59,6 +61,11 @@

3D view

right drag, arrowstranslate in the plane of the view under the mouse
right drag, WASD, arrowstranslate in the plane of the view under the mouse
ctrl + right drag, q / erotate about the cursor, in the plane of the view under the mouse
shiftfine keyboard steps (a tenth)
ctrl + zundo
right dragpan
wheelzoom
+

Saving

+ + + +
transformthe name this is saved under; change it to save the alignment as a new transform
*on the save button and in the title, an alignment that differs from the one last saved
diff --git a/cortex/webgl/aligner.py b/cortex/webgl/aligner.py index 383155bf8..e9df6fc7b 100644 --- a/cortex/webgl/aligner.py +++ b/cortex/webgl/aligner.py @@ -21,6 +21,7 @@ import mimetypes import os import queue +import re import time import uuid import warnings @@ -45,6 +46,39 @@ #: The two view modes of the page, as its `view` control names them MODES = dict(outline="mesh + slices", projected="data on surface") +#: A transform name has to serve as a directory name in the filestore +XFM_NAME = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]*$") + + +def check_xfm_name(name: str) -> str: + """Return `name` if it can be used as the name of a transform. + + The name becomes a directory in the filestore, so anything else is + refused here rather than reaching the filesystem. + + Parameters + ---------- + name : str + The name to check. + + Returns + ------- + name : str + The name, unchanged. + + Raises + ------ + ValueError + If the name is empty or holds anything but letters, digits, '.', + '_' and '-', or does not start with a letter or a digit. + """ + if not isinstance(name, str) or XFM_NAME.match(name) is None: + raise ValueError( + "%r is not a usable transform name: use letters, digits, '.', '_' and " + "'-', starting with a letter or a digit" % (name,) + ) + return name + def reference_frame(nii) -> npt.NDArray[np.float64]: """The voxel-to-world matrix the aligner works in for a reference image. @@ -480,20 +514,25 @@ def post(self): xfm = np.asarray(json.loads(self.get_argument("xfm")), dtype=float) if xfm.shape != (4, 4): raise ValueError("expected a 4x4 matrix, got shape %s" % (xfm.shape,)) - # The masks were cut with the alignment being replaced, so - # they are wrong from here on; db.save_xfm also refuses to - # write over a transform that still has them. - dropped = clear_masks(subject, xfmname) - db.save_xfm(subject, xfmname, xfm, xfmtype="coord", reference=reference) + # The page can save the alignment under another name, which + # creates a transform of that name rather than changing the + # one that was loaded. + name = check_xfm_name(self.get_argument("name", xfmname).strip()) + # The masks of the transform being written were cut with the + # alignment being replaced, so they are wrong from here on; + # db.save_xfm also refuses to write over a transform that + # still has them. + dropped = clear_masks(subject, name) + db.save_xfm(subject, name, xfm, xfmtype="coord", reference=reference) except Exception as exc: self.write(json.dumps(dict(status="error", message="not saved: %s" % exc))) return - message = "saved transform %s for %s" % (xfmname, subject) + message = "saved transform %s for %s" % (name, subject) if len(dropped) > 0: message += "; deleted %d stale mask%s (%s)" % ( len(dropped), "" if len(dropped) == 1 else "s", ", ".join(dropped)) print(message) - self.write(json.dumps(dict(status="ok", message=message, masks_deleted=dropped))) + self.write(json.dumps(dict(status="ok", message=message, name=name, masks_deleted=dropped))) class WebApp(serve.WebApp): disconnect_on_close = close_on_disconnect diff --git a/cortex/webgl/resources/css/aligner.css b/cortex/webgl/resources/css/aligner.css index 41d502b00..09e425162 100644 --- a/cortex/webgl/resources/css/aligner.css +++ b/cortex/webgl/resources/css/aligner.css @@ -103,3 +103,68 @@ white-space:nowrap; color:#fc9; } + +/* Colormap dropdown: a strip of the colormap beside its name. + mriview.css hides the select2 control and leaves only its dropdown + visible, which is how the viewer opens its picker from the colorbar. The + aligner uses the control itself, so these put select2's own layout back. */ +.select2-container { + visibility:visible; + position:relative; +} + +.aligner-cmap { + display:flex; + align-items:center; +} +.aligner-cmap img { + flex:none; + width:64px; + height:11px; + margin-right:8px; + border:1px solid #555; +} +.aligner-cmap-name { + overflow:hidden; + text-overflow:ellipsis; + white-space:nowrap; +} +/* the closed control sits in a dat.GUI cell, so its strip is smaller */ +.select2-selection__rendered .aligner-cmap img { + width:40px; + height:9px; + margin-right:6px; +} +.select2-container--default .select2-selection--single { + background:#303030; + border:1px solid #555; + border-radius:0; + height:20px; +} +.select2-container--default .select2-selection--single .select2-selection__rendered { + color:#eee; + line-height:18px; + padding-left:4px; +} +.select2-container--default .select2-selection--single .select2-selection__arrow { + height:18px; +} +.select2-dropdown { + background:#1a1a1a; + border-color:#555; + color:#ccc; +} +.select2-container--default .select2-results__option { + padding:2px 6px; +} +.select2-container--default .select2-results__option[aria-selected=true] { + background:#333; +} +.select2-container--default .select2-results__option--highlighted[aria-selected] { + background:#0a6; +} +.select2-search--dropdown .select2-search__field { + background:#303030; + border:1px solid #555; + color:#eee; +} diff --git a/cortex/webgl/resources/js/aligner.js b/cortex/webgl/resources/js/aligner.js index beb5f3a40..c14f6c79c 100644 --- a/cortex/webgl/resources/js/aligner.js +++ b/cortex/webgl/resources/js/aligner.js @@ -171,6 +171,13 @@ var aligner = (function(module) { this.planeCoord = [0, 0, 0]; this._mode = MODES.outline; + //The transform the page saves to. It starts as the one being edited + //and can be changed, which saves the alignment as a new transform. + this._xfmName = config.xfmname; + this._savedXfm = this.xfm.clone(); + this._savedName = this._xfmName; + this._dirtyShown = false; + this._title = document.title; this._cmapName = config.cmap; this._showSurf = [true, true]; this._translateStep = 1; @@ -337,10 +344,10 @@ var aligner = (function(module) { var masks = this.config.masks || []; if (masks.length > 0 && !this.config.view_only) { $(panel).find("#aligner-masks").text( - "Saving deletes the " + masks.length + " cached mask" + - (masks.length == 1 ? "" : "s") + " of this transform (" + - masks.join(", ") + "). Data already masked with them has to be " + - "masked again from the volumes.").show(); + "Saving over " + this.config.xfmname + " deletes its " + masks.length + + " cached mask" + (masks.length == 1 ? "" : "s") + " (" + masks.join(", ") + + "). Data already masked with them has to be masked again from the " + + "volumes. Saving under another name leaves them alone.").show(); } this.statusElement = $(panel).find("#aligner-status"); figure.gui.open(); @@ -539,12 +546,17 @@ var aligner = (function(module) { this._cmapName = names.indexOf("gray") >= 0 ? "gray" : names[0]; this.volUniforms.colormap.value = this.colormaps[this._cmapName]; + //the name field sits above the save button, so that the alignment can + //be saved as a new transform var top = {}; - if (!this.config.view_only) + if (!this.config.view_only) { + top.transform = {action: [this, "setXfmName"]}; top.save = {action: this.save.bind(this)}; + } top.undo = {action: this.undo.bind(this)}; top.view = {action: [this, "setMode", [MODES.outline, MODES.projected]]}; this.ui.add(top); + this._updateDirty(); var vol = this.config.volume; this.ui.addFolder("image", false).add({ @@ -574,9 +586,45 @@ var aligner = (function(module) { "translate (mm)": {action: [this, "setTranslateStep", 0.05, 10, 0.05]}, "rotate (deg)": {action: [this, "setRotateStep", 0.05, 10, 0.05]}, }); + this._previewColormaps(); this.schedule(); }; + //Draws a strip of each colormap beside its name in the colormap dropdown. + //dat.GUI renders a plain select, so select2 takes it over to draw the + //options, the same library the viewer's colormap picker uses. Picking one + //writes the value into the select and fires a jQuery event; dat.GUI + //listens for the native one, so the pick is passed on as such and the + //control goes on working the way the others do. + module.Aligner.prototype._previewColormaps = function() { + var folder = this.ui["image"]; + var control = folder === undefined ? undefined : folder._controls.colormap; + if (control === undefined || control.__select === undefined || $.fn.select2 === undefined) + return; + + var select = control.__select; + var colormaps = this.colormaps; + var draw = function(state) { + if (!state.id) + return state.text; + var texture = colormaps[state.id]; + var row = $(""); + if (texture !== undefined && texture.image !== undefined) + row.append($("").attr("src", texture.image.src)); + row.append($("").text(state.text)); + return row; + }; + + this._cmapSelect = $(select).select2({ + templateResult: draw, + templateSelection: draw, + width: "100%", + }); + this._cmapSelect.on("select2:select", function() { + select.dispatchEvent(new Event("change")); + }); + }; + //Updates the value shown by a menu control without running its action module.Aligner.prototype._syncControl = function(folder, name, value) { var menu = this.ui[folder]; @@ -721,10 +769,47 @@ var aligner = (function(module) { module.Aligner.prototype._xfmChanged = function() { this.brain.matrix.copy(this.xfm); this.brain.matrixWorldNeedsUpdate = true; + this._updateDirty(); this.dispatchEvent({type: "xfm"}); this.schedule(); }; + //The name of the transform the page saves to. Changing it saves the + //alignment as a new transform, leaving the one it was loaded from alone. + module.Aligner.prototype.setXfmName = function(name) { + if (name === undefined) + return this._xfmName; + this._xfmName = String(name).trim(); + this._updateDirty(); + }; + + //Whether the alignment on screen is the one that was last saved. An undo + //back to that alignment counts as saved again, which is why this compares + //the matrices rather than merely noting that something was moved. + module.Aligner.prototype.isDirty = function() { + if (this._xfmName !== this._savedName) + return true; + var now = this.xfm.elements, saved = this._savedXfm.elements; + for (var i = 0; i < 16; i++) { + if (now[i] !== saved[i]) + return true; + } + return false; + }; + + //Marks unsaved changes with an asterisk, on the save button and in the + //title, so that a page left open does not look like a saved alignment. + module.Aligner.prototype._updateDirty = function() { + var dirty = this.isDirty(); + if (dirty === this._dirtyShown) + return; + this._dirtyShown = dirty; + document.title = (dirty ? "* " : "") + this._title; + var button = this.ui._controls.save; + if (button !== undefined) + button.name(dirty ? "save *" : "save"); + }; + //Translates the surfaces by a world vector (mm) module.Aligner.prototype.translate = function(vector) { this.pushUndo(); @@ -757,14 +842,23 @@ var aligner = (function(module) { this.showStatus("view only: the transform is not saved", true); return "view only"; } - this.showStatus("saving..."); + //what is being saved, rather than what is on screen when the answer + //comes back: the alignment can be moved on while the request is out + var name = this._xfmName; + var saved = this.xfm.clone(); + this.showStatus("saving " + name + "..."); $.ajax({ type: "POST", url: "save", - data: {xfm: JSON.stringify(this.getXfm())}, + data: {xfm: JSON.stringify(this.getXfm()), name: name}, dataType: "json", }).done(function(resp) { this.showStatus(resp.message, resp.status != "ok"); + if (resp.status == "ok") { + this._savedXfm = saved; + this._savedName = name; + this._updateDirty(); + } }.bind(this)).fail(function() { this.showStatus("saving failed: no answer from the server", true); }.bind(this)); @@ -830,6 +924,10 @@ var aligner = (function(module) { return; this._cmapName = name; this.volUniforms.colormap.value = this.colormaps[name]; + //keep the dropdown showing the colormap that is in effect when the + //change came from somewhere else, a menu set from python for instance + if (this._cmapSelect !== undefined && this._cmapSelect.val() != name) + this._cmapSelect.val(name).trigger("change.select2"); this.schedule(); }; module.Aligner.prototype.setFlip = function(flip) { @@ -1222,17 +1320,19 @@ var aligner = (function(module) { var rstep = this._rotateStep * fine * Math.PI / 180; var toViewer = view.look.clone().negate(); var handled = true; + //WASD moves the mesh the same way the arrows do. Both upper and lower + //case, since shift is the fine-step modifier and shift+w arrives as W. switch (key) { - case "ArrowLeft": + case "ArrowLeft": case "a": case "A": this.translate(view.right.clone().multiplyScalar(-tstep).toArray()); break; - case "ArrowRight": + case "ArrowRight": case "d": case "D": this.translate(view.right.clone().multiplyScalar(tstep).toArray()); break; - case "ArrowUp": + case "ArrowUp": case "w": case "W": this.translate(view.up.clone().multiplyScalar(tstep).toArray()); break; - case "ArrowDown": + case "ArrowDown": case "s": case "S": this.translate(view.up.clone().multiplyScalar(-tstep).toArray()); break; case "q": case "Q": diff --git a/cortex/webgl/resources/js/menu.js b/cortex/webgl/resources/js/menu.js index 137a704f8..549381c47 100644 --- a/cortex/webgl/resources/js/menu.js +++ b/cortex/webgl/resources/js/menu.js @@ -109,11 +109,13 @@ var jsplot = (function (module) { return folder; } module.Menu.prototype._add = function(gui, name, desc) { + var ctrl; if (desc.action instanceof Function) { //A button that runs a function (IE Reset) this[name] = desc.action; - if (!desc.hidden) - gui.add(desc, "action").name(name); + if (!desc.hidden) + //keep the controller, so that the button can be renamed later + ctrl = gui.add(desc, "action").name(name); } else if ( desc.action instanceof Array) { var obj = desc.action[0][desc.action[1]]; if (obj instanceof Function) { @@ -128,7 +130,7 @@ var jsplot = (function (module) { newargs.push(desc.action[i]); //a color picker, for a method that gets and sets a css color string - var ctrl = desc.color ? gui.addColor(this, name) : gui.add.apply(gui, newargs); + ctrl = desc.color ? gui.addColor(this, name) : gui.add.apply(gui, newargs); ctrl.onChange(function(name) { parent[method](this[name]); this.dispatchEvent({type:"update"}); @@ -147,7 +149,7 @@ var jsplot = (function (module) { }.bind(this, name) }; } else if (!desc.hidden) { - var ctrl = gui.add.apply(gui, desc.action).name(name); + ctrl = gui.add.apply(gui, desc.action).name(name); ctrl.onChange(function() { this.dispatchEvent({type:"update"}); }.bind(this)); diff --git a/cortex/webgl/template.html b/cortex/webgl/template.html index bfabfb158..e36a20b59 100644 --- a/cortex/webgl/template.html +++ b/cortex/webgl/template.html @@ -131,6 +131,8 @@ +{% block css %} +{% end %}