fix: make the sphinx-gallery scraper capture every shown figure by larsoner · Pull Request #5701 · plotly/plotly.py

larsoner · 2026-08-11T15:11:26Z

Link to issue

Closes #4722
Closes #4959
Closes sphinx-gallery/sphinx-gallery#1238
Closes sphinx-gallery/sphinx-gallery#1632

Description of change

Fix sphinx-gallery thumbnail generation!

Demo

Testing strategy

Added some unit tests to make sure it works. It uses SG (added to dev_optional so CIs install it) to make sure the contract holds.

Additional information (optional)

I am a SG maintainer. I used Claude Opus 5 to draft the changes here but I iterated with it a lot and have reviewed the final diff.

Guidelines

I have reviewed the pull request guidelines and the Code of Conduct and confirm that this PR follows them.
I have added an entry to the changelog if needed (not required for documentation PRs).

sr-murthy · 2026-08-12T09:06:08Z

Tried it (replaced plotly dep in project TOML with a ref to your fork branch of Plotly), cleared out the docs build folder, and ran Sphinx build again (make -C docs html), but still get the same placeholder thumbs (in the docs auto_examples/images/thumb folder - the full size figures are correctly drawn in the HTMLs.

I can see that the Plotly version in my env. is your branch because the UV log has the following line:

...
DEBUG Requirement already installed: plotly==6.9.0 (from git+https://github.com/larsoner/plotly.py@5ef36f39d12315835d211c39ae5ff2357c06a70b)
...

Conf changes for Plotly PNG rendering are as described here. Folder structure for the examples folder containing the Python files, is also as recommended, with auto_examples adjacent to examples. Here is the gallery conf. in docs/conf.py:

# Sphinx gallery conf.
sphinx_gallery_conf = {
    "examples_dirs": "sources/plot-gallery/examples",  # `docs`-relative path to Python scripts
    "gallery_dirs": "sources/plot-gallery/auto_examples",  # `docs`-relative path to gallery outputs
    "image_scrapers": ("matplotlib", "plotly.io._sg_scraper.plotly_sg_scraper",),
}

Wouldn't it be simpler for Sphinx gallery to support a post-build hook (a custom function maybe) that created the thumbnails by resizing and exporting the full sized figures into PNGs in the appropriate location?

…ly",)

Probe Kaleido/browser availability once per build; when unavailable, emit a single sphinx warning (suppressible via suppress_warnings = ["plotly.sg_scraper"]), embed shown figures inline in the rst instead of via files (sphinx-gallery requires an image file for every image path consumed), and let examples fall back to placeholder thumbnails. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- The _repr_html_ fallback (hit when the default renderer is not a mimetype renderer, e.g. sphinx_gallery_png) now sizes like the html renderers (default_height=525) instead of height:100%, which collapses or overflows in containers with no set height. - The scraper now embeds shown figures inline in the rst, wrapped in the same output_subarea div sphinx-gallery wraps captured HTML reprs in, so themes can style both kinds of embed with one hook; it no longer writes .html files next to the images. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

A figure whose inline script draws while the page is still being parsed can be sized to a container whose width changes by the time loading finishes (e.g. pydata-sphinx-theme's secondary sidebar comes after the article in the DOM), leaving it clipped until a window resize. Append a per-figure script to the _repr_html_ fallback and the sphinx-gallery scraper embeds that calls Plotly.Plots.resize once on window load. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Move the dark-theme styling out of sphinx-gallery: sphinx-gallery should not carry plotly-specific CSS, so ship a scoped style with each embed (repr fallback and scraper) instead. The white padded card only shows on dark pages (data-theme toggles or OS preference); on light pages it is invisible. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

larsoner · 2026-08-12T16:41:18Z

Wouldn't it be simpler for Sphinx gallery to support a post-build hook (a custom function maybe) that created the thumbnails by resizing the full sized PNGs into the appropriate location?

I don't think we should rework / discuss the SG API contracts here -- it'll be a much bigger discussion if needed, and require an understanding of design decisions and code evolution over the years where we settled on the existing design.

Instead I'd like to see if we can get things working properly for you using this branch. Can you make clean and try again with the latest version of the PR? I pushed some fixe -- the previous code required some SG changes and fig.show() to be used etc. but this version should work, even with the simpler:

    "image_scrapers": ("matplotlib", "plotly"),

I confirmed this works in sphinx-gallery/sphinx-gallery#1635 (see the updated example render) so hopefully it works for you now!

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Keep only the minimal sizing fix in basedatatypes (_repr_html_ fallback sizes like the html renderers). The dark-page card and the after-load resize now come from a single idempotent fix-up block the scraper appends to any code block that displayed a figure: repr-captured embeds and the scraper's own embeds both sit in an output_subarea div, so one :has() rule styles both, and one guarded listener resizes every figure. No more per-figure uuid/script plumbing outside scraper code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

sr-murthy · 2026-08-12T16:59:01Z

Thx, I'll try it out, and let you know.

sr-murthy · 2026-08-13T08:32:53Z

@larsoner The changes seem to work (Plotly pointing to rev. e3b95636) for my plot gallery, so thank you. But please note that thumbnail generation will only work if the build environment has both kaleido and a Chromium-based browser installed, and this needs to be made clear in the documentation (more details in point 4 below).

I've done several local and remote (RTD) builds to check consistency and reproducibility. There are a few points you may wish to note:

The build does seem to have been slowed down significantly - I think the initial build (after clearing the build folder) took almost a minute, but subsequent builds now are averaging about 30 seconds. Usually these builds, at least for my project, take less than 10 seconds.
My guess is that the build times would go up in line with the number of plots in the gallery [?]
I see that Chrome is used - I see two separate instantiations of it - to enable the kaleido exports for generating thumbnails. This would obviously fail if either the user or build environment did not have Chrome (or a Chromium-based browser) or did not have kaleido installed: in this case you get the following warning:

WARNING: plotly static image export is unavailable, so example thumbnails will fall back to a placeholder image. Static export requires Kaleido and a Chromium-based browser; see https://plotly.com/python/static-image-export/ for installation instructions. The failure was: ValueError: 
Image export using the "kaleido" engine requires the Kaleido package,
which can be installed using pip:

and the result is placeholder thumbnails, but the correct full sized figures are still drawn in the HTML pages. It's probably a good idea to note this somewhere in the Sphinx gallery docs. RTD builds had placeholder thumbnails because although kaleido is part of dependencies and is installed, the build environment initially did not have a Chromium-based browser installed - this required a pre-install step to be added to the RTD YML:

...
  jobs:
    pre_install:
      - uv run plotly_get_chrome -y

Once this is done, RTD builds are fine.

P. S. I had an earlier version of the gallery working, that used statically generated images, and with some scripting it would have come very close to what I have now. But Sphinx gallery is a good solution, so it's fine.

sr-murthy · 2026-08-13T08:47:26Z

BTW in the gallery conf. it is enough to have:

    {
        ...
        "image_scrapers": ("plotly",),
    }

You don't need matplotlib if you're only using Plotly.

larsoner · 2026-08-13T12:50:27Z

I see that Chrome is used

Yeah we need some way to get the thumbnail image from the rendered HTML page. Likely also the slowdown cause but I'll check.

It's probably a good idea to note this somewhere in the Sphinx gallery docs

Yes you're right!

sr-murthy · 2026-08-13T13:09:05Z

I've removed the point about opaque thumbnails on-hover - I'm guessing this could be fixed with the theme CSS [?], but anyway it's not an essential.

The vnd.plotly mimetype renderers produce no text/html bundle, so _repr_html_ takes the fallback branch, which now sizes like the html renderers (525px default). Also point the no-static-export warning at the plotly_get_chrome command, which is the fix when kaleido is installed but no browser is available (e.g. on ReadTheDocs). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Each static image export launches and tears down Chrome via kaleido (~1.6 s per figure; rendering itself is ~50 ms). Start kaleido's global sync server once from the availability probe so every export in the build reuses one browser; kaleido stops it atexit. Carries the same kopts (plotlyjs/mathjax/headers defaults) the per-call path would pass, and scopes away the resulting 'kopts ignored' warning. Brings the sphinx-gallery demo build from ~7 s to ~1 s of plotly example time. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

larsoner · 2026-08-13T14:17:39Z

Okay I pushed a change to reuse a single kaleido server instead of spinning one up for every figure. Can you see if that's faster?

sr-murthy · 2026-08-13T14:41:43Z

That appears to be quicker (no duplicate Chrome instances) - about 16 seconds total. That's fine.

camdecoster · 2026-08-13T19:42:03Z

Thanks for the PR! Before we review, could you please provide some testing steps to show that these changes fix the underlying issues?

sr-murthy · 2026-08-13T20:01:53Z

Thanks for the PR! Before we review, could you please provide some testing steps to show that these changes fix the underlying issues?

Is that me? My testing procedure was:

Re-install Plotly in the local env. pointing to this PR branch (rev. 6894c4b).
Clear out the Sphinx build folder(docs/_build), and build again (make -C docs html).
Check the plot gallery page (thumbnails, and figures in the figure pages).

I did about 10 builds locally, and several on RTD also. All fine.

larsoner · 2026-08-13T20:19:11Z

Quickest way to get something you can look at would maybe be to run sphinx gallery's own doc build on the branch from this PR : sphinx-gallery/sphinx-gallery#1635 . But that's also exactly what CircleCI does over there so you can view its result here. That plus @sr-murthy testing is maybe enough already?

@sr-murthy do you have a branch you could open in whatever repo you've been working on that could show it working as well?

If it helps for review, I could alternatively put together a tiny reproduction sphinx setup and upload that somewhere.

sr-murthy · 2026-08-13T20:30:53Z

@larsoner Yes, in fact it's this one.

https://github.com/ISARICResearch/IsaricAnalytics/tree/plot-gallery

RTD build on the branch:

https://isaricanalytics--18.org.readthedocs.build/en/18/sources/visualisation/index.html

The problem prior to the fix was that the thumbnails were placeholder images as displayed here. The figures inside the figure pages were never an issue, so they render properly as before.

On CircleCI Linux, kaleido's shared sync-server browser renders once and then its process exits (choreographer's watchdog then closes the channels and the next export fails the whole build) — with both the machine image's browser and a fresh Chrome for Testing, while per-export browsers work fine there. On an export failure with the server running, stop it, log at info level, and retry the export with kaleido's regular one-browser-per-call path for the rest of the build. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

camdecoster · 2026-08-25T00:41:15Z

FYI, we're going to be releasing v7 soon and that will update some of the files that you're editing. After that release, could you please merge changes and update your branch to address the conflicts?

larsoner · 2026-08-25T02:41:49Z

Sure, just ping me when it's time!

camdecoster · 2026-08-26T17:06:28Z

v7 has been released.

larsoner · 2026-08-27T06:40:18Z

Okay merged main into the branch, and the sphinx-gallery companion PR scrapes correctly I think (including thumbnails).

camdecoster · 2026-08-27T23:48:00Z

Could you please give me some testing steps? Ideally, something I can use on main and see it failing, then on this branch and see it working.

larsoner · 2026-08-28T15:31:19Z

You can run a variant of https://gist.github.com/larsoner/e0decb518fd029daa5e8c3e8c3c3d084 (WARNING: creates a venv and pip installs stuff there, adjust to taste) to see:

To see things not working, you can also look at the current sphinx-gallery plotly gallery (placeholder thumbnail), then what CircleCI produced from sphinx-gallery/sphinx-gallery#1635 , which uses this branch.

camdecoster · 2026-08-31T19:37:14Z

Thanks for that. I'll take a look and follow up.

sr-murthy · 2026-09-10T17:57:36Z

Any idea of when this may be resolved? We're currently waiting on a new Plotly release with this fix.

camdecoster · 2026-09-10T17:59:42Z

Hello again! There have been other libraries issues taking my time, so I haven't gotten around to it yet. But it is on my list. Sorry for the delay!

camdecoster

This seems like a good update, but I'd like you to make some changes. Additionally, could you please address the merge conflict in the changelog?

camdecoster · 2026-09-11T02:22:24Z

+
+    Instead of displaying the figure, this renderer queues it in
+    ``plotly.io._base_renderers.sphinx_gallery_figures``;
+    :func:`plotly.io._sg_scraper.plotly_sg_scraper` then writes each queued


We shouldn't be advertising a function in a private file. Could you add a public name for reset_renderer? Also, could you remove the :func: tags and use double backticks instead? That's what we use elsewhere.

camdecoster · 2026-09-11T02:24:36Z

+)
+

 def plotly_sg_scraper(block, block_vars, gallery_conf, **kwargs):


It appears that gallery_conf is no longer used. Is this intentional?

Yeah it's part of the sphinx-gallery contract that this block, block_vars, gallery_conf are always passed to the scraper (the scraper can choose to use them or not)

camdecoster · 2026-09-11T02:26:29Z

+    figure to the gallery's image directory, both as an interactive HTML file
+    and as a static image used for the gallery thumbnail.


I don't think this description is correct with the changes you made. Could you update it?

camdecoster · 2026-09-11T02:30:46Z

+
+        log = getLogger(__name__).info
+    except Exception:
+        log = logging.getLogger(__name__).info


Let's use the warning log level for this exception.

Suggested change

log = logging.getLogger(__name__).info

log = logging.getLogger(__name__).warning

This one I'm less convinced about... sphinx-gallery runs one example and hit this warning, which then failed the build because of warnings-as-error (which I'd recommend nowadays everyone build with) despite the example actually rendering just fine:

https://app.circleci.com/jobs/github/mne-tools/mne-python/87980

https://output.circle-artifacts.com/output/job/c71ff42d-093f-4c49-9cf0-3c2fe2073636/artifacts/0/html/auto_plotly_examples/index.html

So to me since it's fully recovered from info seems like the better choice here. But I'll push a commit with a different warning type at least in case you disagree...

camdecoster · 2026-09-11T14:51:59Z

    """Scrape Plotly figures for galleries of examples using
    sphinx-gallery.

-    Examples should use ``plotly.io.show()`` to display the figure with
-    the custom sphinx_gallery renderer.
+    Examples should use ``plotly.io.show()`` (or the equivalent
+    ``fig.show()``) to display the figure with the custom
+    ``sphinx_gallery_png`` renderer, which is made the default renderer as a
+    side effect of importing this module. Parallel builds on sphinx-gallery
+    0.21 and earlier also need :func:`reset_renderer`, since their worker
+    processes never import it.
+
+    Every figure shown that way is embedded in the page as interactive HTML,
+    and written to the gallery image directory as a static image, which
+    sphinx-gallery uses to generate the thumbnail of the example.
+
+    A figure that is instead displayed by making it the last expression of a
+    code block (sphinx-gallery's repr capture) gets a static image too, so
+    that it can also serve as the thumbnail; its HTML is embedded by
+    sphinx-gallery itself.

-    Since the sphinx_gallery renderer generates both html and static png
-    files, we simply crawl these files and give them the appropriate path.
+    Static image export requires Kaleido and a Chromium-based browser (the
+    ``plotly_get_chrome`` command installs one); when unavailable, a warning
+    is emitted once per build and the examples fall back to placeholder
+    thumbnails, with the interactive figures unaffected.


This seems pretty verbose for a docstring. Could you trim it down a bit?

camdecoster · 2026-09-11T14:53:36Z

+    """Select the ``sphinx_gallery_png`` renderer, as a sphinx-gallery resetter.
+
+    Importing this module selects that renderer too, but the worker processes
+    of a parallel build never import it: ``conf.py`` is read only in the
+    parent process, and ``image_scrapers=("plotly",)`` reaches a worker as a
+    plain string that sphinx-gallery resolves only after the example has run.
+    The example therefore executes with the default ``browser`` renderer,
+    whose ``fig.show()`` serves the figure from a local web server and waits
+    for a browser that a worker does not have, hanging the build.
+
+    Naming this function in ``reset_modules`` runs it in every worker, before
+    each example::
+
+        sphinx_gallery_conf = {
+            ...
+            "image_scrapers": ("plotly",),
+            "reset_modules": ("matplotlib", "plotly.io._sg_scraper.reset_renderer"),
+        }
+
+    Passing it by name rather than importing it keeps ``sphinx_gallery_conf``
+    picklable, which Sphinx needs in order to cache the environment.
+
+    Up to and including sphinx-gallery 0.21, ``image_scrapers`` is resolved
+    only after an example has run, which is what makes this necessary. From
+    0.22 it is resolved beforehand and selects the renderer itself, so this
+    resetter is redundant there -- but harmless, and still worth configuring
+    if the documentation is also built against older sphinx-gallery.

    Parameters
    ----------
-    figure_list : list
-        List of strings of the figures' absolute paths.
-    sources_dir : str
-        absolute path of Sphinx documentation sources
+    gallery_conf : dict
+        Contains the configuration of Sphinx-Gallery (unused).
+    fname : str
+        Name of the example file about to be executed (unused).
+    when : str or None
+        Whether sphinx-gallery is resetting before or after the example
+        (unused). Sphinx-gallery passes it by keyword, and only to resetters
+        that accept it.
+    """


This is way too much for a single line of code. This is the kind of content that would be better located on a documentation page. Could you please make the docstring short and to the point?

camdecoster · 2026-09-11T16:55:21Z

I'm reluctant to change the defaults for all renderers like this. Could you figure out another way to address what this change was intended to fix? What about changing the renderer to "sphinx_gallery_png+sphinx_gallery"?

Rather than change the name, I reverted the renderer defaults change (seemed cleaner)

* commit '2e2d1f0c8': (38 commits) Switch to alternate fence syntax docs: keep troubleshooting paragraph on one line docs: move Jupyter troubleshooting section docs: add Jupyter blank-render troubleshooting skip changelog check if draft, and display messages for all outcomes Apply suggestions from code review update changelog instructions in CONTRIBUTING.md Revert "add dummy changelog entry" add dummy changelog entry add CI workflow to make sure changelog gets updated with every PR update changelog update years in license and README update instructions for updateplotlyjs command update plotly.js to v4.1.0 update changelog do not use full_html for google colab renderer Add changelog entry Add yticks test Export tick text for FixedFormatter tick labels Update uneven tick spacing warning to reflect explicit tickvals export ...

larsoner · 2026-09-11T20:12:20Z

Okay I think I addressed the comments!

fix: make the sphinx-gallery scraper capture every shown figure

5ef36f3

larsoner mentioned this pull request Aug 12, 2026

Fix Plotly-specific thumbnail generation + working examples sphinx-gallery/sphinx-gallery#1632

Open

feat: thumbnails for repr-displayed figures and image_scrapers=("plot…

a933f1b

…ly",)

larsoner force-pushed the sgthumb branch from 9392929 to a933f1b Compare August 12, 2026 14:24

larsoner and others added 2 commits August 12, 2026 09:40

refactor: cache the static-export probe instead of a module global

0795b26

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

larsoner mentioned this pull request Aug 12, 2026

Document modern plotly scraper usage sphinx-gallery/sphinx-gallery#1635

Draft

2 tasks

larsoner and others added 3 commits August 12, 2026 10:36

larsoner and others added 2 commits August 12, 2026 11:45

test: compact the sphinx-gallery scraper tests

9b6cceb

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

sr-murthy mentioned this pull request Aug 13, 2026

Plot gallery ISARICResearch/IsaricAnalytics#18

Open

larsoner and others added 2 commits August 13, 2026 08:24

robertclaus requested a review from camdecoster August 19, 2026 19:26

robertclaus assigned camdecoster Aug 19, 2026

FIX: More

2d2adff

camdecoster requested changes Sep 11, 2026

View reviewed changes

larsoner added 2 commits September 11, 2026 16:02

Address review

ab67a5e

FIX: Subtype

618afc9

		)


		def plotly_sg_scraper(block, block_vars, gallery_conf, **kwargs):

		figure to the gallery's image directory, both as an interactive HTML file
		and as a static image used for the gallery thumbnail.

	log = logging.getLogger(__name__).info
	log = logging.getLogger(__name__).warning

Uh oh!

Uh oh!

Conversation

larsoner commented Aug 11, 2026

Link to issue

Description of change

Demo

Testing strategy

Additional information (optional)

Guidelines

Uh oh!

sr-murthy commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

larsoner commented Aug 12, 2026

Uh oh!

sr-murthy commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sr-murthy commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sr-murthy commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

larsoner commented Aug 13, 2026

Uh oh!

sr-murthy commented Aug 13, 2026

Uh oh!

larsoner commented Aug 13, 2026

Uh oh!

sr-murthy commented Aug 13, 2026

Uh oh!

camdecoster commented Aug 13, 2026

Uh oh!

sr-murthy commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

larsoner commented Aug 13, 2026

Uh oh!

sr-murthy commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

camdecoster commented Aug 25, 2026

Uh oh!

larsoner commented Aug 25, 2026

Uh oh!

camdecoster commented Aug 26, 2026

Uh oh!

larsoner commented Aug 27, 2026

Uh oh!

camdecoster commented Aug 27, 2026

Uh oh!

larsoner commented Aug 28, 2026

Uh oh!

camdecoster commented Aug 31, 2026

Uh oh!

sr-murthy commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

camdecoster commented Sep 10, 2026

Uh oh!

camdecoster left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

sr-murthy commented Aug 12, 2026 •

edited

Loading

sr-murthy commented Aug 12, 2026 •

edited

Loading

sr-murthy commented Aug 13, 2026 •

edited

Loading

sr-murthy commented Aug 13, 2026 •

edited

Loading

sr-murthy commented Aug 13, 2026 •

edited

Loading

sr-murthy commented Aug 13, 2026 •

edited

Loading

sr-murthy commented Sep 10, 2026 •

edited

Loading