Lazy code colorization

[skatkov]

Please see the prior PR #1788

In that PR it was correctly established that eager generation of RDoc::Parser::RubyColorizer::ColoredToken objects creates a heavy performance/system burden for all generators, but not all generators need these objects.

My suggested approach in that PR was to establish a generator property that would stop this eager code colorization process. This approach is not functional, because rdoc itself can switch generators, but use the same @store object.

In this PR I want to suggest an alternative solution:

• Instead of doing this eagerly, do code colorization lazily (only if stream_token was actually called).
• This would also ensure, that no new properties or methods will be introduced. And this will help us keep backward compatibility, while improving performance.

Solution

This change stores a copied method-source slice in a deferred token stream and materializes the existing mutable token array only when token_stream is accessed. It also switches the initial file scan from Prism.parse_lex to Prism.parse, preserves indentation and heredoc boundaries, and keeps the existing public token-stream behavior for HTML and third-party generators.

Benchmark

I have done a benchmark on google-client-api gem (one of the biggest and popular gems in the ecosystem). This could be found here:
https://gist.github.com/skatkov/7e509742651586c1b993a8b15b10a8da

RI

After Parse

Variant	RSS	Peak RSS	Colored	Deferred	Raw strings	Raw source	Prism nodes	Prism tokens	Heap slots
released	958.8 MiB	965.6 MiB	3900930	0	0	0.0 MiB	0	0	9998568
current	478.2 MiB	478.2 MiB	0	46571	617	57.0 MiB	0	0	2433708

After Generate

Variant	RSS	Peak RSS	Colored	Deferred	Raw strings	Raw source	Prism nodes	Prism tokens	Heap slots
released	1356.7 MiB	1391.4 MiB	3900930	0	0	0.0 MiB	0	0	11901244
current	719.7 MiB	755.3 MiB	0	46571	617	57.0 MiB	0	0	4336429

Variant	Parse	Generate	Total	GNU peak RSS
released	24.31s	30.40s	58.18s	1391.4 MiB
current	14.62s	28.59s	45.43s	755.3 MiB

Parse RSS change: -50.1%
Generated RSS change: -47.0%
Peak RSS change: -45.7%
Parse time change: -39.8%
Generate time change: -5.9%
Total time change: -21.9%
Outputs identical: yes

Aliki, 100 files

./benchmark.rb --format aliki --limit 100

After Parse

Variant	RSS	Peak RSS	Colored	Deferred	Raw strings	Raw source	Prism nodes	Prism tokens	Heap slots
released	96.3 MiB	96.9 MiB	253392	0	0	0.0 MiB	0	0	708268
current	65.4 MiB	65.4 MiB	0	2701	2701	0.9 MiB	0	0	206955

After Generate

Variant	RSS	Peak RSS	Colored	Deferred	Raw strings	Raw source	Prism nodes	Prism tokens	Heap slots
released	219.1 MiB	220.4 MiB	253392	0	0	0.0 MiB	0	0	1037598
current	210.3 MiB	210.3 MiB	253392	0	0	0.0 MiB	0	0	1038957

Variant	Parse	Generate	Total	GNU peak RSS
released	1.40s	44.09s	46.10s	220.4 MiB
current	0.90s	44.56s	46.04s	209.5 MiB

Parse RSS changed by -32.1%, generated RSS by -4.0%, peak RSS by -4.9%,
parse time by -35.7%, generation time by +1.1%, and total time by -0.1%.
Outputs were identical.

Results

According to the benchmark, we can see that the lazy approach brings significant memory improvements even for generators that actually output highlighted code.

In cases of 'RI', generator that has no use for highlighted code, we can see significant memory improvements (~50%), but also ~20% speed improvements.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[skatkov]

@tompng this is available for review. What's your opinion on this approach?

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.