Skip to content

Port LCM domain-model codegen to a Roslyn source generator - #404

Open
hahn-kev wants to merge 5 commits into
masterfrom
claude/lcm-source-generators-port-b8c8b2
Open

hahn-kev wants to merge 5 commits into
masterfrom
claude/lcm-source-generators-port-b8c8b2

Conversation

@hahn-kev

@hahn-kev hahn-kev commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Does what it says, this is the blessed way to generate source code at build time, and avoids any race conditions or parallel build issues.

Mostly vibe coded, though I have read all the code and more or less understand the new code.


AI summary

Ports LCM's domain-model code generation from the LcmGenerate MSBuild task to a modern Roslyn incremental source generator, reusing the existing NVelocity templates and model wrappers unchanged.

Before: an MSBuild Task shelled out to a child MSBuild process (GenerateModel.proj) to run NVelocity — a workaround for the task DLL being locked by Visual Studio — writing 9 Generated*.cs files into the source tree (gitignored, regenerated every build). This depended on locating an msbuild/dotnet executable, used Directory.SetCurrentDirectory and process-global state, and cluttered the tree.

After: a new SIL.LCModel.SourceGenerators project (IIncrementalGenerator) produces the domain model at compile time, directly into the compilation. No child process, no DLL-locking workaround, no on-disk generated files.

Approach — reuse, don't rewrite:

  • The 49 NVelocity templates and the 10 model-wrapper classes moved into the generator unchanged (git-tracked as renames).
  • Only the host was rewritten: LcmModelGenerator reads MasterLCModel.xml, HandGenerated.xml and IntPropTypeOverrides.xml as AdditionalFiles and emits via AddSource; LcmGenerateImpl accumulates outputs in memory instead of writing files; EmbeddedTemplateLoader (a custom NVelocity ResourceLoader) serves the templates from embedded resources, replacing the filesystem #parse + current-directory mechanism.
  • NVelocity 1.2.0 is a single self-contained, dependency-free DLL that runs inside the Roslyn analyzer host; it is flowed alongside the generator via GetDependencyTargetPaths.
  • The model wrappers reach their override lists through the object graph (Property → Class → CellarModule → Model, which holds the generator) rather than any process-global state, so generation is fully instance-local and lock-free.

Scope: only the domain-model generator (Generator A) is ported. The kernel IDL importer (IdlImp, which shells out to cl.exe/gcc to preprocess native COM IDL) is a poor fit for source generators and stays as-is in SIL.LCModel.Build.Tasks.

Compatibility: the generated code is compiled into SIL.LCModel.dll as before, so downstream NuGet consumers are unaffected and no public API changes. Downstream packaging/content files are left untouched.

Notable findings while porting:

  • NVelocity's ExtendedProperties treats commas as list separators, so the loader class name must be passed as Type; Assembly (semicolon), matching NVelocity's own escaped-comma convention.
  • The EmitCompilerGeneratedFiles on-disk emission differs from the old output only by a 3-byte UTF-8 BOM; the compiler-consumed content is byte-for-byte identical.

Test plan

  • Golden-file diff (primary correctness gate): all 9 generated files are byte-for-byte identical to the previous MSBuild-task output.
  • Runs in the real Roslyn analyzer host: SIL.LCModel builds with the generator active (NVelocity loads and executes in-process).
  • Full solution builds clean across all target frameworks (net462 / netstandard2.0 / net8.0).
  • SIL.LCModel.Tests: 1709 passed, 0 failed, 18 skipped.
  • Concurrency stress test: 32 simultaneous generations, repeated 5× (160 runs), every file hash-matching golden — confirms the lock-free design is safe.

This change is Reviewable

hahn-kev and others added 2 commits September 10, 2026 16:57
Replace the LcmGenerate MSBuild task (which ran the NVelocity templates in a
child MSBuild process to avoid VS DLL-locking, writing 9 Generated*.cs files
into the source tree) with a Roslyn incremental source generator that produces
the domain model at compile time.

The new SIL.LCModel.SourceGenerators project reuses the existing NVelocity
templates and model-wrapper classes unchanged; only the host is rewritten:
- LcmModelGenerator (IIncrementalGenerator) reads MasterLCModel.xml,
  HandGenerated.xml and IntPropTypeOverrides.xml as AdditionalFiles and emits
  via AddSource.
- LcmGenerateImpl accumulates outputs in memory instead of writing files.
- EmbeddedTemplateLoader serves the 49 templates from embedded resources,
  replacing the filesystem #parse + Directory.SetCurrentDirectory mechanism.

NVelocity is a single self-contained DLL that runs inside the Roslyn analyzer
host; it is flowed alongside the generator via GetDependencyTargetPaths.

SIL.LCModel.Build.Tasks keeps only the IdlImp task (kernel-interface codegen,
left as-is). Generated output is byte-for-byte identical to the previous task;
SIL.LCModel.Tests pass (1709 passed, 0 failed, 18 skipped).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Property looked up the current generator via NVelocity's process-global
RuntimeSingleton to read the override lists, which forced generation to be
serialized. Store the LcmGenerateImpl on the root Model instead and have
Property reach it through the existing parent chain
(Property -> Class -> CellarModule -> Model).

With that global gone (and the never-read "LcmGenerate.Engine" attribute and
the write-only static Generator field removed), a generation run is entirely
instance-local, so LcmModelRunner no longer needs a lock; the immutable
template set is cached once via Lazy. Verified with a concurrency stress test
(32 simultaneous generations, repeated) all producing byte-identical output,
and the output remains byte-for-byte identical to the previous commit.
SIL.LCModel.Tests pass (1709 passed, 0 failed, 18 skipped).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 10, 2026

Copy link
Copy Markdown

LCM Tests

    16 files  ±0      16 suites  ±0   1m 53s ⏱️ -3s
 2 880 tests ±0   2 860 ✅ ±0   20 💤 ±0  0 ❌ ±0 
11 468 runs  ±0  11 300 ✅ ±0  168 💤 ±0  0 ❌ ±0 

Results for commit 0b659b1. ± Comparison against base commit 28e3487.

♻️ This comment has been updated with latest results.

imnasnainaec

This comment was marked as resolved.

imnasnainaec

This comment was marked as outdated.

@jasonleenaylor

Copy link
Copy Markdown
Contributor

There is one consequence of this that we have to solve first. FieldWorks was using
LcmGenerate to render a C++ header -- Build/mkall.targets:208-213 runs it against
Src/Kernel/CellarConstants.vm.h, and GenerateCellarConstants is a DependsOnTargets
of both FwKernel and Views, so the native build stops without it (on the next
SilLcmVersion bump, not on merge). The good news is that a thin task over your new
engine covers it: I prototyped one and it produced the header byte-for-byte identical to
what we ship today, with no changes to any file in this PR. The only real costs are
putting NVelocity.dll back into the packed tools/ and either linking the engine
sources or making a few members public.

Roughly what it takes:

  • A task project targeting net462, linking
    src/SIL.LCModel.SourceGenerators/**/*.cs but excluding LcmModelGenerator.cs (the
    Roslyn driver pulls in Microsoft.CodeAnalysis.CSharp, which a task should not carry).

  • The task body, which is the whole of it:

    var templates = new Dictionary<string, string>(StringComparer.Ordinal) {
        [Path.GetFileName(TemplateFile)] = File.ReadAllText(TemplateFile)
    };
    var impl = new LcmGenerateImpl(doc, templates);
    impl.SetOutput(OutputFile);
    impl.Process(Path.GetFileName(TemplateFile));
    File.WriteAllText(Path.Combine(OutputDir, OutputFile), impl.Outputs[OutputFile]);
  • Pack the task assembly and NVelocity.dll to tools/net462, and restore the
    UsingTask line in SIL.LCModel.Build.Tasks.props.

  • Nothing changes on the FieldWorks side, except that WorkingDirectory can be dropped
    from the call site now that EmbeddedTemplateLoader has replaced the current-directory
    handling.

hahn-kev and others added 2 commits September 19, 2026 11:36
…nerate task

FieldWorks uses the LcmGenerate MSBuild task to render a C++ header
(CellarConstants.vm.h) from the LCM model, so removing it would break the
FieldWorks native build on the next SilLcmVersion bump (reported by Jason on
the PR). Restore the task, but without going back to the old design.

Move the NVelocity engine (LcmGenerateImpl, EmbeddedTemplateLoader, and the
model wrappers) out of the source generator into a new netstandard2.0 project,
SIL.LCModel.ModelGeneration, whose only dependency is NVelocity. Both the
source generator and Build.Tasks reference it by project reference:

- SIL.LCModel.SourceGenerators keeps the Roslyn driver, the runner and the
  embedded templates; it flows the engine + NVelocity into the analyzer load
  context as analyzer assets.
- SIL.LCModel.Build.Tasks gains a thin LcmGenerate task over the same engine
  (original property surface, so FieldWorks needs no changes) and repacks the
  engine + NVelocity into tools/, with the UsingTask restored.

The engine stays internal, shared via InternalsVisibleTo, rather than
source-linking the files into two assemblies. Domain-model output is byte-for-
byte identical to the original task (verified against a fresh master build);
the LcmGenerate task renders a template end-to-end through MSBuild; the full
solution builds and SIL.LCModel.Tests pass (1709 passed, 0 failed, 18 skipped).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SIL.LCModel.ModelGeneration is IsPackable=false (its assembly is bundled
directly into the analyzer and into SIL.LCModel.Build.Tasks tools/), but the
plain ProjectReference from Build.Tasks emitted a <dependency> on it in the
Build.Tasks nuspec. Consumers pull it transitively (SIL.LCModel -> Core ->
Build.Tasks -> ModelGeneration), so restore fails with NU1101: package
SIL.LCModel.ModelGeneration not found. A full FieldWorks build against a local
package feed surfaced this.

Mark the ModelGeneration project references PrivateAssets="all" in both
Build.Tasks and SourceGenerators so the engine is not advertised as a package
dependency; the DLL continues to flow via the explicit tools/ pack and the
analyzer's GetDependencyTargetPaths. SIL.LCModel still builds with the
generator (ModelGeneration.dll + NVelocity.dll reach the analyzer load context).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@hahn-kev

Copy link
Copy Markdown
Contributor Author

@jasonleenaylor after looking at the issue I decided to fix it by making a new project, both the build tasks and the source generator reference it. Claude tested the new package building against FLEx and it worked fine without any changes, so you shouldn't be affected once this is merged in.

@jasonleenaylor jasonleenaylor left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified with a FieldWorks build LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants