Add multi-file model output

[PaleNeutron]

Closes #88.

This adds a CLI option for writing generated models into a directory, with one Python file per generated model:

• --output-directory <dir> creates the directory if needed
• shared declarations are written to base_model.py
• an empty __init__.py is created
• each generated model is written to its own .py file

The generator API now also accepts generate(multi_file=True) and returns a mapping of module names to generated source strings. Calling generate() without arguments preserves the existing single-file behavior.

Verification:

• uv run pytest
• uv run ruff check .

[coveralls]

coverage: 97.802% (-0.04%) from 97.839% — PaleNeutron:feature/multi-file-output-pr into agronholm:master

[sheinbergon]

@PaleNeutron thank you for your contribution. While there has been a long discussion about this request in the original issue, I feel the a file per module maybe a bit of an exaggeration. I would say a file per schema makes more sense, but feel free to counter that idea. @agronholm wdyt?

[PaleNeutron]

@sheinbergon , we subscribe financial data into a database and the provider put about 2000 tables in one schema. Split models by schema can not solve this problem.

BTW, generally schema count is less than 10 and generate each one in a loop is acceptable, no need to create a new option.

[sheinbergon]

One smal change request

[sheinbergon]

@sheinbergon , we subscribe financial data into a database and the provider put about 2000 tables in one schema. Split models by schema can not solve this problem.

BTW, generally schema count is less than 10 and generate each one in a loop is acceptable, no need to create a new option.

Alright, Fine by me. Submitted one small change request

[sheinbergon]

More changes requested

[sheinbergon]

Passing this flag a function argument is wrong. It also creates the need to overload the generate method which serve as an entrypoint to the core functionality. Just include it as part of the constructor arg options and branch internally

[sheinbergon]

@agronholm aside from the additional comment - fine by me. WDYT?

[sheinbergon]

Better move this to live under the ExitStack

[sheinbergon]

One of the things about code-generation libs was their ability to leave you in a broken state.
Sticking to a single module helped this library avoid that. With the introduction of multi-files, you need to start dealing with stuff like cleaning up residue from past versions and partial success, whih seems to be missing from the current implementaion

[PaleNeutron]

You are right, but what's the best implementation? Check the whole output dir and warning all redundant files?

[PaleNeutron]

We can delete the output dir and create a new one but I think it is a little crude. If user pass / to the output-directory arg...

[sheinbergon]

If the user is rackless, this piece of software is not going to be the one to save him

[agronholm]

Multi-file generation is painfully hard to get right. Limiting it to file-per-schema doesn't really fix that, as foreign keys between schemas are a thing. What is your motivation here @PaleNeutron ?

[agronholm]

Multi-file generation is painfully hard to get right. Limiting it to file-per-schema doesn't really fix that, as foreign keys between schemas are a thing. What is your motivation here @PaleNeutron ?

Oh, I spotted your comment on the issue. Do you have thousands of tables in a single schema then?

[PaleNeutron]

Multi-file generation is painfully hard to get right. Limiting it to file-per-schema doesn't really fix that, as foreign keys between schemas are a thing. What is your motivation here @PaleNeutron ?

Oh, I spotted your comment on the issue. Do you have thousands of tables in a single schema then?

Yes, a subscribed fin-data database.

[agronholm]

I'm unfavorable to adding this to sqlacodegen due to the extra maintainer load it adds (not least due to the complexity of getting the table namings, relationships etc. right), to the benefit benefit of very few users (who always have an option of manually splitting the code).

[PaleNeutron]

I'm unfavorable to adding this to sqlacodegen due to the extra maintainer load it adds (not least due to the complexity of getting the table namings, relationships etc. right), to the benefit benefit of very few users (who always have an option of manually splitting the code).

I think this could still provide a noticeable benefit even for users with around 100 tables. Splitting the generated models into separate modules can improve startup performance quite a bit — in some cases saving around 1–5 seconds on every startup.

The fact that more users are not asking for this feature does not necessarily mean it would not be useful. A 1–5 second delay may be tolerable, but that does not mean the current approach is optimal.

I also do not think resolving relationships is especially complex in SQLAlchemy 2.0. We can use string-based references / forward annotations for relationships, for example:

# Source - https://stackoverflow.com/a/79601366
# Posted by user15630736, modified by community. See post 'Timeline' for change history
# Retrieved 2026-06-07, License - CC BY-SA 4.0

import typing

if typing.TYPE_CHECKING:
  from users.models import User

class Account(BaseModel):
  __tablename__ = "accounts"

  user_id: Mapped[int] = mapped_column(ForeignKey("users.id"), nullable=False)
  user: Mapped["User"] = relationship(back_populates="accounts")

[agronholm]

I'm unfavorable to adding this to sqlacodegen due to the extra maintainer load it adds (not least due to the complexity of getting the table namings, relationships etc. right), to the benefit benefit of very few users (who always have an option of manually splitting the code).

I think this could still provide a noticeable benefit even for users with around 100 tables. Splitting the generated models into separate modules can improve startup performance quite a bit — in some cases saving around 1–5 seconds on every startup.

The fact that more users are not asking for this feature does not necessarily mean it would not be useful. A 1–5 second delay may be tolerable, but that does not mean the current approach is optimal.

Sure, but I have to weigh the maintenance cost against the benefit, and I have no metric other than the number of users asking for this.

I also do not think resolving relationships is especially complex in SQLAlchemy 2.0. We can use string-based references / forward annotations for relationships, for example:

String-based references won't resolve unless the class in question has been imported. SQLAlchemy 2.0 did not change that fact. Lazy imports would handily fix this, but they were only recently introduced in Python 3.15 (still in beta).

[agronholm]

I also do not think resolving relationships is especially complex in SQLAlchemy 2.0. We can use string-based references / forward annotations for relationships, for example:

String-based references won't resolve unless the class in question has been imported. SQLAlchemy 2.0 did not change that fact. Lazy imports would handily fix this, but they were only recently introduced in Python 3.15 (still in beta).

Actually, the mapper configuration step would result in all the lazy imports being triggered, making it pointless.

[nerdstrike]

@agronholm parachuting into this PR/debate. Past experience with using DBIx::Class and its schema generator shows me that splitting models by file is helpful with complex schemata, but becomes more so if you checksum the resulting model and add a boundary beyond which customisations can be made. Then the generator (sqlacodegen or DBIx::Class::Schema::Loader) can safely update the model from a database while leaving supplemental code intact. The consumer of a schema can build up libraries of common code associated with each model without having to maintain multiple libraries/packages.

In my opinion this is the best way to do it, as it creates the smallest numbers of artifacts to maintain and keeps code close to the source - on the flip side it does increase the difficulty for... you.

"If you want custom code, you can just extend the models!" I hear you say. True, but super-not-fun if you've got hundreds of them and the original schema receives frequent changes.

I hope I've made you contemplate a change of heart, because we're getting plenty of value out of sqlacodegen even with the limitations.