Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 37 additions & 5 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,12 @@ db2sql

Two output modes are supported:

* **Dump mode** (default) — write a SQL file (or stream to ``stdout``) that can later be replayed with ``psql -f`` or ``sqlcmd -i``.
* **Migrate mode** — open a live connection to the target database and apply the same DDL and data directly, without an intermediate file. The DDL produced is byte-identical to dump mode: a single ``SqlEmitter`` is the source of truth in both paths.
* ``db2sql dump`` (the default command) — write a SQL file (or stream to ``stdout``) that can later be replayed with ``psql -f`` or ``sqlcmd -i``.
* ``db2sql migrate`` — open a live connection to the target database and apply the same DDL and data directly, without an intermediate file. The DDL produced is byte-identical to dump mode: a single ``SqlEmitter`` is the source of truth in both paths.

Two helper commands round out the CLI: ``db2sql init`` generates a configuration file through an interactive wizard, and ``db2sql validate`` checks one (optionally previewing the export plan) before a long run.

Running ``db2sql`` with dump options but no command is a shorthand for ``db2sql dump`` — both forms are supported and produce identical output.


Installation
Expand Down Expand Up @@ -46,7 +50,7 @@ dump, but without the round-trip through a ``.sql`` file:
.. code-block:: console

# SQLite source → live Postgres target
$ db2sql --driver sqlite --dbname mydb.sqlite migrate \
$ db2sql migrate --driver sqlite --dbname mydb.sqlite \
--target-host localhost --target-port 5432 \
--target-dbname mytarget --target-user postgres --target-password s3cr3t

Expand All @@ -69,15 +73,43 @@ order:

.. code-block:: console

$ db2sql --driver sqlite --dbname mydb.sqlite --on-existing drop -f dump.sql
$ db2sql dump --driver sqlite --dbname mydb.sqlite --on-existing drop -f dump.sql

Pass ``--on-existing truncate`` to produce a *data-only* script: no DDL is
emitted, the dump just ``TRUNCATE``\s every managed table and reloads its
rows. Use it to refresh data into a pre-existing schema:

.. code-block:: console

$ db2sql --driver sqlite --dbname mydb.sqlite --on-existing truncate -f refresh.sql
$ db2sql dump --driver sqlite --dbname mydb.sqlite --on-existing truncate -f refresh.sql


Connecting with a DSN
---------------------

The discrete ``-H`` / ``-P`` / ``-d`` / ``-u`` / ``-p`` flags cover the common
case. When you need something they cannot express — a TLS mode, a charset, an
Oracle ``service_name``, an alternative DBAPI — pass a full SQLAlchemy URL
instead:

.. code-block:: console

# prefer the environment: a DSN on the command line is visible in `ps`
$ export DB2SQL_SOURCE_DSN='postgresql+psycopg2://app:s3cr3t@pg.example.com:5432/mydb?sslmode=require'
$ db2sql dump --driver postgres -f dump.sql

# and its mirror for a live migration
$ export DB2SQL_TARGET_DSN='postgresql+psycopg2://svc@target.internal:5432/stage'
$ db2sql migrate --driver mysql -H mysql.example.com -d mydb -u app -W

A DSN **replaces** the connection rather than merging with it, and the URL
dialect must match ``--driver`` / ``--target``. Passing a DSN together with
``-H`` / ``-d`` / … on the same command line — or declaring both in the same
config file — is rejected as a contradiction; a DSN overriding a connection
that came from a config file or the environment is allowed, and warns about
what it dropped. Passwords are always redacted in log output. See the
`CLI reference <https://python-db2sql.readthedocs.org/en/stable/cli.html>`__
for the full semantics.


Validating a configuration
Expand Down
8 changes: 8 additions & 0 deletions db2sql/const.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,13 @@
"ENV_DB2SQL_HOST",
"ENV_DB2SQL_PORT",
"ENV_DB2SQL_DBNAME",
"ENV_DB2SQL_SOURCE_DSN",
"ENV_DB2SQL_TARGET_HOST",
"ENV_DB2SQL_TARGET_PORT",
"ENV_DB2SQL_TARGET_USER",
"ENV_DB2SQL_TARGET_PASSWORD",
"ENV_DB2SQL_TARGET_DBNAME",
"ENV_DB2SQL_TARGET_DSN",
"ENV_NO_COLOR",
"ENV_CLICOLOR_FORCE",
"ENV_DB2SQL_COLOR_DARK",
Expand Down Expand Up @@ -40,6 +42,9 @@
ENV_DB2SQL_DBNAME = "DB2SQL_DBNAME"
"""Source database name."""

ENV_DB2SQL_SOURCE_DSN = "DB2SQL_SOURCE_DSN"
"""Full SQLAlchemy URL for the source database; overrides the discrete fields."""

ENV_DB2SQL_TARGET_HOST = "DB2SQL_TARGET_HOST"
"""Target database host (live migration)."""

Expand All @@ -55,6 +60,9 @@
ENV_DB2SQL_TARGET_DBNAME = "DB2SQL_TARGET_DBNAME"
"""Target database name (live migration)."""

ENV_DB2SQL_TARGET_DSN = "DB2SQL_TARGET_DSN"
"""Full SQLAlchemy URL for the target database (live migration)."""

ENV_NO_COLOR = "NO_COLOR"
"""Disable ANSI colors."""

Expand Down
26 changes: 24 additions & 2 deletions db2sql/infrastructure/config/loader.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,13 +27,14 @@
str(Path.home() / "db2sql.yml"),
]

_SERVER_FIELDS = {"hostname", "port", "username", "password", "dbname"}
_SERVER_FIELDS = {"hostname", "port", "username", "password", "dbname", "dsn"}
_TARGET_SERVER_FIELDS = {
"target_hostname",
"target_port",
"target_username",
"target_password",
"target_dbname",
"target_dsn",
}
_MIGRATE_FIELDS = {"on_existing", "transaction_mode", "batch_size", "use_transaction"}
_DUMP_FIELDS = {
Expand Down Expand Up @@ -105,9 +106,30 @@ def load_config(config_file: Optional[PathLike] = None) -> AppConfig:
if not data:
return AppConfig()
try:
return AppConfig.model_validate(data)
config = AppConfig.model_validate(data)
except ValidationError as exc:
raise ConfigInvalidError(f"Invalid configuration file {resolved}: {exc}") from exc
_reject_dsn_conflicts(config, resolved)
return config


def _reject_dsn_conflicts(config: AppConfig, source: str) -> None:
"""Refuse a ``dsn`` sitting next to discrete connection keys in the same file.

A DSN replaces the connection rather than merging with it, so declaring
both in one file states two contradictory intents. A DSN passed on the
command line while the file describes a host is a different matter — that
is the documented precedence, and the runner only warns about it.
"""
for section in ("server", "target_server"):
server = getattr(config, section)
shadowed = server.fields_shadowed_by_dsn()
if shadowed:
keys = ", ".join(f"{section}.{name}" for name in shadowed)
raise ConfigInvalidError(
f"Invalid configuration file {source}: {section}.dsn cannot be combined "
f"with {keys} — a DSN replaces the connection, it does not merge with it."
)


def merge_cli_overrides(config: AppConfig, options: Mapping[str, Any]) -> AppConfig:
Expand Down
19 changes: 18 additions & 1 deletion db2sql/infrastructure/config/schema.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

from __future__ import annotations

from typing import Any, Dict, List, Optional
from typing import Any, Dict, List, Optional, Tuple

from pydantic import BaseModel, ConfigDict, Field, field_validator

Expand All @@ -19,8 +19,25 @@ class ServerConfig(BaseModel):
username: Optional[str] = None
password: Optional[str] = None
dbname: Optional[str] = None
dsn: Optional[str] = None
options: Dict[str, str] = Field(default_factory=dict)

def fields_shadowed_by_dsn(self) -> Tuple[str, ...]:
"""Discrete connection fields that ``dsn`` makes irrelevant.

A DSN replaces the whole connection rather than merging with it, so
anything set alongside it is silently unused. Callers report this back
to the user instead of letting the mismatch pass unnoticed.
"""
if not self.dsn:
return ()
return tuple(
name for name in _DISCRETE_CONNECTION_FIELDS if getattr(self, name) is not None
)


_DISCRETE_CONNECTION_FIELDS = ("hostname", "port", "username", "password", "dbname")


class TableOverride(BaseModel):
"""Per-table overrides applied on top of the global dump options."""
Expand Down
11 changes: 3 additions & 8 deletions db2sql/infrastructure/persistence/mssql/reader.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
from db2sql.infrastructure.config import AppConfig
from db2sql.infrastructure.persistence import query_introspection
from db2sql.infrastructure.persistence.errors import SourceReaderError
from db2sql.infrastructure.url import build_url, redact_url


class MSSQLSourceReader:
Expand All @@ -25,20 +26,14 @@ def __init__(self, config: AppConfig, logger: Logger) -> None:

def _ensure_session(self) -> Session:
if self._session is None:
self._logger.info(f"set connection to {self._connection_string}")
self._logger.info(f"set connection to {redact_url(self._connection_string)}")
self._engine = create_engine(self._connection_string)
self._session = sessionmaker(bind=self._engine)()
return self._session

@property
def _connection_string(self) -> str:
server = self._config.server
port = f":{server.port}" if server.port else ""
username = server.username or ""
password = server.password or ""
hostname = server.hostname or ""
dbname = server.dbname or ""
return f"mssql+pymssql://{username}:{password}@{hostname}{port}/{dbname}"
return build_url(self._config.server, "mssql+pymssql")

def collect_metadata(self) -> Database:
database = Database(str(self._config.server.dbname or ""))
Expand Down
21 changes: 10 additions & 11 deletions db2sql/infrastructure/persistence/mysql/reader.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
from db2sql.infrastructure.config import AppConfig
from db2sql.infrastructure.persistence import query_introspection
from db2sql.infrastructure.persistence.errors import SourceReaderError
from db2sql.infrastructure.url import build_url, database_from_url, redact_url


class MySQLSourceReader:
Expand All @@ -25,26 +26,24 @@ def __init__(self, config: AppConfig, logger: Logger) -> None:

def _ensure_session(self) -> Session:
if self._session is None:
self._logger.info(f"set connection to {self._connection_string}")
self._logger.info(f"set connection to {redact_url(self._connection_string)}")
self._engine = create_engine(self._connection_string)
self._session = sessionmaker(bind=self._engine)()
return self._session

@property
def _connection_string(self) -> str:
server = self._config.server
port = f":{server.port}" if server.port else ""
username = server.username or ""
password = server.password or ""
hostname = server.hostname or ""
dbname = server.dbname or ""
return f"mysql+pymysql://{username}:{password}@{hostname}{port}/{dbname}"
return build_url(self._config.server, "mysql+pymysql")

@property
def _database_name(self) -> str:
if not self._config.server.dbname:
raise SourceReaderError("MySQL reader requires server.dbname")
return str(self._config.server.dbname)
# MySQL has no schema layer: the database name doubles as the schema
# every table is filed under, so it must be known even with a DSN.
server = self._config.server
name = database_from_url(server.dsn) if server.dsn else server.dbname
if not name:
raise SourceReaderError("MySQL reader requires server.dbname or a DSN naming it")
return str(name)

def collect_metadata(self) -> Database:
database = Database(self._database_name)
Expand Down
14 changes: 5 additions & 9 deletions db2sql/infrastructure/persistence/oracle/reader.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
from db2sql.infrastructure.config import AppConfig
from db2sql.infrastructure.persistence import query_introspection
from db2sql.infrastructure.persistence.errors import SourceReaderError
from db2sql.infrastructure.url import build_url, redact_url


def _normalize_oracle_type(raw: str) -> str:
Expand Down Expand Up @@ -91,7 +92,7 @@ def __init__(self, config: AppConfig, logger: Logger) -> None:

def _ensure_session(self) -> Session:
if self._session is None:
self._logger.info(f"set connection to {self._connection_string}")
self._logger.info(f"set connection to {redact_url(self._connection_string)}")
self._engine = create_engine(self._connection_string)
self._session = sessionmaker(bind=self._engine)()
return self._session
Expand All @@ -100,16 +101,11 @@ def _ensure_session(self) -> Session:
def _connection_string(self) -> str:
server = self._config.server
options = server.options or {}
driver = options.get("driver", "oracledb")
port = f":{server.port}" if server.port else ""
userinfo = f"{server.username or ''}:{server.password or ''}"
host = server.hostname or ""
scheme = f"oracle+{options.get('driver', 'oracledb')}"
service_name = options.get("service_name")
sid = options.get("sid")
if service_name:
return f"oracle+{driver}://{userinfo}@{host}{port}/?service_name={service_name}"
target = sid or server.dbname or ""
return f"oracle+{driver}://{userinfo}@{host}{port}/{target}"
return build_url(server, scheme, database="", query={"service_name": service_name})
return build_url(server, scheme, database=options.get("sid") or server.dbname or "")

@property
def _schema_filter(self) -> Optional[str]:
Expand Down
11 changes: 3 additions & 8 deletions db2sql/infrastructure/persistence/postgres/reader.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
from db2sql.infrastructure.config import AppConfig
from db2sql.infrastructure.persistence import query_introspection
from db2sql.infrastructure.persistence.errors import SourceReaderError
from db2sql.infrastructure.url import build_url, redact_url

_SYSTEM_SCHEMAS = ("pg_catalog", "information_schema", "pg_toast")

Expand All @@ -27,20 +28,14 @@ def __init__(self, config: AppConfig, logger: Logger) -> None:

def _ensure_session(self) -> Session:
if self._session is None:
self._logger.info(f"set connection to {self._connection_string}")
self._logger.info(f"set connection to {redact_url(self._connection_string)}")
self._engine = create_engine(self._connection_string)
self._session = sessionmaker(bind=self._engine)()
return self._session

@property
def _connection_string(self) -> str:
server = self._config.server
port = f":{server.port}" if server.port else ""
username = server.username or ""
password = server.password or ""
hostname = server.hostname or ""
dbname = server.dbname or ""
return f"postgresql+psycopg2://{username}:{password}@{hostname}{port}/{dbname}"
return build_url(self._config.server, "postgresql+psycopg2")

def collect_metadata(self) -> Database:
database = Database(str(self._config.server.dbname or ""))
Expand Down
12 changes: 8 additions & 4 deletions db2sql/infrastructure/persistence/sqlite/reader.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
from db2sql.infrastructure.config import AppConfig
from db2sql.infrastructure.persistence import query_introspection
from db2sql.infrastructure.persistence.errors import SourceReaderError
from db2sql.infrastructure.url import build_url, redact_url

_DEFAULT_SCHEMA = "public"

Expand All @@ -28,14 +29,17 @@ def __init__(self, config: AppConfig, logger: Logger) -> None:

@property
def _connection_string(self) -> str:
path = self._config.server.options.get("path") or self._config.server.dbname
server = self._config.server
if server.dsn:
return build_url(server, "sqlite", credentials=False)
path = server.options.get("path") or server.dbname
if not path:
raise SourceReaderError("SQLite reader requires server.dbname or options.path")
return f"sqlite:///{path}"
raise SourceReaderError("SQLite reader requires server.dbname, options.path, or a DSN")
return build_url(server, "sqlite", database=str(path), credentials=False)

def _ensure_session(self) -> Session:
if self._session is None:
self._logger.info(f"set connection to {self._connection_string}")
self._logger.info(f"set connection to {redact_url(self._connection_string)}")
self._engine = create_engine(self._connection_string)
self._session = sessionmaker(bind=self._engine)()
return self._session
Expand Down
Loading
Loading