diff --git a/.github/workflows/sql-server-integration.yml b/.github/workflows/sql-server-integration.yml new file mode 100644 index 0000000..66c62fc --- /dev/null +++ b/.github/workflows/sql-server-integration.yml @@ -0,0 +1,127 @@ +name: SQL Server integration + +on: + pull_request: + branches: + - main + paths: + - .github/workflows/sql-server-integration.yml + - .env.example + - docker-compose.yml + - scripts/** + - sql/** + push: + branches: + - main + paths: + - .github/workflows/sql-server-integration.yml + - .env.example + - docker-compose.yml + - scripts/** + - sql/** + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: sql-server-integration-${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + sql-server-integration: + name: Provision, verify, and repeat + runs-on: ubuntu-24.04 + timeout-minutes: 20 + + steps: + - name: Check out repository + uses: actions/checkout@v6 + with: + persist-credentials: false + + - name: Show runner tool versions + shell: pwsh + run: | + Write-Host "PowerShell: $($PSVersionTable.PSVersion)" + Write-Host "Docker server: $(docker version --format '{{.Server.Version}}')" + docker compose version + + - name: Validate PowerShell syntax + shell: pwsh + run: | + $parseFailed = $false + + Get-ChildItem -LiteralPath ./scripts -Filter *.ps1 | ForEach-Object { + $tokens = $null + $parseErrors = $null + + [System.Management.Automation.Language.Parser]::ParseFile( + $_.FullName, + [ref]$tokens, + [ref]$parseErrors + ) | Out-Null + + if ($parseErrors.Count -gt 0) { + $parseFailed = $true + Write-Error "Syntax error in $($_.Name): $($parseErrors.Message -join '; ')" + } + else { + Write-Host "OK: $($_.Name)" + } + } + + if ($parseFailed) { + throw 'At least one PowerShell script contains a syntax error.' + } + + - name: Create temporary CI configuration + shell: pwsh + run: | + $ciPassword = "Ci9!$([guid]::NewGuid().ToString('N'))" + + @( + "MSSQL_SA_PASSWORD=$ciPassword" + 'MSSQL_PID=Developer' + 'MSSQL_PORT=14333' + 'MSSQL_IMAGE=mcr.microsoft.com/mssql/server:2022-CU26-ubuntu-22.04' + ) | Set-Content -LiteralPath ./.env -Encoding utf8NoBOM + + - name: Run complete workflow on a fresh runner + shell: pwsh + run: | + & ./scripts/Initialize-Lab.ps1 -TimeoutSeconds 300 + + - name: Run complete workflow again for repeatability + shell: pwsh + run: | + & ./scripts/Initialize-Lab.ps1 -TimeoutSeconds 300 -SkipImagePull + + - name: Write validation summary + shell: pwsh + run: | + @' + ## SQL Server integration result + + - PowerShell syntax validation: passed + - Fresh SQL Server provisioning: passed + - Relational integrity suite: passed + - Dimensional reporting reconciliation: passed + - Second complete run for repeatability: passed + '@ | Add-Content -LiteralPath $env:GITHUB_STEP_SUMMARY + + - name: Show SQL Server diagnostics on failure + if: failure() + shell: bash + run: | + docker compose --env-file .env -f docker-compose.yml ps --all || true + docker compose --env-file .env -f docker-compose.yml logs --tail 200 sqlserver || true + + - name: Clean up CI containers + if: always() + shell: bash + run: | + if [[ -f .env ]]; then + docker compose --env-file .env -f docker-compose.yml down --remove-orphans || true + fi + rm -f .env diff --git a/README.md b/README.md index d5af3e5..b120466 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,14 @@ # SQL Server Docker Basics -**Microsoft SQL Server 2022 · Docker Desktop · PowerShell 7 · T-SQL · relational modelling · data integrity · dimensional modelling · reporting verification** +[![SQL Server integration](https://github.com/DataTideHH/sql-server-docker-basics/actions/workflows/sql-server-integration.yml/badge.svg)](https://github.com/DataTideHH/sql-server-docker-basics/actions/workflows/sql-server-integration.yml) -This repository is a compact SQL Server analytics lab. It provides a reproducible local database environment, a normalized training model, synthetic seed data, enforced integrity rules, a dimensional reporting layer and a PowerShell workflow for startup, readiness checks, ordered execution and verification. +**Microsoft SQL Server 2022 · Docker Compose · PowerShell 7 · T-SQL · data integrity · dimensional modelling · GitHub Actions** -It is part of my Data/BI portfolio during the IHK retraining program in Data and Process Analysis. The project is deliberately bounded: it demonstrates a reliable path from relational source data to a verified star schema that can later support Power BI and CI-based integration tests. +This repository is a compact SQL Server analytics lab. It provides a reproducible database environment, a normalized training model, deterministic synthetic data, enforced integrity rules, a dimensional reporting layer and executable verification from source tables through the reporting fact. + +The same PowerShell entry point runs locally and in GitHub Actions. The CI workflow provisions SQL Server on a fresh Ubuntu runner and executes the complete lab twice, proving both clean installation and repeatability. + +The project is part of my Data/BI portfolio during the IHK retraining program in Data and Process Analysis. Its scope is deliberately bounded: it demonstrates a technically credible path from relational source data to a tested star schema without presenting the lab as a production data platform. --- @@ -12,65 +16,69 @@ It is part of my Data/BI portfolio during the IHK retraining program in Data and | Area | Current evidence | |---|---| -| SQL Server environment | SQL Server 2022 runs in Docker with an explicit host-port mapping, persistent local volume and healthcheck | -| Provisioning | PowerShell validates Docker and `.env`, starts the service, waits for readiness and executes the SQL workflow in order | -| Relational model | Four related tables with primary keys, foreign keys, uniqueness constraints and validated required values | -| Derived outcome | Pass status and score percentage are derived in `dpa.v_assessment_results` rather than stored redundantly | -| Cross-table integrity | Triggers prevent scores above the assessment maximum and invalid reductions of an existing maximum | -| Negative tests | Seven transaction-based tests prove that invalid writes are rejected and rolled back | -| Reporting model | Three dimensions and one fact table implement a documented star-schema grain | -| Reporting load | A transaction-based upsert synchronizes source values without silently deleting reporting rows | -| Reconciliation | Executable checks compare dimensions and facts with the relational source in both directions | -| SQL client workflow | Documented DataGrip connection and script execution process | +| SQL Server environment | Pinned SQL Server 2022 image, Docker healthcheck, explicit host port and persistent local volume | +| PowerShell provisioning | Docker and configuration validation, health polling, ordered SQL execution and non-zero failure behaviour | +| Relational model | Four normalized source tables with primary keys, foreign keys, uniqueness and required-value constraints | +| Derived outcome | Pass status and score percentage are derived instead of stored redundantly | +| Cross-table integrity | Set-based triggers enforce score limits that span results and assessments | +| Negative tests | Seven rollback-based tests prove invalid writes are rejected | +| Reporting model | Three dimensions and one fact table with a documented learner-assessment grain | +| Reporting load | Transaction-based upsert with preserved surrogate keys and source lineage | +| Reconciliation | Bidirectional checks compare source and reporting rows and values | +| Continuous integration | GitHub Actions runs the complete workflow twice on Ubuntu 24.04 | +| Data safety | Only synthetic data; `.env`, credentials, volumes and dumps remain excluded | ## Review Path -A quick technical review can follow these files in order: +A technical reviewer can inspect the implementation in this order: -1. [`docker-compose.yml`](docker-compose.yml) — pinned SQL Server image, volume, bind mount and healthcheck -2. [`scripts/Initialize-Lab.ps1`](scripts/Initialize-Lab.ps1) — end-to-end local bootstrap -3. [`scripts/SqlServerLab.Common.ps1`](scripts/SqlServerLab.Common.ps1) — shared Docker, configuration and readiness functions -4. [`sql/02_create_schema.sql`](sql/02_create_schema.sql) — relational schema, constraints, triggers and derived-result view -5. [`sql/06_test_integrity_rules.sql`](sql/06_test_integrity_rules.sql) — negative tests for enforced database rules -6. [`sql/07_create_reporting_model.sql`](sql/07_create_reporting_model.sql) — dimensional schema, constraints, indexes and reporting view -7. [`sql/08_load_reporting_model.sql`](sql/08_load_reporting_model.sql) — transaction-based dimensional upsert -8. [`sql/09_verify_reporting_model.sql`](sql/09_verify_reporting_model.sql) — source-to-reporting reconciliation -9. [`docs/reporting-model.md`](docs/reporting-model.md) — grain, dimensions, measures, load behaviour and boundaries +1. [`.github/workflows/sql-server-integration.yml`](.github/workflows/sql-server-integration.yml) — end-to-end CI execution and repeatability test +2. [`docker-compose.yml`](docker-compose.yml) — pinned SQL Server image, volume, bind mount and healthcheck +3. [`scripts/Initialize-Lab.ps1`](scripts/Initialize-Lab.ps1) — complete orchestration entry point +4. [`scripts/SqlServerLab.Common.ps1`](scripts/SqlServerLab.Common.ps1) — shared Docker, configuration and readiness functions +5. [`sql/02_create_schema.sql`](sql/02_create_schema.sql) — relational schema, constraints, triggers and derived-result view +6. [`sql/06_test_integrity_rules.sql`](sql/06_test_integrity_rules.sql) — executable negative tests +7. [`sql/07_create_reporting_model.sql`](sql/07_create_reporting_model.sql) — dimensional schema, constraints, indexes and reporting view +8. [`sql/08_load_reporting_model.sql`](sql/08_load_reporting_model.sql) — transaction-based dimensional upsert +9. [`sql/09_verify_reporting_model.sql`](sql/09_verify_reporting_model.sql) — source-to-reporting reconciliation +10. [`docs/ci-workflow.md`](docs/ci-workflow.md) — CI design, security boundaries and validation evidence --- -## Workflow +## End-to-End Workflow ```text -.env configuration - │ - ▼ +local .env or temporary CI .env + │ + ▼ PowerShell bootstrap - │ - ├── validate Docker and Compose - ├── validate local configuration - ├── pull and start SQL Server - └── wait for container health - │ - ▼ - relational workflow - │ - ├── create or upgrade source schema - ├── insert deterministic synthetic data - ├── enforce constraints and triggers - ├── verify source objects and values - └── run rollback-based negative tests - │ - ▼ - dimensional workflow - │ - ├── create reporting schema - ├── upsert dimensions and fact rows - ├── verify grain and key integrity - └── reconcile reporting data with source data + │ + ├── validate Docker and Compose + ├── validate configuration + ├── pull and start SQL Server + └── wait for container health + │ + ▼ +relational source workflow + │ + ├── create or upgrade source schema + ├── insert deterministic synthetic data + ├── enforce constraints and triggers + ├── verify source objects and values + └── run rollback-based negative tests + │ + ▼ +dimensional reporting workflow + │ + ├── create reporting schema + ├── upsert dimensions and fact rows + ├── verify grain and key integrity + └── reconcile reporting with source ``` -The host connects to SQL Server through port `14333`. Using a non-default host port avoids collisions with a separate local SQL Server installation while keeping the container's internal port at `1433`. +GitHub Actions executes this complete sequence twice in one job. The first run validates provisioning on a fresh runner. The second run uses the same temporary database volume and proves that the setup and load remain repeatable. + +The host connection uses port `14333`, while SQL Server remains on internal port `1433` inside the container. --- @@ -78,29 +86,31 @@ The host connects to SQL Server through port `14333`. Using a non-default host p ### Database environment -- SQL Server 2022 container managed through Docker Compose -- pinned SQL Server 2022 cumulative-update image -- persistent Docker volume for local development -- configurable SQL Server password, edition, image and host port -- healthcheck based on `sqlcmd` -- read-only container mount for repository SQL scripts -- connection through DataGrip or another SQL Server client +- SQL Server 2022 managed through Docker Compose +- pinned cumulative-update image +- persistent named volume for local development +- configurable password, edition, image and host port +- `sqlcmd`-based healthcheck +- repository SQL directory mounted read-only inside the container +- access through DataGrip or another SQL Server client ### PowerShell provisioning -The PowerShell workflow provides: +The shared PowerShell workflow provides: - Docker CLI, engine and Compose validation - `.env` presence and value checks -- rejection of the public password placeholder -- SQL Server image pull with an optional skip switch -- container startup and health polling -- ordered execution of relational and dimensional SQL scripts -- non-zero exit behaviour for Docker and SQL errors +- rejection of public placeholder passwords +- optional image-pull skipping for later runs +- container startup and SQL Server health polling +- ordered execution of source and reporting scripts +- non-zero failure propagation from Docker and `sqlcmd` - source-model verification - rollback-based negative integrity tests - reporting-model reconciliation -- safe stop and container-removal commands that preserve the named volume +- safe stop and container removal without automatic volume deletion + +The scripts require PowerShell 7 and are validated on Windows 11 locally and Ubuntu 24.04 in GitHub Actions. ### Relational source model @@ -116,19 +126,19 @@ The `dpa_training` database contains four source tables in the `dpa` schema: The source schema enforces: - surrogate identity keys -- stable business codes for modules and learners +- stable business codes - primary and foreign keys - unique module and learner codes - unique assessment names within a module - one result per learner and assessment - non-empty required text values -- valid assessment maxima and pass thresholds -- non-negative result scores +- valid maxima and pass thresholds +- non-negative scores - cross-table score limits through set-based triggers -### Derived outcome view +### Derived result view -`dpa.v_assessment_results` joins each result to its assessment and derives: +`dpa.v_assessment_results` derives: - `score_percentage` from `score / max_score` - `passed` from `score >= pass_score` @@ -137,7 +147,7 @@ The earlier stored `passed` column was removed only after a controlled upgrade c ### Executable integrity tests -`sql/06_test_integrity_rules.sql` deliberately attempts invalid writes and expects SQL Server to reject them: +`sql/06_test_integrity_rules.sql` attempts seven invalid writes: 1. set `max_score` to zero 2. set `pass_score` above `max_score` @@ -145,9 +155,9 @@ The earlier stored `passed` column was removed only after a controlled upgrade c 4. set a result score above the assessment maximum 5. lower an assessment maximum below an existing result 6. create a duplicate learner-assessment result -7. duplicate an assessment name within the same module +7. duplicate an assessment name within one module -Each test runs inside a transaction and rolls back any open transaction before evaluating the expected error. The suite fails closed when an expected error is missing. +Each test runs in a transaction, rolls back any open transaction and checks the expected SQL Server error class. The suite fails closed when an expected error is missing. ### Dimensional reporting model @@ -160,7 +170,7 @@ The `reporting` schema contains: | `reporting.dim_assessment` | one row per source assessment | | `reporting.fact_assessment_result` | one row per learner and assessment | -The fact table uses surrogate dimension keys and retains `source_result_id` for lineage. Its measures include: +The fact table uses surrogate dimension keys while retaining `source_result_id` for lineage. It contains: - `score` - `max_score` @@ -169,37 +179,43 @@ The fact table uses surrogate dimension keys and retains `source_result_id` for - persisted `passed` - `loaded_at_utc` -The learner-assessment grain is enforced by a unique constraint. Foreign keys connect every fact row to all three dimensions. +A unique `(assessment_key, learner_key)` constraint enforces the declared fact grain. Foreign keys connect every fact row to all three dimensions. -### Reporting load behaviour +### Reporting load `sql/08_load_reporting_model.sql` performs a transaction-based upsert: -- existing dimension values are updated from the source +- existing dimension attributes are refreshed +- existing surrogate keys are preserved - new source members are inserted -- existing fact rows are refreshed through `source_result_id` -- new fact rows are inserted - every source result must resolve to all dimension keys +- existing facts are refreshed through `source_result_id` +- new facts are inserted -The load does not silently delete reporting rows. Unexpected stale or additional rows are surfaced by verification instead of being removed automatically. +The load does not silently remove reporting rows. Unexpected stale or additional rows are surfaced by reconciliation instead. ### Reporting verification `sql/09_verify_reporting_model.sql` checks: -- presence of all reporting tables and the reporting view -- exact source-to-dimension and source-to-fact row-count parity -- bidirectional value equality for all dimensions -- absence of orphaned dimension keys -- uniqueness of the learner-assessment fact grain +- presence of all reporting objects +- exact source-to-dimension and source-to-fact count parity +- bidirectional dimension equality +- absence of orphaned keys +- uniqueness of the fact grain - bidirectional equality between source results and fact values -- enabled and trusted reporting constraints and foreign keys -- presence of both persisted computed measures +- enabled and trusted constraints and foreign keys +- persisted computed measures -A successful verification reports: +Validated values for the included dataset are: -| Metric | Validated value | +| Metric | Value | |---|---:| +| modules | 5 | +| learners | 5 | +| assessments | 5 | +| source results | 25 | +| passed results | 21 | | module dimension rows | 5 | | learner dimension rows | 5 | | assessment dimension rows | 5 | @@ -209,21 +225,26 @@ A successful verification reports: | source reconciliation verified | 1 | | reporting model verified | 1 | -### Validation evidence +--- + +## GitHub Actions Integration -The reporting increment was validated locally on Windows 11 with PowerShell 7.6.4 and Docker Desktop using the Linux engine. +The workflow at [`.github/workflows/sql-server-integration.yml`](.github/workflows/sql-server-integration.yml) runs for relevant pull requests and pushes to `main`, and can also be started manually. -The first successful run created and loaded the reporting model on the existing persistent database volume. The complete initialization was then executed a second time. Both runs produced: +It performs: -- `5 / 5 / 5` dimension rows -- `25` fact rows -- `21` passed fact rows -- average score percentage `68.12` -- source reconciliation result `1` -- reporting-model verification result `1` -- no duplicated source or reporting rows +- checkout with read-only repository permission +- PowerShell, Docker and Compose version reporting +- syntax parsing of every PowerShell script +- runtime generation of a temporary CI-only `.env` +- one complete initialization on a fresh Ubuntu runner +- a second complete initialization for repeatability +- SQL Server diagnostics on failure +- unconditional container and temporary-file cleanup -The repeated run confirms that the reporting schema and upsert load are repeatable for the included dataset. +The first pull-request run completed successfully on GitHub-hosted Ubuntu 24.04.4. Both complete database runs passed, including source verification, seven negative integrity tests and reporting reconciliation. + +The CI workflow uses no repository secrets and receives no write permission. See [`docs/ci-workflow.md`](docs/ci-workflow.md) for details. --- @@ -231,12 +252,16 @@ The repeated run confirms that the reporting schema and upsert load are repeatab ```text sql-server-docker-basics/ +├── .github/ +│ └── workflows/ +│ └── sql-server-integration.yml ├── README.md ├── LICENSE ├── .env.example ├── .gitignore ├── docker-compose.yml ├── docs/ +│ ├── ci-workflow.md │ ├── datagrip-workflow.md │ ├── project-notes.md │ └── reporting-model.md @@ -259,10 +284,6 @@ sql-server-docker-basics/ ├── 08_load_reporting_model.sql ├── 09_verify_reporting_model.sql └── examples/ - ├── README.md - ├── 01_basic_checks.sql - ├── 02_training_queries.sql - └── 03_data_quality_checks.sql ``` --- @@ -271,11 +292,10 @@ sql-server-docker-basics/ ### Prerequisites -- Windows 11 -- PowerShell 7 -- Docker Desktop with the Linux engine running +- Windows 11 or another environment with PowerShell 7 +- Docker Desktop or Docker Engine with Compose - Git -- optional: a SQL Server client such as DataGrip +- optional: DataGrip or another SQL Server client ### 1. Create the local environment file @@ -284,7 +304,7 @@ Copy-Item .env.example .env notepad.exe .env ``` -Replace the password placeholder before starting the lab. The `.env` file is ignored by Git and must remain local. +Replace the password placeholder before starting the lab. `.env` is ignored by Git and must remain local. ### 2. Initialize the complete lab @@ -306,18 +326,7 @@ For later runs, the image pull can be skipped: & ".\scripts\Test-ReportingModel.ps1" ``` -### 4. Run selected SQL scripts - -```powershell -& ".\scripts\Invoke-SqlScript.ps1" -Path @( - "sql/08_load_reporting_model.sql", - "sql/09_verify_reporting_model.sql" -) -``` - -Only SQL files inside the repository's `sql/` directory are accepted. - -### 5. Connect with a SQL client +### 4. Connect with a SQL client | Setting | Value | |---|---| @@ -328,48 +337,48 @@ Only SQL files inside the repository's `sql/` directory are accepted. | Password | local value from `.env` | | Database | `dpa_training` | -The detailed DataGrip procedure is documented in [`docs/datagrip-workflow.md`](docs/datagrip-workflow.md). +See [`docs/datagrip-workflow.md`](docs/datagrip-workflow.md) for the detailed client procedure. -### 6. Stop the lab +### 5. Stop the lab -Stop the running container while preserving it and the database volume: +Preserve the container and volume: ```powershell & ".\scripts\Stop-Lab.ps1" ``` -Remove the container and Compose network while preserving the named database volume: +Remove the container and Compose network while preserving the named volume: ```powershell & ".\scripts\Stop-Lab.ps1" -RemoveContainer ``` -The workflow deliberately does not remove the named volume. +The normal workflow deliberately does not remove the named volume. --- ## Current Boundaries -This repository does not currently claim: +This repository does not claim: - production deployment or high availability - production user and role design -- a general-purpose automated migration framework +- a general-purpose migration framework - slowly changing dimension handling -- incremental or deletion-aware warehouse loading -- CI-based SQL Server integration tests +- incremental, watermark-based or deletion-aware warehouse loading +- database backup and restore automation - a finished Power BI report - Azure or Microsoft Fabric deployment -- a macOS or Linux bootstrap equivalent to the PowerShell workflow +- a deployment or release pipeline -The dimensional model is a compact reporting layer for the synthetic training dataset, not a production data warehouse. +The star schema is a compact reporting layer for one deterministic synthetic dataset, not a production enterprise warehouse. The GitHub Actions workflow is an integration test, not a deployment pipeline. ## Next Development Steps -1. run the complete database workflow in GitHub Actions -2. document and validate a Power BI connection against the reporting layer -3. build a compact Power BI report with explicit KPI definitions -4. evaluate versioned migrations and incremental load strategies when the project scope creates a real need +1. document and validate a Power BI connection against the reporting layer +2. build a compact Power BI report with explicit KPI definitions +3. add screenshots or a short demo for portfolio review +4. evaluate versioned migrations and incremental loading when project scope creates a concrete need --- @@ -383,4 +392,4 @@ Only synthetic training data belongs in this repository. The following remain ex - SQL Server database volumes - local exports and backup files -See [`docs/project-notes.md`](docs/project-notes.md) for design decisions and [`docs/reporting-model.md`](docs/reporting-model.md) for the dimensional model specification. +See [`docs/project-notes.md`](docs/project-notes.md), [`docs/reporting-model.md`](docs/reporting-model.md) and [`docs/ci-workflow.md`](docs/ci-workflow.md) for design details and validation evidence. diff --git a/docs/ci-workflow.md b/docs/ci-workflow.md new file mode 100644 index 0000000..0074da7 --- /dev/null +++ b/docs/ci-workflow.md @@ -0,0 +1,140 @@ +# GitHub Actions Integration Workflow + +## Purpose + +The workflow in `.github/workflows/sql-server-integration.yml` runs the complete SQL Server lab on a fresh GitHub-hosted runner. It reuses the repository's Docker Compose and PowerShell entry points instead of maintaining a separate CI-only provisioning implementation. + +The CI goal is to prove that a clean machine can: + +1. validate the PowerShell scripts +2. start the pinned SQL Server container +3. create and verify the relational source model +4. execute the negative integrity suite +5. create and load the dimensional reporting model +6. reconcile reporting data with the source +7. execute the complete workflow a second time without duplication or failure + +## Triggers + +The workflow runs on: + +- pull requests to `main` when workflow, Compose, environment-template, PowerShell or SQL files change +- pushes to `main` for the same relevant paths +- manual `workflow_dispatch` runs + +Documentation-only changes do not start the comparatively expensive SQL Server container job unless the workflow file itself is part of the change. + +## Runner and Permissions + +The job uses: + +- `ubuntu-24.04` +- `actions/checkout@v6` +- `contents: read` +- a 20-minute job timeout +- concurrency cancellation for superseded runs on the same ref + +The workflow does not require repository secrets or write permissions. + +## Temporary Configuration + +The local workflow expects a `.env` file. CI creates one at runtime with: + +- a generated SQL Server system-administrator password +- the Developer edition +- host port `14333` +- the repository's pinned SQL Server image + +The generated password is not committed. The temporary `.env` file is removed during the final cleanup step. + +## Execution Sequence + +```text +checkout + │ + ├── report PowerShell, Docker and Compose versions + ├── parse every scripts/*.ps1 file + ├── create temporary .env + │ + ▼ +first complete initialization + │ + ├── pull pinned SQL Server image + ├── start container and wait for health + ├── create or upgrade relational model + ├── load deterministic source data + ├── create and load star schema + ├── verify source model + ├── execute seven negative integrity tests + └── reconcile reporting model + │ + ▼ +second complete initialization + │ + └── repeat the same workflow with -SkipImagePull + │ + ▼ +cleanup +``` + +The second initialization is part of the same job and uses the same temporary Docker volume. This proves repeatability after the first load rather than only proving that a clean installation works once. + +## Failure Diagnostics + +When a previous step fails, the workflow prints: + +- Docker Compose service and container state +- the latest 200 SQL Server service log lines + +This preserves useful evidence in the Actions log while still allowing the final cleanup step to execute. + +## Cleanup Behaviour + +The final step always attempts to: + +- stop and remove the Compose containers +- remove the Compose network +- remove the temporary `.env` + +The command does not use Docker's volume-removal option. On a GitHub-hosted runner, the virtual machine and its remaining local storage are discarded after the job. + +## Validation Evidence + +The first pull-request run completed successfully on 28 July 2026. + +Observed CI characteristics: + +- GitHub-hosted Ubuntu 24.04.4 runner +- repository token limited to read access +- all PowerShell scripts parsed successfully +- first complete SQL Server workflow succeeded on a fresh runner +- second complete workflow succeeded for repeatability +- failure diagnostics were not needed +- cleanup completed successfully + +The database assertions executed inside both complete runs included: + +- 5 modules +- 5 learners +- 5 assessments +- 25 source results +- 21 derived passed results +- seven passed negative integrity tests +- 5 module dimension rows +- 5 learner dimension rows +- 5 assessment dimension rows +- 25 fact rows +- 21 passed fact rows +- average score percentage `68.12` +- source reconciliation result `1` +- reporting-model verification result `1` + +## Boundaries + +This workflow is an integration test for the repository's local lab. It is not: + +- a production deployment pipeline +- a database backup strategy +- a release pipeline +- a cloud-hosted SQL Server deployment +- a substitute for versioned production migrations diff --git a/docs/project-notes.md b/docs/project-notes.md index 45b8c44..b8967b0 100644 --- a/docs/project-notes.md +++ b/docs/project-notes.md @@ -2,105 +2,108 @@ ## Purpose -This project provides a small, inspectable SQL Server environment for practical Data/BI work. It combines a Docker-based runtime with a normalized training database, synthetic data, enforced integrity rules, reporting queries, a dimensional reporting layer, PowerShell-based provisioning and documented client access. +This project provides a small, inspectable SQL Server environment for practical Data/BI work. It combines a Docker-based runtime with a normalized training database, synthetic data, enforced integrity rules, a dimensional reporting layer, PowerShell-based provisioning and end-to-end GitHub Actions integration. -The repository is intentionally limited to one local SQL Server workflow. It is not a production deployment template or a general-purpose data warehouse, but the implemented parts should still be reproducible, understandable and safe to publish. +The repository is not a production deployment template or enterprise data warehouse. Its implemented parts are intended to be reproducible, understandable, testable and safe to publish. ## Current Implementation -The repository currently includes: - -- a SQL Server 2022 container managed with Docker Compose -- a pinned SQL Server cumulative-update image -- a persistent local Docker volume -- configurable credentials, image and host port through `.env` -- a Docker healthcheck based on `sqlcmd` -- a PowerShell bootstrap for validation, startup, readiness and ordered SQL execution -- source-model verification and rollback-based negative integrity tests -- the `dpa_training` database and normalized `dpa` source schema -- a derived-result view for score percentages and pass status +The repository includes: + +- SQL Server 2022 managed with Docker Compose +- a pinned cumulative-update image +- a persistent named volume for local use +- runtime configuration through `.env` +- a `sqlcmd`-based healthcheck +- a PowerShell bootstrap for validation, startup, readiness and ordered execution +- a normalized `dpa` source schema +- a derived-result view for percentages and pass status +- named constraints and set-based cross-table triggers +- seven rollback-based negative integrity tests - a `reporting` star schema with three dimensions and one fact table - a transaction-based reporting upsert - executable source-to-reporting reconciliation -- documented DataGrip and reporting-model workflows +- a GitHub Actions workflow that executes the complete lab twice +- documentation for DataGrip, the reporting model and CI ## Design Decisions ### Pinned SQL Server image -The Compose file uses a named SQL Server 2022 cumulative-update image rather than the floating `2022-latest` tag. +The Compose file uses an explicit SQL Server 2022 cumulative-update image instead of the floating `2022-latest` tag. -This makes local runs more predictable and keeps image changes explicit in version control. Updating the image remains a deliberate maintenance task. +This keeps image changes reviewable and makes local and CI runs more predictable. Updating the image remains a deliberate maintenance task. ### Non-default host port -The container exposes SQL Server on host port `14333` while keeping the standard internal port `1433`. +The container exposes SQL Server on host port `14333` while retaining internal port `1433`. -This avoids conflicts with another local SQL Server installation and makes the Docker-based instance easier to identify in client configurations. +This avoids collisions with another local SQL Server installation and makes the Docker instance easy to identify in client configurations. ### Local credentials -The SQL Server password is supplied through a local `.env` file. The repository contains only `.env.example`; `.env` is excluded through `.gitignore`. +The SQL Server password is supplied through `.env`. The repository contains only `.env.example`; `.env` is excluded through `.gitignore`. -The bootstrap checks that the password value exists and rejects the public placeholder. Credentials must not be copied into SQL scripts, documentation, screenshots or committed client configuration. +The bootstrap verifies that the password exists and rejects public example values. Credentials must not be copied into scripts, documentation, screenshots or committed client settings. ### Container-side SQL execution -The repository's `sql/` directory is mounted read-only at `/workspace/sql` inside the container. PowerShell invokes the container's `sqlcmd` installation rather than requiring a separate host installation. +The repository's `sql/` directory is mounted read-only at `/workspace/sql`. PowerShell invokes the container's `sqlcmd` binary rather than requiring a host installation. -This reduces host prerequisites and ensures that setup and verification use the SQL tooling shipped with the selected container image. +This reduces host prerequisites and ensures that setup and verification use the SQL tooling included in the selected container image. ### Readiness before execution -Docker reporting a running container does not guarantee that SQL Server is ready to accept connections. The Compose healthcheck executes `SELECT 1`, and the PowerShell bootstrap waits for a healthy state before running setup scripts. +A running container is not necessarily ready to accept SQL connections. Docker Compose therefore uses a `SELECT 1` healthcheck, and the PowerShell bootstrap waits for the container to become healthy before executing database scripts. -An unhealthy, exited or timed-out container causes the workflow to stop and print recent service logs. +An unhealthy, exited, dead or timed-out container causes a non-zero failure and prints recent service logs. ### Synthetic data -The included learner and assessment data is synthetic. Stable learner and module codes make joins and repeatable inserts easier to inspect without using personal information. +All learner, module, assessment and result data is synthetic. Stable codes support repeatable joins and inserts without exposing personal information. ### Derived pass status -The first model stored `passed` in `dpa.assessment_results` even though it was completely determined by `score` and the related assessment's `pass_score`. +The first source model stored `passed` even though the value was completely determined by `score` and the related `pass_score`. -The hardened source model removes that redundancy. `dpa.v_assessment_results` derives both `passed` and `score_percentage`. Before dropping the old column, the schema upgrade checks that every stored value agrees with the source score and threshold. A mismatch stops the upgrade instead of silently discarding contradictory data. +The hardened model removes that redundancy. `dpa.v_assessment_results` derives both `passed` and `score_percentage`. Before the stored column is removed, the controlled upgrade checks that every existing value agrees with the source score and threshold. ### Check constraints and cross-table rules -Rules that depend on columns from the same row are implemented as named and trusted check constraints: +Rules that depend on one row are implemented as named and trusted check constraints: -- required text values cannot be blank +- required text cannot be blank - `max_score` must be positive - `pass_score` must be between zero and `max_score` - result scores cannot be negative -Rules that compare values across tables cannot be expressed as ordinary SQL Server check constraints. Two set-based triggers therefore enforce that: +Rules that compare values across tables require set-based triggers: -- a result score cannot exceed its assessment maximum +- a result cannot exceed its assessment maximum - an assessment maximum cannot be reduced below an existing result -The triggers inspect all affected rows through the `inserted` pseudo-table rather than assuming one-row statements. +The triggers evaluate all rows in the `inserted` pseudo-table and fail the complete statement on a violation. ### Business-key uniqueness The source model enforces: -- unique learner and module codes +- unique learner codes +- unique module codes - one result per learner and assessment - one assessment name per module -The last rule allows the same descriptive assessment name in different modules while preventing ambiguous duplicates inside one module. +The last rule permits the same descriptive assessment name in different modules while preventing ambiguous duplicates within one module. ### Executable negative tests -`sql/06_test_integrity_rules.sql` attempts seven invalid writes and confirms the expected database error for each one. Every test uses a transaction and rolls back any open transaction before checking the outcome. +`sql/06_test_integrity_rules.sql` attempts seven invalid writes and confirms the expected SQL Server error class for each one. -The suite is designed to fail closed. A missing or unexpected error code causes the script to throw instead of treating an unverified write as success. +Every test uses a transaction and rolls back any open transaction before evaluating the outcome. The suite fails closed: missing or unexpected errors cause the script to throw. ### Dimensional grain -The reporting model uses this declared fact grain: +The declared fact grain is: > one row per learner and assessment @@ -110,17 +113,17 @@ The reporting model uses this declared fact grain: - `reporting.dim_learner` - `reporting.dim_assessment` -A unique constraint on `(assessment_key, learner_key)` enforces the grain. `source_result_id` is retained as a lineage key back to the normalized source. +A unique constraint on `(assessment_key, learner_key)` enforces the grain. `source_result_id` provides lineage back to the normalized source. ### Surrogate keys and source lineage -Each dimension uses an identity-based surrogate key for star-schema relationships while also retaining its source identifier. +Each dimension uses an identity-based surrogate key while retaining its source identifier. -This separates reporting relationships from source primary keys without losing traceability. Unique constraints on the source identifiers prevent one source member from being loaded more than once. +This separates reporting relationships from source primary keys without losing traceability. Unique constraints on source identifiers prevent duplicate dimension members. ### Reporting measures -The fact table stores the additive source measures: +The fact table stores: - `score` - `max_score` @@ -131,95 +134,145 @@ It derives and persists: - `score_percentage` - `passed` -Persisting the computed values makes their definitions explicit in SQL Server and allows them to be included in reporting-oriented indexes. Required SQL session options are set explicitly in the schema and load scripts because indexed persisted computed columns depend on those settings. +Persisted computed columns make the definitions explicit and allow reporting indexes to include the results. The required SQL session options are set explicitly in schema and load scripts. ### Transaction-based upsert -`sql/08_load_reporting_model.sql` updates existing reporting rows and inserts new ones inside one transaction. +`sql/08_load_reporting_model.sql` updates existing reporting rows and inserts new rows inside one transaction. The load: -- refreshes dimension attributes from the source -- preserves existing surrogate keys +- refreshes dimension attributes +- preserves surrogate keys - inserts new source members -- resolves every source result to all three dimension keys -- refreshes or inserts fact rows through `source_result_id` +- resolves every source result to all three dimensions +- refreshes or inserts facts through `source_result_id` -The load does not automatically delete reporting rows that no longer exist in the source. This is deliberate: silent deletion is inappropriate for the current learning workflow. Exact reconciliation detects stale or additional rows and fails the run instead. +The load does not automatically delete reporting rows. Exact reconciliation exposes stale or additional rows instead of silently removing them. ### Bidirectional reconciliation -`sql/09_verify_reporting_model.sql` uses row counts and bidirectional `EXCEPT` comparisons to prove that source and reporting values agree. +`sql/09_verify_reporting_model.sql` combines row counts with bidirectional `EXCEPT` checks. The verification covers: -- all three dimensions -- fact measures and lineage identifiers +- all dimensions +- fact measures and lineage - learner-assessment grain - orphaned foreign keys - trusted constraints and foreign keys - persisted computed measures -Checking both directions matters. A source-minus-reporting comparison alone would detect missing rows but not unexpected additional reporting rows. +Checking both directions detects both missing reporting rows and unexpected additional reporting rows. ### Idempotent setup where practical -Database, schema, table and seed scripts use existence checks or `CREATE OR ALTER` where appropriate. Re-running the normal setup does not recreate existing source records or duplicate reporting rows. +Database, schema, table and seed scripts use existence checks or `CREATE OR ALTER` where appropriate. Re-running the complete workflow does not duplicate source records or reporting facts. -The reporting upsert updates timestamps and values but preserves the star-schema row counts and surrogate-key relationships for the included dataset. +The reporting upsert refreshes values and timestamps while preserving dimension keys and row counts for the current dataset. ### Controlled schema evolution -The source schema contains one explicit upgrade from the earlier stored `passed` column to the derived-result view. The reporting schema was introduced as an additive extension. +The source schema contains one explicit upgrade from a stored `passed` column to the derived-result view. The reporting schema was introduced as an additive extension. -This remains a bounded local workflow, not a general-purpose migration framework. Future non-trivial schema evolution should use a versioned migration approach rather than accumulating ad hoc upgrade blocks indefinitely. +This remains a bounded lab workflow, not a general-purpose migration framework. Future non-trivial evolution should use versioned migrations rather than accumulating ad hoc upgrade blocks. + +### GitHub Actions runner choice + +The integration workflow uses `ubuntu-24.04` rather than a floating runner label. + +This makes the operating-system generation explicit while still using a supported GitHub-hosted image. The same PowerShell scripts used locally run directly on the Linux runner; SQL execution remains inside the SQL Server container. + +### Restricted CI permissions + +The workflow declares: + +```yaml +permissions: + contents: read +``` + +No write permission or repository secret is required. The workflow only checks out source code and runs the local lab. + +### Temporary CI configuration + +CI generates a fresh SQL Server administrator password and writes a temporary `.env` at runtime. + +The password is not committed, and the temporary file is removed during cleanup. This avoids storing a reusable database credential in GitHub repository secrets for a disposable integration-test database. + +### Fresh-run and repeatability coverage + +The CI job invokes the complete `Initialize-Lab.ps1` workflow twice: + +1. first run with image pull on a fresh runner +2. second run against the same temporary database with `-SkipImagePull` + +The first run proves clean provisioning. The second proves that schema creation, seed logic, reporting upsert and all verification steps remain repeatable. + +### Failure diagnostics + +When the job fails, GitHub Actions prints: + +- Compose service and container state +- the latest 200 SQL Server log lines + +The diagnostics step precedes unconditional cleanup so failure evidence remains available in the Actions log. + +### CI cleanup + +The final workflow step always attempts to remove the Compose containers and network and then deletes the temporary `.env`. + +The command deliberately does not use the Docker volume-removal option. GitHub discards the hosted runner after the job, while the repository's normal safety rule remains explicit. ### Separation of workflow and examples The numbered scripts in `sql/` form one connected database workflow: 1. create the database -2. create or upgrade the relational source schema -3. insert synthetic source records -4. create the dimensional reporting schema +2. create or upgrade the source schema +3. insert synthetic source data +4. create the dimensional schema 5. load the reporting model 6. run reporting-oriented source queries 7. verify the source model 8. test source integrity rules -9. reconcile the reporting model with the source +9. reconcile reporting with source -The scripts in `sql/examples/` remain standalone learning and diagnostic examples. +Scripts under `sql/examples/` remain standalone learning and diagnostic examples. -### Safe stop behaviour +### Safe local stop behaviour -The stop script preserves the named SQL Server data volume. Its optional removal mode removes the container and Compose network, but it deliberately does not add Docker's volume-removal flag. +The local stop script preserves the named SQL Server volume. Its optional removal mode removes only the container and Compose network. -Destroying the local database volume remains a separate manual action and is not part of the normal workflow. +Destroying the local volume remains a separate manual operation and is not part of the normal workflow. ## Validation Evidence ### Source-model integrity increment -The integrity increment was validated locally on Windows 11 with PowerShell 7 and Docker Desktop using the Linux engine. +The source integrity increment was validated locally on Windows 11 with PowerShell 7 and Docker Desktop using the Linux engine. -Observed results after upgrading the existing database: +Observed results: -- all seed statements reported `0 rows affected` -- 5 modules, 5 learners, 5 assessments and 25 results remained present -- 21 passed results were derived from score and threshold -- setup verification returned `1` -- integrity-rule verification returned `1` -- seven invalid writes were rejected with their expected error classes -- the derived-outcome test returned `1` -- the complete integrity suite returned `1` +- all repeated seed statements returned `0 rows affected` +- 5 modules +- 5 learners +- 5 assessments +- 25 results +- 21 derived passed results +- setup verification `1` +- integrity-rule verification `1` +- seven rejected invalid writes +- derived-outcome test `1` +- complete integrity suite `1` -The complete initialization was executed a second time with the same successful results. +A second complete run produced the same results. ### Dimensional reporting increment -The reporting increment was validated locally on Windows 11 with PowerShell 7.6.4 and Docker Desktop using the Linux engine against the existing persistent database volume. +The reporting increment was validated locally on Windows 11 with PowerShell 7.6.4 and Docker Desktop against the existing persistent database volume. -The first successful load produced: +Both complete runs produced: - 5 module dimension rows - 5 learner dimension rows @@ -227,12 +280,32 @@ The first successful load produced: - 25 fact rows - 21 passed fact rows - average score percentage `68.12` -- source reconciliation result `1` -- reporting-model verification result `1` +- source reconciliation `1` +- reporting-model verification `1` +- no duplicate source or reporting rows + +### GitHub Actions integration increment + +The first pull-request CI run completed successfully on 28 July 2026. + +Observed runner and permission evidence: + +- GitHub-hosted Ubuntu 24.04.4 +- repository token limited to `contents: read` +- `actions/checkout@v6` +- every PowerShell script parsed successfully + +Observed execution evidence: -The full initialization was then executed again. The second run produced the same counts and verification results without duplicate source or reporting rows. +- fresh-run initialization succeeded +- source verification succeeded +- all seven negative integrity tests succeeded +- reporting reconciliation succeeded +- second complete initialization succeeded +- failure diagnostics were skipped because no failure occurred +- cleanup succeeded -This confirms repeatability for the included deterministic dataset and current upsert strategy. +This provides an independent clean-machine check in addition to the local Windows validation. ## Data Handling Rules @@ -241,8 +314,9 @@ Allowed content: - schema and setup scripts - synthetic records - public training examples +- workflow definitions - documentation -- reproducible query results that contain no private data +- reproducible outputs containing no private data Excluded content: @@ -250,32 +324,32 @@ Excluded content: - personal or customer data - private database dumps - Docker database volumes -- local exports and backup files -- screenshots that expose local credentials or unrelated private information +- local exports and backups +- screenshots exposing credentials or unrelated private information ## Current Boundaries -The repository does not currently include: +The repository does not include: - production deployment or high availability - production user and role design - a general-purpose automated migration framework -- slowly changing dimension handling +- slowly changing dimensions - incremental, watermark-based or deletion-aware loading -- SQL Server integration tests in CI +- backup and restore automation - cloud deployment - a finished Power BI report -- equivalent bootstrap scripts for macOS or Linux +- a release or deployment pipeline -The reporting model is a compact dimensional layer for one synthetic dataset, not a production enterprise warehouse. +The reporting model remains a compact dimensional layer for one synthetic dataset. The GitHub Actions workflow is an integration test for that lab, not a production deployment mechanism. ## Planned Extensions The next useful increments are: -1. end-to-end SQL Server integration tests in GitHub Actions -2. a reviewed Power BI connection based on the reporting layer -3. a compact Power BI report with explicit KPI definitions -4. a versioned migration and incremental-load approach when further project scope creates a real need +1. a reviewed Power BI connection based on the reporting layer +2. a compact Power BI report with explicit KPI definitions +3. portfolio screenshots or a short demo +4. versioned migrations and incremental loading when additional scope creates a real need -Each increment should remain separately reviewable and include a concrete verification step. +Each increment should remain separately reviewable and include concrete verification evidence.