Extending EU model to include ES (Spain) variant

.github/workflows/SimPathsBuild.yml

@@ -42,4 +42,8 @@ jobs:
       uses: actions/upload-artifact@v4
       with:
         name: actual-simulation-results
-        path: output/INTEGRATION_TESTS_TRAINING/csv/
+        # One folder per country. Default if-no-files-found: warn is wanted here —
+        # no output at all means the run died before simulating.
+        path: |
+          output/INTEGRATION_TESTS_TRAINING_PL/csv/
+          output/INTEGRATION_TESTS_TRAINING_ES/csv/

.gitignore

@@ -194,6 +194,8 @@ input/~$scenario_uprating_factor.xlsx
 /input/PL/InitialPopulations/*.csv
 /input/EL/EUROMODoutput/*.txt
 /input/EL/InitialPopulations/*.csv
+/input/ES/EUROMODoutput/*.txt
+/input/ES/InitialPopulations/*.csv
 /input/*/EUROMODoutput/training/.DS_Store
 /input/*/InitialPopulations/training/.DS_Store
 !/input/*/EUROMODoutput/training/*.txt
@@ -245,10 +247,13 @@ input/tax_donor_population_HU.csv
 input/tax_donor_population_PL.csv
 input/tax_donor_population_IT.csv
 input/tax_donor_population_EL.csv
+input/tax_donor_population_ES.csv
 
-# Integration-test golden CSVs for the REAL-DATA baseline — generated locally
+# Integration-test golden CSVs for the REAL-DATA baselines — generated locally
 # per developer because real EUROMOD input data is not shareable. Do not commit.
-# The TRAINING-DATA baseline under expected_training/ IS committed. See
-# expected/README.md and expected_training/README.md for details.
-src/test/java/simpaths/integrationtest/expected/*.csv
-!src/test/java/simpaths/integrationtest/expected_training/*.csv
+# The TRAINING-DATA baselines under expected_training_<CC>/ ARE committed and are
+# what CI diffs against. See each folder's README.md for details.
+src/test/java/simpaths/integrationtest/expected_PL/*.csv
+src/test/java/simpaths/integrationtest/expected_ES/*.csv
+!src/test/java/simpaths/integrationtest/expected_training_PL/*.csv
+!src/test/java/simpaths/integrationtest/expected_training_ES/*.csv

CLAUDE.md

@@ -1,10 +1,27 @@
-can you# CLAUDE.md
+# CLAUDE.md
 
 This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
 
 ## Project Overview
 
-SimPaths is a JAS-mine-based microsimulation model that projects individual and household life course events (career, family, health, finances) for EU countries. This repository covers Greece (EL), Italy (IT), Hungary (HU), and Poland (PL). It integrates with EUROMOD for tax/benefit policy simulation.
+SimPaths is a JAS-mine-based microsimulation model that projects individual and household life course events (career, family, health, finances) for EU countries. This repository covers Greece (EL), Italy (IT), Spain (ES), Hungary (HU), and Poland (PL). It integrates with EUROMOD for tax/benefit policy simulation.
+
+Spain (ES) is being added. **The Java code treats ES as a first-class country** — `Country.ES("Spain", 13)`, `Region.ES1`–`ES7` (NUTS-1), `Labour.CATEGORY_ES_1`–`3` (ids 51–53), Spanish state-pension rules in `Parameters.getStatePensionAge` (statutory ages given in years and months), an ES block in `BenefitUnit.getRegressionValue`, and `Person.Regressors.ES1`–`ES7`. The earlier `Parameters.dataCountryString()` seam that redirected `ES`→`PL` has been removed, so nothing falls back to Poland's structures any more.
+
+**Most of `input/ES/` is now genuine Spanish data** (verified Aug 2026):
+
+- `InitialPopulations/population_initial_ES_2011..2024.csv` — real Spanish populations on the current 44-column camelCase schema, `demRgn ∈ {1…7}`, so all seven NUTS-1 regions resolve. The earlier `ES_TBC/` staging folder is gone.
+- `InitialPopulations/training/population_initial_ES_2011..2024.csv` — the shareable ES training subset (committed, added Aug 2026), so `-t true` and the ES integration test both work.
+- `EUROMODoutput/es_2005..2025_std.txt` — genuine Spanish EUROMOD output (`dct = 13`, 344 columns); the tax-unit identifier is `tu_nucfam_HeadID`, mapped in `input/system_bu_names.xlsx`. The leftover `EUROMODoutput/pl/` folder is gone.
+- Eleven of the twelve `reg_*.xlsx` carry ES coefficients with `ES1`–`ES6` region dummies (`reg_wages`, `reg_employmentSelection` and `reg_fertility` also include `ES7`; `reg_RMSE` has no region dummies). `align_popProjections.xlsx` is keyed on `ES1`–`ES7`, and `time_series_factor.xlsx` is Spain's own uprating series, no longer Poland's.
+- `input/DatabaseCountryYear.xlsx` includes ES, and `src/main/resources/images/ES.png` supplies the GUI flag.
+
+**Still outstanding before `-c ES` is trustworthy:**
+
+- Three files under `input/ES/` remain byte-identical PL clones and still need Spanish values: `reg_labourSupplyUtility.xlsx` (so the ES block in `BenefitUnit` is still estimated on Polish coefficients), `align_educLevel.xlsx`, `social_care_parameters.xlsx`.
+- `alignment_adjustment_factors.xlsx` is **expected** to be identical across countries — do not flag it as a stale PL clone. All 13 data sheets are zero-filled, and zero is the neutral cold start for a fresh calibration. Aligned runs search for the adjustment path and overwrite the in-memory map only; the workbook is never written back, so calibrated values are copied in by hand if you want to reuse them (see the file's own `Info` sheet).
+- `scenario_retirementAgeFixed.xlsx` was rebuilt with Spain's schedule (65 to 2017, 66 for 2018–2023, 67 from 2024, gender-neutral) to match `Parameters.getStatePensionAge` case `"ES"`; it is no longer a PL clone.
+- The `Region.ES1`–`ES7` code↔name mapping is still unverified against the EUROMOD ES country report (TODO at `Region.java:22`).
 
 **Data access**: The input data is not freely shareable. Training data is provided for development, but results from training data should not be interpreted beyond development purposes. Contact maintainers via GitHub issues for real data access.
 
@@ -33,7 +50,7 @@ CLI help: `java -jar singlerun.jar -h` or `java -jar multirun.jar -h`
 
 ### Key CLI flags
 
-- `-c <CC>` country code (`EL`, `IT`, `HU`, `PL`); `-s` start year; `-e` end year; `-p` population size; `-g true|false` show GUI.
+- `-c <CC>` country code (`EL`, `IT`, `ES`, `HU`, `PL`); `-s` start year; `-e` end year; `-p` population size; `-g true|false` show GUI.
 - `-t true|false` (`--training`) — use the training-data subset under `input/<CC>/InitialPopulations/training/` and `EUROMODoutput/training/` (uses `TaxDonorParserTraining`). On `multirun.jar` this **overrides** `parameter_args.trainingFlag` from the YAML config.
 - `singlerun.jar -Setup` — setup phase only (build the H2 input DB, no simulation). Multi-run equivalent is `-DBSetup`.
 - `multirun.jar -r <seed>` random seed, `-n <N>` max runs, `-f` output to file, `-config <file.yml>` custom config (default `config/default.yml`).
@@ -71,6 +88,57 @@ CLI help: `java -jar singlerun.jar -h` or `java -jar multirun.jar -h`
 4. **Alignment**: Mahalanobis-distance resampling adjusts distributions to match targets (YAML configs in `config/alignment_*.yml`)
 5. **Collection**: `SimPathsCollector` exports CSV statistics and optional DB snapshots to timestamped `output/` subdirectories
 
+### Statistics output files
+
+Six domain CSVs are written to `output/<run>/csv/`, one row per simulated year unless noted:
+
+| File | Cols | Contents | Toggle |
+|------|------|----------|--------|
+| `WealthIncomeStatistics.csv` | 31 | Gini, income percentiles, median EDI, S-Index, plus income and wealth by age band | `persistWealthIncomeStatistics` |
+| `DemographicStatistics.csv` | 9 | Partnership rates, dependent children, population counts by age band | `persistDemographicStatistics` |
+| `HealthStatistics.csv` | 6 | Mean self-rated health and disability shares by age band | `persistHealthStatistics` |
+| `LabourStatistics.csv` | 10 | Transitions and participation (16–64), full-time / part-time shares by age band | `persistLabourStatistics` |
+| `AlignmentStatistics.csv` | 36 | Alignment adjustment factors, simulated shares, target shares | `persistAlignmentStatistics` |
+| `HealthByGender.csv` | 10 | Self-rated health and disability, ages 16–64, Total / Male / Female — **3 rows per year** | `persistHealthByGender` |
+
+Age bands throughout are 18–29, 30–54 and 55–74. `DemographicStatistics.csv` carries the
+population counts that are the denominator for the age-band statistics in the other three
+wide files, so keep it enabled when interpreting them.
+
+Two rules in JAS-mine's `microsim.data.ExportCSV` (verified against 4.3.24) govern all of this:
+
+1. **The filename is the runtime type handed to `DataExport`.** For a `Collection` it is
+   `getSimpleName()`; for a bare object it is `getSimpleName() + <PanelEntityKey id>`. That
+   trailing `1` is why these files used to be `Statistics1.csv`, `Statistics21.csv`,
+   `EmploymentStatistics1.csv`, `HealthStatistics1.csv` and
+   `AlignmentAdjustmentFactors1.csv`. They are now wrapped in `List.of(...)` in
+   `SimPathsCollector.buildObjects()` so they take the collection branch. Renaming an output
+   therefore means renaming its class.
+2. **CSV columns are Java field names, sorted alphabetically** (a `TreeSet` over non-`@Transient`
+   fields). The `@Column` annotation names only the H2 database column — renaming it changes
+   nothing in the CSV, while renaming a field renames *and reorders* the CSV column.
+
+The four wide outputs are fed from one shared traversal of the population,
+`AgeBandAggregates`, cached per simulated year in `SimPathsCollector.ageBands()`. Each output
+has an independent toggle and its own dump event, so none may assume another has run.
+
+Twelve columns of the former `Statistics21.csv` were calibration loss-function terms, not
+statistics: a hard-coded pooled-2019-UKHLS target was subtracted from the simulated value
+invisibly, so shares were reported negative and expenditure below zero. Nothing read them and
+the targets were meaningless for the EU countries, so they were deleted —
+`labNoWork*Share` (recoverable as 1 − full-time − part-time), `x*Avg`, `xToLeisureRatio` and
+`statYDisp*Avg`. `statYDispGrossOfLosses*Avg` is a level and survives. `statYLab*Avg` was
+renamed `statYLabWeeklyPerWorker*Avg` because, unlike every income column beside it, it is
+weekly, unequivalised and per worker rather than monthly, equivalised and per capita.
+
+**When renaming an exported entity**, an IDE rename does not reach everything: the class in
+`data/statistics/`, the `SimPathsCollector` fields / `Processes` constants / `onEvent` cases /
+`buildObjects` / `buildSchedule` / `@GUIparameter` toggles and accessors, the
+`persistence.xml` entity list, the `persist*` **YAML config keys** (resolved via
+`getDeclaredField`, so a stale key is silently ignored rather than an error), the integration
+test paths and method names, the golden CSVs (including the `id_<ClassName>` header label),
+the Stata validation do-files under `validation/`, and this file.
+
 ### Data Inputs
 
 - `input/input.mv.db` — H2 database with processed EU-SILC starting population
@@ -80,12 +148,13 @@ CLI help: `java -jar singlerun.jar -h` or `java -jar multirun.jar -h`
 - `input/DatabaseCountryYear.xlsx` — Cross-country/year index
 - `config/default.yml` — Default multi-run parameters (population size, year range, run count)
 - `config/alignment_*.yml` — Staged alignment configurations
-- `config/test_create_database.yml`, `config/test_run.yml` — Configs used by the integration test
+- `config/test_create_database_<CC>.yml`, `config/test_run_<CC>.yml` — Configs used by that country's integration test (`PL`, `ES`)
 
 ### Repository layout (beyond `src/`)
 
 - `scripts/` — shell wrappers for batch multi-runs (`run_alignment_multiruns.sh`, `run_multiruns-alignPopOFF.sh`, `run_TEST_multiruns.sh`, …)
-- `input_processing/` — Stata do-files that prepare model inputs upstream of the Java pipeline (master conditions, regression-estimate cleaning, lag-structure generation)
+- `input_processing/` — Stata do-files that prepare model inputs upstream of the Java pipeline (master conditions, regression-estimate cleaning, lag-structure generation). For ES: `90_cleaning_excel_inputs.do` reads `reg_estimates_ES_toClean/` and writes `reg_estimates_ES_Cleaned/`; `91_add_dct_to_EUROMOD_training.do` stamps `dct` onto the training donor files.
+- `input/<CC>/DoFilesTargets/` — Stata do-files that build that country's alignment-target workbooks from the initial populations (`01`–`05` for retirement, in-school, disability, partnership and employment) plus `91_plot_targets_from_xlsx.do` for the target plots
 - `tools/generate_simpaths_eu_variable_codebook.py` — variable codebook generator
 - `validation/` — Stata validation against EU-SILC/EUROMOD targets
 - `documentation/` — supplementary documentation
@@ -107,7 +176,29 @@ JUnit 5 + Mockito. Tests in `src/test/java/simpaths/`:
 - `experiment/SimPathsMultiRunTest` — Multi-run configuration
 - `experiment/PersonTest` — Person entity logic
 - `data/MahalanobisDistanceTest` — Statistical matching
-- `integrationtest/RunSimPathsIntegrationTest` — End-to-end run using `config/test_create_database.yml` + `config/test_run.yml`
+- `integrationtest/SimPathsIntegrationTestBase` — the shared, country-agnostic machinery; a country test is a subclass naming only its two configs
+- `integrationtest/RunSimPathsPLIntegrationTest` — end-to-end run for **Poland**, `config/test_create_database_PL.yml` + `config/test_run_PL.yml` (`trainingFlag: true`)
+- `integrationtest/RunSimPathsESIntegrationTest` — the same run for **Spain**, `config/test_create_database_ES.yml` + `config/test_run_ES.yml` (`trainingFlag: true`)
+
+The integration tests are excluded from `mvn test` (surefire) and run under failsafe:
+
+```bash
+mvn clean package -DskipTests                       # the tests shell out to multirun.jar, so build it first
+mvn verify -Dit.test=RunSimPathsPLIntegrationTest   # Poland only
+mvn verify -Dit.test=RunSimPathsESIntegrationTest   # Spain only
+mvn verify                                          # both, sequentially
+```
+
+Output folders and golden-file folders are both derived from the country code and `parameter_args.trainingFlag`, so adding a country needs no path wiring:
+
+```
+output/INTEGRATION_TESTS[_TRAINING]_<CC>/csv/                    # produced by the run
+src/test/java/simpaths/integrationtest/expected[_training]_<CC>/ # diffed against
+```
+
+Both countries default to `trainingFlag: true`, so both baselines are **committed** and CI-checked; the real-data baselines are gitignored and captured locally per developer (see each `expected*` folder's README). The flag lives only in the run config — the test passes it to the `-DBSetup` step as `-t`, so the `test_create_database_<CC>.yml` files do not repeat it.
+
+Because each country has its own folders, either test can be run on its own, in any order. They do share `input/input.mv.db` and `input/DatabaseCountryYear.xlsx`, which each test rebuilds in its own `-DBSetup` step, so never run them concurrently.
 
 ## Branch Conventions
 

README.md

@@ -85,7 +85,8 @@ $ mvn clean package -DskipTests
 Unit tests run by default as part of the `test` / `package` phases. Integration tests (which exercise a full end-to-end simulation) are bound to the `verify` phase:
 ```
 $ mvn verify                                          # run unit + integration tests
-$ mvn verify -Dit.test=RunSimPathsIntegrationTest     # run just the integration test
+$ mvn verify -Dit.test=RunSimPathsPLIntegrationTest   # run just the Polish integration test
+$ mvn verify -Dit.test=RunSimPathsESIntegrationTest   # run just the Spanish integration test
 ```
 
 #### Single run

config/default.yml

@@ -53,8 +53,12 @@ collector_args:
 #    calculateGiniCoefficients: false
 #    exportToDatabase: false
 #    exportToCSV: true
-#    persistStatistics: true
-#    persistStatistics2: true
+#    persistWealthIncomeStatistics: true
+#    persistDemographicStatistics: true
+#    persistHealthStatistics: true
+#    persistHealthByGender: true
+#    persistLabourStatistics: true
+#    persistAlignmentStatistics: true
 #    persistPersons: false
 #    persistBenefitUnits: false
 #    persistHouseholds: false

config/test_create_database_ES.yml

@@ -0,0 +1,17 @@
+# FOR TESTING — SPAIN (ES)
+# CONFIG FILE TO CREATE NEW DATABASE OBJECTS
+#
+# Used by RunSimPathsESIntegrationTest to set up a fresh input H2 database from
+# the Spanish input files before running the deterministic simulation scenario
+# defined in test_run_ES.yml.
+#
+# Mirrors config/test_create_database_PL.yml in every setting except the country,
+# so the two country baselines stay comparable.
+
+# Arguments of the SimPathsMultiRun object overridden by the command-line
+countryString: "Spain"
+executeWithGui: false
+randomSeed: 606
+startYear: 2019
+endYear: 2030
+popSize: 30000

config/test_create_database.yml → config/test_create_database_PL.yml

@@ -1,9 +1,9 @@
 # FOR TESTING
 # CONFIG FILE TO CREATE NEW DATABASE OBJECTS
 #
-# Used by the integration test (RunSimPathsIntegrationTest) to set up a fresh
-# input H2 database from the EUROMOD Polish input files before running the
-# deterministic simulation scenario defined in test_run.yml.
+# Used by RunSimPathsPLIntegrationTest to set up a fresh input H2 database from
+# the EUROMOD Polish input files before running the deterministic simulation
+# scenario defined in test_run_PL.yml.
 
 # Arguments of the SimPathsMultiRun object overridden by the command-line
 countryString: "Poland"

config/test_run_ES.yml

@@ -0,0 +1,29 @@
+# FOR TESTING — SPAIN (ES)
+# CONFIG FILE FOR A DETERMINISTIC, SHORT-HORIZON SIMULATION RUN
+#
+# Used by RunSimPathsESIntegrationTest. Identical to config/test_run_PL.yml
+# apart from the country.
+
+countryString: "Spain"
+maxNumberOfRuns: 2
+executeWithGui: false
+randomSeed: 100
+startYear: 2019
+endYear: 2022
+popSize: 20000
+integrationTest: true
+
+parameter_args:
+  trainingFlag: true    # → INTEGRATION_TESTS_TRAINING_ES/ + expected_training_ES/ (committed)
+  # trainingFlag: false # → INTEGRATION_TESTS_ES/          + expected_ES/          (not committed)
+
+collector_args:
+    persistWealthIncomeStatistics: true
+    persistDemographicStatistics: true
+    persistAlignmentStatistics: true
+    persistLabourStatistics: true
+    persistHealthStatistics: true
+    persistHealthByGender: true
+    persistPersons: false
+    persistBenefitUnits: false
+    persistHouseholds: false