Test and Coverage Comparison Tools

The following two Python scripts are used for comparing test results and code coverage across different runs. These are especially useful in comparison of AMD based runs against H100 runs, as well as in continuous integration pipelines.

cucim_compare_junit.py Compare two XML test reports
cucim_compare_coverage.py Compare two coverage.py XML reports

Note that these scripts are integrated into the ../run_amd script as part of the Python tests and the comparison reports are automatically written to the files junit-cucim_report.txt and cucim-coverage_report.txt, respectively, after each run.

Features
Requirements
Code Coverage
- C++ Code Coverage
- Python Code Coverage
Usage
Examples
Diagnostics
Next Steps

Features

cucim_compare_junit.py

Detects test regressions, fix-ups, additions, and removals.
Prints a summary of total tests, test suite name, run time, host, and path.
Ensures test suite consistency between runs.

cucim_compare_coverage.py

Compares overall line and branch coverage rates.
Detects coverage regressions, improvements, and file-level changes.
Displays detailed metadata: version, run timestamp, line and branch statistics.
Handles divergent source paths gracefully.

Requirements

Python 3.7+
Standard libraries only (no third-party dependencies)
LLVM tools for C++ coverage (llvm package)

Code Coverage

hipCIM supports code coverage for both C++ and Python components to help ensure comprehensive testing.

C++ Code Coverage

hipCIM includes integrated code coverage support for C++ components using LLVM coverage tools.

Prerequisites

Install LLVM tools for coverage analysis:

sudo apt update && sudo apt install -y llvm

Generate Coverage Reports

Single Command

# Generate coverage with debug build (more accurate)
./run_amd cpp_coverage

# Or with release build
./run_amd cpp_coverage release

This command will:

Run all C++ tests with coverage enabled
Generate coverage data (.profraw files)
Create HTML coverage report at coverage_report/index.html
Display coverage summary in terminal

View Coverage Results

Terminal Summary: After running coverage, you'll see a summary showing file-by-file coverage percentages.

HTML Report:

# View locally (if GUI available)
firefox coverage_report/index.html

# View remotely (on server)
cd coverage_report
python3 -m http.server 8080
# Open http://your-server-ip:8080 in browser

The HTML report provides:

Interactive file-by-file coverage visualization
Line-by-line coverage highlighting (green=covered, red=not covered)
Function-level coverage statistics
Detailed source code view with coverage annotations

Generated Files:

coverage_report/index.html - Main HTML coverage report
merged.profdata - Merged coverage profile data
code_cov_*.profraw - Raw coverage data from test runs

Coverage Configuration

The coverage system is integrated into the CMake build system through the ENABLE_CODE_COVERAGE option:

Automatically detects compiler type (hipcc/clang vs gcc)
Applies appropriate coverage flags (-fprofile-instr-generate -fcoverage-mapping for LLVM)
Excludes test and benchmark code from coverage metrics
Works with existing build and test infrastructure

Understanding C++ Coverage Metrics

The C++ coverage report generated by LLVM's llvm-cov or gcov/lcov shows how much of each file was executed when running your test suite. Each row corresponds to one source/header file with the following columns:

Column	Meaning
Function Coverage	Percentage of functions in that file that were executed during tests. Example: `100.00% (3/3)` means all 3 functions were executed at least once.
Line Coverage	Percentage of individual source lines executed during tests. Example: `78.87% (56/71)` means 56 out of 71 relevant lines ran.
Region Coverage	A "region" is a block of code with a single entry and exit (e.g., part of a function, loop body, or conditional branch). Region coverage is finer-grained than line coverage. Example: `86.84% (33/38)` means 33 out of 38 regions were executed.
Branch Coverage	Percentage of conditional branches (like `if` or `switch` cases) taken during tests. Example: `50.00% (10/20)` means only half of all possible true/false or case outcomes were exercised.

How to Interpret:

High function but low branch coverage → Your tests are calling functions but not covering all decision paths inside them.
0% coverage → That file's code never ran during tests (often happens for unused dependencies or untested modules).
High coverage across all columns → Strong, thorough testing.

Common Patterns in Your Report:

Many absl/* files have 0.00% coverage — likely third-party code not directly exercised by your tests.
Some Catch2 internal files show partial coverage (e.g., 60–80% line coverage) because only certain test scenarios hit their code paths.
Files like catch_get_random_seed.cpp have 100% across all metrics, meaning all lines, functions, and branches were tested.

Python Code Coverage

Python code coverage is automatically generated when running Python tests through the run_amd script and saved as XML reports that can be compared using the cucim_compare_coverage.py tool described below.

Usage

Test Comparison

Script: cucim_compare_junit.py

Arguments

python cucim_compare_junit.py --baseline BASELINE_XML --report TEST_XML [--skip SKIPLIST]

Argument	Description	Required
`--baseline`	Path to baseline XML file	Yes
`--report`	Path to current test XML file	Yes
`--skip`	Path to plain text file containing list of files to skip from comparison	No

Output

Metadata for both baseline and test runs
Total tests
Diagnostics on:
- Test suite mismatch
- Total tests mismatch
Summary of:
- New passing tests
- New failing tests
- Fixed tests
- Removed tests

Coverage Comparison

Script: cucim_compare_coverage.py

Arguments

python cucim_compare_coverage.py --baseline BASELINE_XML --report COVERAGE_XML

Argument	Description	Required
`--baseline`	Path to baseline coverage XML	Yes
`--report`	Path to current coverage XML	Yes

Output

Metadata for both baseline and test runs:
- Source path
- Coverage version
- Timestamp of run
- Line & branch statistics
- Complexity
Overall line rate delta
Regressions and progressions at the file level
New or removed files in test run

Baselines

A collection of known test results from prior executions, including those from competitive systems, is maintained under the directory test_data/baseline/. These baselines serve as reference points for comparison against current test or coverage results.

Naming Conventions

To simplify identification and tracking, test result files adhere to a structured naming format that embeds the timestamp, host system, and versioning information. This standardization aids reproducibility and clarity.

Unit Test Results

Format: junit-cucim_YYYYMMDD_HHMMSS_hostname.xml

Example: junit-cucim_20250320_071825_rocm-framework-h100-sxm-1.xml
Coverage Results

Format: cucim-coverage_VERSION_TIMESTAMP.xml

Example: cucim-coverage_v1.3_1742457165573.xml
Expected Failures (Skip List)

Format: junit-cucim_skiplist_YYYYMMDD_HHMMSS_hostname.txt

Example: junit-cucim_skiplist_20250320_071825_rocm-framework-h100-sxm-1.txt

Symbolic Links for Default Baselines

To streamline comparisons, symbolic links are maintained to always point to the most recent test artifacts. These are used as the default baselines by the test automation scripts unless explicitly overridden via command-line options.

Latest unit test result
```
junit-cucim_latest.xml
```
Latest coverage result
```
cucim-coverage_latest.xml
```
Latest skip list
```
junit-cucim_skiplist_latest.txt
```

Examples

Compare Test Reports

python cucim_compare_junit.py                \
   --baseline junit-cucim.xml                \
   --report AMD_lifescience/junit-cucim.xml

Sample output:

===== Test Report Comparison Summary =====

Baseline Report:
  File        : junit-cucim.xml
  Suite Name  : pytest
  Tests       : 10626
  Timestamp   : 2025-03-20T07:18:25.157139+00:00
  Hostname    : rocm-framework-h100-sxm-1

Current Report:
  File        : AMD_lifescience/junit-cucim.xml
  Suite Name  : pytest
  Tests       : 10617
  Timestamp   : 2025-04-29T12:34:40.305180+00:00
  Hostname    : hpe-hq-08

WARNING: Total test count differs: 10626 vs. 10617

===== Test Classification =====
  Total Tests           : 10627
  Skipped Test(s)       : 0
  Regression(s)         : 4840 (45.54%)
  Progression(s)        : 47 (0.44%)
  Known Failure(s)      : 0
  Changed Failure(s)    : 3
  Missing Test(s)       : 10
  Extra Test(s)         : 1

Missing Test Case(s):
  src.cucim.skimage.restoration.tests.test_j_invariant::test_calibrate_denoiser
  src.cucim.skimage.restoration.tests.test_j_invariant::test_calibrate_denoiser_extra_output
  src.cucim.skimage.restoration.tests.test_j_invariant::test_calibrate_denoiser_tv
  ...

Extra Test Case(s):
  tests.unit.clara.converter.test_converter::test_image_converter_stripe_4096x4096_256_jpeg

Regression(s):
  src.cucim.skimage._vendored.tests.test_morphology::test_binary_axes[binary_closing-0-origin0-0]
  src.cucim.skimage._vendored.tests.test_morphology::test_binary_axes[binary_closing-0-origin0-1]
  src.cucim.skimage._vendored.tests.test_morphology::test_binary_axes[binary_closing-0-origin1-0]
  ...

Progression(s):
  tests.performance.clara.test_read_region_memory_usage::test_read_random_region_cpu_memleak[testimg_tiff_stripe_4096x4096_256_deflate]
  tests.performance.clara.test_read_region_memory_usage::test_read_random_region_cpu_memleak[testimg_tiff_stripe_4096x4096_256_jpeg]
  tests.performance.clara.test_read_region_memory_usage::test_read_random_region_cpu_memleak[testimg_tiff_stripe_4096x4096_256_raw]
  ...

Changed Failure(s):
  tests.performance.clara.test_read_region_memory_usage::test_read_region_cuda_memleak
  tests.unit.clara.test_image_cache::test_get_shared_memory_cache
  ...

Compare Coverage Reports

python cucim_compare_coverage.py                \
   --baseline cucim-coverage.xml                \
   --report AMD_lifescience/cucim-coverage.xml

Sample output:

==== Coverage Report Comparison ====

Baseline File: cucim-coverage.xml
  Source Path:     /home/goplanid/cucim/python/cucim
  Coverage Ver:    7.7.0
  Timestamp:       2025-03-20 13:22:45
  Lines Valid:     10,077
  Lines Covered:   9,384
  Line Rate:       93.12%
  Branches Valid:  0
  Branches Covered:0
  Branch Rate:     0.00%
  Complexity:      0.0

Report File: AMD_lifescience/cucim-coverage.xml
  Source Path:     /home/dgoplani/dgoplani_cucim/hipCIM/python/cucim
  Coverage Ver:    7.8.0
  Timestamp:       2025-04-29 18:48:16
  Lines Valid:     11,000
  Lines Covered:   8,524
  Line Rate:       77.49%
  Branches Valid:  0
  Branches Covered:0
  Branch Rate:     0.00%
  Complexity:      0.0

Line Rate Delta: -15.63%


New Files:
  - src/cucim/clara/converter/tiff.py: 90.29%
  - src/cucim/skimage/measure/tests/test_regionprops.py: 12.80%

Regressions:
  - src/cucim/skimage/_shared/_gradient.py: 96.55% vs. 93.10%
  - src/cucim/skimage/_shared/coord.py: 100.00% vs. 93.02%
  ...

Progressions:
  - src/cucim/clara/__init__.py: 42.86% vs. 100.00%
  - src/cucim/clara/cache/__init__.py: 0.00% vs. 100.00%

Diagnostics

Test Comparison will emit diagnostics if:

Test suite names differ → "ERROR: Test suite name mismatch"
Total test count differs → "WARNING: Total test count mismatch"

Coverage Comparison:

Does not raise errors for source path mismatches
Prints line rate delta and categorizes changes

Next Steps

Potential enhancement(s):

Export diffs as JSON/XML for downstream tools
Write comparison report to a file based on CLI option
Integrate with CI/CD (e.g., GitHub Actions, GitLab, Jenkins)
Colorized terminal output

Navigation Menu

Test and Coverage Comparison Tools

Table of Contents

Features

cucim_compare_junit.py

cucim_compare_coverage.py

Requirements

Code Coverage

C++ Code Coverage

Prerequisites

Generate Coverage Reports

View Coverage Results

Coverage Configuration

Understanding C++ Coverage Metrics

Python Code Coverage

Usage

Test Comparison

Arguments

Output

Coverage Comparison

Arguments

Output

Baselines

Naming Conventions

Symbolic Links for Default Baselines

Examples

Compare Test Reports

Compare Coverage Reports

Diagnostics

Next Steps

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

Test and Coverage Comparison Tools

Table of Contents

Features

cucim_compare_junit.py

cucim_compare_coverage.py

Requirements

Code Coverage

C++ Code Coverage

Prerequisites

Generate Coverage Reports

View Coverage Results

Coverage Configuration

Understanding C++ Coverage Metrics

Python Code Coverage

Usage

Test Comparison

Arguments

Output

Coverage Comparison

Arguments

Output

Baselines

Naming Conventions

Symbolic Links for Default Baselines

Examples

Compare Test Reports

Compare Coverage Reports

Diagnostics

Next Steps