Skip to content

Add FOM unit metadata to results #69

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yoshifuminakamura merged 1 commit into from
Jul 16, 2026
Merged

Add FOM unit metadata to results #69

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 18 additions & 7 deletions docs/guides/add-app.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -258,14 +258,14 @@ case "$system" in
mpiexec -n $((nodes * numproc_node)) ./main [args] > output
# 結果解析
FOM=$(grep "performance" output | awk '{print $2}')
bk_emit_result --fom "$FOM" --fom-version v1.0 --exp test \
bk_emit_result --fom "$FOM" --fom-unit s --fom-version v1.0 --exp test \
--nodes "$nodes" --numproc-node "$numproc_node" --nthreads "$nthreads" >> ../results/result
;;
MiyabiG|MiyabiC)
# MPI実行（Miyabi）
mpirun -n $((nodes * numproc_node)) ./main [args] > output
FOM=$(grep "performance" output | awk '{print $2}')
bk_emit_result --fom "$FOM" --fom-version v1.0 --exp test \
bk_emit_result --fom "$FOM" --fom-unit s --fom-version v1.0 --exp test \
--nodes "$nodes" --numproc-node "$numproc_node" --nthreads "$nthreads" >> ../results/result
;;
*)
Expand All @@ -283,7 +283,7 @@ sync
### 結果フォーマット
`results/result` の各行は以下の形式：
```
FOM:5.752 FOM_version:DDSolverJacobi Exp:CASE0 node_count:1 numproc_node:4 nthreads:12
FOM:5.752 FOM_unit:s FOM_version:DDSolverJacobi Exp:CASE0 node_count:1 numproc_node:4 nthreads:12
SECTION:compute_kernel time:0.30
SECTION:communication time:0.20
OVERLAP:compute_kernel,communication time:0.05
Expand All @@ -297,7 +297,7 @@ OVERLAP:compute_kernel,communication time:0.05
source "${PWD}/scripts/bk_functions.sh"

# FOM出力
bk_emit_result --fom 5.752 --fom-version DDSolverJacobi --exp CASE0 \
bk_emit_result --fom 5.752 --fom-unit s --fom-version DDSolverJacobi --exp CASE0 \
--nodes 1 --numproc-node 4 --nthreads 12 >> results/result

# FOM内訳（オプション）
Expand All @@ -308,17 +308,28 @@ bk_emit_overlap compute_kernel,communication 0.05 >> results/result

**bk_emit_result の引数：**
- `--fom 数値` - 性能指標（必須）
- `--fom-unit 文字列` - FOM の単位（推奨。例: `s`, `GB/s`, `GFLOPS`, `token/s`）
- `--fom-version 文字列` - バージョン情報
- `--exp 文字列` - 実験名
- `--nodes 数値` - ノード数
- `--numproc-node 数値` - ノードあたりプロセス数
- `--nthreads 数値` - プロセスあたりスレッド数
- `--confidential 文字列` - 機密データ（チーム限定表示）

省略された引数は出力に含まれません。`--fom` のみが必須です。
省略された引数は出力に含まれません。`--fom` のみが必須ですが、FOM の意味を誤読しないよう `--fom-unit` も原則として指定してください。

### Performance Analysis データ
詳細データは `results/padata[0-9].tgz` として保存：
### 最低限必要な出力

新しい app を BenchKit に接続する最低ラインは、`run.sh` が `results/result` に少なくとも `FOM:<数値>` 相当の結果を書けることです。
ただし FOM には単位が含まれないため、`FOM_unit:s` のように単位も出してください。
多くのアプリでは経過時間の `s` で十分ですが、システムソフトウェアやライブラリでは `GB/s`、`GFLOPS`、`token/s` などになることがあります。
`bk_emit_result --fom ... --fom-unit ...` を使うと、FOM、単位、実験名、ノード数、プロセス数、スレッド数を同じ形式で出力できます。

`source_info` は必須ではありませんが、Git などから source を取得する app では `bk_fetch_source` を使って `results/source_info.env` を残すことを推奨します。
section / overlap / profiler archive は、詳細分析や推定を使う場合の任意拡張です。

### Performance Analysis データ（任意）
詳細データがある場合は `results/padata[0-9].tgz` として保存：
```bash
# PAデータの作成例
mkdir -p pa
Expand Down
3 changes: 2 additions & 1 deletion docs/guides/add-estimation-to-app.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ myapp_declare_estimation_layout() {

### 最小構成

最小構成で必要なのは FOM の出力です。
最小構成で必要なのは FOM の出力です。FOM の数値だけでは単位が分からないため、`--fom-unit` も原則として指定してください。多くの app は経過時間の `s` ですが、ライブラリや system software では `GB/s`、`GFLOPS`、`token/s` などになることがあります。

```bash
source "${PWD}/scripts/bk_functions.sh"
Expand All @@ -169,6 +169,7 @@ mkdir -p results && > results/result

bk_emit_result \
--fom "$FOM" \
--fom-unit "$FOM_UNIT" \
--fom-version "$FOM_VERSION" \
--exp "$EXP" \
--nodes "$nodes" \
Expand Down
4 changes: 2 additions & 2 deletions programs/LQCD_dw_solver/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,14 +54,14 @@ get_fom () {
FOM2=`get_etime_total $LOG`
FOM_O=`echo $FOM2 $FOM1 | awk '{print $1-$2}'`
if [ $# -eq 1 ]; then
bk_emit_result --fom "$FOM2" --fom-version LQCD_dw_solver --exp total --nodes "$nodes" --numproc-node "$numproc_node" --nthreads "$nthreads"
bk_emit_result --fom "$FOM2" --fom-unit s --fom-version LQCD_dw_solver --exp total --nodes "$nodes" --numproc-node "$numproc_node" --nthreads "$nthreads"
bk_emit_section solver "$FOM1"
bk_emit_section other "$FOM_O"
else
# 第2引数をExp名として使用
# 注意: 暫定的にtarget情報をExpに付け加えます。
TARGET=$(echo "$2" | sed 's/target: //' | sed 's/ $//')
bk_emit_result --fom "$FOM2" --fom-version LQCD_dw_solver --exp "total_${TARGET}" --nodes "$nodes" --numproc-node "$numproc_node" --nthreads "$nthreads"
bk_emit_result --fom "$FOM2" --fom-unit s --fom-version LQCD_dw_solver --exp "total_${TARGET}" --nodes "$nodes" --numproc-node "$numproc_node" --nthreads "$nthreads"
bk_emit_section solver "$FOM1"
bk_emit_section other "$FOM_O"
fi
Expand Down
1 change: 1 addition & 0 deletions programs/MHDTurbulence/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ fomtcc=$(awk "BEGIN {printf \"%.6e\", 1.0/$tcc}")
{
bk_emit_result \
--fom "$fomtcc" \
--fom-unit cell/s \
--fom-version "cell_updates_per_sec" \
--exp "KH" \
--nodes "$nodes" \
Expand Down
1 change: 1 addition & 0 deletions programs/ffb/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@ fi

bk_emit_result \
--fom "$fom" \
--fom-unit s \
--fom-version "67.01" \
--exp "cavity" \
--nodes "$nodes" \
Expand Down
2 changes: 1 addition & 1 deletion programs/genesis-nonbonded-kernels/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ case "$system" in
#printf "result%d: %.3f\n" "$index" "$fom_val" >> ${resultsdir}/result
#total_fom=$(awk -v a="$total_fom" -v b="$fom_val" 'BEGIN{printf("%.6f", a + b)}')

bk_emit_result --fom "$fom_val" --exp "$name" --nodes "$nodes" --numproc-node "$numproc_node" --nthreads "$nthreads" >> ${resultsdir}/result
bk_emit_result --fom "$fom_val" --fom-unit s --exp "$name" --nodes "$nodes" --numproc-node "$numproc_node" --nthreads "$nthreads" >> ${resultsdir}/result
done

;;
Expand Down
2 changes: 1 addition & 1 deletion programs/genesis/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -515,7 +515,7 @@ if [[ -z "$fom_val" ]]; then
fi

{
bk_emit_result --fom "$fom_val" --exp "$exp" --nodes "$nodes" --numproc-node "$numproc_node" --nthreads "$nthreads"
bk_emit_result --fom "$fom_val" --fom-unit s --exp "$exp" --nodes "$nodes" --numproc-node "$numproc_node" --nthreads "$nthreads"
genesis_emit_estimation_data_from_log "$output" "$fom_val"
} >> ${resultsdir}/result
# if information is requierd
Expand Down
4 changes: 2 additions & 2 deletions programs/qws/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ print_results() {
./check.sh "$outfile" "data/$exp"
local fom
fom=$(grep etime "$outfile" | awk 'NR==2{printf("%5.3f\n",$5)}')
bk_emit_result --fom "$fom" --fom-version DDSolverJacobi --exp "$exp" --nodes "$nodes" --numproc-node "$np" --nthreads "$nthreads"
bk_emit_result --fom "$fom" --fom-unit s --fom-version DDSolverJacobi --exp "$exp" --nodes "$nodes" --numproc-node "$np" --nthreads "$nthreads"
qws_emit_estimation_data_from_fom "$fom"
}

Expand Down Expand Up @@ -78,7 +78,7 @@ case "$system" in
# FugakuLN retired; previous LN smoke run kept for reference.
# FugakuLN)
# echo 'dummy call for CI test: QWS program: ./main 32 6 4 3 1 1 1 1 -1 -1 6 50'
# bk_emit_result --fom 123.56 --fom-version dummy --exp CheckingPrivateRepo --nodes "$nodes" --numproc-node "$numproc_node" --nthreads "$nthreads" >> ../results/result
# bk_emit_result --fom 123.56 --fom-unit s --fom-version dummy --exp CheckingPrivateRepo --nodes "$nodes" --numproc-node "$numproc_node" --nthreads "$nthreads" >> ../results/result
# emit_qws_dummy_padata ../results/padata0.tgz
# ;;
RIKYU)
Expand Down
1 change: 1 addition & 0 deletions programs/salmon/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -231,6 +231,7 @@ fi
{
bk_emit_result \
--fom "${total_elapsed}" \
--fom-unit s \
--fom-version "total_elapsed_time_s" \
--exp "Si-1-1-1" \
--nodes "${nodes}" \
Expand Down
6 changes: 3 additions & 3 deletions programs/scale-letkf/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ case "$system" in
elapse_letkf=$elapse

FOM=$(echo "$elapse_scale $elapse_letkf" | awk '{printf "%.3f\n", $1 + $2}')
bk_emit_result --fom "$FOM" --fom-version SCALE-LETKF --exp SC23_128x128 --nodes "$nodes" --numproc-node 4 --nthreads 12 >> ../results/result
bk_emit_result --fom "$FOM" --fom-unit s --fom-version SCALE-LETKF --exp SC23_128x128 --nodes "$nodes" --numproc-node 4 --nthreads 12 >> ../results/result
;;
75)
echo "copy essential files ... `date`"
Expand Down Expand Up @@ -102,7 +102,7 @@ case "$system" in
elapse_letkf=$elapse

FOM=$(echo "$elapse_scale $elapse_letkf" | awk '{printf "%.3f\n", $1 + $2}')
bk_emit_result --fom "$FOM" --fom-version SCALE-LETKF --exp SC23_1280x1280 --nodes "$nodes" --numproc-node 4 --nthreads 12 >> ../results/result
bk_emit_result --fom "$FOM" --fom-unit s --fom-version SCALE-LETKF --exp SC23_1280x1280 --nodes "$nodes" --numproc-node 4 --nthreads 12 >> ../results/result
;;
esac
;;
Expand Down Expand Up @@ -160,7 +160,7 @@ case "$system" in
elapse_letkf=$elapse

FOM=$(echo "$elapse_scale $elapse_letkf" | awk '{printf "%.3f\n", $1 + $2}')
bk_emit_result --fom "$FOM" --fom-version SCALE-LETKF --exp SC23_128x128 --nodes "$nodes" --numproc-node 12 --nthreads 1 >> ../results/result
bk_emit_result --fom "$FOM" --fom-unit s --fom-version SCALE-LETKF --exp SC23_128x128 --nodes "$nodes" --numproc-node 12 --nthreads 1 >> ../results/result
;;
esac
;;
Expand Down
3 changes: 2 additions & 1 deletion result_server/templates/_results_table.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,8 @@
{% elif key in ["fom_version", "exp", "source_hash"] %}
<td title="{{ row[key] }}">{{ row[key] }}</td>
{% elif key == "fom" %}
<td title="{{ row[key] }}">{{ "%.3f"|format(row[key]|float) if row[key] not in [None, '', 'N/A', 'null', 'nan'] else row[key] }}</td>
{% set fom_value = "%.3f"|format(row[key]|float) if row[key] not in [None, '', 'N/A', 'null', 'nan'] else row[key] %}
<td title="{{ row[key] }}{% if row.fom_unit %} {{ row.fom_unit }}{% endif %}">{{ fom_value }}{% if row.fom_unit %} {{ row.fom_unit }}{% endif %}</td>
{% endif %}
{% if key == "fom" %}
<td class="center-cell">
Expand Down
4 changes: 3 additions & 1 deletion result_server/tests/test_portal_list_templates.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ def test_results_template_renders_table_note():
{"label": "Timestamp", "key": "timestamp", "tooltip": "Date and time when benchmark execution completed and results were automatically submitted to server", "tooltip_class": "tooltip-left"},
{"label": "SYSTEM", "key": "system", "tooltip": "Computing system name"},
{"label": "CODE", "key": "code"},
{"label": "FOM", "key": "fom", "tooltip": "Figure of Merit - Benchmark performance metric value, typically elapsed time in seconds for main section"},
{"label": "FOM", "key": "fom", "tooltip": "Figure of Merit - Benchmark performance metric value with its unit when available"},
{"label": "Exp", "key": "exp", "tooltip": "Experimental conditions (filtered by CODE)"},
{"label": "Profiler / PA", "key": "profile_summary", "tooltip": "Profiler tool, level, report summary, and PA data download access"},
{"label": "CI", "key": "ci_summary", "tooltip": "CI trigger source and pipeline ID"},
Expand All @@ -32,6 +32,7 @@ def test_results_template_renders_table_note():
"system": "Fugaku",
"code": "qws",
"fom": 1.234,
"fom_unit": "s",
"exp": "CASE0",
"json_link": "/results/result0.json",
"data_link": "/results/padata0.tgz",
Expand Down Expand Up @@ -87,6 +88,7 @@ def test_results_template_renders_table_note():
assert "CI" in html
assert "padata" in html
assert "#10" in html
assert "1.234 s" in html


def test_results_template_renders_ncu_options_tooltip():
Expand Down
5 changes: 3 additions & 2 deletions result_server/tests/test_results_loader.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -272,7 +272,7 @@ def test_existing_columns_unchanged(self, flask_app, tmp_dir):
{"label": "CODE", "key": "code"},
{"label": "Branch/Hash", "key": "source_hash", "tooltip": "Source code branch name and short commit hash (git) or short md5sum (file archive)"},
{"label": "Exp", "key": "exp", "tooltip": "Experimental conditions (filtered by CODE)"},
{"label": "FOM", "key": "fom", "tooltip": "Figure of Merit - Benchmark performance metric value, typically elapsed time in seconds for main section"},
{"label": "FOM", "key": "fom", "tooltip": "Figure of Merit - Benchmark performance metric value with its unit when available"},
{"label": "FOM version", "key": "fom_version", "tooltip": "Version identifier for the FOM measurement section - helps identify which code region was measured when users modify the timing boundaries"},
{"label": "SYSTEM", "key": "system", "tooltip": "Computing system name"},
{"label": "Nodes", "key": "nodes"},
Expand All @@ -289,7 +289,7 @@ def test_existing_row_fields_preserved(self, flask_app, tmp_dir):
uid = str(uuid.uuid4())
_write_json(tmp_dir, f"result_20250101_120000_{uid}.json", {
"code": "mycode", "system": "mysys", "Exp": "myexp",
"FOM": 99.9, "FOM_version": "v1", "node_count": 4,
"FOM": 99.9, "FOM_unit": "s", "FOM_version": "v1", "node_count": 4,
"numproc_node": "48", "nthreads": "12",
})

Expand All @@ -302,6 +302,7 @@ def test_existing_row_fields_preserved(self, flask_app, tmp_dir):
assert row["system"] == "mysys"
assert row["exp"] == "myexp"
assert row["fom"] == 99.9
assert row["fom_unit"] == "s"
assert row["timestamp"] == "2025-01-01 12:00:00"
assert row["numproc_node"] == "48"
assert row["nthreads"] == "12"
Expand Down
1 change: 1 addition & 0 deletions result_server/utils/result_table_rows.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ def build_result_table_row(json_filename, result_data, padata_filenames):
"code": result_data.get("code", "N/A"),
"exp": result_data.get("Exp", "N/A"),
"fom": result_data.get("FOM", "N/A"),
"fom_unit": result_data.get("FOM_unit") or "",
"fom_version": result_data.get("FOM_version", "N/A"),
"system": result_data.get("system", "N/A"),
"nodes": result_data.get("node_count", "N/A"),
Expand Down
2 changes: 1 addition & 1 deletion result_server/utils/results_loader.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
{"label": "CODE", "key": "code"},
{"label": "Branch/Hash", "key": "source_hash", "tooltip": "Source code branch name and short commit hash (git) or short md5sum (file archive)"},
{"label": "Exp", "key": "exp", "tooltip": "Experimental conditions (filtered by CODE)"},
{"label": "FOM", "key": "fom", "tooltip": "Figure of Merit - Benchmark performance metric value, typically elapsed time in seconds for main section"},
{"label": "FOM", "key": "fom", "tooltip": "Figure of Merit - Benchmark performance metric value with its unit when available"},
{"label": "FOM version", "key": "fom_version", "tooltip": "Version identifier for the FOM measurement section - helps identify which code region was measured when users modify the timing boundaries"},
{"label": "SYSTEM", "key": "system", "tooltip": "Computing system name"},
{"label": "Nodes", "key": "nodes"},
Expand Down
17 changes: 15 additions & 2 deletions scripts/bk_functions.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
#
# Named arguments:
# --fom <value> (required, numeric)
# --fom-unit <value> (recommended, no spaces; e.g. s, GB/s, GFLOPS)
# --fom-version <value> (optional)
# --exp <value> (optional)
# --nodes <value> (optional)
Expand All @@ -18,14 +19,16 @@
# Unknown arguments are silently ignored for future extensibility.
#
# Output format (omitted optional args produce no key:value pair):
# FOM:<value> [FOM_version:<version>] [Exp:<experiment>] [node_count:<nodes>]
# [numproc_node:<numproc_node>] [nthreads:<nthreads>] [confidential:<value>]
# FOM:<value> [FOM_unit:<unit>] [FOM_version:<version>] [Exp:<experiment>]
# [node_count:<nodes>] [numproc_node:<numproc_node>] [nthreads:<nthreads>]
# [confidential:<value>]
#
# Exit codes:
# 0 - success
# 1 - missing or invalid --fom
bk_emit_result() {
_bk_fom=""
_bk_fom_unit=""
_bk_fom_version=""
_bk_exp=""
_bk_nodes=""
Expand All @@ -45,6 +48,12 @@ bk_emit_result() {
_bk_fom="$1"
_bk_fom_set=1
;;
--fom-unit)
shift
if [ $# -gt 0 ]; then
_bk_fom_unit="$1"
fi
;;
--fom-version)
shift
if [ $# -gt 0 ]; then
Expand Down Expand Up @@ -136,6 +145,10 @@ bk_emit_result() {
# Build output line
_bk_output="FOM:${_bk_fom}"

if [ -n "$_bk_fom_unit" ]; then
_bk_output="${_bk_output} FOM_unit:${_bk_fom_unit}"
fi

if [ -n "$_bk_fom_version" ]; then
_bk_output="${_bk_output} FOM_version:${_bk_fom_version}"
fi
Expand Down
10 changes: 9 additions & 1 deletion scripts/result.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ build_source_info_block() {
source_info_block=$(build_source_info_block)

# Function to write a Result_JSON file for one FOM block
# Arguments: $1=index, uses global vars: code, system, fom, fom_version, exp, node_count, numproc_node, description, confidential, sections_json, overlaps_json
# Arguments: $1=index, uses global vars: code, system, fom, fom_unit, fom_version, exp, node_count, numproc_node, description, confidential, sections_json, overlaps_json
write_result_json() {
local idx="$1"
local fom_breakdown_block=""
Expand Down Expand Up @@ -252,6 +252,7 @@ write_result_json() {
"code": "$code",
"system": "$system",
"FOM": "$fom",
"FOM_unit": "$fom_unit",
"FOM_version": "$fom_version",
"Exp": "$exp",
"node_count": "$node_count",
Expand Down Expand Up @@ -283,6 +284,7 @@ EOF
i=0
in_fom_block=false
fom=""
fom_unit=""
fom_version="null"
exp="null"
description="null"
Expand All @@ -308,6 +310,12 @@ while IFS= read -r line; do
fom=null
fi

if echo "$line" | grep -q 'FOM_unit:'; then
fom_unit=$(echo "$line" | grep -Eo 'FOM_unit:[ ]*[^ ]*' | head -n1 | awk -F':' '{print $2}' | sed 's/^ *//')
else
fom_unit=""
fi

node_count_line=$(echo $line | grep -Eo 'node_count:[ ]*[0-9]*' | head -n1 | awk -F':' '{print $2}' | sed 's/^ *//')
if [ -n "$node_count_line" ]; then
node_count=${node_count_line}
Expand Down
Loading
Loading