Add device/edge CPU, network latency, switch penalty to offloading_decisions

.gitignore

@@ -112,4 +112,7 @@ src/server/mosquitto_broker/data/
 datasets/
 sciot-fomo-models-export/
 
-data/models/debug/*.json
+data
+client-sciot.json
+.python-version
+uv.lock

DEPENDENCY_PROFILES.md

@@ -159,6 +159,16 @@ uv sync --extra client-static --no-dev
 
 After the second command, the environment represents the static client rather than a combined server/client installation. Specify multiple `--extra` arguments when both are needed.
 
+## Dependency on specific python versions
+On some hardware, there may be problems in running the dependency chain from `uv`. In case that happens, it is possible to pin the python version to a specific one (e.g. 3.12) with the following commands:
+
+```bash
+uv python install 3.12
+uv python pin 3.12
+rm -rf .venv
+uv sync
+```
+
 ## Running Server and Client from One Checkout
 
 Use named project environments to keep the installations fully isolated:

docs/CONFIGURATION.md

@@ -77,19 +77,20 @@ The server writes inference-cycle CSV output under
 mirrored to both CSV and JSONL:
 
 - `data/results/offloading_decisions*.csv`
-- `data/results/offloading_decisions_<inference_timestamp>.jsonl`
+- `data/results/offloading_decisions_<first_inference_timestamp>.jsonl`
 
 Each output stream can be enabled or disabled as needed (both default to `true`);
 a disabled stream skips its event construction entirely, so there is no runtime
 cost. Stable scalar metrics are emitted as normal CSV columns; variable-length
-values such as per-layer timing lists and offloading candidates are preserved as
-JSON-encoded CSV cells and as native JSON values in JSONL:
+values such as per-layer timing and offloading cost lists are preserved as
+JSON-encoded CSV cells. JSONL batches keep shared layer sizes once in file
+metadata and store scalar per-inference compute timing fields:
 
 ```yaml
 evaluation:
   outputs:
     inference_cycles: true       # inference_cycles_*.csv (structured evaluation)
-    offloading_decisions: true   # decision CSV and inference-timestamped JSONL mirror
+    offloading_decisions: true   # decision CSV and compact JSONL batches
 ```
 
 ### Splitting
@@ -107,10 +108,12 @@ evaluation:
 A new CSV segment is opened at the **first** criterion that trips. CSV segment
 filenames embed a zero-padded index and segment start time, e.g.
 `offloading_decisions.seg0001_20260706_101500.csv`. The JSONL mirror does not use
-CSV-style segments: every offloading decision uses its own inference decision
-timestamp directly in the filename:
-`offloading_decisions_20260706_101500.jsonl`. Fractional seconds are added only
-if two decisions have the same second-level inference timestamp.
+CSV-style segments: it writes compact batches named from the first inference
+decision timestamp, for example `offloading_decisions_20260706_101500.jsonl`.
+Each JSONL batch starts with shared metadata such as layer sizes and then appends
+per-inference rows until adding another row would exceed 950 KiB. Fractional
+seconds and numeric suffixes are added only when a new batch would otherwise
+collide with an existing filename.
 
 ### Firestore publishing
 
@@ -139,14 +142,16 @@ Documents are written below:
 - `sciot_results/{run_document}/offloading_decisions/{document_id}`
 
 For offloading decisions, the Firestore document id and payload reference the
-same inference-timestamped JSONL file written locally, such as
+local JSONL batch file containing the buffered inferences, such as
 `offloading_decisions_20260630_105214.jsonl`, via
-`offloading_decisions_jsonl_file` and `offloading_decisions_jsonl_path`.
+`offloading_decisions_jsonl_file` and `offloading_decisions_jsonl_path`. The
+latest pending offloading-decision JSONL file is uploaded as `jsonl_content` on
+each flush and deleted locally after the Firestore write succeeds.
 
 Publishing is buffered and best-effort: local CSV files are written immediately,
-then queued Firestore rows are flushed every `flush_interval_seconds` seconds
-(five minutes by default). Firestore errors are logged and do not stop local CSV
-writes or inference.
+then queued Firestore rows and the latest pending offloading-decision batch are
+flushed every `flush_interval_seconds` seconds (five minutes by default).
+Firestore errors are logged and do not stop local CSV writes or inference.
 
 Static-client validation covers:
 

docs/DASHBOARD_TELEMETRY.md

@@ -22,7 +22,7 @@ Both files are generated runtime artifacts and should not be committed.
 | evaluations CSV | `model_dir` | string | Model profile/directory when available. |
 | decision CSV | `server_start_timestamp` | ISO-8601 timestamp | Time at which the server process started. |
 | decision CSV | `offloading_algorithm_class` | string | Concrete offloading algorithm class used for the decision. |
-| decision CSV | `selected_layer` | layer index | Layer selected for the next request. |
+| decision CSV | `offloading_layer_index` | layer index | Layer selected for the next request. |
 | decision CSV | `estimated_total_ms` | milliseconds | Estimated cost for the selected strategy after dashboard normalization. |
 | decision CSV | `observed_total_time_ms` | milliseconds | Observed end-to-end time for the processed request. |
 | decision CSV | `error_ms` | milliseconds | Observed total minus selected estimated total. |
@@ -37,9 +37,9 @@ The current dashboard is intentionally transitional:
 - generated output stays under `data/results/`.
 
 New telemetry should prefer versioned CSV rows with explicit units. Stable scalar
-metrics should be normal CSV columns; variable-length structures should be
-JSON-encoded inside CSV cells so all per-cycle details stay available to
-analysis and dashboard consumers.
+metrics should be normal CSV columns; per-layer vectors should be JSON-encoded
+inside CSV cells so all per-cycle details stay available to analysis and
+dashboard consumers.
 
 ## Startup command
 

docs/FIRESTORE_TELEMETRY.md

@@ -43,21 +43,28 @@ The default layout groups telemetry by server run:
 `inference_cycles_<timestamp>.csv` filename. Inference-cycle document ids are
 derived from the event schema, request id, and event timestamp so repeated
 publishes overwrite the same telemetry row instead of creating random duplicates.
-Offloading-decision document ids start with the concrete inference-timestamped
-JSONL filename, for example `offloading_decisions_20260630_105214.jsonl`,
-followed by the request/timestamp context.
+Offloading-decision document ids are the concrete JSONL batch filename, for
+example `offloading_decisions_20260630_105214.jsonl`. Each Firestore flush keeps
+only the latest pending offloading-decision batch, so multiple inferences written
+before the same flush produce one Firestore write for the current JSONL file.
 
 Each offloading-decision payload also includes:
 
 - `offloading_decisions_jsonl_file`, e.g.
   `offloading_decisions_20260630_105214.jsonl`
-- `offloading_decisions_jsonl_path`, the local path written by the server
+- `offloading_decisions_jsonl_path`, the local batch path written by the server
+- `last_request_id` and `last_event_timestamp`, identifying the latest inference
+  appended to that batch before upload
+- `jsonl_content`, the uploaded batch file content
 
 ## Runtime behavior
 
-- Local CSV and inference-timestamp-named JSONL files are still written.
+- Local CSV and first-inference-timestamp-named JSONL batch files are still written.
 - Firestore events are queued in memory and flushed every
   `flush_interval_seconds` seconds, which defaults to five minutes.
+- Offloading decisions are uploaded as the latest JSONL batch file in the queue,
+  then the uploaded local JSONL file is deleted after the Firestore write
+  succeeds.
 - Firestore failures are logged and do not block inference.
 - If the Firestore package is missing from the runtime environment, publishing
   is disabled after a warning.

docs/OFFLOADING_DECISION_EVENTS.md

@@ -4,13 +4,16 @@ SCIoT records each offloading decision in two local representations so runtime
 behavior can be inspected without blocking inference:
 
 - CSV segments derived from `data/results/offloading_decisions.csv`;
-- JSON Lines files named
+- JSON Lines batch files named
   `data/results/offloading_decisions_<inference_timestamp>.jsonl`.
 
-Each row/line is the same `offloading-decision.v1` event. Stable scalar values
-are emitted as normal CSV columns; the variable-length `candidates` list is
-preserved as a JSON-encoded CSV cell. JSONL retains the complete native JSON
-structure.
+Each CSV row is an expanded `offloading-decision.v1` event. Stable scalar values
+are emitted as normal CSV columns; per-layer cost vectors are JSON-encoded CSV
+cells. JSONL uses a compact `offloading-decision-batch.v1` representation: the
+first line stores repeated file-level metadata such as `model_dir`, `strategy`,
+and `layer_sizes_bytes`, while later lines store individual inference records.
+Files are capped at 950 KiB by default; once adding another inference would
+exceed that limit, a new timestamped JSONL file is opened.
 
 ## Schema: `offloading-decision.v1`
 
@@ -24,35 +27,70 @@ structure.
 | `model_dir` | string | Model artifact directory/profile used for the decision. |
 | `request_id` | string | Correlation ID linking the client request, prediction, and decision event. |
 | `strategy` | string | Algorithm family, such as `static` or `adaptive_risk`. |
-| `selected_layer` | integer | Selected offloading layer; `-1` means local-only fallback. |
+| `offloading_layer_index` | integer | Selected offloading layer; `-1` means local-only fallback. |
 | `selection_reason` | string | Why the layer was selected, such as `lowest_estimated_cost`, `forced_local_inference`, or `fallback_missing_model_metadata`. |
 | `avg_speed_bytes_per_second` | number | Network speed estimate used by the algorithm. |
-| `candidates` | array | Candidate strategies and cost components considered by the algorithm. |
+| `device_cpu_percent` | number | Device-side CPU load (0-100) reported by the client for this request. |
+| `edge_cpu_percent` | number | Edge (server host) CPU load (0-100) sampled when the decision was made. |
+| `network_latency_ms` | number | Observed network round-trip time for this request, in milliseconds. |
+| `switch_penalty` | number | Hysteresis penalty in milliseconds applied for switching away from the previously selected layer; `0` for strategies without switch hysteresis (e.g. `static`). |
+| `cpu_idle` | number | Edge host cumulative idle CPU time from `psutil.cpu_times()`. |
+| `cpu_user` | number | Edge host cumulative user CPU time from `psutil.cpu_times()`. |
+| `cpu_system` | number | Edge host cumulative system CPU time from `psutil.cpu_times()`. |
+| `cpu_count` | integer | Edge host logical CPU count from `psutil.cpu_count()`. |
+| `cpu_freq` | array | Per-CPU current frequencies from `psutil.cpu_freq(percpu=True)`. |
+| `load_avg_1_min` | number | One-minute load average from `psutil.getloadavg()`. |
+| `load_avg_5_min` | number | Five-minute load average from `psutil.getloadavg()`. |
+| `load_avg_15_min` | number | Fifteen-minute load average from `psutil.getloadavg()`. |
+| `virtual_memory_total` | integer | Total memory in bytes from `psutil.virtual_memory()`. |
+| `virtual_memory_available` | integer | Available memory in bytes from `psutil.virtual_memory()`. |
+| `device_compute_cost` | array | Device-side compute cost in milliseconds by layer index. `device_compute_cost[12]` is the value for layer 12. |
+| `network_cost` | array | Network transfer cost in milliseconds by layer index, computed as `(layer_size_bytes / avg_speed_bytes_per_second) * 1000` when layer size metadata is available. This array is stored in CSV and reconstructed by readers for batched JSONL. |
+| `edge_compute_cost` | array | Edge-side compute cost in milliseconds by layer index. |
+| `estimated_total_cost` | array | Total estimated cost in milliseconds by layer index (`device_compute_cost + network_cost + edge_compute_cost` when all components are available). |
 | `observed_*_ms` | number | Observed request timing columns in milliseconds. |
 
-Each candidate includes:
+The selected layer is available as the top-level `offloading_layer_index` field.
+Persisted events no longer include the verbose
+`candidates` list and never carry a redundant `selected` boolean. Empty or
+unevaluated layer slots are written as `null` in the cost arrays. Readers still
+recognize older candidate-list documents for backward compatibility.
+Cost-array entries are per-layer values only; they do not include the cost of
+previous device layers or later edge layers.
+Persisted time values use milliseconds and are rounded to at most three decimal
+places.
 
-| Field | Type | Description |
-| --- | --- | --- |
-| `strategy` | string | Candidate strategy, such as `split`, `device_only`, or `edge_only`. |
-| `offloading_layer_index` | integer | Candidate layer; `-1` represents edge-only/local fallback semantics depending on strategy. |
-| `device_compute_cost` | number | Estimated device-side compute cost. |
-| `network_cost` | number | Estimated network transfer cost. |
-| `edge_compute_cost` | number | Estimated edge-side compute cost. |
-| `estimated_total_cost` | number | Total estimated cost used for comparison. |
-| `considered_for_selection` | boolean | Whether the candidate participates in the final choice. |
-| `selected` | boolean | Whether this candidate matches the selected layer. |
-
-Adaptive candidates may also include `adaptive_score`, `risk_penalty`,
-`switch_penalty`, `network_speed_ewma`, `network_uncertainty`,
-`device_uncertainty`, `edge_uncertainty`, `device_cpu_percent`,
-`device_load_factor`, `raw_device_compute_cost`, and `device_load_penalty`.
+## JSONL Batch Schema: `offloading-decision-batch.v1`
+
+Each batch file begins with one metadata record:
+
+```json
+{"schema_version":"offloading-decision-batch.v1","event_schema_version":"offloading-decision.v1","server_start_timestamp":"2026-07-07T10:00:00+02:00","offloading_algorithm_class":"OffloadingAlgo","model_dir":"FOMO_96_CUT","strategy":"static","layer_sizes_bytes":[100.0,200.0,300.0]}
+```
+
+Subsequent lines are inference records:
+
+```json
+{"timestamp":"2026-06-26T00:00:00+00:00","device_id":"device_01","request_id":"ABCD","offloading_layer_index":12,"selection_reason":"lowest_estimated_cost","avg_speed_bytes_per_second":12345.0,"network_latency_ms":9.2,"cpu_idle":100.111,"cpu_user":20.222,"cpu_system":3.333,"cpu_count":8,"cpu_freq":[2400.123,2399.988],"load_avg_1_min":1.234,"load_avg_5_min":2.346,"load_avg_15_min":3.457,"virtual_memory_total":16000,"virtual_memory_available":4000,"acquisition_time_ms":15.0,"device_compute_time_ms":20.0,"edge_compute_time_ms":30.0,"network_time_ms":10.0,"total_time_ms":75.0}
+```
+
+In JSONL batches, device and edge compute are stored once as
+`device_compute_time_ms` and `edge_compute_time_ms`. The duplicate
+`device_compute_cost` and `edge_compute_cost` aliases, plus `estimated_total_cost`,
+are intentionally omitted. `network_cost` is also not repeated when
+`layer_sizes_bytes` is available in the metadata. Consumers can compute
+`network_cost[layer]` as
+`(layer_sizes_bytes[layer] / avg_speed_bytes_per_second) * 1000`. Observed timing
+values are flattened onto the inference record as top-level `*_time_ms` fields,
+including `acquisition_time_ms`; the old `observed` wrapper is not written to
+JSONL batches. The Python reader `iter_offloading_decisions` performs network-cost
+reconstruction for batched JSONL.
 
 ## Example
 
 ```csv
-schema_version,server_start_timestamp,offloading_algorithm_class,timestamp,device_id,model_dir,request_id,strategy,selected_layer,selection_reason,avg_speed_bytes_per_second,candidates,observed_total_time_ms
-offloading-decision.v1,2026-07-07T10:00:00+02:00,OffloadingAlgo,2026-06-26T00:00:00+00:00,device_01,FOMO_96_CUT,ABCD,static,12,lowest_estimated_cost,12345.0,"[{""strategy"":""split"",""offloading_layer_index"":12,""estimated_total_cost"":0.06,""selected"":true}]",75.0
+schema_version,server_start_timestamp,offloading_algorithm_class,timestamp,device_id,model_dir,request_id,strategy,offloading_layer_index,selection_reason,avg_speed_bytes_per_second,device_cpu_percent,edge_cpu_percent,network_latency_ms,switch_penalty,cpu_idle,cpu_user,cpu_system,cpu_count,cpu_freq,load_avg_1_min,load_avg_5_min,load_avg_15_min,virtual_memory_total,virtual_memory_available,device_compute_cost,network_cost,edge_compute_cost,estimated_total_cost,observed_total_time_ms
+offloading-decision.v1,2026-07-07T10:00:00+02:00,OffloadingAlgo,2026-06-26T00:00:00+00:00,device_01,FOMO_96_CUT,ABCD,static,12,lowest_estimated_cost,12345.0,34.5,18.0,9.2,0.0,100.111,20.222,3.333,8,"[2400.123,2399.988]",1.234,2.346,3.457,16000,4000,"[null,null,20.0]","[null,null,10.0]","[null,null,30.0]","[null,null,60.0]",75.0
 ```
 
 ## Privacy and retention
@@ -87,12 +125,14 @@ than a single file. When both split criteria are `0`, the legacy behaviour appli
 instead: a single `offloading_decisions.csv` rotated by size, keeping up to three
 backups (`.csv.1`, `.csv.2`, `.csv.3`).
 
-JSONL mirror files always put the inference decision timestamp immediately after
-the basename, for example `offloading_decisions_20260630_105214.jsonl`. Each
-decision opens its own JSONL file, so Firestore can reference the exact local
-artifact for that inference. If two decisions share the same second-level
-timestamp, fractional seconds are added to avoid a collision. Legacy `.jsonl.1`,
-`.jsonl.2`, and `.jsonl.3` names are not produced.
+JSONL batch files put the first inference decision timestamp immediately after
+the basename, for example `offloading_decisions_20260630_105214.jsonl`. Multiple
+inferences with matching file-level metadata append to that file until the next
+row would exceed 950 KiB. Metadata changes, such as a different model directory,
+strategy, algorithm class, server start timestamp, or layer-size vector, also
+start a new file. If a new batch would collide with an existing second-level
+timestamped filename, fractional seconds and then a numeric suffix are added.
+Legacy `.jsonl.1`, `.jsonl.2`, and `.jsonl.3` names are not produced.
 
 If a segment cannot be written, the background I/O worker logs the failure (with a
 traceback) and continues processing later tasks.
@@ -108,7 +148,7 @@ python scripts/analysis/offloading_decision_summary.py \
   --output data/results/offloading_decision_summary.csv
 ```
 
-The summary includes selected layer, strategy, estimated total milliseconds,
+The summary includes offloading layer, strategy, estimated total milliseconds,
 observed total milliseconds, and error milliseconds.
 
 > Note: with segment splitting enabled the base `offloading_decisions.csv` is not

docs/config/server.full.yaml

@@ -95,9 +95,10 @@ verbose: false
 debug_cprofiler: false
 
 # Splitting of evaluation CSV outputs (offloading decisions and
-# inference_cycles_*.csv). JSONL decision mirrors use one inference-timestamped
-# file per decision. A new CSV segment file is opened at the first criterion
-# that trips; segments are never deleted. 0 disables a criterion.
+# inference_cycles_*.csv). JSONL decision mirrors aggregate multiple inferences
+# per file until the size cap and can be deleted after successful Firestore
+# upload. A new CSV segment file is opened at the first criterion that trips.
+# 0 disables a criterion.
 evaluation:
   # Enable/disable each CSV output as needed (default: both true).
   outputs:

pyproject.toml

@@ -15,7 +15,7 @@ readme = "README.md"
 license = { file = "LICENSE" }
 keywords = ["flask-app", "esp32", "split-computing"]
 
-requires-python = ">=3.13"
+requires-python = ">=3.12"
 dependencies = [
     # Shared by the server and every Python client profile.
     "numpy",
@@ -135,6 +135,7 @@ enabled = true
 [tool.uv]
 required-environments = [
     "sys_platform == 'darwin' and platform_machine == 'arm64'",
+    "sys_platform == 'darwin' and platform_machine == 'x86_64'",
     "sys_platform == 'linux' and platform_machine == 'x86_64'",
     "sys_platform == 'linux' and platform_machine == 'aarch64'",
 ]