audio: tensorflow: keyword detect - #11092
Draft
singalsu wants to merge 12 commits into
Draft
Conversation
singalsu
commented
Aug 14, 2026
| CONFIG_COMP_GOOGLE_RTC_AUDIO_PROCESSING=m | ||
| CONFIG_GOOGLE_RTC_AUDIO_PROCESSING_MOCK=y | ||
| CONFIG_COMP_STFT_PROCESS=y | ||
| CONFIG_SOF_STAGING=y |
Collaborator
Author
There was a problem hiding this comment.
Note: Untested by me.
| CONFIG_COMP_MULTIBAND_DRC=y | ||
| CONFIG_COMP_VOLUME_WINDOWS_FADE=y | ||
| CONFIG_FORMAT_CONVERT_HIFI3=n | ||
| CONFIG_SOF_STAGING=y |
Collaborator
Author
There was a problem hiding this comment.
Note: Build works only with gcc.
| CONFIG_COMP_ARIA=y | ||
| CONFIG_COMP_CROSSOVER=y | ||
| CONFIG_COMP_DRC=y | ||
| CONFIG_COMP_TESTER=m |
Collaborator
Author
There was a problem hiding this comment.
This patch is here to enable testing, but can't be merged due to modifying too much normal build. Also this version failed to boot on ARL-S.
singalsu
commented
Aug 14, 2026
See thesofproject#10562 Previously binding two DP (Data Processing) scheduled components was rejected with IPC4_INVALID_REQUEST. This patch adds support for DP-to-DP binding by creating a dual ring buffer configuration where each DP module gets its own ring buffer on either side of the intermediate comp_buffer. Data flow for DP-to-DP: src_DP -> ring_buf_src -> comp_buffer -> ring_buf_sink -> sink_DP Changes in helper.c: - Remove the DP-to-DP bind rejection in ipc_comp_connect(). - Add src_is_dp, sink_is_dp, and dp_to_dp flags to detect the DP-to-DP case. - Create a second ring_buffer allocated from the source module's mod_alloc_ctx for the source side of the comp_buffer. - Refcount the DP vregion for each created ring_buffer via vregion_get(), with a NULL alloc guard. Changes in audio_buffer.c: - Change audio_buffer_attach_secondary_buffer() from a global rejection to per-side checks, allowing both secondary_buffer_sink and secondary_buffer_source to be set simultaneously. - Add a dual-secondary sync path in audio_buffer_sync_secondary_buffer() that cascades data through: input ring_buffer -> comp_buffer -> output ring_buffer, with rate-limiting applied on the output side. Changes in ring_buffer.c: - Release the DP vregion in ring_buffer_free() via vregion_put() and free the mod_alloc_ctx when the refcount reaches zero, matching the pattern used in comp_buffer_free(). Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
See thesofproject#11097 The two stft_process source files pulled in libc's <assert.h>, which under a newlib-based Zephyr build (e.g. MTL with COMMON_LIBC_MALLOC_ARENA_SIZE set) expands assert() to a __assert_no_args() call whose implementation is not linked into the SOF firmware image. Enabling COMP_STFT_PROCESS on such a build therefore fails at link time with an "undefined reference to __assert_no_args" error. Every other SOF audio component that uses assert() includes <rtos/panic.h> instead, which maps assert() to Zephyr's __ASSERT_NO_MSG in firmware builds and to sof_panic() in the posix testbench. Do the same in stft_process-generic.c and stft_process-hifi3.c. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
singalsu
force-pushed
the
tflm_mfcc_development
branch
from
August 14, 2026 09:45
8992437 to
15aff1e
Compare
Wire the TFLM sources into the SOF firmware build for statically linked firmware. This unblocks non-LLEXT targets where TFLM has to live in the base image, using the Zephyr SDK GCC HiFi3 toolchain. Also add the requantize helper in speech.cc that unpacks the IEEE 754 mantissa directly instead of calling frexpf, so the file pulls in no libm symbols on the minimal-libc SOF build. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
This model was trained with scripts
- sof_tflm_generate_keyword_dataset.sh
with parameters MAX_SAMPLES=1000.
- sof_tflm_train_pipeline.sh
with parameters N_SILENCE=1500, N_UNKNOWN=4000, EPOCHS=40
The verification report by sof_tflm_verify.py is:
samples: 2751 overall accuracy: 0.9807
class support precision recall f1
--------------------------------------------
silence 225 0.9694 0.9867 0.9780
unknown 1176 0.9773 0.9881 0.9827
banana 450 0.9779 0.9822 0.9800
mango 450 0.9863 0.9622 0.9741
orange 450 0.9932 0.9756 0.9843
confusion matrix (rows=true, cols=pred):
silence unknown banana mango orange
silence 222 3 0 0 0
unknown 7 1162 2 2 3
banana 0 8 442 0 0
mango 0 9 8 433 0
orange 0 7 0 4 439
Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
singalsu
force-pushed
the
tflm_mfcc_development
branch
from
August 14, 2026 12:14
15aff1e to
80b518b
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a TensorFlow Lite Micro (TFLM) keyword-detection (“Wake-on-Voice”) path integrated with MFCC feature extraction and KPB draining, including new topology2 overlays for HDA and SoundWire capture sources plus supporting build, scheduling, and tooling updates.
Changes:
- Add topology2 capture overlays (HDA analog, SDW jack, SDW DMIC) that tap an existing capture endpoint into
SRC -> KPB -> (WoV drain + MFCC/micsel/TFLM detect)and register new topology build targets. - Add a new
tflmclytopology widget class and pipeline templates (host-gateway-micsel-mfcc-tflm-capture,src-kpb-be) and MFCC mel40 blob variants used by the WoV path and benchmarks. - Update firmware/runtime to support DP scheduler init, DP-to-DP bindings via dual ring-buffers, and replace the TFLM model/label plumbing with retraining scripts and new generated headers.
Reviewed changes
Copilot reviewed 49 out of 50 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/topology/topology2/sof-hda-generic.conf | Adds optional HDA mic WoV overlay include and required include files |
| tools/topology/topology2/platform/intel/sdw-jack-tflm-kpb.conf | New SDW jack WoV branch overlay using SRC+KPB and MFCC/TFLM detect |
| tools/topology/topology2/platform/intel/sdw-dmic-tflm-kpb.conf | New SDW DMIC WoV branch overlay using SRC+KPB and MFCC/TFLM detect |
| tools/topology/topology2/platform/intel/sdw-dmic-audio-feature.conf | Routes MFCC output through new tflmcly widget for audio-feature capture |
| tools/topology/topology2/platform/intel/hda-mic-tflm-kpb.conf | New HDA analog WoV overlay (adds WoV drain + detect PCMs) |
| tools/topology/topology2/platform/intel/dmic1-mfcc.conf | Removes redundant MFCC class include (now assumed included by parent) |
| tools/topology/topology2/include/pipelines/cavs/src-kpb-be.conf | New pipeline class providing SRC->KPB backend capture branch |
| tools/topology/topology2/include/pipelines/cavs/host-gateway-tdfb-drc-capture.conf | Adjusts DRC widget pin count for capture pipeline |
| tools/topology/topology2/include/pipelines/cavs/host-gateway-micsel-mfcc-tflm-capture.conf | New detect pipeline class (micsel->mfcc->tflmcly->host) |
| tools/topology/topology2/include/components/tflm.conf | Defines topology2 widget class for TFLM classifier (tflmcly) |
| tools/topology/topology2/include/components/mfcc/mel40.conf | Adds 40-bin MFCC blob variant |
| tools/topology/topology2/include/components/mfcc/mel40_compress.conf | Adds compressed 40-bin MFCC blob variant for WoV path |
| tools/topology/topology2/include/components/kpb.conf | Updates KPB UUID used by topology2 widget definition |
| tools/topology/topology2/include/common/common_definitions.conf | Adds new feature flags for WoV overlays (SDW jack, SDW DMIC, HDA mic) |
| tools/topology/topology2/include/common/abi.conf | Adds SOF ABI manifest blob include |
| tools/topology/topology2/include/bench/mfcc_controls_playback.conf | Adds mel40 option to MFCC bench controls |
| tools/topology/topology2/include/bench/mfcc_controls_capture.conf | Adds mel40 option to MFCC bench controls |
| tools/topology/topology2/development/tplg-targets.cmake | Registers new HDA and SDW WoV topology2 build targets |
| tools/topology/topology2/development/tplg-targets-bench.cmake | Adds mel40 MFCC bench target/config |
| tools/topology/topology2/cavs-sdw.conf | Includes new pipeline classes and adds WoV overlay include keys |
| tools/topology/topology2/cavs-benchmark-hda.conf | Adds mfccmel40 bench configs via existing mfccmel includes |
| tools/rimage/config/tgl.toml.h | Includes TFLM component TOML when enabled |
| src/platform/intel/cavs/platform.c | Initializes DP scheduler when configured |
| src/ipc/ipc4/helper.c | Enables DP-to-DP binding with dual ring-buffers and vregion refcounting |
| src/audio/tensorflow/tune/sof_tflm_verify.py | New off-device verifier for quantized model accuracy metrics |
| src/audio/tensorflow/tune/sof_tflm_train.py | New training script to emit drop-in model/labels C artifacts |
| src/audio/tensorflow/tune/sof_tflm_train_pipeline.sh | New end-to-end dataset→features→train pipeline wrapper |
| src/audio/tensorflow/tune/sof_tflm_prepare_silence_unknown.sh | New Speech Commands v2 silence/unknown dataset prep script |
| src/audio/tensorflow/tune/sof_tflm_generate_keyword_dataset.sh | New Piper sample-generator keyword dataset synthesis script |
| src/audio/tensorflow/tune/sof_tflm_generate_keyword_dataset_piper_tts.sh | New single-speaker piper-tts keyword dataset synthesis script |
| src/audio/tensorflow/tune/sof_tflm_dataset.py | New loader for SOF MFCC hop wire-format into model windows |
| src/audio/tensorflow/tune/sof_mfcc_extract_features.sh | New testbench-based feature extraction runner producing .raw hops |
| src/audio/tensorflow/tflm-classify.c | Major rework: MFCC-hop parsing, requantization, inference loop, KPB trigger plumbing |
| src/audio/tensorflow/speech.h | Switches labels/categories to generated header; adds quantization params and arena query APIs |
| src/audio/tensorflow/speech.cc | Switches to new model data header; expands op resolver; exposes input quant params and arena usage |
| src/audio/tensorflow/sof_tflm_quantized_model_data.h | Adds generated model header for new model symbol |
| src/audio/tensorflow/sof_tflm_labels.h | Adds generated labels header used by runtime |
| src/audio/tensorflow/README.md | Large documentation update describing WoV/TFLM architecture and training flow |
| src/audio/tensorflow/micro_speech_quantized_model_data.h | Removes old model header include |
| src/audio/tensorflow/CMakeLists.txt | Updates build gating for nnlib-hifi4, adds required TFLM kernels, and toolchain libc shim for static builds |
| src/audio/stft_process/stft_process-hifi3.c | Replaces assert.h include with panic header |
| src/audio/stft_process/stft_process-generic.c | Replaces assert.h include with panic header |
| src/audio/mfcc/tune/setup_mfcc.m | Adds mel40 and mel40_compress MFCC blob generation |
| src/audio/mfcc/mfcc.c | Adjusts DP prepare sizing to consume full MFCC hop per DP wake |
| src/audio/buffers/ring_buffer.c | Adds vregion_put on ring_buffer free to match new refcounting |
| src/audio/buffers/audio_buffer.c | Allows secondary buffers on both sides and adds DP-to-DP sync logic |
| scripts/tensorflow-clone.sh | Makes BASE_DIR configurable via environment override |
| app/boards/intel_adsp_cavs25.conf | Enables C++/TFLM static build and increases heap/malloc arena sizes |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Introduce a TensorFlow Lite Micro-based keyword-detection component that runs an int8-quantized micro-speech model on the mel features from the MFCC component. The key-phrase buffer (KPB) sits ahead of MFCC in the pipeline; on a positive keyword the TFLM component posts a KPB trigger notification so the host can drain the buffered audio. Also includes tensor byte-sizing fixes for the int8 input path, per-hop mel feature requantization that strips the MFCC header and avoids float math on the DSP, and stream-shutdown event logging so the host sees a clean end-of-capture on trigger. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
The on-device TFLM wake-word path (KPB -> SRC -> MFCC -> tflmcly) needs a 40-bin/20 ms-hop mel spectrogram to match TFLM micro_speech's front-end shape (FEATURE_SIZE=40, FEATURE_STRIDE_MS=20, FEATURE_DURATION_MS=30). Two variants are required: mel40.conf for the plain audio-buffer path used for off-target sanity checking and mel40_compress.conf for the compress PCM output actually consumed by the wake-word inference component. Add both export blocks to setup_mfcc.m so the blobs stay in lock-step under any future front-end tuning change, and ship the initial generated mel40.conf and mel40_compress.conf so the Wake-on-Voice topology graphs that consume them have matching blobs to reference. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
When the MFCC component runs in the DP (Data Processing) scheduling domain, the DP thread is woken on ibs, which is roughly one FFT hop of input. Sizing the processing capacity to only dev->frames means the DP thread has to re-enter many times per LL tick to nibble through one hop, adding unnecessary scheduling overhead. Raise the max_frames argument passed to mfcc_setup() to at least cd->config->frame_shift when the component is scheduled in the DP domain, so a single DP invocation drains one whole hop. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Add topology graphs and IPC4 ABI manifest bits for the Wake-on-Voice pipeline on HDA and SoundWire jack inputs, plus a bench variant for off-target training and validation. Register the KPB IPC4 UUID, attach the mel40 MFCC binary config blob as a real config, and give the ARL target the HDA_MIC_MFCC_PARAMS it was missing. Wire the dual-path KPB routing so the always-on inference branch and the host capture branch share the same buffer. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Enable KPB, TFLM, MFCC, Gain and Volume on the cAVS 2.5 (TGL/ADL/RPL) board defconfig and size its DP scheduler heap for the TFLM working set. Register the DP scheduler in the cAVS platform init so the DP domain pipelines that host MFCC and TFLM come up cleanly, and add tflmcly.toml to the TGL rimage manifest so the signed image lists the new modules. Also make the tensorflow static CMake wiring portable across xtensa targets: hardcoded /home/lrg zephyr-sdk paths, mcpu names and include directories are replaced with SOC_TOOLCHAIN_NAME and ZEPHYR_SDK_INSTALL_DIR expansions, the -mllvm text-section-literals flag is feature-detected, and HiFi4-only kernels are gated on TENSORFLOW_HAVE_NNLIB_HIFI4. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Document the TFLM Wake-on-Voice architecture: pipeline graph, module UUIDs, topology entry points, static vs LLEXT build variants, MFCC config blob layout, KPB trigger flow, and the on-device vs off-device verification loop. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Add an end-to-end training pipeline for TFLM wake-word models
under src/audio/mfcc/tune and src/audio/tensorflow/tune:
- Piper-TTS driven keyword dataset generator with per-file
Gaussian gain jitter capped 3 dB below full scale so the
augmented WAVs directly reflect the level distribution the
model will train on without clipping.
- Batch feature extractor that runs the SOF testbench MFCC on
the WAVs and writes raw mel40 features.
- Silence and unknown-class preparation from Speech Commands v2
with the same level-jitter policy (silence exempt).
- Python trainer that fits an int8-quantized DS-CNN on the mel
features, emits the tflite/C-array pair, and auto-writes a
sof_tflm_labels.h header so the runtime labels match the
training set without hand-editing.
- Off-device verifier that runs the tflite model against a
directory of WAVs and reports per-class scores.
Refreshes the tensorflow install and training recipe and fixes
int8 calibration to use the correct scale and zero-point.
Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
The existing sof_tflm_generate_keyword_dataset.sh is hard-wired to the English multi-speaker piper-sample-generator (LibriTTS-R). It cannot train a positive class in any other language because the phonemizer inside that checkpoint only knows English graphemes. Add a companion script that drives the regular piper-tts package against any single-speaker voice from rhasspy/piper-voices (Finnish fi_FI-harri, Swedish sv_SE-nst, etc.). To compensate for the missing speaker-embedding axis it (a) randomizes Piper noise-scale / noise-w per utterance while cycling several length-scale values and (b) fans each synthesized clip out into sox pitch-cents plus pitch-preserving tempo copies. Output layout, IR augmentation hook, and gain jitter match the English generator so sof_tflm_train_pipeline.sh consumes it unchanged. Update the tune README with a language-agnostic pipeline table entry, a non-English setup section. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
singalsu
force-pushed
the
tflm_mfcc_development
branch
from
August 14, 2026 15:04
80b518b to
2f16184
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 50 out of 51 changed files in this pull request and generated 1 comment.
Suppressed comments (1)
src/audio/tensorflow/README.md:293
- The README states the shipped model was retrained “currently against the hey_linux keyword”, but the generated label header checked in by this PR (sof_tflm_labels.h) contains the example labels {silence, unknown, banana, mango, orange}. Please align the documentation with the actual model/labels committed in this PR (or vice versa) so readers don’t follow the wrong training/validation expectations.
The stock model only recognizes `yes`/`no` (plus `silence`/`unknown`). The
shipped `sof_tflm_quantized_model_data.{cc,h}` was retrained end-to-end
against real SOF mel40 features (currently against the `hey_linux`
keyword) using the scripts under [./tune/](./tune/). This section
documents that exact recipe so the model can be reproduced, a different
Comment on lines
+1
to
+3
| # mfcc widget class is already included at the top of the parent | ||
| # sof-hda-generic.conf, so no need to pull include/components/mfcc.conf | ||
| # here again. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.