Skip to content

spectrogram and inversespectrogram x86 conv1d and vulkan implementation - #6908

Open
futz12 wants to merge 9 commits into
Tencent:masterfrom
futz12:stft-istft
Open

futz12 wants to merge 9 commits into
Tencent:masterfrom
futz12:stft-istft

Conversation

@futz12

@futz12 futz12 commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Implement fast STFT/ISTFT for the Spectrogram and InverseSpectrogram
layers on both x86 and Vulkan. The existing naive CPU implementations
are kept untouched as reference.

x86:

  • STFT is expressed as an embedded Convolution1D: the DFT basis is
    precomputed at load_param into a weight matrix (kernel = n_fft,
    stride = hop_len, out channels = 2 * freqs), so the whole forward is a
    single optimized conv1d instead of a per-frame FFT loop.
  • ISTFT is expressed as a 1x1 Convolution1D (2n_fft -> 2n_fft, the
    inverse-DFT basis including window), followed by overlap-add with
    window^2 normalization.

Vulkan:

  • Dedicated shaders: STFT launches one invocation per (frame, bin);
    ISTFT uses a gather-style overlap-add with window^2 normalization.

Both paths honor the existing layer parameters: center padding
(constant / replicate / reflect), onesided, power (magnitude) mode and
the normalized flag.

Files:

  • src/layer/x86/spectrogram_x86.cpp/.h, inversespectrogram_x86.cpp/.h
  • src/layer/vulkan/spectrogram_vulkan.cpp/.h, inversespectrogram_vulkan.cpp/.h
  • src/layer/vulkan/shader/spectrogram_conv.comp, inversespectrogram_conv.comp

Motivation: torchaudio-style spectrogram/ISTFT are hot spots in
voice/audio pipelines (e.g. RVC); these implementations avoid the naive
per-frame loops and keep the computation on-device.

x86: stft/istft as embedded convolution1d with precomputed dft basis weights
vulkan: dedicated shaders, per-thread (frame, bin) for stft, gather overlap-add for istft
naive implementations untouched as reference

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1be6de8ff6

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/layer/x86/spectrogram_x86.cpp
Comment thread src/layer/vulkan/shader/inversespectrogram_conv.comp Outdated
Comment thread src/layer/x86/inversespectrogram_x86.cpp Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 921a1418ea

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/layer/vulkan/inversespectrogram_vulkan.cpp Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a9b0bb916e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/layer/vulkan/spectrogram_vulkan.cpp

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 651106c5ec

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/layer/x86/spectrogram_x86.cpp Outdated
Comment thread src/layer/vulkan/spectrogram_vulkan.cpp

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 238011434e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/layer/vulkan/shader/inversespectrogram_conv.comp
@github-actions

Copy link
Copy Markdown

Please enable github action in YOUR FORKED REPO to make code-format workflow work

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant