mkdbg

A simple mixed CPU/GPU debugger for x86_64 Linux with a DAP interface. It allows debugging CUDA code in any editor that supports the DAP protocol.

I wrote it for fun and to deepen my understanding of debuggers.

Disclaimer: This is a personal project developed in my own time. It is not affiliated with, endorsed by, or supported by NVIDIA Corporation. Any views expressed here are my own and do not reflect those of my employer.

It has support for:

• Interrupt/resume
• Stop on signal (CPU)/ GPU exception
• Stack trace (CPU+GPU)
• Breakpoints (CPU (software and hardware)+GPU)
• Stepping (CPU+GPU)
• Disassembly (CPU+GPU)
• Thread information
• Variable inspection (GPU only, limited to basic types)
• Register inspection and modification (CPU+GPU)
• Reading and writing memory (CPU+GPU)

Implemented using:

• ptrace()
• CUDA debugger API
• LLVM
• libunwind
• cppdap

Requirements:

• Fairly recent x86_64 Linux
• CUDA toolkit >= 13.2
• LLVM development libraries >= 19
• libunwind >= 1.6.2
• Using libc other than GNU libc will likely not work

Building

Assuming that CUDA toolkit is installed in /usr/local/cuda:

$ cmake -S . -B build -DCMAKE_CUDA_ARCHITECTURES=native -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc
$ cmake --build build

If the compiler complains about missing cudadebugger.h header, then these flags need to be added to CMake invocation:

-DCMAKE_CXX_FLAGS='-isystem /usr/local/cuda/extras/Debugger/include/'
-DCMAKE_CUDA_FLAGS='-isystem /usr/local/cuda/extras/Debugger/include/'

Running

To start debugging, you need to set up your editor to run build/mkdbg-dap PROGRAM ARGS.... The debugger communicates via stdin/stdout. The file emacs-dape-cfg.el contains an example configuration for Emacs dape client.

The debugger will stop right after the program image is loaded into memory, i.e. right after exec*(). After stopping, you can simply set your breakpoints, resume the program, pause/interrupt etc. as you wish. CUDA debugging will initialize automatically when the program calls cuInit().

To run with debug logging, set SPDLOG_LEVEL=trace or SPDLOG_LEVEL=debug environment variable when running the debugger.

Debugging CUDA

The debugger shows the client a single "CUDA thread", which is focused on a single device, SM, warp and lane. You can normally select this thread as any other CPU thread. When the CUDA thread is selected, a special "CUDA focus" scope is available. In this scope, these synthetic variables are present:

• dev (selected device)

• sm

• warp

• lane

• kernel - the focused kernel number; switch to "CUDA kernels" scope to see kernel number->name mapping.

You can modify these variables to switch focus to other threads and kernels/workloads. The debugger will not allow focusing on invalid warps; the focus is either always valid or it does not exist, in which case no stack trace is shown.

These read-only helper variables are available in the "CUDA focus" scope:

• numSms - number of SMs on the selected device
• validWarps - shows which warps are valid on this SM
• validLanes - shows valid lanes on the selected warp
• threadIdx - CUDA thread index for the selected lane
• blockIdx - CUDA block index for the selected warp
• gridDim - dimensions of the grid for the current warp
• blockDim - dimensions of the block for the current warp

Supported DAP methods

Request	Supported	Notes
BreakpointLocationsRequest	✓ yes
ConfigurationDoneRequest	✓ yes
ContinueRequest	✓ yes
DisassembleRequest	✓ yes	CPU disassembly is not symbolized
ExceptionInfoRequest	✓ yes
InitializeRequest	✓ yes
LaunchRequest	✓ yes
LoadedSourcesRequest	✓ yes
ModulesRequest	✓ yes
NextRequest	✓ yes	Only instruction granularity supported
PauseRequest	✓ yes
ReadMemoryRequest	✓ yes
ScopesRequest	✓ yes
SetBreakpointsRequest	✓ yes
SetExceptionBreakpointsRequest	✓ yes
SetInstructionBreakpointsRequest	✓ yes
SetVariableRequest	✓ yes
StackTraceRequest	✓ yes
StepInRequest	✓ yes	Only instruction granularity supported
StepOutRequest	✓ yes
ThreadsRequest	✓ yes
VariablesRequest	✓ yes
WriteMemoryRequest	✓ yes
AttachRequest	✗ no
CancelRequest	✗ no
CompletionsRequest	✗ no
DataBreakpointInfoRequest	✗ no
DisconnectRequest	✗ no
EvaluateRequest	✗ no
GotoRequest	✗ no
GotoTargetsRequest	✗ no
LocationsRequest	✗ no
RestartFrameRequest	✗ no
RestartRequest	✗ no
ReverseContinueRequest	✗ no
SetDataBreakpointsRequest	✗ no
SetExpressionRequest	✗ no
SetFunctionBreakpointsRequest	✗ no
SourceRequest	✗ no
StepBackRequest	✗ no
StepInTargetsRequest	✗ no
TerminateRequest	✓ no
TerminateThreadsRequest	✗ no

License

MIT License

Copyright (c) 2026 Michał Krzywkowski <k.michal@zoho.com>

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.