Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
af9f3c6
docs: mkdocs 전용 정의목록을 GitHub 이 읽는 문법으로 (26곳)
Aug 21, 2026
f5b5de7
fix(run_frcnn): SparseR-CNN 이 마지막 단계 박스 정제를 건너뛰고 있었다
Aug 24, 2026
4e157d0
fix(frcnn_wrap): groie 거절 사유를 실제 남은 한계로 바꾼다
Aug 24, 2026
67ae1aa
docs(frcnn_wrap): groie 거절 사유 갱신 — 남은 벽은 6D energy 하나
Aug 24, 2026
106d175
fix(frcnn_wrap): groie 거절 삭제 — 컴파일 게이트가 이유를 말한다
Aug 24, 2026
752103b
feat(verify): 마스크 로짓 축 — 코드는 공통, 게이트는 groie 하나
Aug 25, 2026
0ca1bdb
feat(verify): box_all.py — one-stage 박스 전수 드라이버를 저장소에 둔다
Aug 25, 2026
44a5657
fix(frcnn_wrap): FastRCNN 은 rpn_head 가 없다 — 한 줄이 그걸 직접 읽고 있었다
Aug 25, 2026
7aae19d
fix(append_head_weights): head 가중치도 GTX_GGUF_F32 를 따른다 — 섞이면 세그폴트
Aug 25, 2026
e77467f
fix(verify_postproc): 러너와 기준값에 **같은 픽셀**을 준다 — 각자 jpg 를 열고 있었다
Aug 25, 2026
3a1ded8
feat(verify_postproc_roi): --workers — 가용 메모리를 읽어 스스로 낮춘다
Aug 25, 2026
5eb6eab
feat(frontend): GLIP 을 이미지 하나만 받는 모듈로 감싼다 — 융합헤드까지 한 그래프로
Aug 27, 2026
94b95e0
feat(mmseg): 세그멘테이션 계열을 잴 수 있게 한다 — 래퍼 + 하네스, depthwise dilation 수정
Aug 27, 2026
a3a1465
feat(mmpose): 포즈 계열을 잴 수 있게 한다 — 래퍼 + 하네스, 러너가 비정방 입력을 받는다
Aug 27, 2026
74f65c2
fix(verify): 계열별 입력 크기를 쓰고, 메모리 가드를 넣는다 + GLIP 박스 디코드 참조 구현
Aug 27, 2026
8d05551
fix(nn): conv 을 축별로 받게 넓힌다 — 비대칭 conv 이 조용히 대칭으로 돌고 있었다
Aug 27, 2026
af6c66b
refactor(verify): 하네스를 생태계 축으로 재편한다 — frontend/ 와 같은 축으로
Aug 27, 2026
e67ee1e
feat(verify): 층별 프로브를 저장소로 들인다 — 어디서 값이 벌어지는지 가른다
Aug 27, 2026
6ec6bb5
fix(nn): PReLU 를 정확히 낸다 — 0.25 근사는 랜덤 가중치에서만 맞았다
Aug 27, 2026
7f52009
fix(frontend): cascade decode head 를 부른다 — 「범위 밖」이 아니라 하네스가 못 부른 것이었다
Aug 28, 2026
fc069b4
perf(verify): mmseg 스윕을 병렬로 돌린다 — 계열당 52초 중 추론은 6초뿐이다
Aug 28, 2026
61b9763
feat(verify): MaskFormer 계열 프론트엔드 글루 + 층별 프로브
Aug 28, 2026
0a57698
feat(verify): 계열 무관 프로브(bb/neck) + mmpose 메모리 게이트에 락
Aug 28, 2026
1c1b19a
mmseg 하네스: emanet metafile 오타 예외·san 멀티모달·point_rend cascade predict
Aug 31, 2026
9dfef8b
feat(frontend/runtime): as_f32 헬퍼 · PointRend/SAN 프론트엔드 shim · 하네스 단계 표시
Aug 31, 2026
363278f
feat(nn): conv_2d_wt_dw — 동적 weight 의 depthwise conv (DMNet DCM)
Aug 31, 2026
e447522
feat(nn): conv_2d_depthwise_ex — 축별 strip conv (MSCAN 1x7/7x1)
Aug 31, 2026
eb772bb
verify: 못 잰 것을 실패로 적던 자리 셋을 고친다
Sep 1, 2026
727dfd4
verify: 계열별 입력 크기 표를 만들고 dense_head 경로를 고친다
Sep 1, 2026
c2ff2ea
runner: vision-cli 가 비정사각 입력을 받게 한다
Sep 1, 2026
7f611da
frontend: 세그·포즈 래퍼가 mmdet 규약을 따르게 한다
Sep 1, 2026
b135765
install_arch: 클래스 수 자동판별이 한 번도 동작한 적 없었다
Sep 1, 2026
0700eb5
tools: add_model.sh — 컴파일·등록·재빌드를 한 번에
Sep 2, 2026
2eccc05
install_arch: g2c 가 남긴 meta.json 을 기본값으로 읽는다
Sep 2, 2026
d9125f9
Revert "install_arch: g2c 가 남긴 meta.json 을 기본값으로 읽는다"
Sep 2, 2026
ae3a325
Revert "tools: add_model.sh — 컴파일·등록·재빌드를 한 번에"
Sep 2, 2026
787f090
cli: v8~v12 의 붙은 출력(4+nc)을 잘라 쓴다
Sep 2, 2026
aeda36e
preprocess: YOLO 계열에 letterbox 를 넣는다
Sep 2, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
166 changes: 92 additions & 74 deletions docs/mmdet-detectors.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,17 +80,19 @@ python tools/frontend/mmdet/mmdet_to_pt.py \

Outputs:

`backbone.pt`
: The backbone and neck as a traceable module. The head is kept as an attribute so that its
weights are preserved in the `state_dict`, but it does not participate in the forward pass.

`backbone.postproc.h`
: Everything the C++ side needs to reconstruct the head and decode its output: anchor
generator settings, bbox coder statistics, head convolution layout, and the pre-processing
normalisation taken from the config's `data_preprocessor` — emitted as a generated
`mmdet_params()` function. These values are constants once the architecture is chosen, so
they are compiled into the runner rather than read at run time.
See [Configuration reference](#configuration-reference).
**`backbone.pt`**

The backbone and neck as a traceable module. The head is kept as an attribute so that its
weights are preserved in the `state_dict`, but it does not participate in the forward pass.

**`backbone.postproc.h`**

Everything the C++ side needs to reconstruct the head and decode its output: anchor
generator settings, bbox coder statistics, head convolution layout, and the pre-processing
normalisation taken from the config's `data_preprocessor` — emitted as a generated
`mmdet_params()` function. These values are constants once the architecture is chosen, so
they are compiled into the runner rather than read at run time.
See [Configuration reference](#configuration-reference).

Those two are what you read. Two more land beside them, and they are not optional: saving a
module pickles its classes by module name, so whatever opens the `.pt` has to be able to
Expand Down Expand Up @@ -160,15 +162,16 @@ bash tools/build/build_mmdet_cpp.sh output/MMDetBackbone backbone.postproc.h
The script compiles three translation units together, with the generated parameters
included as a header, and links them against `libvisioncpp`:

- `tools/verify/backbone/run_mmdet.cpp` — the runner,
- `tools/verify/mmdet/backbone/run_mmdet.cpp` — the runner,
- `tools/detect/head.cpp` — the head component,
- `output/MMDetBackbone/MMDetBackbone.cpp` — the generated graph,
- `backbone.postproc.h` — the generated parameters.

`build_mmdet_cpp.sh <gen_dir> [params.h] [arch_name]`
: `gen_dir` is the directory holding the generated `.cpp`, `.h` and `.gguf`. The parameters
header is found in `gen_dir` when it is there, and named explicitly otherwise.
`arch_name` defaults to the base name of the `.cpp` found there.
**`build_mmdet_cpp.sh <gen_dir> [params.h] [arch_name]`**

`gen_dir` is the directory holding the generated `.cpp`, `.h` and `.gguf`. The parameters
header is found in `gen_dir` when it is there, and named explicitly otherwise.
`arch_name` defaults to the base name of the `.cpp` found there.

The library is looked up in `build/`, which is where [Building](../README.md#building) puts it.
If you configured elsewhere, name that directory:
Expand All @@ -194,22 +197,26 @@ output/MMDetBackbone/run_mmdet \
run_mmdet <gguf> <input> <output> [size=512]
```

`<gguf>`
: Weights produced in step 2.
**`<gguf>`**

Weights produced in step 2.

**`<input>`**

`<input>`
: An image (`.jpg`, `.jpeg`, `.png`, `.bmp`) or a pre-processed tensor (`.bin`).
Images are resized and normalised in-process using `preprocess()`; the mean, standard
deviation and channel order are compiled in. A `.bin` file is taken as-is and must
contain `3 × size × size` `float32` values in CWHN order.
An image (`.jpg`, `.jpeg`, `.png`, `.bmp`) or a pre-processed tensor (`.bin`).
Images are resized and normalised in-process using `preprocess()`; the mean, standard
deviation and channel order are compiled in. A `.bin` file is taken as-is and must
contain `3 × size × size` `float32` values in CWHN order.

`<output>`
: Where to write the result. The extension decides what is written: `.bin` gives raw
detections, anything else gives the input image with the boxes drawn on it.
**`<output>`**

`[size]`
: Input resolution. Must match the value passed to `--size` in step 1 and the shape the graph
was compiled for.
Where to write the result. The extension decides what is written: `.bin` gives raw
detections, anything else gives the input image with the boxes drawn on it.

**`[size]`**

Input resolution. Must match the value passed to `--size` in step 1 and the shape the graph
was compiled for.

### Output

Expand Down Expand Up @@ -256,7 +263,7 @@ import numpy as np
d = np.fromfile("boxes.bin", dtype="float32").reshape(-1, 6)
```

`tools/verify/draw_boxes.py` draws such a file afterwards, with class names and scores as text.
`tools/verify/common/draw_boxes.py` draws such a file afterwards, with class names and scores as text.


## Models whose head survives tracing
Expand Down Expand Up @@ -409,35 +416,40 @@ struct detection {

### Pre-processing

`std::vector<float> preprocess(uint8_t const* img, int img_h, int img_w, int img_c, int out_size, float const mean[3], float const std[3], bool to_rgb, int* out_w = nullptr, int* out_h = nullptr)`
: Resize to `out_size × out_size` and normalise to `(v - mean) / std`, optionally swapping
channel order. Returns a CWHN `float32` tensor ready for the graph input.
**`std::vector<float> preprocess(uint8_t const* img, int img_h, int img_w, int img_c, int out_size, float const mean[3], float const std[3], bool to_rgb, int* out_w = nullptr, int* out_h = nullptr)`**

Resize to `out_size × out_size` and normalise to `(v - mean) / std`, optionally swapping
channel order. Returns a CWHN `float32` tensor ready for the graph input.

### Dense heads

<a id="detect_anchor"></a>
`std::vector<detection> detect_anchor(cls_scores, bbox_preds, feat_hw, det_params const& p, score_factors = nullptr)`
: Anchor-based decoding: anchor generation, delta decoding, per-level top-k, score
thresholding and NMS. Used by RetinaNet, ATSS, PAA and other delta-coded heads.
`score_factors` is the optional centerness/IoU branch. MMDetection thresholds and takes
top-k on the class score **alone** and multiplies the factor in afterwards, so passing it
here rather than folding it into `cls_scores` keeps the surviving set the same.

`std::vector<detection> detect_fcos(cls_scores, bbox_preds, centerness, feat_hw, fcos_params const& p)`
: Anchor-free distance decoding. `centerness` may be empty — GFL and VFNet fold quality into
the class score and have no such branch. `bbox_preds` are already pixel distances: the head
component applies the DFL integral — DFL is Distribution Focal Loss, which predicts each
box edge as a distribution over bins and recovers the distance by integrating it — plus the
stride multiply and the exponent, so this function
must not apply a stride again. `point_offset` is 0.5 for FCOS and 0 for the heads built on
an `AnchorGenerator`.

`std::vector<detection> detect_yolox(cls, box, obj, feat_hw, yolox_params const& p)`
: Grid-based decoding with an objectness branch; score is `sigmoid(cls) * sigmoid(obj)`.

`std::vector<detection> detect_detr(float const* cls, float const* bbox, detr_params const& p)`
: Set prediction. Takes query logits and normalised `cxcywh` boxes, applies top-k, and
performs no NMS. Set `use_sigmoid` for Deformable-DETR-style heads.
**`std::vector<detection> detect_anchor(cls_scores, bbox_preds, feat_hw, det_params const& p, score_factors = nullptr)`**

Anchor-based decoding: anchor generation, delta decoding, per-level top-k, score
thresholding and NMS. Used by RetinaNet, ATSS, PAA and other delta-coded heads.
`score_factors` is the optional centerness/IoU branch. MMDetection thresholds and takes
top-k on the class score **alone** and multiplies the factor in afterwards, so passing it
here rather than folding it into `cls_scores` keeps the surviving set the same.

**`std::vector<detection> detect_fcos(cls_scores, bbox_preds, centerness, feat_hw, fcos_params const& p)`**

Anchor-free distance decoding. `centerness` may be empty — GFL and VFNet fold quality into
the class score and have no such branch. `bbox_preds` are already pixel distances: the head
component applies the DFL integral — DFL is Distribution Focal Loss, which predicts each
box edge as a distribution over bins and recovers the distance by integrating it — plus the
stride multiply and the exponent, so this function
must not apply a stride again. `point_offset` is 0.5 for FCOS and 0 for the heads built on
an `AnchorGenerator`.

**`std::vector<detection> detect_yolox(cls, box, obj, feat_hw, yolox_params const& p)`**

Grid-based decoding with an objectness branch; score is `sigmoid(cls) * sigmoid(obj)`.

**`std::vector<detection> detect_detr(float const* cls, float const* bbox, detr_params const& p)`**

Set prediction. Takes query logits and normalised `cxcywh` boxes, applies top-k, and
performs no NMS. Set `use_sigmoid` for Deformable-DETR-style heads.

`det_params` carries the anchor generator (`strides`, `octave_base_scale`, `octave_scales`,
`ratios`, `center_offset`), the bbox coder (`means`, `stds`), and the test-time thresholds
Expand All @@ -446,32 +458,38 @@ image.

### Two-stage components

`std::vector<detection> detect_rpn(rpn_cls, rpn_bbox, feat_hw, rpn_params const& p)`
: Region proposals with their objectness scores. NMS runs **per level**, not per class —
MMDetection passes level ids to `batched_nms` — and there is no pre-NMS score threshold.
The returned `label` is the level the proposal came from.
**`std::vector<detection> detect_rpn(rpn_cls, rpn_bbox, feat_hw, rpn_params const& p)`**

Region proposals with their objectness scores. NMS runs **per level**, not per class —
MMDetection passes level ids to `batched_nms` — and there is no pre-NMS score threshold.
The returned `label` is the level the proposal came from.

`std::vector<float> rpn_proposals(rpn_cls, rpn_bbox, feat_hw, rpn_params const& p)`
: The same computation with the scores dropped: `M × 4` boxes in image coordinates,
`M ≤ max_per_img`, which is the form the RoIAlign stage takes.
**`std::vector<float> rpn_proposals(rpn_cls, rpn_bbox, feat_hw, rpn_params const& p)`**

`std::vector<float> roi_align(feats, feat_hw, float const* rois, int m, roi_align_params const& p)`
: MMCV-compatible RoIAlign (`aligned = true`, adaptive `sampling_ratio`). Level assignment
follows `clamp(floor(log2(sqrt(w*h) / finest_scale + 1e-6)), 0, L-1)`.
Returns `M × C × out × out` in NCHW order.
The same computation with the scores dropped: `M × 4` boxes in image coordinates,
`M ≤ max_per_img`, which is the form the RoIAlign stage takes.

`std::vector<detection> detect_roi(float const* scores, float const* bbox_deltas, float const* proposals, int n, roi_params const& p)`
: Final RoI-head decoding: class-wise delta decoding and per-class NMS. `scores` are
post-softmax with background last. Set `class_agnostic` when `bbox_pred` has four columns
instead of `num_classes * 4`.
**`std::vector<float> roi_align(feats, feat_hw, float const* rois, int m, roi_align_params const& p)`**

MMCV-compatible RoIAlign (`aligned = true`, adaptive `sampling_ratio`). Level assignment
follows `clamp(floor(log2(sqrt(w*h) / finest_scale + 1e-6)), 0, L-1)`.
Returns `M × C × out × out` in NCHW order.

**`std::vector<detection> detect_roi(float const* scores, float const* bbox_deltas, float const* proposals, int n, roi_params const& p)`**

Final RoI-head decoding: class-wise delta decoding and per-class NMS. `scores` are
post-softmax with background last. Set `class_agnostic` when `bbox_pred` has four columns
instead of `num_classes * 4`.

### Masks and keypoints

`std::vector<uint8_t> paste_mask(float const* mask_logit, int mh, int mw, detection const& box, float thr = 0.5f, int* out_h = nullptr, int* out_w = nullptr)`
: Sigmoid, resize to the box, threshold. Returns a binary mask covering the box.
**`std::vector<uint8_t> paste_mask(float const* mask_logit, int mh, int mw, detection const& box, float thr = 0.5f, int* out_h = nullptr, int* out_w = nullptr)`**

Sigmoid, resize to the box, threshold. Returns a binary mask covering the box.

**`std::vector<float> decode_keypoints(float const* heatmap, int k, int hm_h, int hm_w, float stride)`**

`std::vector<float> decode_keypoints(float const* heatmap, int k, int hm_h, int hm_w, float stride)`
: Per-keypoint argmax over a heatmap; returns `k × 3` as `(x, y, score)`.
Per-keypoint argmax over a heatmap; returns `k × 3` as `(x, y, score)`.

### Building blocks

Expand Down
54 changes: 31 additions & 23 deletions docs/using-the-cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,36 +23,44 @@ The command selects the model, `-m` says which weights to load, `-i` and `-o` ar

## Options

`-m, --model <file>`
: The `.gguf` weights. Omit it and each command looks for its own default name —
`MobileSAM-F16.gguf`, `BiRefNet-lite-F16.gguf`, and so on — under `models/`,
`$VISION_MODEL_DIR`, `$XDG_DATA_HOME/visioncpp`, `~/.local/share/visioncpp` and the
install directory, in that order.
**`-m, --model <file>`**

`-i, --input <image> [<image> ...]`
: Input image. `migan` takes two — the image and the mask.
The `.gguf` weights. Omit it and each command looks for its own default name —
`MobileSAM-F16.gguf`, `BiRefNet-lite-F16.gguf`, and so on — under `models/`,
`$VISION_MODEL_DIR`, `$XDG_DATA_HOME/visioncpp`, `~/.local/share/visioncpp` and the
install directory, in that order.

`-o, --output <file>`
: Output file. Defaults to `output.png`. Images are always written as **PNG**, whatever
the name says — `-o out.jpg` produces a PNG file called `out.jpg`, which some viewers
refuse to open. Give it a `.png` name.
**`-i, --input <image> [<image> ...]`**

`-p, --prompt <x> [<y> ...]`
: Prompt for models that take one. `sam` accepts a point (`x y`) or a box
(`x1 y1 x2 y2`) in pixels, origin top-left.
Input image. `migan` takes two — the image and the mask.

`-b, --backend <cpu|gpu>`
: Which device to run on. Defaults to automatic — GPU if the build has Vulkan and a device is
available, CPU otherwise.
**`-o, --output <file>`**

`--composite <file>`
: Also write the input image combined with the resulting mask, instead of the mask alone.
Output file. Defaults to `output.png`. Images are always written as **PNG**, whatever
the name says — `-o out.jpg` produces a PNG file called `out.jpg`, which some viewers
refuse to open. Give it a `.png` name.

`--tile <size>`
: Split large inputs into tiles of this size. Used by `esrgan` to keep memory bounded.
**`-p, --prompt <x> [<y> ...]`**

`-h, --help`
: Print the command list and exit.
Prompt for models that take one. `sam` accepts a point (`x y`) or a box
(`x1 y1 x2 y2`) in pixels, origin top-left.

**`-b, --backend <cpu|gpu>`**

Which device to run on. Defaults to automatic — GPU if the build has Vulkan and a device is
available, CPU otherwise.

**`--composite <file>`**

Also write the input image combined with the resulting mask, instead of the mask alone.

**`--tile <size>`**

Split large inputs into tiles of this size. Used by `esrgan` to keep memory bounded.

**`-h, --help`**

Print the command list and exit.

## Getting weights

Expand Down
Loading
Loading