Depth Align

This project is comprised of two steps, with the end goal being to generate depth information for missing objects directly aligned with structure from motions images.

1. Generate depths from images using DepthAnything3

2. Align generated depths against structure from motion depths

STEP 1: GENERATE DEPTHS

The depth command takes in either an image path or a directory of images and an output directory to save estimated depths to. The depths saved from this step are the raw model outputs, with no normalization or post processing.

Additional parameters include --model_name for changing which depth anything model is used, and --device to specify if you would rather use CPU instead of GPU for model predictions.

Model depths are saved as 32bit numpy npy files.

Generate depths using single image inference

depth-align depth --images "data\images" --output_dir "output_depths\depths"

Generate depths in batches using 5 frames at a time

depth-align depth --images "data\images" --output_dir "output_depths\multiframe-depths" --batch_size 5

Generate depths in batches using 5 frames at a time and camera property data

depth-align depth ^
--images "data\images" ^
--output_dir "output_depths\multiframe-informed-depths" ^
--intrinsics_path "data\intrinsics.txt" ^
--extrinsics_path "data\extrinsics.txt" ^
--batch_size 5

A proper study has not been done to identify which performs best. Based off of visualizations, it appears using multiframe inference and camera properties is helpful

Useful DepthAnything3 references

• https://github.com/ByteDance-Seed/Depth-Anything-3/tree/main
• https://github.com/ByteDance-Seed/Depth-Anything-3/blob/main/docs/API.md#-inference-method

Useful camera properties parsing code

• https://github.com/VG-TechCenter/scripts-for-colmap/blob/main/read_write_model.py

STEP 2: ALIGN DEPTHS

The hough align command takes as input true depths and estimated depths, and attempts to align the estimated depths to the true depths.

Align 8bit depths using hough transform

depth-align hough_align ^
  --true_depths "data\resized_sfm_8bit_depth" ^
  --estimated_depths "output_depths\depths" ^
  --output_dir "output_alignments\8bit_hough_s100_n141120" ^
  --sfm_images_dir "data\resized_sfm_images" ^
  --true_images_dir "data\resized_images" ^
  --sensitivity 100 ^
  --num_pixels 141120

Align 8bit depths using hough transform

depth-align hough_align ^
  --true_depths "data\resized_sfm_8bit_depth" ^
  --estimated_depths "output_depths\multiframe-informed-depths" ^
  --output_dir "output_alignments\8bit_multiframe-informed-hough_s100_n141120" ^
  --sfm_images_dir "data\resized_sfm_images" ^
  --true_images_dir "data\resized_images" ^
  --sensitivity 100 ^
  --num_pixels 141120

Align 32bit depths using hough transform

depth-align hough_align ^
  --true_depths "data\resized_sfm_32bit_depth" ^
  --estimated_depths "output_depths\multiframe-informed-depths" ^
  --output_dir "output_alignments\32bit_multiframe-informed-hough_s100_n141120-32bit" ^
  --sfm_images_dir "data\resized_sfm_images" ^
  --true_images_dir "data\resized_images" ^
  --sensitivity 100 ^
  --num_pixels 141120