admin/Media_Depth
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Rename DA3 directory and document system status

Browse Source
This commit is contained in:
Codex
2026-05-20 12:40:53 +08:00
parent 4a0aebb2bd
commit e8a4ec8ad0
99 changed files with 267 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

465
Depth-Anything-V3-main/docs/API.md Normal file
View File

@@ -0,0 +1,465 @@
# 📚 DepthAnything3 API Documentation
## 📑 Table of Contents
1. [📖 Overview](#overview)
2. [💡 Usage Examples](#usage-examples)
3. [🔧 Core API](#core-api)
- [DepthAnything3 Class](#depthanything3-class)
- [inference() Method](#inference-method)
4. [⚙️ Parameters](#parameters)
- [Input Parameters](#input-parameters)
- [Pose Alignment Parameters](#pose-alignment-parameters)
- [Feature Export Parameters](#feature-export-parameters)
- [Rendering Parameters](#rendering-parameters)
- [Processing Parameters](#processing-parameters)
- [Export Parameters](#export-parameters)
5. [📤 Export Formats](#export-formats)
6. [↩️ Return Value](#return-value)
## 📖 Overview
This documentation provides comprehensive API reference for DepthAnything3, including usage examples, parameter specifications, export formats, and advanced features. It covers both basic pose and depth estimation workflows and advanced pose-conditioned processing with multiple export capabilities.
## 💡 Usage Examples
Here are quick examples to get you started:
### 🚀 Basic Depth Estimation
```python
from depth_anything_3.api import DepthAnything3
# Initialize and run inference
model = DepthAnything3.from_pretrained("depth-anything/DA3NESTED-GIANT-LARGE").to("cuda")
prediction = model.inference(["image1.jpg", "image2.jpg"])
```
### 📷 Pose-Conditioned Depth Estimation
```python
import numpy as np
# With camera parameters for better consistency
prediction = model.inference(
image=["image1.jpg", "image2.jpg"],
extrinsics=extrinsics_array, # (N, 4, 4)
intrinsics=intrinsics_array # (N, 3, 3)
)
```
### 📤 Export Results
```python
# Export depth data and 3D visualization
prediction = model.inference(
image=image_paths,
export_dir="./output",
export_format="mini_npz-glb"
)
```
### 🔍 Feature Extraction
```python
# Export intermediate features from specific layers
prediction = model.inference(
image=image_paths,
export_dir="./output",
export_format="feat_vis",
export_feat_layers=[0, 1, 2] # Export features from layers 0, 1, 2
)
```
### ✨ Advanced Export with Gaussian Splatting
```python
# Export multiple formats including Gaussian Splatting
# Note: infer_gs=True requires da3-giant or da3nested-giant-large model
model = DepthAnything3(model_name="da3-giant").to("cuda")
prediction = model.inference(
image=image_paths,
extrinsics=extrinsics_array,
intrinsics=intrinsics_array,
export_dir="./output",
export_format="npz-glb-gs_ply-gs_video",
align_to_input_ext_scale=True,
infer_gs=True, # Required for gs_ply and gs_video exports
)
```
### 🎨 Advanced Export with Feature Visualization
```python
# Export with intermediate feature visualization
prediction = model.inference(
image=image_paths,
export_dir="./output",
export_format="mini_npz-glb-depth_vis-feat_vis",
export_feat_layers=[0, 5, 10, 15, 20],
feat_vis_fps=30,
)
```
### 📐 Using Ray-Based Pose Estimation
```python
# Use ray-based pose estimation instead of camera decoder
prediction = model.inference(
image=image_paths,
export_dir="./output",
export_format="glb",
use_ray_pose=True, # Enable ray-based pose estimation
)
```
### 🎯 Reference View Selection
```python
# For multi-view inputs, automatically select the best reference view
prediction = model.inference(
image=image_paths,
ref_view_strategy="saddle_balanced", # Default: balanced selection
)
# For video sequences, use middle frame as reference
prediction = model.inference(
image=video_frames,
ref_view_strategy="middle", # Good for temporally ordered inputs
)
```
## 🔧 Core API
### 🔨 DepthAnything3 Class
The main API class that provides depth estimation capabilities with optional pose conditioning.
#### 🎯 Initialization
```python
from depth_anything_3 import DepthAnything3
# Initialize the model with a model name
model = DepthAnything3(model_name="da3-large")
model = model.to("cuda") # Move to GPU
```
**Parameters:**
- `model_name` (str, default: "da3-large"): The name of the model preset to use.
- **Available models:**
- 🦾 `"da3-giant"` - 1.15B params, any-view model with GS support
- ⭐ `"da3-large"` - 0.35B params, any-view model (recommended for most use cases)
- 📦 `"da3-base"` - 0.12B params, any-view model
- 🪶 `"da3-small"` - 0.08B params, any-view model
- 👁️ `"da3mono-large"` - 0.35B params, monocular depth only
- 📏 `"da3metric-large"` - 0.35B params, metric depth with sky segmentation
- 🎯 `"da3nested-giant-large"` - 1.40B params, nested model with all features
### 🚀 inference() Method
The primary inference method that processes images and returns depth predictions.
```python
prediction = model.inference(
image=image_list,
extrinsics=extrinsics_array, # Optional
intrinsics=intrinsics_array, # Optional
align_to_input_ext_scale=True, # Whether to align predicted poses to input scale
infer_gs=True, # Enable Gaussian branch for gs exports
use_ray_pose=False, # Use ray-based pose estimation instead of camera decoder
ref_view_strategy="saddle_balanced", # Reference view selection strategy
render_exts=render_extrinsics, # Optional renders for gs_video
render_ixts=render_intrinsics, # Optional renders for gs_video
render_hw=(height, width), # Optional renders for gs_video
process_res=504,
process_res_method="upper_bound_resize",
export_dir="output_directory", # Optional
export_format="mini_npz",
export_feat_layers=[], # List of layer indices to export features from
conf_thresh_percentile=40.0, # Confidence threshold percentile for depth map in GLB export
num_max_points=1_000_000, # Maximum number of points to export in GLB export
show_cameras=True, # Whether to show cameras in GLB export
feat_vis_fps=15, # Frames per second for feature visualization in feat_vis export
export_kwargs={} # Optional, additional arguments to export functions. export_format:key:val, see 'Parameters/Export Parameters' for details
)
```
## ⚙️ Parameters
### 📸 Input Parameters
#### `image` (required)
- **Type**: `List[Union[np.ndarray, Image.Image, str]]`
- **Description**: List of input images. Can be numpy arrays, PIL Images, or file paths.
- **Example**:
```python
# From file paths
image = ["image1.jpg", "image2.jpg", "image3.jpg"]
# From numpy arrays
image = [np.array(img1), np.array(img2)]
# From PIL Images
image = [Image.open("image1.jpg"), Image.open("image2.jpg")]
```
#### `extrinsics` (optional)
- **Type**: `Optional[np.ndarray]`
- **Shape**: `(N, 4, 4)` where N is the number of input images
- **Description**: Camera extrinsic matrices (world-to-camera transformation). When provided, enables pose-conditioned depth estimation mode.
- **Note**: If not provided, the model operates in standard depth estimation mode.
#### `intrinsics` (optional)
- **Type**: `Optional[np.ndarray]`
- **Shape**: `(N, 3, 3)` where N is the number of input images
- **Description**: Camera intrinsic matrices containing focal length and principal point information. When provided, enables pose-conditioned depth estimation mode.
### 🎯 Pose Alignment Parameters
#### `align_to_input_ext_scale` (default: True)
- **Type**: `bool`
- **Description**: When True the predicted extrinsics are replaced with the input
ones and the depth maps are rescaled to match their metric scale. When False the
function returns the internally aligned poses computed via Umeyama alignment.
#### `infer_gs` (default: False)
- **Type**: `bool`
- **Description**: Enable Gaussian Splatting branch for gaussian splatting exports. Required when using `gs_ply` or `gs_video` export formats.
#### `use_ray_pose` (default: False)
- **Type**: `bool`
- **Description**: Use ray-based pose estimation instead of camera decoder for pose prediction. When True, the model uses ray prediction heads to estimate camera poses; when False, it uses the camera decoder approach.
#### `ref_view_strategy` (default: "saddle_balanced")
- **Type**: `str`
- **Description**: Strategy for selecting the reference view from multiple input views. Options: `"first"`, `"middle"`, `"saddle_balanced"`, `"saddle_sim_range"`. Only applied when number of views ≥ 3. See [detailed documentation](funcs/ref_view_strategy.md) for strategy comparisons.
- **Available strategies**:
- `"saddle_balanced"`: Selects view with balanced features across multiple metrics (recommended default)
- `"saddle_sim_range"`: Selects view with largest similarity range
- `"first"`: Always uses first view (not recommended, equivalent to no reordering for views < 3)
- `"middle"`: Uses middle view (recommended for video sequences)
### 🔍 Feature Export Parameters
#### `export_feat_layers` (default: [])
- **Type**: `List[int]`
- **Description**: List of layer indices to export intermediate features from. Features are stored in the `aux` dictionary of the Prediction object with keys like `feat_layer_0`, `feat_layer_1`, etc.
### 🎥 Rendering Parameters
These arguments are only used when exporting Gaussian-splatting videos (include
`"gs_video"` in `export_format`). They describe an auxiliary camera trajectory
with ``M`` views.
#### `render_exts` (optional)
- **Type**: `Optional[np.ndarray]`
- **Shape**: `(M, 4, 4)`
- **Description**: Camera extrinsics for the synthesized trajectory. If omitted,
the exporter falls back to the predicted poses.
#### `render_ixts` (optional)
- **Type**: `Optional[np.ndarray]`
- **Shape**: `(M, 3, 3)`
- **Description**: Camera intrinsics for each rendered frame. Leave `None` to
reuse the input intrinsics.
#### `render_hw` (optional)
- **Type**: `Optional[Tuple[int, int]]`
- **Description**: Explicit output resolution `(height, width)` for the rendered
frames. Defaults to the input resolution when not provided.
### ⚡ Processing Parameters
#### `process_res` (default: 504)
- **Type**: `int`
- **Description**: Base resolution for processing. The model will resize images to this resolution for inference.
#### `process_res_method` (default: "upper_bound_resize")
- **Type**: `str`
- **Description**: Method for resizing images to the target resolution.
- **Options**:
- `"upper_bound_resize"`: Resize so that the specified dimension (504) becomes the longer side
- `"lower_bound_resize"`: Resize so that the specified dimension (504) becomes the shorter side
- **Example**:
- Input: 1200×1600 → Output: 378×504 (with `process_res=504`, `process_res_method="upper_bound_resize"`)
- Input: 504×672 → Output: 504×672 (no change needed)
### 📦 Export Parameters
#### `export_dir` (optional)
- **Type**: `Optional[str]`
- **Description**: Directory path where exported files will be saved. If not provided, no files will be exported.
#### `export_format` (default: "mini_npz")
- **Type**: `str`
- **Description**: Format for exporting results. Supports multiple formats separated by `-`.
- **Example**: `"mini_npz-glb"` exports both mini_npz and glb formats.
#### 🌐 GLB Export Parameters
These parameters are passed directly to the `inference()` method and only apply when `export_format` includes `"glb"`.
##### `conf_thresh_percentile` (default: 40.0)
- **Type**: `float`
- **Description**: Lower percentile for adaptive confidence threshold. Points below this confidence percentile will be filtered out from the point cloud.
##### `num_max_points` (default: 1,000,000)
- **Type**: `int`
- **Description**: Maximum number of points in the exported point cloud. If the point cloud exceeds this limit, it will be downsampled.
##### `show_cameras` (default: True)
- **Type**: `bool`
- **Description**: Whether to include camera wireframes in the exported GLB file for visualization.
#### 🎨 Feature Visualization Parameters
These parameters are passed directly to the `inference()` method and only apply when `export_format` includes `"feat_vis"`.
##### `feat_vis_fps` (default: 15)
- **Type**: `int`
- **Description**: Frame rate for the output video when visualizing features across multiple images.
#### ✨🎥 3DGS and 3DGS Video Parameters
These parameters are passed directly to the `inference()` method and only apply when `export_format` includes `"gs_ply"` or `"gs_video"`.
##### `export_kwargs` (default: `{}`)
- Type: `dict[str, dict[str, Any]]`
- Description: Per-format extra arguments passed to export functions, mainly for `"gs_ply"` and `"gs_video"`.
- Access pattern: `export_kwargs[export_format][key] = value`
- Example:
```python
{
"gs_ply": {
"gs_views_interval": 1,
},
"gs_video": {
"trj_mode": "interpolate_smooth",
"chunk_size": 1,
"vis_depth": None,
},
}
```
## 📤 Export Formats
The API supports multiple export formats for different use cases:
### 📊 `mini_npz`
- **Description**: Minimal NPZ format containing essential data
- **Contents**: `depth`, `conf`, `exts`, `ixts`
- **Use case**: Lightweight storage for depth data with camera parameters
### 📦 `npz`
- **Description**: Full NPZ format with comprehensive data
- **Contents**: `depth`, `conf`, `exts`, `ixts`, `image`, etc.
- **Use case**: Complete data export for advanced processing
### 🌐 `glb`
- **Description**: 3D visualization format with point cloud and camera poses
- **Contents**:
- Point cloud with colors from original images
- Camera wireframes for visualization
- Confidence-based filtering and downsampling
- **Use case**: 3D visualization, inspection, and analysis
- **Features**:
- Automatic sky depth handling
- Confidence threshold filtering
- Background filtering (black/white)
- Scene scale normalization
- **Parameters** (passed via `inference()` method directly):
- `conf_thresh_percentile` (float, default: 40.0): Lower percentile for adaptive confidence threshold. Points below this confidence percentile will be filtered out.
- `num_max_points` (int, default: 1,000,000): Maximum number of points in the exported point cloud. If exceeded, points will be downsampled.
- `show_cameras` (bool, default: True): Whether to include camera wireframes in the exported GLB file for visualization.
### ✨ `gs_ply`
- **Description**: Gaussian Splatting point cloud format
- **Contents**: 3DGS data in PLY format. Compatible with standard 3DGS viewers such as [SuperSplat](https://superspl.at/editor) (recommended), [SPARK](https://sparkjs.dev/viewer/).
- **Use case**: Gaussian Splatting reconstruction
- **Requirements**: Must set `infer_gs=True` when calling `inference()`. Only supported by `da3-giant` and `da3nested-giant-large` models.
- **Additional configs**, provided via `export_kwargs` (see [Export Parameters](#export-parameters)):
- `gs_views_interval`: Export to 3DGS every N views, default: `1`.
### 🎥 `gs_video`
- **Description**: Rasterized 3DGS to obtain videos
- **Contents**: A video of 3DGS-rasterized views using either provided viewpoints or a predefined camera trajectory.
- **Use case**: Video rendering for Gaussian Splatting
- **Requirements**: Must set `infer_gs=True` when calling `inference()`. Only supported by `da3-giant` and `da3nested-giant-large` models.
- **Note**: Can optionally use `render_exts`, `render_ixts`, and `render_hw` parameters in `inference()` method to specify novel viewpoints.
- **Additional configs**, provided via `export_kwargs` (see [Export Parameters](#export-parameters)):
- `extrinsics`: Optional world-to-camera poses for novel views. Falls back to the predicted poses of input views if not provided. (Alternatively, use `render_exts` parameter in `inference()`)
- `intrinsics`: Optional camera intrinsics for novel views. Falls back to the predicted intrinsics of input views if not provided. (Alternatively, use `render_ixts` parameter in `inference()`)
- `out_image_hw`: Optional output resolution `H x W`. Falls back to input resolution if not provided. (Alternatively, use `render_hw` parameter in `inference()`)
- `chunk_size`: Number of views rasterized per batch. Default: `8`.
- `trj_mode`: Predefined camera trajectory for novel-view rendering.
- `color_mode`: Same as `render_mode` in [gsplat](https://docs.gsplat.studio/main/apis/rasterization.html#gsplat.rasterization).
- `vis_depth`: How depth is combined with RGB. Default: `hcat` (horizontal concatenation).
- `enable_tqdm`: Whether to display a tqdm progress bar during rendering.
- `output_name`: File name of the rendered video.
- `video_quality`: Video quality to save. Default: `high`.
- `high`: High quality video (default)
- `medium`: Medium quality video (balance of storage space and quality)
- `low`: Low quality video (fewer storage space)
### 🔍 `feat_vis`
- **Description**: Feature visualization format
- **Contents**: PCA-visualized intermediate features from specified layers
- **Use case**: Model interpretability and feature analysis
- **Note**: Requires `export_feat_layers` to be specified
- **Parameters** (passed via `inference()` method directly):
- `feat_vis_fps` (int, default: 15): Frame rate for the output video when visualizing features across multiple images.
### 🎨 `depth_vis`
- **Description**: Depth visualization format
- **Contents**: Color-coded depth maps alongside original images
- **Use case**: Visual inspection of depth estimation quality
### 🔗 Multiple Format Export
You can export multiple formats simultaneously by separating them with `-`:
```python
# Export both mini_npz and glb formats
export_format = "mini_npz-glb"
# Export multiple formats
export_format = "npz-glb-gs_ply"
```
## ↩️ Return Value
The `inference()` method returns a `Prediction` object with the following attributes:
### 📊 Core Outputs
- **depth**: `np.ndarray` - Estimated depth maps with shape `(N, H, W)` where N is the number of images, H is height, and W is width.
- **conf**: `np.ndarray` - Confidence maps with shape `(N, H, W)` indicating prediction reliability (optional, depends on model).
### 📷 Camera Parameters
- **extrinsics**: `np.ndarray` - Camera extrinsic matrices with shape `(N, 3, 4)` representing world-to-camera transformations. Only present if camera poses were estimated or provided as input.
- **intrinsics**: `np.ndarray` - Camera intrinsic matrices with shape `(N, 3, 3)` containing focal length and principal point information. Only present if poses were estimated or provided as input.
### 🎁 Additional Outputs
- **processed_images**: `np.ndarray` - Preprocessed input images with shape `(N, H, W, 3)` in RGB format (0-255 uint8).
- **aux**: `dict` - Auxiliary outputs including:
- `feat_layer_X`: Intermediate features from layer X (if `export_feat_layers` was specified)
- `gaussians`: 3D Gaussian Splats data (if `infer_gs=True`)
### 💻 Usage Example
```python
prediction = model.inference(image=["img1.jpg", "img2.jpg"])
# Access depth maps
depth_maps = prediction.depth # shape: (2, H, W)
# Access confidence
if hasattr(prediction, 'conf'):
confidence = prediction.conf
# Access camera parameters (if available)
if hasattr(prediction, 'extrinsics'):
camera_poses = prediction.extrinsics # shape: (2, 4, 4)
if hasattr(prediction, 'intrinsics'):
camera_intrinsics = prediction.intrinsics # shape: (2, 3, 3)
# Access intermediate features (if export_feat_layers was set)
if hasattr(prediction, 'aux') and 'feat_layer_0' in prediction.aux:
features = prediction.aux['feat_layer_0']
```

654
Depth-Anything-V3-main/docs/CLI.md Normal file
View File

@@ -0,0 +1,654 @@
# 🚀 Depth Anything 3 Command Line Interface
## 📋 Table of Contents
- [📖 Overview](#overview)
- [⚡ Quick Start](#quick-start)
- [📚 Command Reference](#command-reference)
- [🤖 auto - Auto Mode](#auto---auto-mode)
- [🖼️ image - Single Image Processing](#image---single-image-processing)
- [🗂️ images - Image Directory Processing](#images---image-directory-processing)
- [🎬 video - Video Processing](#video---video-processing)
- [📐 colmap - COLMAP Dataset Processing](#colmap---colmap-dataset-processing)
- [🔧 backend - Backend Service](#backend---backend-service)
- [🎨 gradio - Gradio Application](#gradio---gradio-application)
- [🖼️ gallery - Gallery Server](#gallery---gallery-server)
- [⚙️ Parameter Details](#parameter-details)
- [💡 Usage Examples](#usage-examples)
## 📖 Overview
The Depth Anything 3 CLI provides a comprehensive command-line toolkit supporting image depth estimation, video processing, COLMAP dataset handling, and web applications.
The backend service enables cache model to GPU so that we do not need to reload model for each command.
## ⚡ Quick Start
The CLI can run fully offline or connect to the backend for cached weights and task scheduling:
```bash
# 🔧 Start backend service (optional, keeps model resident in GPU memory)
da3 backend --model-dir depth-anything/DA3NESTED-GIANT-LARGE
# 🚀 Use auto mode to process input
da3 auto path/to/input --export-dir ./workspace/scene001
# ♻️ Reuse backend for next job
da3 auto path/to/video.mp4 \
--export-dir ./workspace/scene002 \
--use-backend \
--backend-url http://localhost:8008
```
Each export directory contains `scene.glb`, `scene.jpg`, and optional extras such as `depth_vis/` or `gs_video/` depending on the requested format.
## 📚 Command Reference
### 🤖 auto - Auto Mode
Automatically detect input type and dispatch to the appropriate handler.
**Usage:**
```bash
da3 auto INPUT_PATH [OPTIONS]
```
**Input Type Detection:**
- 🖼️ Single image file (.jpg, .png, .jpeg, .webp, .bmp, .tiff, .tif)
- 📁 Image directory
- 🎬 Video file (.mp4, .avi, .mov, .mkv, .flv, .wmv, .webm, .m4v)
- 📐 COLMAP directory (containing `images/` and `sparse/` subdirectories)
**Parameters:**
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `INPUT_PATH` | str | Required | Input path (image, directory, video, or COLMAP) |
| `--model-dir` | str | Default model | Model directory path |
| `--export-dir` | str | `debug` | Export directory |
| `--export-format` | str | `glb` | Export format (supports `mini_npz`, `glb`, `feat_vis`, etc., can be combined with hyphens) |
| `--device` | str | `cuda` | Device to use |
| `--use-backend` | bool | `False` | Use backend service for inference |
| `--backend-url` | str | `http://localhost:8008` | Backend service URL |
| `--process-res` | int | `504` | Processing resolution |
| `--process-res-method` | str | `upper_bound_resize` | Processing resolution method |
| `--export-feat` | str | `""` | Export features from specified layers, comma-separated (e.g., `"0,1,2"`) |
| `--auto-cleanup` | bool | `False` | Automatically clean export directory without confirmation |
| `--fps` | float | `1.0` | [Video] Frame sampling FPS |
| `--sparse-subdir` | str | `""` | [COLMAP] Sparse reconstruction subdirectory (e.g., `"0"` for `sparse/0/`) |
| `--align-to-input-ext-scale` | bool | `True` | [COLMAP] Align prediction to input extrinsics scale |
| `--use-ray-pose` | bool | `False` | Use ray-based pose estimation instead of camera decoder |
| `--ref-view-strategy` | str | `saddle_balanced` | Reference view selection strategy: `first`, `middle`, `saddle_balanced`, `saddle_sim_range`. See [docs](funcs/ref_view_strategy.md) |
| `--conf-thresh-percentile` | float | `40.0` | [GLB] Lower percentile for adaptive confidence threshold |
| `--num-max-points` | int | `1000000` | [GLB] Maximum number of points in the point cloud |
| `--show-cameras` | bool | `True` | [GLB] Show camera wireframes in the exported scene |
| `--feat-vis-fps` | int | `15` | [FEAT_VIS] Frame rate for output video |
**Examples:**
```bash
# 🖼️ Auto-process an image
da3 auto path/to/image.jpg --export-dir ./output
# 🎬 Auto-process a video
da3 auto path/to/video.mp4 --fps 2.0 --export-dir ./output
# 🔧 Use backend service
da3 auto path/to/input \
--export-format mini_npz-glb \
--use-backend \
--backend-url http://localhost:8008 \
--export-dir ./output
```
---
### 🖼️ image - Single Image Processing
Process a single image for camera pose and depth estimation.
**Usage:**
```bash
da3 image IMAGE_PATH [OPTIONS]
```
**Parameters:**
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `IMAGE_PATH` | str | Required | Input image file path |
| `--model-dir` | str | Default model | Model directory path |
| `--export-dir` | str | `debug` | Export directory |
| `--export-format` | str | `glb` | Export format |
| `--device` | str | `cuda` | Device to use |
| `--use-backend` | bool | `False` | Use backend service for inference |
| `--backend-url` | str | `http://localhost:8008` | Backend service URL |
| `--process-res` | int | `504` | Processing resolution |
| `--process-res-method` | str | `upper_bound_resize` | Processing resolution method |
| `--export-feat` | str | `""` | Export feature layer indices (comma-separated) |
| `--auto-cleanup` | bool | `False` | Automatically clean export directory |
| `--use-ray-pose` | bool | `False` | Use ray-based pose estimation instead of camera decoder |
| `--ref-view-strategy` | str | `saddle_balanced` | Reference view selection strategy. See [docs](funcs/ref_view_strategy.md) |
| `--conf-thresh-percentile` | float | `40.0` | [GLB] Confidence threshold percentile |
| `--num-max-points` | int | `1000000` | [GLB] Maximum number of points |
| `--show-cameras` | bool | `True` | [GLB] Show cameras |
| `--feat-vis-fps` | int | `15` | [FEAT_VIS] Video frame rate |
**Examples:**
```bash
# ✨ Basic usage
da3 image path/to/image.png --export-dir ./output
# ⚡ With backend acceleration
da3 image path/to/image.png \
--use-backend \
--backend-url http://localhost:8008 \
--export-dir ./output
# 🔍 Export feature visualization
da3 image image.jpg \
--export-format feat_vis \
--export-feat "9,19,29,39" \
--export-dir ./results
```
---
### 🗂️ images - Image Directory Processing
Process a directory of images for batch depth estimation.
**Usage:**
```bash
da3 images IMAGES_DIR [OPTIONS]
```
**Parameters:**
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `IMAGES_DIR` | str | Required | Directory path containing images |
| `--image-extensions` | str | `png,jpg,jpeg` | Image file extensions to process (comma-separated) |
| `--model-dir` | str | Default model | Model directory path |
| `--export-dir` | str | `debug` | Export directory |
| `--export-format` | str | `glb` | Export format |
| `--device` | str | `cuda` | Device to use |
| `--use-backend` | bool | `False` | Use backend service for inference |
| `--backend-url` | str | `http://localhost:8008` | Backend service URL |
| `--process-res` | int | `504` | Processing resolution |
| `--process-res-method` | str | `upper_bound_resize` | Processing resolution method |
| `--export-feat` | str | `""` | Export feature layer indices |
| `--auto-cleanup` | bool | `False` | Automatically clean export directory |
| `--use-ray-pose` | bool | `False` | Use ray-based pose estimation instead of camera decoder |
| `--ref-view-strategy` | str | `saddle_balanced` | Reference view selection strategy. See [docs](funcs/ref_view_strategy.md) |
| `--conf-thresh-percentile` | float | `40.0` | [GLB] Confidence threshold percentile |
| `--num-max-points` | int | `1000000` | [GLB] Maximum number of points |
| `--show-cameras` | bool | `True` | [GLB] Show cameras |
| `--feat-vis-fps` | int | `15` | [FEAT_VIS] Video frame rate |
**Examples:**
```bash
# 📁 Process directory (defaults to png/jpg/jpeg)
da3 images ./image_folder --export-dir ./output
# 🎯 Custom extensions
da3 images ./dataset --image-extensions "png,jpg,webp" --export-dir ./output
# 🔧 Use backend service
da3 images ./dataset \
--use-backend \
--backend-url http://localhost:8008 \
--export-dir ./output
```
---
### 🎬 video - Video Processing
Process video by extracting frames for depth estimation.
**Usage:**
```bash
da3 video VIDEO_PATH [OPTIONS]
```
**Parameters:**
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `VIDEO_PATH` | str | Required | Input video file path |
| `--fps` | float | `1.0` | Frame extraction sampling FPS |
| `--model-dir` | str | Default model | Model directory path |
| `--export-dir` | str | `debug` | Export directory |
| `--export-format` | str | `glb` | Export format |
| `--device` | str | `cuda` | Device to use |
| `--use-backend` | bool | `False` | Use backend service for inference |
| `--backend-url` | str | `http://localhost:8008` | Backend service URL |
| `--process-res` | int | `504` | Processing resolution |
| `--process-res-method` | str | `upper_bound_resize` | Processing resolution method |
| `--export-feat` | str | `""` | Export feature layer indices |
| `--auto-cleanup` | bool | `False` | Automatically clean export directory |
| `--use-ray-pose` | bool | `False` | Use ray-based pose estimation instead of camera decoder |
| `--ref-view-strategy` | str | `saddle_balanced` | Reference view selection strategy. See [docs](funcs/ref_view_strategy.md) |
| `--conf-thresh-percentile` | float | `40.0` | [GLB] Confidence threshold percentile |
| `--num-max-points` | int | `1000000` | [GLB] Maximum number of points |
| `--show-cameras` | bool | `True` | [GLB] Show cameras |
| `--feat-vis-fps` | int | `15` | [FEAT_VIS] Video frame rate |
**Examples:**
```bash
# ✨ Basic video processing
da3 video path/to/video.mp4 --export-dir ./output
# ⚙️ Control frame sampling and resolution
da3 video path/to/video.mp4 \
--fps 2.0 \
--process-res 1024 \
--export-dir ./output
# 🔧 Use backend service
da3 video path/to/video.mp4 \
--use-backend \
--backend-url http://localhost:8008 \
--export-dir ./output
```
---
### 📐 colmap - COLMAP Dataset Processing
Run pose-conditioned depth estimation on COLMAP data.
**Usage:**
```bash
da3 colmap COLMAP_DIR [OPTIONS]
```
**Parameters:**
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `COLMAP_DIR` | str | Required | COLMAP directory containing `images/` and `sparse/` subdirectories |
| `--sparse-subdir` | str | `""` | Sparse reconstruction subdirectory (e.g., `"0"` for `sparse/0/`) |
| `--align-to-input-ext-scale` | bool | `True` | Align prediction to input extrinsics scale |
| `--model-dir` | str | Default model | Model directory path |
| `--export-dir` | str | `debug` | Export directory |
| `--export-format` | str | `glb` | Export format |
| `--device` | str | `cuda` | Device to use |
| `--use-backend` | bool | `False` | Use backend service for inference |
| `--backend-url` | str | `http://localhost:8008` | Backend service URL |
| `--process-res` | int | `504` | Processing resolution |
| `--process-res-method` | str | `upper_bound_resize` | Processing resolution method |
| `--export-feat` | str | `""` | Export feature layer indices |
| `--auto-cleanup` | bool | `False` | Automatically clean export directory |
| `--use-ray-pose` | bool | `False` | Use ray-based pose estimation instead of camera decoder |
| `--ref-view-strategy` | str | `saddle_balanced` | Reference view selection strategy. See [docs](funcs/ref_view_strategy.md) |
| `--conf-thresh-percentile` | float | `40.0` | [GLB] Confidence threshold percentile |
| `--num-max-points` | int | `1000000` | [GLB] Maximum number of points |
| `--show-cameras` | bool | `True` | [GLB] Show cameras |
| `--feat-vis-fps` | int | `15` | [FEAT_VIS] Video frame rate |
**Examples:**
```bash
# 📐 Process COLMAP dataset
da3 colmap ./colmap_dataset --export-dir ./output
# 🎯 Use specific sparse subdirectory and align scale
da3 colmap ./colmap_dataset \
--sparse-subdir 0 \
--align-to-input-ext-scale \
--export-dir ./output
# 🔧 Use backend service
da3 colmap ./colmap_dataset \
--use-backend \
--backend-url http://localhost:8008 \
--export-dir ./output
```
---
### 🔧 backend - Backend Service
Start model backend service with integrated gallery.
**Usage:**
```bash
da3 backend [OPTIONS]
```
**Parameters:**
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `--model-dir` | str | Default model | Model directory path |
| `--device` | str | `cuda` | Device to use |
| `--host` | str | `127.0.0.1` | Host address to bind to |
| `--port` | int | `8008` | Port number to bind to |
| `--gallery-dir` | str | Default gallery dir | Gallery directory path (optional) |
**Features:**
- 🎯 Keeps model resident in GPU memory
- 🔌 Provides REST inference API
- 📊 Integrated dashboard and status monitoring
- 🖼️ Optional gallery browser (if `--gallery-dir` is provided)
**Available Endpoints:**
- 🏠 `/` - Home page
- 📊 `/dashboard` - Dashboard
- ✅ `/status` - API status
- 🖼️ `/gallery/` - Gallery browser (if enabled)
**Examples:**
```bash
# 🚀 Basic backend service
da3 backend --model-dir depth-anything/DA3NESTED-GIANT-LARGE
# 🖼️ Backend with gallery
da3 backend \
--model-dir depth-anything/DA3NESTED-GIANT-LARGE \
--device cuda \
--host 0.0.0.0 \
--port 8008 \
--gallery-dir ./workspace
# 💻 Use CPU
da3 backend --model-dir depth-anything/DA3NESTED-GIANT-LARGE --device cpu
```
---
### 🎨 gradio - Gradio Application
Launch Depth Anything 3 Gradio interactive web application.
**Usage:**
```bash
da3 gradio [OPTIONS]
```
**Parameters:**
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `--model-dir` | str | Required | Model directory path |
| `--workspace-dir` | str | Required | Workspace directory path |
| `--gallery-dir` | str | Required | Gallery directory path |
| `--host` | str | `127.0.0.1` | Host address to bind to |
| `--port` | int | `7860` | Port number to bind to |
| `--share` | bool | `False` | Create a public link |
| `--debug` | bool | `False` | Enable debug mode |
| `--cache-examples` | bool | `False` | Pre-cache all example scenes at startup |
| `--cache-gs-tag` | str | `""` | Tag to match scene names for high-res+3DGS caching |
**Examples:**
```bash
# 🎨 Basic Gradio application
da3 gradio \
--model-dir depth-anything/DA3NESTED-GIANT-LARGE \
--workspace-dir ./workspace \
--gallery-dir ./gallery
# 🌐 Enable sharing and debug
da3 gradio \
--model-dir depth-anything/DA3NESTED-GIANT-LARGE \
--workspace-dir ./workspace \
--gallery-dir ./gallery \
--share \
--debug
# ⚡ Pre-cache examples
da3 gradio \
--model-dir depth-anything/DA3NESTED-GIANT-LARGE \
--workspace-dir ./workspace \
--gallery-dir ./gallery \
--cache-examples \
--cache-gs-tag "dl3dv"
```
---
### 🖼️ gallery - Gallery Server
Launch standalone Depth Anything 3 Gallery server.
**Usage:**
```bash
da3 gallery [OPTIONS]
```
**Parameters:**
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `--gallery-dir` | str | Default gallery dir | Gallery root directory |
| `--host` | str | `127.0.0.1` | Host address to bind to |
| `--port` | int | `8007` | Port number to bind to |
| `--open-browser` | bool | `False` | Open browser after launch |
**Note:**
The gallery expects each scene folder to contain at least `scene.glb` and `scene.jpg`, with optional subfolders such as `depth_vis/` or `gs_video/`.
**Examples:**
```bash
# 🖼️ Basic gallery server
da3 gallery --gallery-dir ./workspace
# 🌐 Custom host and port
da3 gallery \
--gallery-dir ./workspace \
--host 0.0.0.0 \
--port 8007
# 🚀 Auto-open browser
da3 gallery --gallery-dir ./workspace --open-browser
```
---
## ⚙️ Parameter Details
### 🔧 Common Parameters
- **`--export-dir`**: Output directory, defaults to `debug`
- **`--export-format`**: Export format, supports combining multiple formats with hyphens:
- 📦 `mini_npz`: Compressed NumPy format
- 🎨 `glb`: glTF binary format (3D scene)
- 🔍 `feat_vis`: Feature visualization
- Example: `mini_npz-glb` exports both formats
- **`--process-res`** / **`--process-res-method`**: Control preprocessing resolution strategy
- `process-res`: Target resolution (default 504)
- `process-res-method`: Resize method (default `upper_bound_resize`)
- **`--auto-cleanup`**: Remove existing export directory without confirmation
- **`--use-backend`** / **`--backend-url`**: Reuse running backend service
- ⚡ Reduces model loading time
- 🌐 Supports distributed processing
- **`--export-feat`**: Layer indices for exporting intermediate features (comma-separated)
- Example: `"9,19,29,39"`
### 🎨 GLB Export Parameters
- **`--conf-thresh-percentile`**: Lower percentile for adaptive confidence threshold (default 40.0)
- Used to filter low-confidence points
- **`--num-max-points`**: Maximum number of points in point cloud (default 1,000,000)
- Controls output file size and performance
- **`--show-cameras`**: Show camera wireframes in exported scene (default True)
### 🔍 Feature Visualization Parameters
- **`--feat-vis-fps`**: Frame rate for feature visualization output video (default 15)
### 🎬 Video-Specific Parameters
- **`--fps`**: Video frame extraction sampling rate (default 1.0 FPS)
- Higher values extract more frames
### 📐 COLMAP-Specific Parameters
- **`--sparse-subdir`**: Sparse reconstruction subdirectory
- Empty string uses `sparse/` directory
- `"0"` uses `sparse/0/` directory
- **`--align-to-input-ext-scale`**: Align prediction to input extrinsics scale (default True)
- Ensures depth estimation is consistent with COLMAP scale
---
## 💡 Usage Examples
### 1️⃣ Basic Workflow
```bash
# 🔧 Start backend service
da3 backend --model-dir depth-anything/DA3NESTED-GIANT-LARGE --host 0.0.0.0 --port 8008
# 🖼️ Process single image
da3 image image.jpg --export-dir ./output1 --use-backend
# 🎬 Process video
da3 video video.mp4 --fps 2.0 --export-dir ./output2 --use-backend
# 📐 Process COLMAP dataset
da3 colmap ./colmap_data --export-dir ./output3 --use-backend
```
### 2️⃣ Using Auto Mode
```bash
# 🤖 Auto-detect and process
da3 auto ./unknown_input --export-dir ./output
# ⚡ With backend acceleration
da3 auto ./unknown_input \
--use-backend \
--backend-url http://localhost:8008 \
--export-dir ./output
```
### 3️⃣ Multi-Format Export
```bash
# 📦 Export both NPZ and GLB formats
da3 auto assets/examples/SOH \
--export-format mini_npz-glb \
--export-dir ./workspace/soh
# 🔍 Export feature visualization
da3 image image.jpg \
--export-format feat_vis \
--export-feat "9,19,29,39" \
--export-dir ./results
```
### 4️⃣ Advanced Configuration
```bash
# ⚙️ Custom resolution and point cloud density
da3 image image.jpg \
--process-res 1024 \
--num-max-points 2000000 \
--conf-thresh-percentile 30.0 \
--export-dir ./output
# 📐 COLMAP advanced options
da3 colmap ./colmap_data \
--sparse-subdir 0 \
--align-to-input-ext-scale \
--process-res 756 \
--export-dir ./output
```
### 5️⃣ Batch Processing Workflow
```bash
# 🔧 Start backend
da3 backend \
--model-dir depth-anything/DA3NESTED-GIANT-LARGE \
--device cuda \
--host 0.0.0.0 \
--port 8008 \
--gallery-dir ./workspace
# 🔄 Batch process multiple scenes
for scene in scene1 scene2 scene3; do
da3 auto ./data/$scene \
--export-dir ./workspace/$scene \
--use-backend \
--auto-cleanup
done
# 🖼️ Launch gallery to view results
da3 gallery --gallery-dir ./workspace --open-browser
```
### 6️⃣ Web Applications
```bash
# 🎨 Launch Gradio application
da3 gradio \
--model-dir depth-anything/DA3NESTED-GIANT-LARGE \
--workspace-dir workspace/gradio \
--gallery-dir ./gallery \
--host 0.0.0.0 \
--port 7860 \
--share
```
### 7️⃣ Transformer Feature Visualization
```bash
# 🔍 Export Transformer features
# 📦 Combined with numerical output
da3 auto video.mp4 \
--export-format glb-feat_vis \
--export-feat "11,21,31" \
--export-dir ./debug \
--use-backend
```
---
## 📝 Notes
1. **🔧 Backend Service**: Recommended for processing multiple tasks to improve efficiency
2. **💾 GPU Memory**: Be mindful of GPU memory usage when processing high-resolution inputs
3. **📁 Export Directory**: Use `--auto-cleanup` to avoid manual confirmation for deletion
4. **🔀 Format Combination**: Multiple export formats can be combined with hyphens (e.g., `mini_npz-glb-feat_vis`)
5. **📐 COLMAP Data**: Ensure COLMAP directory structure is correct (contains `images/` and `sparse/` subdirectories)
---
## ❓ Getting Help
View detailed help for any command:
```bash
# 📖 View main help
da3 --help
# 🔍 View specific command help
da3 auto --help
da3 image --help
da3 backend --help
```

183
Depth-Anything-V3-main/docs/funcs/ref_view_strategy.md Normal file
View File

@@ -0,0 +1,183 @@
# 📐 Reference View Selection Strategy
## 📖 Overview
Reference view selection is a component in multi-view depth estimation. When processing multiple input views, the model needs to determine which view should serve as the primary reference frame for depth prediction, defining the world coordinate system.
Different reference view will leads to different reconstruction results. This is a known consideration in multi-view geometry and was analyzed in [PI3](https://arxiv.org/abs/2507.13347). The choice of reference view can affect the quality and consistency of depth predictions across the scene.
## 🚀 Our Simple Solution: Automatic Reference View Selection
DA3 provides a simple approach to address this through **automatic reference view selection** based on **class tokens**. Instead of relying on heuristics or manual selection, the model analyzes the class token features from all input views and intelligently selects the most suitable reference frame.
---
## 🎨 Available Strategies
### 1. ⚖️ `saddle_balanced` (Recommended, Default)
**Philosophy:**
Select a view that achieves balance across multiple feature metrics. This strategy looks for a "middle ground" view that is neither too similar nor too different from other views, making it a stable reference point.
**How it works:**
1. Extracts and normalizes class tokens from all views
2. Computes three complementary metrics for each view:
- **Similarity score**: Average cosine similarity with other views
- **Feature norm**: L2 norm of the original features
- **Feature variance**: Variance across feature dimensions
3. Normalizes each metric to [0, 1] range
4. Selects the view closest to 0.5 (median) across all three metrics
### 2. 🎢 `saddle_sim_range`
**Philosophy:**
Select a view with the largest similarity range to other views. This identifies "saddle point" views that are highly similar to some views but dissimilar to others, making them information-rich anchor points.
**How it works:**
1. Computes pairwise cosine similarity between all views
2. For each view, calculates the range (max - min) of similarities to other views
3. Selects the view with the maximum similarity range
---
### 3. 1️⃣ `first` (Not Recommended)
**Philosophy:**
Always use the first view in the input sequence as the reference.
**How it works:**
Simply returns index 0.
**When to use:**
- ⛔ **Not recommended** in general
- 🔧 Only use when you have manually pre-sorted your views and know the first view is optimal
- 🐛 Debugging or baseline comparisons
---
### 4. ⏸️ `middle`
**Philosophy:**
Select the view in the middle of the input sequence.
**How it works:**
Returns the view at index `S // 2` where S is the number of views.
**When to use:**
- ⏱️ **Only recommended when input images are temporally ordered**
- 🎬 Video sequences (e.g., **DA3-LONG** setting)
- 📹 Sequential captures where the middle frame likely has the most stable viewpoint
**Specific use case: DA3-LONG** 🎬
In video-based depth estimation scenarios (like DA3-LONG), where inputs are consecutive frames, `middle` is often the **optimal choice** because that it has maximum overlap with all other frames.
## 💻 Usage
### 🐍 Python API
```python
from depth_anything_3 import DepthAnything3
model = DepthAnything3.from_pretrained("depth-anything/DA3NESTED-GIANT-LARGE")
# Use default (saddle_balanced)
prediction = model.inference(
images,
ref_view_strategy="saddle_balanced"
)
# For video sequences, consider using middle
prediction = model.inference(
video_frames,
ref_view_strategy="middle" # Good for temporal sequences
)
# For complex scenes with wide baselines
prediction = model.inference(
images,
ref_view_strategy="saddle_sim_range"
)
```
### 🖥️ Command Line Interface
```bash
# Default (saddle_balanced)
da3 auto input/ --export-dir output/
# Explicitly specify strategy
da3 auto input/ --ref-view-strategy saddle_balanced
# For video processing
da3 video input.mp4 --ref-view-strategy middle
# For wide-baseline multi-view
da3 images captures/ --ref-view-strategy saddle_sim_range
```
---
### 🎯 When Selection Is Applied
Reference view selection is applied when:
- 3️⃣ Number of views S ≥ 3
---
## 💡 Recommendations
### 📋 Quick Guide
| Scenario | Recommended Strategy | Rationale |
|----------|---------------------|-----------|
| **Default / Unknown** | `saddle_balanced` | Robust, balanced, works well across diverse scenarios |
| **Video frames** | `middle` | Temporal coherence, stable middle frame |
| **Wide-baseline multi-view** | `saddle_sim_range` | Maximizes information coverage |
| **Pre-sorted inputs** | `first` | Use only if you've manually optimized ordering |
| **Single image** | `first` | Automatically used (no reordering needed for S ≤ 2) |
### ✨ Best Practices
1. 🎯 **Start with defaults**: `saddle_balanced` works well in most cases
2. 🎬 **Consider your input type**: Use `middle` for videos, `saddle_balanced` for photos
3. 🔬 **Experiment if needed**: Try different strategies if results are suboptimal
4. 📊 **Monitor performance**: Check `glb` quality and consistency across views.
---
## 🔧 Technical Details
### 🎚️ Selection Threshold
The reference view selection is only triggered when:
```python
num_views >= 3 # At least 3 views required
```
For 1-2 views, no reordering is performed (equivalent to using `first`).
### ⚙️ Implementation
The selection happens at layer `alt_start - 1` in the vision transformer, before the first global attention layer. This ensures the selected reference view influences the entire depth prediction pipeline.
---
## ❓ FAQ
**Q: 🤔 Why is this feature provided?**
A: The model can handle any view order, but this feature provides automatic optimization for reference view selection, which can help improve depth prediction quality in multi-view scenarios.
**Q: ⏱️ Does this add computational cost?**
A: The overhead is totally negligible.
**Q: 🎮 Can I manually specify which view to use as reference?**
A: Not directly through this parameter. You can pre-sort your input images to place your preferred reference view first and use `ref_view_strategy="first"`.
**Q: ⚙️ What happens if I don't specify this parameter?**
A: The default `saddle_balanced` strategy is used automatically.
**Q: 📊 Is this feature used in the DA3 paper benchmarks?**
A: No, the paper used `first` as the default strategy for all multi-view experiments. The current default has been updated to `saddle_balanced` for better robustness.