admin/Seg_Data_Server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

first commit

Browse Source
This commit is contained in:
admin
2026-05-20 15:05:35 +08:00
commit ac09b26253
2048 changed files with 189478 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

46
Seg_All_In_One_MMSeg/projects/Adabins/README.md Normal file
View File

@@ -0,0 +1,46 @@
# AdaBins: Depth Estimation Using Adaptive Bins
## Reference
> [AdaBins: Depth Estimation Using Adaptive Bins](https://arxiv.org/abs/2011.14141)
## Introduction
<a href="https://github.com/shariqfarooq123/AdaBins">Official Repo</a>
<a href="https://github.com/open-mmlab/mmsegmentation/tree/dev-1.x/projects/Adabins">Code Snippet</a>
## <img src="https://user-images.githubusercontent.com/34859558/190043857-bfbdaf8b-d2dc-4fff-81c7-e0aac50851f9.png" width="25"/> Abstract
We address the problem of estimating a high quality dense depth map from a single RGB input image. We start out with a baseline encoder-decoder convolutional neural network architecture and pose the question of how the global processing of information can help improve overall depth estimation. To this end, we propose a transformer-based architecture block that divides the depth range into bins whose center value is estimated adaptively per image. The final depth values are estimated as linear combinations of the bin centers. We call our new building block AdaBins. Our results show a decisive improvement over the state-of-the-art on several popular depth datasets across all metrics.We also validate the effectiveness of the proposed block with an ablation study and provide the code and corresponding pre-trained weights of the new state-of-the-art model.
Our main contributions are the following:
- We propose an architecture building block that performs global processing of the scene’s information.We propose to divide the predicted depth range into bins where the bin widths change per image. The final depth estimation is a linear combination of the bin center values.
- We show a decisive improvement for supervised single image depth estimation across all metrics for the two most popular datasets, NYU and KITTI.
- We analyze our findings and investigate different modifications on the proposed AdaBins block and study their effect on the accuracy of the depth estimation.
<div align="center">
<img src="https://github.com/open-mmlab/mmsegmentation/assets/15952744/915bcd5a-9dc2-4602-a6e7-055ff5d4889f" width = "1000" />
</div>
## <img src="https://user-images.githubusercontent.com/34859558/190044217-8f6befc2-7f20-473d-b356-148e06265205.png" width="25"/> Performance
### NYU and KITTI
| Model | Encoder | Training epoch | Batchsize | Train Resolution | δ1 | δ2 | δ3 | REL | RMS | RMS log | params(M) | Links |
| ------------- | --------------- | -------------- | --------- | ---------------- | ----- | ----- | ----- | ----- | ----- | ------- | --------- | ----------------------------------------------------------------------------------------------------------------------- |
| AdaBins_nyu | EfficientNet-B5 | 25 | 16 | 416x544 | 0.903 | 0.984 | 0.997 | 0.103 | 0.364 | 0.044 | 78 | [model](https://download.openmmlab.com/mmsegmentation/v0.5/adabins/adabins_efficient_b5_nyu_third-party-f68d6bd3.pth) |
| AdaBins_kitti | EfficientNet-B5 | 25 | 16 | 352x764 | 0.964 | 0.995 | 0.999 | 0.058 | 2.360 | 0.088 | 78 | [model](https://download.openmmlab.com/mmsegmentation/v0.5/adabins/adabins_efficient-b5_kitty_third-party-a1aa6f36.pth) |
## Citation
```bibtex
@article{10.1109/cvpr46437.2021.00400,
author = {Bhat, S. A. and Alhashim, I. and Wonka, P.},
title = {Adabins: depth estimation using adaptive bins},
journal = {2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2021},
doi = {10.1109/cvpr46437.2021.00400}
}
```

4
Seg_All_In_One_MMSeg/projects/Adabins/backbones/__init__.py Normal file
View File

@@ -0,0 +1,4 @@
# Copyright (c) OpenMMLab. All rights reserved.
from .adabins_backbone import AdabinsBackbone
__all__ = ['AdabinsBackbone']

141
Seg_All_In_One_MMSeg/projects/Adabins/backbones/adabins_backbone.py Normal file
View File

@@ -0,0 +1,141 @@
import timm
import torch
import torch.nn as nn
import torch.nn.functional as F
from mmcv.cnn import ConvModule, build_conv_layer
from mmengine.model import BaseModule
from mmseg.registry import MODELS
class UpSampleBN(nn.Module):
""" UpSample module
Args:
skip_input (int): the input feature
output_features (int): the output feature
norm_cfg (dict, optional): Config dict for normalization layer.
Default: dict(type='BN', requires_grad=True).
act_cfg (dict, optional): The activation layer of AAM:
Aggregate Attention Module.
"""
def __init__(self,
skip_input,
output_features,
norm_cfg=dict(type='BN'),
act_cfg=dict(type='LeakyReLU')):
super().__init__()
self._net = nn.Sequential(
ConvModule(
in_channels=skip_input,
out_channels=output_features,
kernel_size=3,
stride=1,
padding=1,
bias=True,
norm_cfg=norm_cfg,
act_cfg=act_cfg,
),
ConvModule(
in_channels=output_features,
out_channels=output_features,
kernel_size=3,
stride=1,
padding=1,
bias=True,
norm_cfg=norm_cfg,
act_cfg=act_cfg,
))
def forward(self, x, concat_with):
up_x = F.interpolate(
x,
size=[concat_with.size(2),
concat_with.size(3)],
mode='bilinear',
align_corners=True)
f = torch.cat([up_x, concat_with], dim=1)
return self._net(f)
class Encoder(nn.Module):
""" the efficientnet_b5 model
Args:
basemodel_name (str): the name of base model
"""
def __init__(self, basemodel_name):
super().__init__()
self.original_model = timm.create_model(
basemodel_name, pretrained=True)
# Remove last layer
self.original_model.global_pool = nn.Identity()
self.original_model.classifier = nn.Identity()
def forward(self, x):
features = [x]
for k, v in self.original_model._modules.items():
if k == 'blocks':
for ki, vi in v._modules.items():
features.append(vi(features[-1]))
else:
features.append(v(features[-1]))
return features
@MODELS.register_module()
class AdabinsBackbone(BaseModule):
""" the backbone of the adabins
Args:
basemodel_name (str):the name of base model
num_features (int): the middle feature
num_classes (int): the classes number
bottleneck_features (int): the bottleneck features
conv_cfg (dict): Config dict for convolution layer.
"""
def __init__(self,
basemodel_name,
num_features=2048,
num_classes=128,
bottleneck_features=2048,
conv_cfg=dict(type='Conv')):
super().__init__()
self.encoder = Encoder(basemodel_name)
features = int(num_features)
self.conv2 = build_conv_layer(
conv_cfg,
bottleneck_features,
features,
kernel_size=1,
stride=1,
padding=1)
self.up1 = UpSampleBN(
skip_input=features // 1 + 112 + 64, output_features=features // 2)
self.up2 = UpSampleBN(
skip_input=features // 2 + 40 + 24, output_features=features // 4)
self.up3 = UpSampleBN(
skip_input=features // 4 + 24 + 16, output_features=features // 8)
self.up4 = UpSampleBN(
skip_input=features // 8 + 16 + 8, output_features=features // 16)
self.conv3 = build_conv_layer(
conv_cfg,
features // 16,
num_classes,
kernel_size=3,
stride=1,
padding=1)
def forward(self, x):
features = self.encoder(x)
x_block0, x_block1, x_block2, x_block3, x_block4 = features[
3], features[4], features[5], features[7], features[10]
x_d0 = self.conv2(x_block4)
x_d1 = self.up1(x_d0, x_block3)
x_d2 = self.up2(x_d1, x_block2)
x_d3 = self.up3(x_d2, x_block1)
x_d4 = self.up4(x_d3, x_block0)
out = self.conv3(x_d4)
return out

32
Seg_All_In_One_MMSeg/projects/Adabins/configs/_base_/datasets/nyu.py Normal file
View File

@@ -0,0 +1,32 @@
dataset_type = 'NYUDataset'
data_root = 'data/nyu'
test_pipeline = [
dict(dict(type='LoadImageFromFile', to_float32=True)),
dict(dict(type='LoadDepthAnnotation', depth_rescale_factor=1e-3)),
dict(
type='PackSegInputs',
meta_keys=('img_path', 'depth_map_path', 'ori_shape', 'img_shape',
'pad_shape', 'scale_factor', 'flip', 'flip_direction',
'category_id'))
]
val_dataloader = dict(
batch_size=1,
num_workers=4,
persistent_workers=True,
sampler=dict(type='DefaultSampler', shuffle=False),
dataset=dict(
type=dataset_type,
data_root=data_root,
test_mode=True,
data_prefix=dict(
img_path='images/test', depth_map_path='annotations/test'),
pipeline=test_pipeline))
test_dataloader = val_dataloader
val_evaluator = dict(
type='DepthMetric', max_depth_eval=10.0, crop_type='nyu_crop')
test_evaluator = val_evaluator
val_cfg = dict(type='ValLoop')
test_cfg = dict(type='TestLoop')

15
Seg_All_In_One_MMSeg/projects/Adabins/configs/_base_/default_runtime.py Normal file
View File

@@ -0,0 +1,15 @@
default_scope = 'mmseg'
env_cfg = dict(
cudnn_benchmark=True,
mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),
dist_cfg=dict(backend='nccl'),
)
vis_backends = [dict(type='LocalVisBackend')]
visualizer = dict(
type='SegLocalVisualizer', vis_backends=vis_backends, name='visualizer')
log_processor = dict(by_epoch=False)
log_level = 'INFO'
load_from = None
resume = False
tta_model = dict(type='SegTTAModel')

35
Seg_All_In_One_MMSeg/projects/Adabins/configs/_base_/models/Adabins.py Normal file
View File

@@ -0,0 +1,35 @@
# model settings
norm_cfg = dict(type='SyncBN', requires_grad=True)
data_preprocessor = dict(
type='SegDataPreProcessor',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
bgr_to_rgb=True,
pad_val=0,
seg_pad_val=255)
model = dict(
type='DepthEstimator',
data_preprocessor=data_preprocessor,
# pretrained='open-mmlab://resnet50_v1c',
backbone=dict(
type='AdabinsBackbone',
basemodel_name='tf_efficientnet_b5_ap',
num_features=2048,
num_classes=128,
bottleneck_features=2048,
),
decode_head=dict(
type='AdabinsHead',
in_channels=128,
n_query_channels=128,
patch_size=16,
embedding_dim=128,
num_heads=4,
n_bins=256,
min_val=0.001,
max_val=10,
norm='linear'),
# model training and testing settings
train_cfg=dict(),
test_cfg=dict(mode='whole'))

15
Seg_All_In_One_MMSeg/projects/Adabins/configs/adabins/adabins_efficient_b5_4x16_25e_NYU_416x544.py Normal file
View File

@@ -0,0 +1,15 @@
_base_ = [
'../_base_/models/Adabins.py', '../_base_/datasets/nyu.py',
'../_base_/default_runtime.py'
]
custom_imports = dict(
imports=['projects.Adabins.backbones', 'projects.Adabins.decode_head'],
allow_failed_imports=False)
crop_size = (416, 544)
data_preprocessor = dict(size=crop_size)
norm_cfg = dict(type='SyncBN', requires_grad=True)
model = dict(
data_preprocessor=data_preprocessor,
backbone=dict(),
decode_head=dict(),
)

12
Seg_All_In_One_MMSeg/projects/Adabins/configs/adabins/adabins_efficient_b5_4x16_25e_kitti_352x704.py Normal file
View File

@@ -0,0 +1,12 @@
_base_ = ['../_base_/models/Adabins.py']
custom_imports = dict(
imports=['projects.Adabins.backbones', 'projects.Adabins.decode_head'],
allow_failed_imports=False)
crop_size = (352, 704)
data_preprocessor = dict(size=crop_size)
norm_cfg = dict(type='SyncBN', requires_grad=True)
model = dict(
data_preprocessor=data_preprocessor,
backbone=dict(),
decode_head=dict(min_val=0.001, max_val=80),
)

4
Seg_All_In_One_MMSeg/projects/Adabins/decode_head/__init__.py Normal file
View File

@@ -0,0 +1,4 @@
# Copyright (c) OpenMMLab. All rights reserved.
from .adabins_head import AdabinsHead
__all__ = ['AdabinsHead']

179
Seg_All_In_One_MMSeg/projects/Adabins/decode_head/adabins_head.py Normal file
View File

@@ -0,0 +1,179 @@
from typing import List, Tuple
import torch
import torch.nn as nn
import torch.nn.functional as F
from mmcv.cnn import build_conv_layer
from torch import Tensor
from mmseg.registry import MODELS
class PatchTransformerEncoder(nn.Module):
"""the Patch Transformer Encoder.
Args:
in_channels (int): the channels of input
patch_size (int): the path size
embedding_dim (int): The feature dimension.
num_heads (int): the number of encoder head
conv_cfg (dict): Config dict for convolution layer.
"""
def __init__(self,
in_channels,
patch_size=10,
embedding_dim=128,
num_heads=4,
conv_cfg=dict(type='Conv')):
super().__init__()
encoder_layers = nn.TransformerEncoderLayer(
embedding_dim, num_heads, dim_feedforward=1024)
self.transformer_encoder = nn.TransformerEncoder(
encoder_layers, num_layers=4) # takes shape S,N,E
self.embedding_convPxP = build_conv_layer(
conv_cfg,
in_channels,
embedding_dim,
kernel_size=patch_size,
stride=patch_size)
self.positional_encodings = nn.Parameter(
torch.rand(500, embedding_dim), requires_grad=True)
def forward(self, x):
embeddings = self.embedding_convPxP(x).flatten(
2) # .shape = n,c,s = n, embedding_dim, s
embeddings = embeddings + self.positional_encodings[:embeddings.shape[
2], :].T.unsqueeze(0)
# change to S,N,E format required by transformer
embeddings = embeddings.permute(2, 0, 1)
x = self.transformer_encoder(embeddings) # .shape = S, N, E
return x
class PixelWiseDotProduct(nn.Module):
"""the pixel wise dot product."""
def __init__(self):
super().__init__()
def forward(self, x, K):
n, c, h, w = x.size()
_, cout, ck = K.size()
assert c == ck, 'Number of channels in x and Embedding dimension ' \
'(at dim 2) of K matrix must match'
y = torch.matmul(
x.view(n, c, h * w).permute(0, 2, 1),
K.permute(0, 2, 1)) # .shape = n, hw, cout
return y.permute(0, 2, 1).view(n, cout, h, w)
@MODELS.register_module()
class AdabinsHead(nn.Module):
"""the head of the adabins,include mViT.
Args:
in_channels (int):the channels of the input
n_query_channels (int):the channels of the query
patch_size (int): the patch size
embedding_dim (int):The feature dimension.
num_heads (int):the number of head
n_bins (int):the number of bins
min_val (float): the min width of bin
max_val (float): the max width of bin
conv_cfg (dict): Config dict for convolution layer.
norm (str): the activate method
align_corners (bool, optional): Geometrically, we consider the pixels
of the input and output as squares rather than points.
"""
def __init__(self,
in_channels,
n_query_channels=128,
patch_size=16,
embedding_dim=128,
num_heads=4,
n_bins=100,
min_val=0.1,
max_val=10,
conv_cfg=dict(type='Conv'),
norm='linear',
align_corners=False,
threshold=0):
super().__init__()
self.out_channels = n_bins
self.align_corners = align_corners
self.norm = norm
self.num_classes = n_bins
self.min_val = min_val
self.max_val = max_val
self.n_query_channels = n_query_channels
self.patch_transformer = PatchTransformerEncoder(
in_channels, patch_size, embedding_dim, num_heads)
self.dot_product_layer = PixelWiseDotProduct()
self.threshold = threshold
self.conv3x3 = build_conv_layer(
conv_cfg,
in_channels,
embedding_dim,
kernel_size=3,
stride=1,
padding=1)
self.regressor = nn.Sequential(
nn.Linear(embedding_dim, 256), nn.LeakyReLU(), nn.Linear(256, 256),
nn.LeakyReLU(), nn.Linear(256, n_bins))
self.conv_out = nn.Sequential(
build_conv_layer(conv_cfg, in_channels, n_bins, kernel_size=1),
nn.Softmax(dim=1))
def forward(self, x):
# n, c, h, w = x.size()
tgt = self.patch_transformer(x.clone()) # .shape = S, N, E
x = self.conv3x3(x)
regression_head, queries = tgt[0,
...], tgt[1:self.n_query_channels + 1,
...]
# Change from S, N, E to N, S, E
queries = queries.permute(1, 0, 2)
range_attention_maps = self.dot_product_layer(
x, queries) # .shape = n, n_query_channels, h, w
y = self.regressor(regression_head) # .shape = N, dim_out
if self.norm == 'linear':
y = torch.relu(y)
eps = 0.1
y = y + eps
elif self.norm == 'softmax':
return torch.softmax(y, dim=1), range_attention_maps
else:
y = torch.sigmoid(y)
bin_widths_normed = y / y.sum(dim=1, keepdim=True)
out = self.conv_out(range_attention_maps)
bin_widths = (self.max_val -
self.min_val) * bin_widths_normed # .shape = N, dim_out
bin_widths = F.pad(
bin_widths, (1, 0), mode='constant', value=self.min_val)
bin_edges = torch.cumsum(bin_widths, dim=1)
centers = 0.5 * (bin_edges[:, :-1] + bin_edges[:, 1:])
n, dim_out = centers.size()
centers = centers.view(n, dim_out, 1, 1)
pred = torch.sum(out * centers, dim=1, keepdim=True)
return bin_edges, pred
def predict(self, inputs: Tuple[Tensor], batch_img_metas: List[dict],
test_cfg, **kwargs) -> Tensor:
"""Forward function for testing, only ``pam_cam`` is used."""
pred = self.forward(inputs)[-1]
final = torch.clamp(pred, self.min_val, self.max_val)
final[torch.isinf(final)] = self.max_val
final[torch.isnan(final)] = self.min_val
return final