# Seg Data Server Net Modular web control plane for the existing Seg image segmentation workspace. The platform keeps the current training and analysis scripts as the compute core, then adds: - a FastAPI backend for catalog discovery, job orchestration, logs, results, GPU status, and weight management; - a React/Vite frontend for launching jobs and inspecting progress; - a unified `weights/` area with a generated manifest for `.pt`, `.pth`, `.onnx`, and `.engine` assets. ## Layout ```text Seg_Data_Server_Net/ backend/ FastAPI API, job runner, module wrappers frontend/ React + Vite operator UI envs/ conda environment specs for task and MMSeg runtimes scripts/ helper scripts for running services and syncing weights weights/ copied model weights and manifest.json ``` ## Quick Start ```bash cd Seg_Data_Server_Net cp .env.example .env # Create or repair the two runtime environments, then verify imports. scripts/bootstrap_conda_envs.sh # Backend. The script loads .env and uses SEG_BACKEND_CONDA_ENV/PORT/HOST. scripts/run_backend.sh # Frontend. The script loads .env, installs npm packages, and starts Vite. scripts/run_frontend.sh ``` Open the Vite URL shown in the terminal. The frontend expects the backend at `http://localhost:8010` by default. ## Deployment and Environment The expected deployment layout keeps the original algorithm workspace next to this web project: ```text /home/wkmgc/Desktop/Data_Disk_1/Seg/ Seg/ existing algorithms, datasets, logs, and raw outputs Seg_Data_Server_Net/ this FastAPI + React control plane ``` Clone or update the web project from Gitea: ```bash git clone https://gitea.huijutec.cn/admin/Seg_Data_Server_Net.git cd Seg_Data_Server_Net cp .env.example .env ``` Edit `.env` for the server before first boot. Relative paths are resolved from the `Seg_Data_Server_Net/` project root; absolute paths are safest when moving to another machine. ```bash SEG_SOURCE_ROOT=../Seg SEG_DATA_SERVER_ROOT=. SEG_BACKEND_DB=var/seg_data_server.sqlite3 SEG_BACKEND_LOG_DIR=var/job_logs SEG_BACKEND_HOST=0.0.0.0 SEG_BACKEND_PORT=8010 SEG_BACKEND_RELOAD=1 SEG_TASK_CONDA_ENV=seg_smp SEG_MMSEG_CONDA_ENV=seg_mmcv SEG_BACKEND_CONDA_ENV=seg_smp SEG_WEIGHT_MODE=copy SEG_ENABLE_SHELL_TASKS=1 SEG_VALIDATE_DEEP=1 SEG_FRONTEND_HOST=0.0.0.0 SEG_FRONTEND_PORT=5173 VITE_API_BASE=http://localhost:8010 ``` Environment variables used during deployment: | Variable | Purpose | | --- | --- | | `SEG_SOURCE_ROOT` | Path to the original `Seg/` algorithm workspace. | | `SEG_DATA_SERVER_ROOT` | Runtime root for this web project. Keep `.` for normal deployments. | | `SEG_BACKEND_DB` | SQLite database used for datasets, jobs, and profiles. | | `SEG_BACKEND_LOG_DIR` | Directory for job logs streamed through SSE. | | `SEG_BACKEND_HOST` / `SEG_BACKEND_PORT` | FastAPI listen address. | | `SEG_BACKEND_RELOAD` | Set `1` for development reload, `0` for long-running production workers. | | `SEG_BACKEND_CONDA_ENV` | Conda env used to run FastAPI. | | `SEG_TASK_CONDA_ENV` | Default env for dataset, SegModel, YOLO, visual, and analysis jobs. | | `SEG_MMSEG_CONDA_ENV` | Dedicated env for full MMSeg/mmcv jobs. | | `SEG_WEIGHT_MODE` | `copy` or `reflink` when syncing weights. | | `SEG_ENABLE_SHELL_TASKS` | Enables execution of the wrapped legacy shell/Python tasks. | | `SEG_VALIDATE_DEEP` | Enables deep acceptance by default for local agent runs. | | `SEG_FRONTEND_HOST` / `SEG_FRONTEND_PORT` | Vite development server listen address. | | `VITE_API_BASE` | Backend URL embedded into the frontend build or used by the dev server. | Install system prerequisites first: Git, Conda or Miniconda, Node.js/npm, a working NVIDIA driver and `nvidia-smi` for GPU discovery, and enough free disk space for the copied weights. Full weight sync currently needs about 35 GB plus normal training output space. Create the Python runtimes with the bundled bootstrap script: ```bash scripts/bootstrap_conda_envs.sh all ``` This creates `seg_smp` for the backend, SegModel, YOLO, dataset tools, and general analysis jobs, plus `seg_mmcv` for full MMSeg/mmcv execution. To repair only one environment: ```bash scripts/bootstrap_conda_envs.sh task scripts/bootstrap_conda_envs.sh mmseg ``` Install and build the frontend once during deployment: ```bash cd frontend npm install npm run build cd .. ``` Synchronize weights locally after the original `Seg/` directory is present. The command copies `.pt`, `.pth`, `.onnx`, and `.engine` files into `weights/files/` and writes `weights/manifest.json`. ```bash python scripts/sync_weights.py --mode copy --hash ``` If the deployment filesystem supports copy-on-write reflinks and disk space is tight, set `SEG_WEIGHT_MODE=reflink` or pass `--mode reflink`; otherwise keep the default `copy` mode. Weight binaries must stay out of normal Git commits. Start the services: ```bash # Terminal 1 scripts/run_backend.sh # Terminal 2 scripts/run_frontend.sh ``` Defaults are `0.0.0.0:8010` for the backend and `0.0.0.0:5173` for the frontend development server. Override backend binding with `SEG_BACKEND_HOST` and `SEG_BACKEND_PORT`; override the Vite dev server with `SEG_FRONTEND_HOST` and `SEG_FRONTEND_PORT`. For long-running production service management, disable backend reload and start the same script from systemd, supervisor, or Docker Compose so `.env` and the configured conda environments are used consistently: ```bash SEG_BACKEND_RELOAD=0 scripts/run_backend.sh ``` For a static frontend deployment, set `VITE_API_BASE` to the public backend URL before building, then serve `frontend/dist/` with Nginx or another static file server. A quick local preview is: ```bash cd frontend npm run build npm run preview -- --host 0.0.0.0 --port 4173 ``` If systemd is used, a minimal backend unit can call the script directly: ```ini [Service] WorkingDirectory=/home/wkmgc/Desktop/Data_Disk_1/Seg/Seg_Data_Server_Net Environment=SEG_BACKEND_RELOAD=0 ExecStart=/home/wkmgc/Desktop/Data_Disk_1/Seg/Seg_Data_Server_Net/scripts/run_backend.sh Restart=always ``` Validate a deployment before handing it to operators: ```bash curl http://127.0.0.1:8010/api/health curl http://127.0.0.1:8010/api/system/readiness PYTHONPATH=backend conda run -n seg_smp python scripts/verify_runtime_envs.py --refresh PYTHONPATH=backend conda run -n seg_smp python scripts/run_agents.py --build scripts/check_no_weight_git.sh ``` For a fast non-training validation pass, run agents with `PYTHONPATH=backend conda run -n seg_smp python scripts/run_agents.py --no-deep`. Add `--live`, `--acceptance`, or `--real` only after the backend and frontend are running and you want HTTP endpoint, smoke, or real-workspace checks. The browser dashboard exposes the same readiness, coverage, GPU, weight, result, and agent checks through the UI. The web UI includes a dataset bench for creating upload workspaces, uploading images/labels/masks, and jumping into the existing rename, PNG conversion, resize, pair-check, label rebuild, transparent overlay, stitch, and video-frame jobs. Selecting an uploaded dataset fills task JSON with its images, labels, and masks directories. The dataset panel validates image/label/mask pairing, checks YOLO txt labels and mask dimensions, and can generate a `dataset.yaml` for the `yolo.train_custom` task. The selected upload dataset also exposes direct YOLO custom train, predict, and heatmap actions; custom outputs are written under `var/custom_yolo_runs` and are scanned by the results dashboard. When a dataset is selected, the dataset panel shows its custom YOLO `best.pt`, prediction previews, heatmap previews, and inline training curve previews. Segmentation previews, YOLO heatmaps, and loss/metric artifacts are grouped on the results dashboard. The artifact browser loads the full result scan, can filter by model family and artifact role, and YOLO-style `results.csv` files are parsed into lightweight training curves. Job APIs and the SSE log stream also expose structured progress parsed from YOLO, MMSeg/MMEngine, SegModel-style epoch logs, and generic tqdm percentages, so the queue and live log panel can show stage, epoch/iteration, and percent without changing the original training scripts. Starting any web job or dataset YOLO shortcut automatically opens its live log; the SSE stream resumes from the current log size after the initial tail so existing lines are not duplicated in the panel. The selected-job panel also shows reproducibility diagnostics: command, cwd, PID, exit code, log size, error text, and the exact task params submitted. The task builder also reads `GET /api/system/gpus` and lets an operator choose CPU or one or more GPUs before launch. Selected GPUs are passed to the backend as `gpus`, exported as `CUDA_VISIBLE_DEVICES`, and reflected into YOLO/visual `device` parameters and MMSeg config-generation `gpu_count/gpu_ids`. The same job launcher reads `GET /api/system/envs` and provides an Auto/manual conda environment selector. Auto keeps the backend defaults (`seg_smp` for general SegModel/YOLO/dataset tasks and `seg_mmcv` for MMSeg); manual mode sends `conda_env` with the job request for custom algorithm environments. The coverage panel calls `GET /api/coverage` and verifies that the user-facing scripts from the existing `Seg/` workspace are mapped to web jobs. MMSeg vendored internals, docs, build outputs, converters, and config templates are classified as supporting artifacts rather than direct web actions. The runtime panel calls `GET /api/system/readiness` and verifies the conda imports required for the backend/task environment and the full MMSeg/mmcv environment. Command-line verification is available with `PYTHONPATH=backend conda run -n seg_smp python scripts/verify_runtime_envs.py --refresh`. The capability matrix calls `GET /api/capabilities` and summarizes readiness for Dataset, SegModel, YOLO, MMSeg, visual tools, analysis, and system operations, including task coverage, runtime status, uploaded datasets, heatmap/segmentation artifacts, training curves, and weight manifest status. The same panel can run `POST /api/acceptance/smoke`, a lightweight live smoke that creates an upload dataset, uploads a label, downloads it through the artifact API, runs a mock job, checks SSE log streaming, and executes one legacy image/label overlay job on tiny generated PNGs. It also launches `yolo.predict_custom` and `yolo.heatmap_custom` through the normal job queue against the uploaded sample image, proving that upload datasets can produce browsable segmentation and heatmap artifacts. It also runs model family readiness checks: a SegModel/SMP forward pass, a YOLO segmentation prediction on a tiny image, MMSeg config parsing, and local MMSeg pretrained weight discovery. MMSeg full-model readiness is validated in `SEG_MMSEG_CONDA_ENV` by importing `mmcv._ext` and building a local MMSeg `EncoderDecoder` from the existing config tree. `POST /api/acceptance/real` runs the same operator-facing path on existing non-synthetic workspace files. It discovers a real `DataSet_Own/*_Ori` image with a matching mask, discovers a real YOLO image/txt pair from `Seg_All_In_One_YoloModel/Yolo数据集构建/Data`, uploads those files through the dataset API, validates YOLO and mask readiness, generates `dataset.yaml`, runs YOLO prediction and heatmap jobs, and runs the legacy stack job on the uploaded real image/mask pair. The latest report is available from `GET /api/acceptance/real/latest` and is shown in the coverage panel as `真实数据`. For stronger runtime proof, `POST /api/acceptance/deep` runs minimal training loops for the three model families: one SegModel optimizer step, one YOLO segmentation epoch on a synthetic 64x64 dataset, one YOLO GradCAM heatmap generation pass from the trained tiny checkpoint, and one MMSeg optimizer step through the full `mmcv._ext` runtime. It also writes tiny SegModel mask/loss artifacts, YOLO heatmap/results artifacts, and MMSeg loss artifacts under `var/acceptance/deep_*`, so the normal results and curve dashboards can prove each model family produced browsable output. The latest report is available from `GET /api/acceptance/deep/latest` and is surfaced in the coverage panel. Current `seg_smp` uses `mmcv-lite` because no `torch 2.6/cu124` full `mmcv` wheel is available on this machine and `nvcc` is not installed for source builds. A dedicated `seg_mmcv` environment is used for MMSeg tasks and has `torch 2.1.2+cu121`, `mmcv 2.1.0`, `mmsegmentation 1.2.2`, and NumPy 1.26. The reproducible specs live in `envs/seg_smp.yml` and `envs/seg_mmcv.yml`; the bootstrap script uses the same pinned package sources: ```bash scripts/bootstrap_conda_envs.sh all scripts/bootstrap_conda_envs.sh task scripts/bootstrap_conda_envs.sh mmseg ``` ## Weight Sync The current workspace contains tens of GB of pretrained and trained weights. They are copied into `weights/files/` and indexed in `weights/manifest.json`. The web weight panel reads `GET /api/weights`, shows family/role counts and sample source paths, and can run `POST /api/weights/verify` to check the local manifest entries without putting the large files in Git. ```bash cd Seg_Data_Server_Net python scripts/sync_weights.py --mode copy --hash ``` Weights must remain local to the deployment machine. Do not push `.pt`, `.pth`, `.onnx`, `.engine`, or `weights/files/` into Gitea. The repository stores only code, `weights/manifest.json`, and helper scripts. Before pushing, run: ```bash scripts/check_no_weight_git.sh ``` If a deployment machine needs weights, run the sync command locally on that machine after cloning the code. ## Job Types The backend exposes all current Seg capabilities as job types. Examples: - `dataset.rename`, `dataset.resize`, `dataset.pair`, `dataset.rebuild_labels`, `dataset.stack`, `dataset.stitch`, `dataset.video_frames`, `dataset.yolo_check_pairs`, `dataset.yolo_stack`, `dataset.yolo_txt_sort`, `dataset.yolo_convert_png`, `dataset.yolo_resize` - `segmodel.train`, `segmodel.batch_train`, `segmodel.predict`, `segmodel.batch_predict`, `segmodel.flops`, `segmodel.params_flops`, `segmodel.benchmark`, `segmodel.raw_mask_check` - `yolo.train`, `yolo.batch_train`, `yolo.predict`, `yolo.batch_predict`, `yolo.train_custom`, `yolo.predict_custom`, `yolo.heatmap`, `yolo.heatmap_custom`, `yolo.compare`, `yolo.raw_mask_check`, `yolo.video_visible` - `mmseg.generate_data`, `mmseg.generate_alg`, `mmseg.train`, `mmseg.metrics`, `mmseg.flops_fps`, `mmseg.draw`, `mmseg.extract_loss_miou` - `visual.train`, `visual.inference`, `visual.fps`, `visual.yolo11_heatmap_v1`, `visual.yolo11_heatmap_v2`, `visual.deal_labels` - `analysis.all`, `system.backup`, `mock.echo` Use `GET /api/catalog` to inspect supported models, algorithms, datasets, and task types discovered from the existing `Seg/` workspace. Use `GET /api/coverage` to inspect script-to-task coverage and task buildability. Use `GET /api/capabilities` to inspect the grouped full-function readiness matrix used by the web dashboard and agents. Use `GET /api/results?limit=1000` to inspect browsable artifacts and `GET /api/results/curves?limit=100` to inspect parsed training curves discovered from YOLO, SegModel, MMSeg, visual-tool, and analysis output directories. Use `GET /api/agents/evaluate` and `GET /api/agents/validate` to surface the same evaluation and validation feedback shown in the web dashboard Agent panel. ## Agents Run the local evaluation and validation agents before publishing changes: ```bash PYTHONPATH=backend conda run -n seg_smp python scripts/run_agents.py --build ``` The validation agent checks catalog coverage, the grouped capability matrix, the `seg_smp` task env, the `seg_mmcv` MMSeg env, runtime import readiness, GPU visibility, no-weight Git safety, backend tests, frontend build, and live backend/frontend endpoints when the services are running. With live validation enabled it also runs the lightweight acceptance smoke above. By default it also runs the deep training acceptance; set `SEG_VALIDATE_DEEP=0` when a quick non-training validation pass is needed. The web dashboard calls validation in light mode by default: `/api/agents/validate?run_build=false&run_acceptance=false&run_deep=false`. Pass `run_live=true`, `run_acceptance=true`, `run_real=true`, or `run_deep=true` only when you explicitly want the agent to launch live endpoint or heavier runtime acceptance checks from the browser/API. Smoke and real data acceptance automatically enable the live backend checks because they submit jobs through the API.