feat: 完善视频传播、标注编辑和拆帧闭环

- 接入 SAM2 视频传播能力：新增 /api/ai/propagate，支持用当前帧 mask/polygon/bbox 作为 seed，通过 SAM2 video predictor 向前、向后或双向传播，并可保存为真实 annotation。 - 接入 SAM3 video tracker：通过独立 Python 3.12 external worker 调用 SAM3 video predictor/tracker，使用本地 checkpoint 与 bbox seed 执行视频级跟踪，并在模型状态中标记 video_track 能力。 - 完善 SAM 模型分发：sam_registry 按 model_id 明确区分 sam2 propagation 与 sam3 video_track，避免两个模型链路混用。 - 打通前端“传播片段”：VideoWorkspace 使用当前选中 mask 和当前 AI 模型调用后端传播接口，传播结果回写并刷新工作区已保存标注。 - 增强 SAM3 本地 checkpoint 配置：新增 sam3_checkpoint_path 配置和 .env.example 示例，状态检查改为基于本地 checkpoint/独立环境/模型包可用性。 - 完善视频拆帧参数：/api/media/parse 支持 parse_fps、max_frames、target_width，后端任务保存帧时间戳、源帧号和 frame_sequence 元数据。 - 增加运行时 schema 兼容处理：启动时为旧 frames 表补充 timestamp_ms 和 source_frame_number 列，避免旧库升级后缺字段。 - 强化 Canvas 标注编辑：补齐多边形闭合、点工具、顶点拖拽、边中点插入、Delete/Backspace 删除、区域合并和重叠去除等交互。 - 增强语义分类联动：选中 mask 后可通过右侧语义分类树更新标签、颜色和 class metadata，并同步到保存/导出链路。 - 增加关键帧时间轴体验：FrameTimeline 显示具体时间信息，并支持键盘左右方向键切换关键帧。 - 完善 AI 交互分割参数：前端保留正向点、反向点、框选和 interactive prompt 的调用状态，支持 SAM2 细化候选区域与 SAM3 bbox 入口。 - 扩展后端/前端 API 类型：新增 propagateMasks、传播请求/响应 schema，并补齐 annotation、导出、模型状态和任务接口的测试覆盖。 - 更新项目文档：同步 README、AGENTS、接口契约、需求冻结、设计冻结、前端元素审计、实施计划和测试计划，标明真实功能边界与剩余风险。 - 增加测试覆盖：补充 SAM2/SAM3 传播、SAM3 状态、媒体拆帧参数、Canvas 编辑、语义标签切换、时间轴、工作区传播和 API 合约测试。 - 加强仓库安全边界：将 sam3权重/ 加入 .gitignore，避免本地模型权重被误提交。验证：npm run test:run；pytest backend/tests；npm run lint；npm run build；python -m py_compile；git diff --check。
2026-05-01 20:27:33 +08:00
parent 689a9ba283
commit 5ab4602535
43 changed files with 2722 additions and 216 deletions
--- a/src/components/CanvasArea.test.tsx
+++ b/src/components/CanvasArea.test.tsx
@@ -65,6 +65,157 @@ describe('CanvasArea', () => {
    }));
  });

+  it('explains that SAM3 point prompts are not supported in the workspace', async () => {
+    useStore.setState({ aiModel: 'sam3' });
+
+    render(<CanvasArea activeTool="point_pos" frame={frame} />);
+    fireEvent.click(screen.getByTestId('konva-stage'));
+
+    expect(apiMock.predictMask).not.toHaveBeenCalled();
+    expect(await screen.findByText(/SAM3 当前工作区只支持框选提示/)).toBeInTheDocument();
+  });
+
+  it('calls SAM3 prediction with a box prompt from the workspace', async () => {
+    useStore.setState({ aiModel: 'sam3' });
+    apiMock.predictMask.mockResolvedValueOnce({
+      masks: [
+        {
+          id: 'sam3-box-mask',
+          pathData: 'M 20 20 L 80 20 L 80 80 Z',
+          label: 'AI Mask',
+          color: '#06b6d4',
+          segmentation: [[20, 20, 80, 20, 80, 80]],
+          bbox: [20, 20, 60, 60],
+          area: 3600,
+        },
+      ],
+    });
+
+    render(<CanvasArea activeTool="box_select" frame={frame} />);
+    const stage = screen.getByTestId('konva-stage');
+    fireEvent.mouseDown(stage, { clientX: 120, clientY: 80 });
+    fireEvent.mouseMove(stage, { clientX: 260, clientY: 200 });
+    fireEvent.mouseUp(stage, { clientX: 260, clientY: 200 });
+
+    await waitFor(() => expect(apiMock.predictMask).toHaveBeenCalledWith({
+      imageId: 'frame-1',
+      imageWidth: 640,
+      imageHeight: 360,
+      model: 'sam3',
+      points: undefined,
+      box: { x1: 120, y1: 80, x2: 260, y2: 200 },
+    }));
+    expect(useStore.getState().masks[0]).toEqual(expect.objectContaining({
+      id: 'sam3-box-mask',
+      metadata: expect.objectContaining({
+        source: 'sam3_box',
+        promptBox: { x1: 120, y1: 80, x2: 260, y2: 200 },
+      }),
+    }));
+  });
+
+  it('refines one SAM2 candidate mask from an initial box with positive and negative points', async () => {
+    apiMock.predictMask
+      .mockResolvedValueOnce({
+        masks: [
+          {
+            id: 'mask-box',
+            pathData: 'M 10 10 L 90 10 L 90 90 Z',
+            label: 'AI Mask',
+            color: '#06b6d4',
+            segmentation: [[10, 10, 90, 10, 90, 90]],
+            bbox: [10, 10, 80, 80],
+            area: 6400,
+          },
+        ],
+      })
+      .mockResolvedValueOnce({
+        masks: [
+          {
+            id: 'mask-refined-pos',
+            pathData: 'M 20 20 L 80 20 L 80 80 Z',
+            label: 'AI Mask',
+            color: '#06b6d4',
+            segmentation: [[20, 20, 80, 20, 80, 80]],
+            bbox: [20, 20, 60, 60],
+            area: 3600,
+          },
+        ],
+      })
+      .mockResolvedValueOnce({
+        masks: [
+          {
+            id: 'mask-refined-neg',
+            pathData: 'M 30 30 L 70 30 L 70 70 Z',
+            label: 'AI Mask',
+            color: '#06b6d4',
+            segmentation: [[30, 30, 70, 30, 70, 70]],
+            bbox: [30, 30, 40, 40],
+            area: 1600,
+          },
+        ],
+      });
+
+    const { rerender } = render(<CanvasArea activeTool="box_select" frame={frame} />);
+    const stage = screen.getByTestId('konva-stage');
+    fireEvent.mouseDown(stage, { clientX: 120, clientY: 80 });
+    fireEvent.mouseMove(stage, { clientX: 260, clientY: 200 });
+    fireEvent.mouseUp(stage, { clientX: 260, clientY: 200 });
+
+    await waitFor(() => expect(apiMock.predictMask).toHaveBeenNthCalledWith(1, {
+      imageId: 'frame-1',
+      imageWidth: 640,
+      imageHeight: 360,
+      model: 'sam2',
+      points: undefined,
+      box: { x1: 120, y1: 80, x2: 260, y2: 200 },
+    }));
+    await waitFor(() => expect(useStore.getState().masks).toHaveLength(1));
+
+    rerender(<CanvasArea activeTool="point_pos" frame={frame} />);
+    fireEvent.click(stage, { clientX: 150, clientY: 100 });
+
+    await waitFor(() => expect(apiMock.predictMask).toHaveBeenNthCalledWith(2, {
+      imageId: 'frame-1',
+      imageWidth: 640,
+      imageHeight: 360,
+      model: 'sam2',
+      points: [{ x: 150, y: 100, type: 'pos' }],
+      box: { x1: 120, y1: 80, x2: 260, y2: 200 },
+    }));
+    expect(useStore.getState().masks).toHaveLength(1);
+    expect(useStore.getState().masks[0]).toEqual(expect.objectContaining({
+      id: 'mask-box',
+      segmentation: [[20, 20, 80, 20, 80, 80]],
+      metadata: expect.objectContaining({
+        source: 'sam2_interactive',
+        promptPointCount: 1,
+      }),
+    }));
+
+    rerender(<CanvasArea activeTool="point_neg" frame={frame} />);
+    fireEvent.click(stage, { clientX: 300, clientY: 150 });
+
+    await waitFor(() => expect(apiMock.predictMask).toHaveBeenNthCalledWith(3, {
+      imageId: 'frame-1',
+      imageWidth: 640,
+      imageHeight: 360,
+      model: 'sam2',
+      points: [
+        { x: 150, y: 100, type: 'pos' },
+        { x: 300, y: 150, type: 'neg' },
+      ],
+      box: { x1: 120, y1: 80, x2: 260, y2: 200 },
+    }));
+    expect(useStore.getState().masks).toHaveLength(1);
+    expect(useStore.getState().masks[0]).toEqual(expect.objectContaining({
+      id: 'mask-box',
+      segmentation: [[30, 30, 70, 30, 70, 70]],
+      points: [[150, 100]],
+      metadata: expect.objectContaining({ promptPointCount: 2 }),
+    }));
+  });
+
  it('renders only masks that belong to the current frame', () => {
    useStore.setState({
      masks: [
@@ -79,6 +230,26 @@ describe('CanvasArea', () => {
    expect(screen.getByText('遮罩数: 1')).toBeInTheDocument();
  });

+  it('publishes the selected mask ids for the ontology panel', async () => {
+    useStore.setState({
+      masks: [
+        {
+          id: 'm1',
+          frameId: 'frame-1',
+          pathData: 'M 0 0 L 10 0 L 10 10 Z',
+          label: 'A',
+          color: '#fff',
+          segmentation: [[0, 0, 10, 0, 10, 10]],
+        },
+      ],
+    });
+
+    render(<CanvasArea activeTool="move" frame={frame} />);
+    fireEvent.click(screen.getByTestId('konva-path'));
+
+    await waitFor(() => expect(useStore.getState().selectedMaskIds).toEqual(['m1']));
+  });
+
  it('renders imported GT seed points for editable point regions', () => {
    useStore.setState({
      masks: [
@@ -164,6 +335,57 @@ describe('CanvasArea', () => {
    }));
  });

+  it('deletes the selected draft mask with Delete when no vertex is selected', () => {
+    useStore.setState({
+      masks: [
+        {
+          id: 'draft-1',
+          frameId: 'frame-1',
+          pathData: 'M 10 10 L 90 10 L 90 40 Z',
+          label: 'Draft',
+          color: '#06b6d4',
+          saveStatus: 'draft',
+          segmentation: [[10, 10, 90, 10, 90, 40]],
+        },
+      ],
+    });
+
+    render(<CanvasArea activeTool="move" frame={frame} />);
+    fireEvent.click(screen.getByTestId('konva-path'));
+    fireEvent.keyDown(window, { key: 'Delete' });
+
+    expect(useStore.getState().masks).toEqual([]);
+    expect(useStore.getState().maskHistory.at(-1)).toEqual([
+      expect.objectContaining({ id: 'draft-1' }),
+    ]);
+  });
+
+  it('deletes the selected saved mask locally and notifies the backend deletion callback', () => {
+    const onDeleteMaskAnnotations = vi.fn();
+    useStore.setState({
+      masks: [
+        {
+          id: 'annotation-99',
+          annotationId: '99',
+          frameId: 'frame-1',
+          pathData: 'M 10 10 L 90 10 L 90 40 Z',
+          label: 'Saved',
+          color: '#06b6d4',
+          saveStatus: 'saved',
+          saved: true,
+          segmentation: [[10, 10, 90, 10, 90, 40]],
+        },
+      ],
+    });
+
+    render(<CanvasArea activeTool="move" frame={frame} onDeleteMaskAnnotations={onDeleteMaskAnnotations} />);
+    fireEvent.click(screen.getByTestId('konva-path'));
+    fireEvent.keyDown(window, { key: 'Backspace' });
+
+    expect(useStore.getState().masks).toEqual([]);
+    expect(onDeleteMaskAnnotations).toHaveBeenCalledWith(['99']);
+  });
+
  it('inserts a polygon vertex from an edge midpoint handle', () => {
    useStore.setState({
      masks: [
@@ -248,9 +470,13 @@ describe('CanvasArea', () => {
    });

    render(<CanvasArea activeTool="area_merge" frame={frame} />);
+    expect(screen.getByText('已选 0')).toBeInTheDocument();
    const paths = screen.getAllByTestId('konva-path');
    fireEvent.click(paths[0]);
+    expect(screen.getByText('已选 1')).toBeInTheDocument();
+    expect(screen.queryAllByTestId('konva-circle')).toHaveLength(0);
    fireEvent.click(paths[1]);
+    expect(screen.getByText('已选 2')).toBeInTheDocument();
    fireEvent.click(screen.getByRole('button', { name: '合并选中' }));

    expect(useStore.getState().masks).toHaveLength(1);
@@ -300,6 +526,45 @@ describe('CanvasArea', () => {
    expect(useStore.getState().masks[1].id).toBe('m2');
  });

+  it('renders inner overlap removal as a hole in the primary mask', () => {
+    useStore.setState({
+      masks: [
+        {
+          id: 'm1',
+          frameId: 'frame-1',
+          pathData: 'M 10 10 L 110 10 L 110 110 L 10 110 Z',
+          label: 'A',
+          color: '#06b6d4',
+          segmentation: [[10, 10, 110, 10, 110, 110, 10, 110]],
+        },
+        {
+          id: 'm2',
+          frameId: 'frame-1',
+          pathData: 'M 40 40 L 80 40 L 80 80 L 40 80 Z',
+          label: 'B',
+          color: '#ff0000',
+          segmentation: [[40, 40, 80, 40, 80, 80, 40, 80]],
+        },
+      ],
+    });
+
+    render(<CanvasArea activeTool="area_remove" frame={frame} />);
+    const paths = screen.getAllByTestId('konva-path');
+    fireEvent.click(paths[0]);
+    fireEvent.click(paths[1]);
+    fireEvent.click(screen.getByRole('button', { name: '从主区域去除' }));
+
+    const [primary] = useStore.getState().masks;
+    expect(primary).toEqual(expect.objectContaining({
+      id: 'm1',
+      area: 8400,
+      bbox: [10, 10, 100, 100],
+      metadata: expect.objectContaining({ hasHoles: true }),
+    }));
+    expect(primary.segmentation).toHaveLength(2);
+    expect(screen.getAllByTestId('konva-path')[0]).toHaveAttribute('data-fill-rule', 'evenodd');
+  });
+
  it('creates a manual rectangle mask that can be undone and redone', () => {
    useStore.setState({
      activeTemplateId: '2',
@@ -329,6 +594,93 @@ describe('CanvasArea', () => {
    expect(useStore.getState().masks).toHaveLength(1);
  });

+  it('creates a manual circle mask from a drag gesture', () => {
+    render(<CanvasArea activeTool="create_circle" frame={frame} />);
+    const stage = screen.getByTestId('konva-stage');
+    fireEvent.mouseDown(stage, { clientX: 120, clientY: 80 });
+    fireEvent.mouseMove(stage, { clientX: 260, clientY: 200 });
+    fireEvent.mouseUp(stage, { clientX: 260, clientY: 200 });
+
+    expect(useStore.getState().masks).toHaveLength(1);
+    expect(useStore.getState().masks[0]).toEqual(expect.objectContaining({
+      frameId: 'frame-1',
+      label: '手工圆形',
+      color: '#06b6d4',
+      saveStatus: 'draft',
+      bbox: [120, 80, 140, 120],
+      metadata: expect.objectContaining({
+        source: 'manual',
+        shape: '圆形',
+      }),
+    }));
+    expect(useStore.getState().masks[0].segmentation?.[0]).toHaveLength(64);
+  });
+
+  it('creates a manual line region from a drag gesture', () => {
+    render(<CanvasArea activeTool="create_line" frame={frame} />);
+    const stage = screen.getByTestId('konva-stage');
+    fireEvent.mouseDown(stage, { clientX: 120, clientY: 80 });
+    fireEvent.mouseMove(stage, { clientX: 260, clientY: 200 });
+    fireEvent.mouseUp(stage, { clientX: 260, clientY: 200 });
+
+    expect(useStore.getState().masks).toHaveLength(1);
+    expect(useStore.getState().masks[0]).toEqual(expect.objectContaining({
+      frameId: 'frame-1',
+      label: '手工线段',
+      color: '#06b6d4',
+      saveStatus: 'draft',
+      metadata: expect.objectContaining({
+        source: 'manual',
+        shape: '线段',
+      }),
+    }));
+    expect(useStore.getState().masks[0].segmentation?.[0]).toHaveLength(8);
+    expect(useStore.getState().masks[0].area).toBeGreaterThan(1000);
+  });
+
+  it('creates an editable point region on click', () => {
+    render(<CanvasArea activeTool="create_point" frame={frame} />);
+    fireEvent.click(screen.getByTestId('konva-stage'), { clientX: 120, clientY: 80 });
+
+    expect(useStore.getState().masks).toHaveLength(1);
+    expect(useStore.getState().masks[0]).toEqual(expect.objectContaining({
+      frameId: 'frame-1',
+      label: '手工点区域',
+      color: '#06b6d4',
+      saveStatus: 'draft',
+      points: [[120, 80]],
+      bbox: expect.arrayContaining([115, 75]),
+      metadata: expect.objectContaining({
+        source: 'manual',
+        shape: '点区域',
+      }),
+    }));
+  });
+
+  it('creates a point region when clicking over an existing mask', () => {
+    useStore.setState({
+      masks: [
+        {
+          id: 'm1',
+          frameId: 'frame-1',
+          pathData: 'M 10 10 L 200 10 L 200 200 Z',
+          label: 'Existing',
+          color: '#06b6d4',
+          segmentation: [[10, 10, 200, 10, 200, 200]],
+        },
+      ],
+    });
+
+    render(<CanvasArea activeTool="create_point" frame={frame} />);
+    fireEvent.click(screen.getByTestId('konva-path'), { clientX: 120, clientY: 80 });
+
+    expect(useStore.getState().masks).toHaveLength(2);
+    expect(useStore.getState().masks[1]).toEqual(expect.objectContaining({
+      metadata: expect.objectContaining({ shape: '点区域' }),
+      points: [[120, 80]],
+    }));
+  });
+
  it('finalizes a clicked polygon with Enter', () => {
    render(<CanvasArea activeTool="create_polygon" frame={frame} />);
    const stage = screen.getByTestId('konva-stage');
@@ -344,6 +696,29 @@ describe('CanvasArea', () => {
    }));
  });

+  it('closes a clicked polygon by clicking the first node again', () => {
+    render(<CanvasArea activeTool="create_polygon" frame={frame} />);
+    const stage = screen.getByTestId('konva-stage');
+    fireEvent.click(stage, { clientX: 120, clientY: 80 });
+    fireEvent.click(stage, { clientX: 220, clientY: 80 });
+    fireEvent.click(stage, { clientX: 180, clientY: 160 });
+
+    const handles = screen.getAllByTestId('konva-circle');
+    expect(handles[0]).toHaveAttribute('data-fill', '#facc15');
+    fireEvent.click(handles[0]);
+
+    expect(useStore.getState().masks).toHaveLength(1);
+    expect(useStore.getState().masks[0]).toEqual(expect.objectContaining({
+      pathData: 'M 120 80 L 220 80 L 180 160 Z',
+      segmentation: [[120, 80, 220, 80, 180, 160]],
+      metadata: expect.objectContaining({
+        source: 'manual',
+        shape: '多边形',
+      }),
+    }));
+    expect(screen.queryAllByTestId('konva-circle')).toHaveLength(0);
+  });
+
  it('applies the selected class to current-frame masks and marks saved masks dirty', () => {
    useStore.setState({
      activeTemplateId: '2',