first commit
This commit is contained in:
186
文档润色流和知识库构建流/claude-scholar/skills/results-report/SKILL.md
Normal file
186
文档润色流和知识库构建流/claude-scholar/skills/results-report/SKILL.md
Normal file
@@ -0,0 +1,186 @@
|
||||
---
|
||||
name: results-report
|
||||
description: This skill should be used when the user asks to "write an experiment report", "summarize experimental results", "do experiment retrospection", "write a results report", "写实验总结报告", "写实验复盘", or mentions turning completed experiment artifacts into a structured, decision-oriented research report. It assumes strict analysis should come from `results-analysis` first.
|
||||
version: 0.1.0
|
||||
tags: [Research, Reporting, Experiments, Obsidian]
|
||||
---
|
||||
|
||||
# Results Report
|
||||
|
||||
Write the **complete post-experiment summary report** after analysis artifacts are ready.
|
||||
|
||||
This skill is for the stage **after** `results-analysis`.
|
||||
|
||||
## Role boundary
|
||||
|
||||
### `results-analysis` does
|
||||
- strict statistics,
|
||||
- real figures,
|
||||
- figure interpretation scaffolding,
|
||||
- stats appendix.
|
||||
|
||||
### `results-report` does
|
||||
- complete experiment wrap-up report,
|
||||
- decision-oriented narrative,
|
||||
- figure-by-figure interpretation inside a coherent structure,
|
||||
- limitations, failure cases, and next actions,
|
||||
- Obsidian write-back into `Results/Reports/`.
|
||||
|
||||
When the task is to create or redesign paper-ready figures/tables themselves, rely on `publication-chart-skill` instead of expanding `results-report` into figure/table production.
|
||||
|
||||
Do not replace strict analysis with confident prose. If the analysis bundle is missing, first identify the blocker and request or produce the missing bundle.
|
||||
|
||||
Hard gate: do not write a complete report when `analysis-report.md`, `stats-appendix.md`, `figure-catalog.md`, provenance, or the primary comparison question is missing. In that case, write only a blocker summary and route back to `results-analysis`.
|
||||
|
||||
## Default output
|
||||
|
||||
The default report is an **internal research report**, not manuscript prose.
|
||||
|
||||
It should be named as:
|
||||
|
||||
```text
|
||||
YYYY-MM-DD--{experiment-line}--r{round}--{purpose}.md
|
||||
```
|
||||
|
||||
Example:
|
||||
- `2026-03-18--freezing--r03--transfer-summary.md`
|
||||
- `2026-03-18--contrastive-adversarial--r02--ablation-report.md`
|
||||
|
||||
The note title should be:
|
||||
|
||||
```text
|
||||
{Experiment Line} / Round {N} / {Purpose} / {YYYY-MM-DD}
|
||||
```
|
||||
|
||||
Read `references/report-naming.md` before finalizing the filename or note title.
|
||||
|
||||
## Required frontmatter
|
||||
|
||||
```yaml
|
||||
---
|
||||
type: results-report
|
||||
date: 2026-03-18
|
||||
experiment_line: freezing
|
||||
round: 3
|
||||
purpose: transfer-summary
|
||||
status: active
|
||||
source_artifacts:
|
||||
- analysis-output/analysis-report.md
|
||||
- analysis-output/stats-appendix.md
|
||||
linked_experiments:
|
||||
- Experiments/Freezing-Study.md
|
||||
linked_results:
|
||||
- Results/Freezing-vs-Adapter.md
|
||||
---
|
||||
```
|
||||
|
||||
## Default report structure
|
||||
|
||||
The report must include all sections below.
|
||||
|
||||
1. **Executive Summary**
|
||||
2. **Experiment Identity and Decision Context**
|
||||
3. **Setup and Evaluation Protocol**
|
||||
4. **Main Findings**
|
||||
5. **Statistical Validation**
|
||||
6. **Figure-by-Figure Interpretation**
|
||||
7. **Failure Cases / Negative Results / Limitations**
|
||||
8. **What Changed Our Belief**
|
||||
9. **Next Actions**
|
||||
10. **Artifact and Reproducibility Index**
|
||||
|
||||
Read `references/report-structure.md` before writing.
|
||||
|
||||
## Workflow
|
||||
|
||||
### 1. Confirm the report object
|
||||
|
||||
Lock these fields first:
|
||||
- date,
|
||||
- experiment line,
|
||||
- round,
|
||||
- purpose,
|
||||
- linked experiment note,
|
||||
- linked durable result note if one already exists.
|
||||
|
||||
If round is unknown, do not silently invent a semantic round. Use `r00` only as a temporary placeholder and state that it should be normalized later.
|
||||
|
||||
### 2. Read the strict analysis bundle
|
||||
|
||||
Minimum required inputs:
|
||||
- `analysis-report.md`
|
||||
- `stats-appendix.md`
|
||||
- `figure-catalog.md`
|
||||
- actual figures, if available
|
||||
|
||||
If these are missing, either generate them first with `results-analysis` or explicitly state which claims cannot be supported.
|
||||
Do not turn loose CSV/Markdown artifacts into a polished report unless the strict analysis bundle has already validated unit of analysis, primary metric, and claim wording.
|
||||
|
||||
### 3. Write the report as a decision object
|
||||
|
||||
This report is not a transcript of outputs.
|
||||
|
||||
Each section must answer a real question:
|
||||
- What did we test?
|
||||
- What changed numerically?
|
||||
- What is actually supported?
|
||||
- What failed or remains uncertain?
|
||||
- What should we do next?
|
||||
|
||||
Read `references/decision-oriented-analysis.md` for the expected reasoning depth.
|
||||
|
||||
### 4. Interpret figures inside the report
|
||||
|
||||
Do not only attach figures.
|
||||
|
||||
For each main figure:
|
||||
- introduce why it is included,
|
||||
- state the key observation,
|
||||
- explain the supported interpretation,
|
||||
- explain the decision implication.
|
||||
|
||||
Read `references/figure-interpretation.md` and `references/statistical-completeness.md` as needed.
|
||||
|
||||
### 5. Choose the write target explicitly
|
||||
|
||||
If the current repo is bound to an Obsidian project knowledge base:
|
||||
- create or update `Results/Reports/{report-name}.md`,
|
||||
- link back to the relevant `Experiments/` note,
|
||||
- update the matching canonical `Results/` note when a durable conclusion is now supported,
|
||||
- append a short trace to today's `Daily/` note,
|
||||
- update `.claude/project-memory/<project_id>.md`.
|
||||
|
||||
If the repo is **not** bound:
|
||||
- write the report as a local markdown artifact in the requested output location or next to the analysis bundle,
|
||||
- keep the same filename contract,
|
||||
- explicitly say that no Obsidian write-back was attempted.
|
||||
|
||||
Use `obsidian-project-kb-core` conventions only for bound repos. Internal experiment reports belong in `Results/Reports/`, not `Writing/`.
|
||||
|
||||
### 6. End with explicit next actions
|
||||
|
||||
The report must end with operational decisions, for example:
|
||||
- stop a weak branch,
|
||||
- schedule one missing ablation,
|
||||
- promote a stable finding into manuscript-facing writing,
|
||||
- update the active plan.
|
||||
|
||||
## Required quality bar
|
||||
|
||||
- The report must be dateable, searchable, and attributable to one experiment line and one round.
|
||||
- The report must cite actual evidence from the analysis bundle.
|
||||
- The report must include negative results when they matter.
|
||||
- The report must separate stable conclusion from tentative interpretation.
|
||||
- The report must say what changed in project belief and what should happen next.
|
||||
- The report must preserve Claim Candidate wording and must not promote `speculative` or `observed` claims into decisive conclusions.
|
||||
|
||||
## Reference files
|
||||
|
||||
Load only what is needed:
|
||||
- `references/report-structure.md`
|
||||
- `references/report-naming.md`
|
||||
- `references/figure-interpretation.md`
|
||||
- `references/statistical-completeness.md`
|
||||
- `references/decision-oriented-analysis.md`
|
||||
- `references/EVIDENCE-PROPAGATION.md`
|
||||
- `examples/example-results-report.md`
|
||||
@@ -0,0 +1,83 @@
|
||||
---
|
||||
type: results-report
|
||||
date: 2026-03-18
|
||||
experiment_line: freezing
|
||||
round: 3
|
||||
purpose: transfer-summary
|
||||
status: active
|
||||
source_artifacts:
|
||||
- analysis-output/analysis-report.md
|
||||
- analysis-output/stats-appendix.md
|
||||
- analysis-output/figure-catalog.md
|
||||
linked_experiments:
|
||||
- Experiments/Freezing-vs-Adapter.md
|
||||
linked_results:
|
||||
- Results/Adapter-Improves-Transfer.md
|
||||
---
|
||||
|
||||
# Freezing / Round 3 / transfer-summary / 2026-03-18
|
||||
|
||||
## Executive Summary
|
||||
- Round 3 tested whether a subject adapter recovers the performance lost by freezing most of the encoder.
|
||||
- Across 5 seeds per condition, the adapter reduced mean WER by **3.8 absolute points** relative to the frozen encoder baseline.
|
||||
- The current evidence supports keeping the adapter branch active, while pure freezing should be deprioritized.
|
||||
|
||||
## Experiment Identity and Decision Context
|
||||
- Experiment line: freezing
|
||||
- Round: 3
|
||||
- Purpose: resolve whether the freezing gap is best handled by lightweight adaptation or by abandoning the freezing branch.
|
||||
- Decision pressure: choose the next transfer branch before scheduling the next low-resource run block.
|
||||
|
||||
## Setup and Evaluation Protocol
|
||||
- Same subject pool and split as rounds 1-2.
|
||||
- 5 seeds per condition.
|
||||
- Primary metric: WER (lower is better).
|
||||
- Compared methods: Full fine-tuning, Subject Adapter, Frozen Encoder.
|
||||
- Statistical unit: seed-level final WER.
|
||||
|
||||
## Main Findings
|
||||
- Subject Adapter: **27.6 ± 1.0 WER**, 95% CI **[26.4, 28.8]**.
|
||||
- Frozen Encoder: **31.4 ± 1.5 WER**, 95% CI **[29.6, 33.2]**.
|
||||
- Full fine-tuning: **25.9 ± 0.8 WER**, 95% CI **[24.9, 26.9]**.
|
||||
- Adapter beats Frozen Encoder in all 5 paired seed comparisons.
|
||||
|
||||
## Statistical Validation
|
||||
- Adapter vs Frozen Encoder: paired Wilcoxon signed-rank test, **p = 0.031**, Holm-corrected **p = 0.047**, matched-rank biserial effect size **r = 0.90**.
|
||||
- Full fine-tuning vs Adapter: paired t-test, **p = 0.11**, Cohen's **d = 0.64**.
|
||||
- Interpretation: the adapter gain over pure freezing is supported at current `n = 5`; the gap to full fine-tuning is directionally consistent but still underpowered.
|
||||
- Unsupported claim boundary: this report does **not** claim generalization beyond the current subject pool or low-resource regime.
|
||||
|
||||
## Figure-by-Figure Interpretation
|
||||
### Figure 1 — Main comparison
|
||||
- Why included: this is the core decision figure.
|
||||
- Evidence carried in: mean WER, 95% CI, and paired-seed comparisons.
|
||||
- Supported interpretation: lightweight subject adaptation closes most of the freezing gap.
|
||||
- Decision implication: future transfer experiments should center on adapter design, not frozen-only variants.
|
||||
|
||||
### Figure 2 — Training dynamics
|
||||
- Why included: to explain stability differences.
|
||||
- Evidence carried in: per-epoch validation traces across seeds.
|
||||
- Supported interpretation: the frozen baseline oscillates more after epoch 8, matching its wider uncertainty interval.
|
||||
- Decision implication: branch weakness is not only lower final accuracy but also worse optimization stability.
|
||||
|
||||
## Failure Cases / Negative Results / Limitations
|
||||
- Full fine-tuning still leads in absolute WER.
|
||||
- The evidence is limited to one subject pool and 5 seeds.
|
||||
- No low-resource stress test or out-of-domain subject split has been run yet.
|
||||
- Adapter width was fixed in this round, so capacity trade-offs remain unresolved.
|
||||
|
||||
## What Changed Our Belief
|
||||
- Before round 3, it was plausible that freezing should be abandoned entirely.
|
||||
- After round 3, the better hypothesis is that freezing alone is too rigid, but freezing plus lightweight adaptation remains viable.
|
||||
|
||||
## Next Actions
|
||||
- Run one low-resource robustness check for the adapter branch.
|
||||
- Add a width ablation around the current best adapter size.
|
||||
- Update the canonical result note for adapter-improves-transfer.
|
||||
|
||||
## Artifact and Reproducibility Index
|
||||
- `analysis-output/analysis-report.md`
|
||||
- `analysis-output/stats-appendix.md`
|
||||
- `analysis-output/figure-catalog.md`
|
||||
- `analysis-output/figures/figure-01-main-comparison.pdf`
|
||||
- `analysis-output/figures/figure-02-training-dynamics.pdf`
|
||||
@@ -0,0 +1,37 @@
|
||||
# Evidence Propagation
|
||||
|
||||
Use this file to keep `results-analysis` outputs aligned with the final report.
|
||||
|
||||
## Mapping rule
|
||||
|
||||
- `analysis-report.md` -> main findings and narrative summary
|
||||
- `analysis-report.md#Claim Candidates` -> claim wording, uncertainty, and decisions that can be carried into the report
|
||||
- `stats-appendix.md` -> test choice, uncertainty, effect size, correction rule
|
||||
- `figure-catalog.md` -> figure purpose and per-figure interpretation scaffolding
|
||||
- figure files -> visual evidence cited in `Figure-by-Figure Interpretation`
|
||||
|
||||
## Minimum statistical carry-over
|
||||
|
||||
Every strong claim in a results report should preserve:
|
||||
- sample size or run/seed count,
|
||||
- metric definition,
|
||||
- uncertainty summary,
|
||||
- test name,
|
||||
- effect size when relevant,
|
||||
- multiple-comparison handling when relevant.
|
||||
|
||||
## Unsupported claim rule
|
||||
|
||||
If the analysis bundle does not support a claim strongly enough, keep the claim tentative and say why.
|
||||
Do not upgrade a suggestive result into a decisive conclusion during report writing.
|
||||
|
||||
## Claim candidate carry-over
|
||||
|
||||
Every claim carried from `results-analysis` should preserve:
|
||||
- the source evidence,
|
||||
- the allowed wording,
|
||||
- the forbidden stronger wording,
|
||||
- uncertainty,
|
||||
- next check or decision.
|
||||
|
||||
Do not convert `speculative` or `observed` claims into decisive conclusions. The `What Changed Our Belief` section should cite either a Claim Candidate or an Evidence Record, not only free-form prose.
|
||||
@@ -0,0 +1,17 @@
|
||||
# Decision-Oriented Analysis
|
||||
|
||||
The purpose of a post-experiment report is not only to record what happened.
|
||||
It should change the project's next decision.
|
||||
|
||||
## Required final questions
|
||||
- What should stop?
|
||||
- What should continue?
|
||||
- What should be tested next?
|
||||
- What should be promoted into a durable result note?
|
||||
- What, if anything, is ready for manuscript-facing writing?
|
||||
|
||||
## Good closing pattern
|
||||
- “This round supports X.”
|
||||
- “It does not yet resolve Y.”
|
||||
- “The main blocker is Z.”
|
||||
- “Therefore the next concrete action is A.”
|
||||
@@ -0,0 +1,17 @@
|
||||
# Figure Interpretation in Results Reports
|
||||
|
||||
A results report should not dump figures.
|
||||
|
||||
For each major figure, write four blocks:
|
||||
- **Why this figure exists**
|
||||
- **What to notice**
|
||||
- **What interpretation is supported**
|
||||
- **What this changes in the project decision**
|
||||
|
||||
## Example micro-structure
|
||||
|
||||
### Figure X
|
||||
- Purpose: compare adapter vs freezing under the same transfer setting.
|
||||
- Observation: adapter improves mean WER and reduces variance.
|
||||
- Interpretation: subject-specific adaptation likely resolves part of the transfer mismatch.
|
||||
- Decision implication: prioritize adapter ablations before expanding frozen-only variants.
|
||||
@@ -0,0 +1,53 @@
|
||||
# Report Naming Standard
|
||||
|
||||
## Filename
|
||||
|
||||
Use:
|
||||
|
||||
```text
|
||||
YYYY-MM-DD--{experiment-line}--r{round}--{purpose}.md
|
||||
```
|
||||
|
||||
Rules:
|
||||
- date must be the report date,
|
||||
- `experiment-line` should be short and stable,
|
||||
- `round` should be zero-padded only if that is already the project convention; otherwise `r3` / `r03` are both acceptable if used consistently,
|
||||
- `purpose` should describe why the report exists, not a vague label like `summary` unless that is truly the purpose.
|
||||
|
||||
Recommended purpose values:
|
||||
- `transfer-summary`
|
||||
- `ablation-report`
|
||||
- `failure-analysis`
|
||||
- `robustness-check`
|
||||
- `round-review`
|
||||
|
||||
## Title
|
||||
|
||||
Use:
|
||||
|
||||
```text
|
||||
{Experiment Line} / Round {N} / {Purpose} / {YYYY-MM-DD}
|
||||
```
|
||||
|
||||
## Frontmatter fields
|
||||
|
||||
Required:
|
||||
- `type: results-report`
|
||||
- `date`
|
||||
- `experiment_line`
|
||||
- `round`
|
||||
- `purpose`
|
||||
- `status`
|
||||
- `source_artifacts`
|
||||
- `linked_experiments`
|
||||
- `linked_results`
|
||||
|
||||
## Placement in Obsidian
|
||||
|
||||
Internal reports go to:
|
||||
|
||||
```text
|
||||
Results/Reports/{filename}
|
||||
```
|
||||
|
||||
Do not put internal experiment reports in `Writing/` unless they are already manuscript/slides/rebuttal material.
|
||||
@@ -0,0 +1,63 @@
|
||||
# Results Report Structure
|
||||
|
||||
## 1. Executive Summary
|
||||
Answer:
|
||||
- what was tested,
|
||||
- what the highest-confidence conclusion is,
|
||||
- what decision this changes.
|
||||
|
||||
## 2. Experiment Identity and Decision Context
|
||||
Answer:
|
||||
- which experiment line this belongs to,
|
||||
- why this round was run,
|
||||
- what prior uncertainty or decision it was meant to resolve.
|
||||
|
||||
## 3. Setup and Evaluation Protocol
|
||||
Answer:
|
||||
- datasets / subjects / splits,
|
||||
- methods compared,
|
||||
- primary metrics,
|
||||
- repeated-run structure,
|
||||
- any deviations from prior protocol.
|
||||
|
||||
## 4. Main Findings
|
||||
Answer:
|
||||
- what changed most,
|
||||
- which comparison matters most,
|
||||
- where the largest gains or failures appear.
|
||||
|
||||
## 5. Statistical Validation
|
||||
Answer:
|
||||
- what evidence supports the major claims,
|
||||
- what tests were used,
|
||||
- where the evidence is weak.
|
||||
|
||||
## 6. Figure-by-Figure Interpretation
|
||||
For each main figure:
|
||||
- why it is shown,
|
||||
- what to notice,
|
||||
- what is supported,
|
||||
- what remains uncertain.
|
||||
|
||||
## 7. Failure Cases / Negative Results / Limitations
|
||||
Answer:
|
||||
- what did not work,
|
||||
- what instability appeared,
|
||||
- what limits the current conclusion.
|
||||
|
||||
## 8. What Changed Our Belief
|
||||
Answer:
|
||||
- which prior hypothesis is strengthened,
|
||||
- weakened,
|
||||
- or still unresolved.
|
||||
|
||||
## 9. Next Actions
|
||||
Answer:
|
||||
- stop / continue / ablate / scale / write.
|
||||
|
||||
## 10. Artifact and Reproducibility Index
|
||||
List:
|
||||
- source artifacts,
|
||||
- figure paths,
|
||||
- scripts/logs,
|
||||
- linked Obsidian notes.
|
||||
@@ -0,0 +1,19 @@
|
||||
# Statistical Completeness for Results Reports
|
||||
|
||||
A report may summarize statistics, but it must not silently weaken them.
|
||||
|
||||
## Always carry forward
|
||||
- sample size / seed count
|
||||
- metric direction
|
||||
- descriptive statistics
|
||||
- uncertainty estimate
|
||||
- test choice
|
||||
- effect size
|
||||
- correction rule when relevant
|
||||
- evidence boundary
|
||||
|
||||
## Never do this in the report
|
||||
- upgrade a trend to a conclusion
|
||||
- omit the sample size
|
||||
- replace effect size with adjectives like “large” without numbers
|
||||
- cite a figure without saying what the uncertainty represents
|
||||
Reference in New Issue
Block a user