first commit
This commit is contained in:
@@ -0,0 +1,37 @@
|
||||
# Evidence Propagation
|
||||
|
||||
Use this file to keep `results-analysis` outputs aligned with the final report.
|
||||
|
||||
## Mapping rule
|
||||
|
||||
- `analysis-report.md` -> main findings and narrative summary
|
||||
- `analysis-report.md#Claim Candidates` -> claim wording, uncertainty, and decisions that can be carried into the report
|
||||
- `stats-appendix.md` -> test choice, uncertainty, effect size, correction rule
|
||||
- `figure-catalog.md` -> figure purpose and per-figure interpretation scaffolding
|
||||
- figure files -> visual evidence cited in `Figure-by-Figure Interpretation`
|
||||
|
||||
## Minimum statistical carry-over
|
||||
|
||||
Every strong claim in a results report should preserve:
|
||||
- sample size or run/seed count,
|
||||
- metric definition,
|
||||
- uncertainty summary,
|
||||
- test name,
|
||||
- effect size when relevant,
|
||||
- multiple-comparison handling when relevant.
|
||||
|
||||
## Unsupported claim rule
|
||||
|
||||
If the analysis bundle does not support a claim strongly enough, keep the claim tentative and say why.
|
||||
Do not upgrade a suggestive result into a decisive conclusion during report writing.
|
||||
|
||||
## Claim candidate carry-over
|
||||
|
||||
Every claim carried from `results-analysis` should preserve:
|
||||
- the source evidence,
|
||||
- the allowed wording,
|
||||
- the forbidden stronger wording,
|
||||
- uncertainty,
|
||||
- next check or decision.
|
||||
|
||||
Do not convert `speculative` or `observed` claims into decisive conclusions. The `What Changed Our Belief` section should cite either a Claim Candidate or an Evidence Record, not only free-form prose.
|
||||
@@ -0,0 +1,17 @@
|
||||
# Decision-Oriented Analysis
|
||||
|
||||
The purpose of a post-experiment report is not only to record what happened.
|
||||
It should change the project's next decision.
|
||||
|
||||
## Required final questions
|
||||
- What should stop?
|
||||
- What should continue?
|
||||
- What should be tested next?
|
||||
- What should be promoted into a durable result note?
|
||||
- What, if anything, is ready for manuscript-facing writing?
|
||||
|
||||
## Good closing pattern
|
||||
- “This round supports X.”
|
||||
- “It does not yet resolve Y.”
|
||||
- “The main blocker is Z.”
|
||||
- “Therefore the next concrete action is A.”
|
||||
@@ -0,0 +1,17 @@
|
||||
# Figure Interpretation in Results Reports
|
||||
|
||||
A results report should not dump figures.
|
||||
|
||||
For each major figure, write four blocks:
|
||||
- **Why this figure exists**
|
||||
- **What to notice**
|
||||
- **What interpretation is supported**
|
||||
- **What this changes in the project decision**
|
||||
|
||||
## Example micro-structure
|
||||
|
||||
### Figure X
|
||||
- Purpose: compare adapter vs freezing under the same transfer setting.
|
||||
- Observation: adapter improves mean WER and reduces variance.
|
||||
- Interpretation: subject-specific adaptation likely resolves part of the transfer mismatch.
|
||||
- Decision implication: prioritize adapter ablations before expanding frozen-only variants.
|
||||
@@ -0,0 +1,53 @@
|
||||
# Report Naming Standard
|
||||
|
||||
## Filename
|
||||
|
||||
Use:
|
||||
|
||||
```text
|
||||
YYYY-MM-DD--{experiment-line}--r{round}--{purpose}.md
|
||||
```
|
||||
|
||||
Rules:
|
||||
- date must be the report date,
|
||||
- `experiment-line` should be short and stable,
|
||||
- `round` should be zero-padded only if that is already the project convention; otherwise `r3` / `r03` are both acceptable if used consistently,
|
||||
- `purpose` should describe why the report exists, not a vague label like `summary` unless that is truly the purpose.
|
||||
|
||||
Recommended purpose values:
|
||||
- `transfer-summary`
|
||||
- `ablation-report`
|
||||
- `failure-analysis`
|
||||
- `robustness-check`
|
||||
- `round-review`
|
||||
|
||||
## Title
|
||||
|
||||
Use:
|
||||
|
||||
```text
|
||||
{Experiment Line} / Round {N} / {Purpose} / {YYYY-MM-DD}
|
||||
```
|
||||
|
||||
## Frontmatter fields
|
||||
|
||||
Required:
|
||||
- `type: results-report`
|
||||
- `date`
|
||||
- `experiment_line`
|
||||
- `round`
|
||||
- `purpose`
|
||||
- `status`
|
||||
- `source_artifacts`
|
||||
- `linked_experiments`
|
||||
- `linked_results`
|
||||
|
||||
## Placement in Obsidian
|
||||
|
||||
Internal reports go to:
|
||||
|
||||
```text
|
||||
Results/Reports/{filename}
|
||||
```
|
||||
|
||||
Do not put internal experiment reports in `Writing/` unless they are already manuscript/slides/rebuttal material.
|
||||
@@ -0,0 +1,63 @@
|
||||
# Results Report Structure
|
||||
|
||||
## 1. Executive Summary
|
||||
Answer:
|
||||
- what was tested,
|
||||
- what the highest-confidence conclusion is,
|
||||
- what decision this changes.
|
||||
|
||||
## 2. Experiment Identity and Decision Context
|
||||
Answer:
|
||||
- which experiment line this belongs to,
|
||||
- why this round was run,
|
||||
- what prior uncertainty or decision it was meant to resolve.
|
||||
|
||||
## 3. Setup and Evaluation Protocol
|
||||
Answer:
|
||||
- datasets / subjects / splits,
|
||||
- methods compared,
|
||||
- primary metrics,
|
||||
- repeated-run structure,
|
||||
- any deviations from prior protocol.
|
||||
|
||||
## 4. Main Findings
|
||||
Answer:
|
||||
- what changed most,
|
||||
- which comparison matters most,
|
||||
- where the largest gains or failures appear.
|
||||
|
||||
## 5. Statistical Validation
|
||||
Answer:
|
||||
- what evidence supports the major claims,
|
||||
- what tests were used,
|
||||
- where the evidence is weak.
|
||||
|
||||
## 6. Figure-by-Figure Interpretation
|
||||
For each main figure:
|
||||
- why it is shown,
|
||||
- what to notice,
|
||||
- what is supported,
|
||||
- what remains uncertain.
|
||||
|
||||
## 7. Failure Cases / Negative Results / Limitations
|
||||
Answer:
|
||||
- what did not work,
|
||||
- what instability appeared,
|
||||
- what limits the current conclusion.
|
||||
|
||||
## 8. What Changed Our Belief
|
||||
Answer:
|
||||
- which prior hypothesis is strengthened,
|
||||
- weakened,
|
||||
- or still unresolved.
|
||||
|
||||
## 9. Next Actions
|
||||
Answer:
|
||||
- stop / continue / ablate / scale / write.
|
||||
|
||||
## 10. Artifact and Reproducibility Index
|
||||
List:
|
||||
- source artifacts,
|
||||
- figure paths,
|
||||
- scripts/logs,
|
||||
- linked Obsidian notes.
|
||||
@@ -0,0 +1,19 @@
|
||||
# Statistical Completeness for Results Reports
|
||||
|
||||
A report may summarize statistics, but it must not silently weaken them.
|
||||
|
||||
## Always carry forward
|
||||
- sample size / seed count
|
||||
- metric direction
|
||||
- descriptive statistics
|
||||
- uncertainty estimate
|
||||
- test choice
|
||||
- effect size
|
||||
- correction rule when relevant
|
||||
- evidence boundary
|
||||
|
||||
## Never do this in the report
|
||||
- upgrade a trend to a conclusion
|
||||
- omit the sample size
|
||||
- replace effect size with adjectives like “large” without numbers
|
||||
- cite a figure without saying what the uncertainty represents
|
||||
Reference in New Issue
Block a user