first commit
This commit is contained in:
@@ -0,0 +1,49 @@
|
||||
# Test: conflicting reviewers
|
||||
|
||||
## Input
|
||||
|
||||
```text
|
||||
Editor decision: Major revision.
|
||||
|
||||
Editor:
|
||||
Please avoid expanding the manuscript substantially; focus on clarifying the central claim and
|
||||
addressing the reviewers' concerns with existing data where possible.
|
||||
|
||||
Reviewer 1:
|
||||
1. The abstract should make a stronger causal claim that X drives Y.
|
||||
|
||||
Reviewer 2:
|
||||
1. The causal language is not supported by the current observational design and should be softened.
|
||||
|
||||
Author notes:
|
||||
- The study is observational.
|
||||
- We can soften the abstract and discussion.
|
||||
- We can add a sentence explaining that the findings support an association, not causality.
|
||||
```
|
||||
|
||||
## Expected behavior
|
||||
|
||||
- Assign editor instruction ID `E.1` and address it before reviewer comments.
|
||||
- Assign reviewer IDs `R1.1` and `R2.1`.
|
||||
- Detect a conflict between Reviewer 1 and Reviewer 2.
|
||||
- Prioritize the editor instruction and the evidentiary limit of the observational design.
|
||||
- Use `SOFTEN_CLAIM` for `R2.1`.
|
||||
- Use `PARTIAL` or `DISAGREE` for the stronger causal-claim request in `R1.1`, with respectful reasoning.
|
||||
- Avoid incompatible promises.
|
||||
- Mark readiness as `draft_with_placeholders` unless exact revised abstract/discussion wording or locations are supplied.
|
||||
|
||||
## Forbidden behavior
|
||||
|
||||
- Do not promise both stronger causal language and softened causal language.
|
||||
- Do not ignore the editor instruction.
|
||||
- Do not claim causality from an observational design.
|
||||
- Do not accuse either reviewer of being wrong.
|
||||
- Do not invent revised abstract or discussion line numbers.
|
||||
|
||||
## Pass/fail checklist
|
||||
|
||||
- [ ] `E.1` appears in the tracker or strategy summary.
|
||||
- [ ] The conflict is surfaced explicitly.
|
||||
- [ ] The chosen response is consistent with the observational design.
|
||||
- [ ] `R1.1` and `R2.1` are both answered.
|
||||
- [ ] No incompatible manuscript-change promises appear.
|
||||
@@ -0,0 +1,47 @@
|
||||
# Test: defensive draft audit
|
||||
|
||||
## Input
|
||||
|
||||
```text
|
||||
Mode requested: audit and revise this draft response.
|
||||
|
||||
Reviewer 1:
|
||||
1. The method description is unclear and does not explain how model calibration was performed.
|
||||
2. The authors should report the software version.
|
||||
|
||||
Author draft:
|
||||
The reviewer clearly misunderstood our method. We already explained the calibration in the paper.
|
||||
We have revised accordingly. The software version is now included.
|
||||
|
||||
Author notes:
|
||||
- Calibration is described in Methods, but the exact paragraph may not be clear.
|
||||
- Software version: v2.3.1.
|
||||
- No line numbers are available yet.
|
||||
```
|
||||
|
||||
## Expected behavior
|
||||
|
||||
- Detect task mode as `audit` or `revise`.
|
||||
- Assign stable IDs `R1.1` and `R1.2`.
|
||||
- Flag the author draft as defensive and insufficiently traceable.
|
||||
- Rewrite the misunderstanding sentence as manuscript-clarity framing.
|
||||
- Treat `R1.1` as `CLARIFY_EXISTING` plus possible `ACCEPT_TEXT`.
|
||||
- Treat `R1.2` as `ACCEPT_TEXT` with supplied version `v2.3.1`.
|
||||
- Use section names rather than invented line numbers.
|
||||
- Mark package readiness as `draft_with_placeholders` or `needs_author_input` until exact Methods location or revised text is supplied.
|
||||
|
||||
## Forbidden behavior
|
||||
|
||||
- Do not retain "The reviewer clearly misunderstood our method."
|
||||
- Do not retain bare "We have revised accordingly."
|
||||
- Do not invent line numbers or a Methods paragraph.
|
||||
- Do not claim the calibration explanation was already sufficient without clarifying the manuscript.
|
||||
- Do not remove the supplied software version.
|
||||
|
||||
## Pass/fail checklist
|
||||
|
||||
- [ ] Defensive language is removed.
|
||||
- [ ] Each reviewer comment receives its own ID.
|
||||
- [ ] Revised response includes manuscript-clarity framing.
|
||||
- [ ] `v2.3.1` is preserved exactly.
|
||||
- [ ] Missing location details remain visible.
|
||||
@@ -0,0 +1,43 @@
|
||||
# Evaluation summary
|
||||
|
||||
`nature-response` is evaluated with synthetic Markdown fixtures. These tests are not executable
|
||||
unit tests; they are behavior contracts for manual and agent review.
|
||||
|
||||
## Status rationale
|
||||
|
||||
Recommended status: `Beta`.
|
||||
|
||||
Rationale:
|
||||
|
||||
- The core rules are defined in `SKILL.md` and modular references.
|
||||
- The skill has synthetic fixtures covering minor revision, major revision with missing evidence,
|
||||
impossible experiment, defensive draft audit, and conflicting reviewers.
|
||||
- Each fixture includes expected behavior, forbidden behavior, and pass/fail criteria.
|
||||
- The examples show expected output shape without using real confidential reviewer comments.
|
||||
- The skill has not yet been validated on real anonymized revision packages, so `Stable` would be premature.
|
||||
|
||||
## Fixture coverage
|
||||
|
||||
| Fixture | Coverage | Key failure prevented |
|
||||
|---|---|---|
|
||||
| `minor-revision.md` | stable IDs, minor comments, missing citation metadata | fabricated citation or line numbers |
|
||||
| `major-revision-missing-evidence.md` | validation request, statistical details, missing evidence | invented results or p-values |
|
||||
| `impossible-experiment.md` | out-of-scope longitudinal evidence | time/funding excuse or fabricated survival data |
|
||||
| `defensive-draft-audit.md` | hostile draft language, vague compliance | accusatory reviewer language |
|
||||
| `conflicting-reviewers.md` | editor priority and incompatible reviewer requests | contradictory manuscript promises |
|
||||
|
||||
## Manual evaluation checklist
|
||||
|
||||
- [x] Every fixture has input, expected behavior, forbidden behavior, and pass/fail checklist.
|
||||
- [x] No fixture uses real reviewer comments.
|
||||
- [x] Examples are synthetic and do not contain confidential review content.
|
||||
- [x] Status remains below `Stable` until real anonymized cases are reviewed.
|
||||
|
||||
## Promotion path to Stable
|
||||
|
||||
Promote from `Beta` to `Stable` only after:
|
||||
|
||||
- at least two real anonymized revision packages are tested with author permission;
|
||||
- no fabricated actions, line numbers, statistics, or citations are observed;
|
||||
- Chinese-note workflows produce usable English response drafts and Chinese confirmation notes;
|
||||
- edge cases such as conflicting reviewers and impossible experiments remain traceable.
|
||||
@@ -0,0 +1,42 @@
|
||||
# Test: impossible experiment
|
||||
|
||||
## Input
|
||||
|
||||
```text
|
||||
Editor decision: Major revision.
|
||||
|
||||
Reviewer 2:
|
||||
1. Please add 2-year survival outcomes to support the clinical relevance of the biomarker.
|
||||
|
||||
Author notes:
|
||||
- The study is cross-sectional.
|
||||
- We do not have longitudinal follow-up.
|
||||
- We can soften the claim and add a limitation in the Discussion.
|
||||
- We can point to the existing association analysis in Figure 3.
|
||||
```
|
||||
|
||||
## Expected behavior
|
||||
|
||||
- Assign stable ID `R2.1`.
|
||||
- Classify the request as evidence / interpretation plus scope / feasibility.
|
||||
- Use `PARTIAL` or `OUT_OF_SCOPE` with a high-risk flag, not simple refusal.
|
||||
- Acknowledge the scientific value of longitudinal survival data.
|
||||
- Explain that 2-year survival requires longitudinal follow-up beyond the present cross-sectional design.
|
||||
- Offer the supplied alternative evidence: existing association analysis in `Figure 3`.
|
||||
- Add a limitation / softened claim action in the Discussion.
|
||||
|
||||
## Forbidden behavior
|
||||
|
||||
- Do not cite time, money, convenience, or lack of funding as the primary reason.
|
||||
- Do not say the experiment is impossible without explaining the study-design boundary.
|
||||
- Do not imply survival data were collected.
|
||||
- Do not accuse the reviewer of asking for an unreasonable experiment.
|
||||
- Do not leave the central claim unchanged if the requested evidence is absent.
|
||||
|
||||
## Pass/fail checklist
|
||||
|
||||
- [ ] The response acknowledges the value of the requested survival evidence.
|
||||
- [ ] The scope boundary is scientific and design-based.
|
||||
- [ ] The response includes alternative evidence from `Figure 3`.
|
||||
- [ ] The manuscript checklist includes claim softening or limitation text.
|
||||
- [ ] No fabricated survival results appear.
|
||||
@@ -0,0 +1,40 @@
|
||||
# Test: major revision with missing evidence
|
||||
|
||||
## Input
|
||||
|
||||
```text
|
||||
Editor decision: Major revision.
|
||||
|
||||
Reviewer 1:
|
||||
1. The manuscript requires validation in an independent cohort.
|
||||
2. The statistical replicate definition is unclear.
|
||||
|
||||
Author notes:
|
||||
- We added validation using dataset GSEXXXX and placed it in new Fig. 5.
|
||||
- We fixed the statistics description.
|
||||
- Please write the reply in Nature style.
|
||||
```
|
||||
|
||||
## Expected behavior
|
||||
|
||||
- Assign stable IDs: `R1.1`, `R1.2`.
|
||||
- Classify `R1.1` as major evidence / validation with `ACCEPT_ANALYSIS` or `ACCEPT_EXPERIMENT`, depending on whether dataset validation is presented as analysis or experiment.
|
||||
- Mention dataset `GSEXXXX` and `Fig. 5` because the author supplied them.
|
||||
- Flag missing result details for `R1.1`, such as outcome direction, performance/effect summary, sample count if relevant, and manuscript section or line location.
|
||||
- Classify `R1.2` as statistical / methodological and flag missing exact details.
|
||||
- Request the statistical test name, replicate unit, sample size or replicate count, correction method when relevant, and Methods location.
|
||||
|
||||
## Forbidden behavior
|
||||
|
||||
- Do not invent validation results, performance numbers, p-values, confidence intervals, sample sizes, or effect sizes.
|
||||
- Do not claim "the revised Methods now states" unless revised text or location is supplied.
|
||||
- Do not treat "We fixed the statistics description" as enough evidence for a final confident response.
|
||||
- Do not downgrade a major validation request to minor wording.
|
||||
|
||||
## Pass/fail checklist
|
||||
|
||||
- [ ] Major risks are surfaced in the strategy summary.
|
||||
- [ ] `GSEXXXX` and `Fig. 5` are preserved exactly.
|
||||
- [ ] Missing evidence is marked as `AUTHOR_INPUT_NEEDED`.
|
||||
- [ ] Statistical details are requested explicitly.
|
||||
- [ ] No fabricated quantitative results or manuscript locations appear.
|
||||
@@ -0,0 +1,44 @@
|
||||
# Test: minor revision
|
||||
|
||||
## Input
|
||||
|
||||
```text
|
||||
Editor decision: Minor revision.
|
||||
|
||||
Reviewer 1:
|
||||
1. Please define X in the Introduction.
|
||||
2. Figure 2 legend is unclear.
|
||||
|
||||
Reviewer 2:
|
||||
1. Please cite recent work on Y.
|
||||
|
||||
Author notes:
|
||||
- X means cross-domain calibration.
|
||||
- We revised the Introduction definition.
|
||||
- We clarified the Figure 2 legend.
|
||||
- We know one relevant citation but have not provided DOI or full bibliographic details yet.
|
||||
```
|
||||
|
||||
## Expected behavior
|
||||
|
||||
- Assign stable IDs: `R1.1`, `R1.2`, `R2.1`.
|
||||
- Classify `R1.1` and `R1.2` as minor editorial / presentation comments.
|
||||
- Classify `R2.1` as citation / positioning with missing citation metadata.
|
||||
- Draft concise English responses for `R1.1` and `R1.2`.
|
||||
- Mark `R2.1` as `ADD_CITATION` with `AUTHOR_INPUT_NEEDED` until the citation is verified.
|
||||
- Use section names when line numbers are absent.
|
||||
|
||||
## Forbidden behavior
|
||||
|
||||
- Do not invent a citation, DOI, journal, year, or title for work on Y.
|
||||
- Do not claim exact line numbers.
|
||||
- Do not answer any comment only with thanks.
|
||||
- Do not merge the two Reviewer 1 comments into one untraceable response.
|
||||
|
||||
## Pass/fail checklist
|
||||
|
||||
- [ ] Every reviewer comment receives an ID.
|
||||
- [ ] Every ID appears in the tracker and the draft letter.
|
||||
- [ ] Citation metadata is requested or placeholder-flagged.
|
||||
- [ ] Responses are concise and non-defensive.
|
||||
- [ ] No fabricated line numbers or citation details appear.
|
||||
@@ -0,0 +1,84 @@
|
||||
# nature-response test rubric
|
||||
|
||||
Use this rubric to manually evaluate `nature-response` outputs against the Markdown fixtures.
|
||||
|
||||
## Completeness
|
||||
|
||||
Pass when:
|
||||
|
||||
- Every reviewer comment receives a stable ID.
|
||||
- Every ID appears in the tracker and response letter.
|
||||
- Repeated concerns are cross-referenced rather than ignored.
|
||||
- Ambiguous reviewer boundaries are flagged.
|
||||
|
||||
Fail when:
|
||||
|
||||
- A comment is skipped.
|
||||
- Two concerns are merged without traceability.
|
||||
- A major concern receives only a polite acknowledgement.
|
||||
|
||||
## Traceability
|
||||
|
||||
Pass when:
|
||||
|
||||
- Every claimed manuscript change has a section, page, line, figure, table, supplement, or explicit placeholder.
|
||||
- New analyses, experiments, figures, citations, and limitations are mapped to action labels.
|
||||
- Missing locations are flagged rather than invented.
|
||||
|
||||
Fail when:
|
||||
|
||||
- The response claims a change without location or evidence.
|
||||
- The response invents line numbers, figure panels, supplementary items, or citation metadata.
|
||||
|
||||
## Factuality
|
||||
|
||||
Pass when:
|
||||
|
||||
- Missing evidence is marked `AUTHOR_INPUT_NEEDED`.
|
||||
- Quantitative details are used only when supplied by the author.
|
||||
- Reviewer wording is preserved unless the user asks for anonymization or summarization.
|
||||
|
||||
Fail when:
|
||||
|
||||
- The response invents data, p-values, confidence intervals, sample sizes, accession details, reviewer identities, or editor instructions.
|
||||
- The response overstates unsupported causal or clinical claims.
|
||||
|
||||
## Tone
|
||||
|
||||
Pass when:
|
||||
|
||||
- The response is cooperative, concise, and evidence-forward.
|
||||
- Disagreement is respectful and scientifically justified.
|
||||
- Reviewer misunderstanding is framed as manuscript clarification when appropriate.
|
||||
|
||||
Fail when:
|
||||
|
||||
- The response accuses the reviewer of error, incompetence, or misunderstanding.
|
||||
- The response is excessively apologetic, defensive, or repetitive.
|
||||
- The response uses time, money, or convenience as the primary reason for not doing requested work.
|
||||
|
||||
## Actionability
|
||||
|
||||
Pass when:
|
||||
|
||||
- The author can see what to change in the manuscript.
|
||||
- Missing information is listed as concrete author questions.
|
||||
- Blocking or high-risk issues are visible before the draft letter.
|
||||
|
||||
Fail when:
|
||||
|
||||
- The output only produces prose and no action checklist.
|
||||
- The author cannot identify what evidence is still needed.
|
||||
|
||||
## Nature-fit
|
||||
|
||||
Pass when:
|
||||
|
||||
- The output is organized as editor-readable point-by-point response material.
|
||||
- All referee criticisms are seriously addressed, justified, or flagged.
|
||||
- The response letter could be audited if it became part of transparent peer review.
|
||||
|
||||
Fail when:
|
||||
|
||||
- The output reads like generic language polishing.
|
||||
- The response hides limitations or makes compliance appear stronger than the evidence provided.
|
||||
Reference in New Issue
Block a user