first commit
This commit is contained in:
@@ -0,0 +1,199 @@
|
||||
# arXiv Literature Search Guide
|
||||
|
||||
## Overview
|
||||
|
||||
This guide provides workflows for discovering and evaluating recent ML research papers on arXiv. Use this when conducting literature reviews, finding related work, or staying updated on recent publications.
|
||||
|
||||
---
|
||||
|
||||
## Search Strategies
|
||||
|
||||
### 1. Keyword-Based Search
|
||||
|
||||
**arXiv Search URL Pattern:**
|
||||
```
|
||||
https://arxiv.org/search/?searchtype=all&query=KEYWORDS&abstracts=show&order=-announced_date_first
|
||||
```
|
||||
|
||||
**Common ML Search Keywords:**
|
||||
- **General ML**: `machine learning`, `deep learning`, `neural networks`
|
||||
- **Specific Areas**: `reinforcement learning`, `transformer`, `attention mechanism`, `graph neural networks`
|
||||
- **Applications**: `computer vision`, `natural language processing`, `reinforcement learning`
|
||||
- **Methods**: `self-supervised learning`, `contrastive learning`, `foundation models`
|
||||
|
||||
**Tips:**
|
||||
- Combine keywords with `+` for AND operation
|
||||
- Use `|` for OR operation
|
||||
- Put phrases in quotes for exact matches: `"attention is all you need"`
|
||||
|
||||
### 2. Category-Based Search
|
||||
|
||||
**Recommended arXiv Categories for ML:**
|
||||
- `cs.LG` (Machine Learning)
|
||||
- `cs.AI` (Artificial Intelligence)
|
||||
- `cs.CV` (Computer Vision and Pattern Recognition)
|
||||
- `cs.CL` (Computation and Language)
|
||||
- `cs.NE` (Neural and Evolutionary Computing)
|
||||
- `stat.ML` (Machine Learning - Statistics)
|
||||
|
||||
**Category Filter URL:**
|
||||
```
|
||||
https://arxiv.org/search/?cat:cs.LG+OR+cat:cs.AI+AND+all:transformer&abstracts=show&order=-announced_date_first
|
||||
```
|
||||
|
||||
### 3. Time-Based Filtering
|
||||
|
||||
**Recent Papers (Last 3 Months):**
|
||||
- Use `order=-announced_date_first` for newest first
|
||||
- Manually filter by submission date
|
||||
- Check paper metadata for submission date
|
||||
|
||||
---
|
||||
|
||||
## Using Chrome MCP for arXiv Search
|
||||
|
||||
When available, use Chrome MCP tools for automated arXiv searching:
|
||||
|
||||
1. **Navigate to arXiv search** using Chrome MCP navigation
|
||||
2. **Extract paper information** from search results:
|
||||
- Paper title
|
||||
- Authors
|
||||
- arXiv ID
|
||||
- Abstract preview
|
||||
- Publication date
|
||||
|
||||
3. **Navigate to individual papers** for detailed review
|
||||
|
||||
---
|
||||
|
||||
## Paper Quality Evaluation
|
||||
|
||||
Evaluate papers using the 5-dimension criteria below:
|
||||
|
||||
| Dimension | Weight | Key Points |
|
||||
|-----------|--------|------------|
|
||||
| **Innovation** | 30% | Novelty of contribution |
|
||||
| **Method Completeness** | 25% | Clarity and reproducibility |
|
||||
| **Experimental Thoroughness** | 25% | Validation depth |
|
||||
| **Writing Quality** | 10% | Clarity of expression |
|
||||
| **Relevance & Impact** | 10% | Domain importance |
|
||||
|
||||
### Scoring Guidelines (1-5 scale)
|
||||
|
||||
**Innovation (30%):**
|
||||
- 5: Breakthrough contribution, major impact
|
||||
- 4: Significant improvement, new insights
|
||||
- 3: Methodological innovation
|
||||
- 2: Incremental improvement
|
||||
- 1: Minor improvements
|
||||
|
||||
**Method Completeness (25%):**
|
||||
- 5: Complete and rigorous, easily reproducible
|
||||
- 4: Very detailed, mostly reproducible
|
||||
- 3: Core method clear, basically reproducible
|
||||
- 2: Lacks key details
|
||||
- 1: Unclear description
|
||||
|
||||
**Experimental Thoroughness (25%):**
|
||||
- 5: Comprehensive multi-dataset, ablation studies
|
||||
- 4: Multiple datasets, reasonable ablations
|
||||
- 3: Main experiments complete
|
||||
- 2: Limited experiments
|
||||
- 1: Minimal validation
|
||||
|
||||
**Writing Quality (10%):**
|
||||
- 5: Excellent clarity and rigor
|
||||
- 4: Clear and well-structured
|
||||
- 3: Understandable
|
||||
- 2: Some ambiguity
|
||||
- 1: Confusing
|
||||
|
||||
**Relevance & Impact (10%):**
|
||||
- 5: Solves important problem, wide impact
|
||||
- 4: Important domain problem
|
||||
- 3: Meaningful contribution
|
||||
- 2: Niche problem
|
||||
- 1: Limited impact
|
||||
|
||||
### Selection Process
|
||||
|
||||
1. **Screen by title/abstract** for relevance
|
||||
2. **Navigate to full paper** for detailed review
|
||||
3. **Score each dimension** (1-5)
|
||||
4. **Calculate weighted total**
|
||||
5. **Rank and select** top papers
|
||||
|
||||
---
|
||||
|
||||
## Extracting Paper Metadata
|
||||
|
||||
**From arXiv Abstract Page (`https://arxiv.org/abs/ARXIV_ID`):**
|
||||
|
||||
- Title (from `<h1>` tag)
|
||||
- Authors (from `.authors` element)
|
||||
- Abstract (from `blockquote.abstract`)
|
||||
- Submission date (from `.dateline`)
|
||||
- arXiv ID (from URL)
|
||||
- Categories (from `.subjects`)
|
||||
- Comments (if present)
|
||||
- Code repository (check abstract for GitHub links)
|
||||
|
||||
---
|
||||
|
||||
## Integration with Citation Workflow
|
||||
|
||||
After finding relevant papers:
|
||||
|
||||
1. **Verify citations** using Semantic Scholar API (see `../citation-workflow.md`)
|
||||
2. **Fetch BibTeX** programmatically via DOI
|
||||
3. **Store in bibliography** with verification status
|
||||
|
||||
---
|
||||
|
||||
## Common Use Cases
|
||||
|
||||
### Finding Related Work
|
||||
|
||||
When writing a paper, use arXiv search to:
|
||||
1. Find recent papers on your topic
|
||||
2. Identify state-of-the-art methods
|
||||
3. Discover competing approaches
|
||||
4. Find baseline comparisons
|
||||
|
||||
### Staying Updated
|
||||
|
||||
Set up regular searches for:
|
||||
- Your specific research area
|
||||
- Competing labs/researchers
|
||||
- New methods in your domain
|
||||
- Conference proceedings (preprints)
|
||||
|
||||
### Literature Reviews
|
||||
|
||||
For comprehensive reviews:
|
||||
1. Start with broad keyword searches
|
||||
2. Filter by recent publications (last 1-3 years)
|
||||
3. Use citation chaining (forward and backward)
|
||||
4. Evaluate and select high-quality papers
|
||||
5. Organize by theme/contribution
|
||||
|
||||
---
|
||||
|
||||
## Tips for Effective Searching
|
||||
|
||||
1. **Use specific keywords** rather than broad terms
|
||||
2. **Combine techniques** (keywords + categories + time filters)
|
||||
3. **Check code availability** (many arXiv papers link to GitHub)
|
||||
4. **Look for citations** to understand impact
|
||||
5. **Read abstracts carefully** before full papers
|
||||
6. **Use paper metrics** (citation count, code stars) as indicators
|
||||
|
||||
---
|
||||
|
||||
## External Resources
|
||||
|
||||
- **arXiv**: https://arxiv.org/
|
||||
- **Semantic Scholar**: https://www.semanticscholar.org/
|
||||
- **Papers With Code**: https://paperswithcode.com/
|
||||
- **Connected Papers**: https://www.connectedpapers.com/
|
||||
- **arXiv API**: http://export.arxiv.org/api_help/
|
||||
@@ -0,0 +1,303 @@
|
||||
# ML Paper Quality Evaluation Criteria
|
||||
|
||||
## Overview
|
||||
|
||||
Use these criteria to evaluate ML research papers found during literature search or when selecting papers for detailed review. The 5-dimension framework provides structured assessment for paper selection and comparison.
|
||||
|
||||
---
|
||||
|
||||
## Evaluation Dimensions
|
||||
|
||||
| Dimension | Weight | Description |
|
||||
|-----------|--------|-------------|
|
||||
| **Innovation** | 30% | Novelty and originality of contribution |
|
||||
| **Method Completeness** | 25% | Clarity, rigor, and reproducibility |
|
||||
| **Experimental Thoroughness** | 25% | Validation depth and analysis quality |
|
||||
| **Writing Quality** | 10% | Clarity and presentation |
|
||||
| **Relevance & Impact** | 10% | Domain importance and potential impact |
|
||||
|
||||
---
|
||||
|
||||
## Detailed Scoring Rubrics
|
||||
|
||||
### 1. Innovation (30%)
|
||||
|
||||
**Score 5 - Breakthrough:**
|
||||
- Proposes entirely new paradigm or framework
|
||||
- Solves long-standing open problem
|
||||
- Major impact expected on the field
|
||||
- Challenges fundamental assumptions
|
||||
|
||||
**Score 4 - Significant Innovation:**
|
||||
- Substantial improvement over existing methods
|
||||
- New insights or perspectives
|
||||
- Novel combination of techniques
|
||||
- Clear advancement over state-of-the-art
|
||||
|
||||
**Score 3 - Methodological Innovation:**
|
||||
- New method or architecture proposed
|
||||
- Some novelty but incremental
|
||||
- Reasonable contribution
|
||||
- Standard type of innovation
|
||||
|
||||
**Score 2 - Incremental Improvement:**
|
||||
- Minor improvements to existing methods
|
||||
- Limited novelty
|
||||
- Small advancement
|
||||
- Mostly derivative
|
||||
|
||||
**Score 1 - Trivial:**
|
||||
- Minimal contribution
|
||||
- Obvious extension
|
||||
- No real innovation
|
||||
- Known results
|
||||
|
||||
**Evaluation Questions:**
|
||||
- Does this paper propose something genuinely new?
|
||||
- Does it advance the state-of-the-art?
|
||||
- Will this influence future work?
|
||||
- Is the contribution significant or marginal?
|
||||
|
||||
---
|
||||
|
||||
### 2. Method Completeness (25%)
|
||||
|
||||
**Score 5 - Complete and Rigorous:**
|
||||
- Full mathematical derivation
|
||||
- All hyperparameters specified
|
||||
- Complete algorithmic details
|
||||
- Easily reproducible
|
||||
- Code available
|
||||
|
||||
**Score 4 - Very Complete:**
|
||||
- Detailed method description
|
||||
- Most important details included
|
||||
- Mostly reproducible
|
||||
- Minor gaps in documentation
|
||||
|
||||
**Score 3 - Reproducible:**
|
||||
- Core method clearly described
|
||||
- Key details present
|
||||
- Can be reproduced with effort
|
||||
- Some ambiguity in details
|
||||
|
||||
**Score 2 - Lacks Details:**
|
||||
- Key details missing
|
||||
- Difficult to reproduce
|
||||
- Incomplete description
|
||||
- Ambiguous in important areas
|
||||
|
||||
**Score 1 - Unclear:**
|
||||
- Method description unclear
|
||||
- Missing critical information
|
||||
- Cannot determine validity
|
||||
- Poorly explained
|
||||
|
||||
**Evaluation Questions:**
|
||||
- Can another researcher reproduce this work?
|
||||
- Are all important details specified?
|
||||
- Is mathematical derivation sound?
|
||||
- Is code available and documented?
|
||||
|
||||
---
|
||||
|
||||
### 3. Experimental Thoroughness (25%)
|
||||
|
||||
**Score 5 - Comprehensive:**
|
||||
- Multiple diverse datasets
|
||||
- Extensive ablation studies
|
||||
- Statistical significance testing
|
||||
- Thorough analysis and discussion
|
||||
- Comparison with strong baselines
|
||||
|
||||
**Score 4 - Very Thorough:**
|
||||
- Multiple datasets
|
||||
- Reasonable ablation studies
|
||||
- Proper baseline comparisons
|
||||
- Good analysis
|
||||
|
||||
**Score 3 - Adequate:**
|
||||
- Main experiments complete
|
||||
- Standard datasets
|
||||
- Basic baselines
|
||||
- Results are credible
|
||||
|
||||
**Score 2 - Limited:**
|
||||
- Limited experiments
|
||||
- Few datasets
|
||||
- Weak baselines
|
||||
- Minimal analysis
|
||||
|
||||
**Score 1 - Insufficient:**
|
||||
- Minimal validation
|
||||
- Toy examples only
|
||||
- No meaningful comparisons
|
||||
- Results not convincing
|
||||
|
||||
**Evaluation Questions:**
|
||||
- Are experiments comprehensive?
|
||||
- Are baselines strong and appropriate?
|
||||
- Are statistical tests used?
|
||||
- Is there ablation analysis?
|
||||
- Are results on standard datasets?
|
||||
|
||||
---
|
||||
|
||||
### 4. Writing Quality (10%)
|
||||
|
||||
**Score 5 - Excellent:**
|
||||
- Clear, precise, well-structured
|
||||
- Logical flow throughout
|
||||
- Professional presentation
|
||||
- High-quality figures
|
||||
- No ambiguity
|
||||
|
||||
**Score 4 - Very Good:**
|
||||
- Clear and well-written
|
||||
- Mostly logical structure
|
||||
- Good presentation
|
||||
- Minor issues
|
||||
|
||||
**Score 3 - Understandable:**
|
||||
- Basically clear
|
||||
- Some organizational issues
|
||||
- Acceptable presentation
|
||||
- Understandable with effort
|
||||
|
||||
**Score 2 - Fair:**
|
||||
- Some confusing sections
|
||||
- Organization problems
|
||||
- Presentation issues
|
||||
- Hard to follow at times
|
||||
|
||||
**Score 1 - Poor:**
|
||||
- Unclear or confusing
|
||||
- Poor organization
|
||||
- Difficult to understand
|
||||
- Major presentation problems
|
||||
|
||||
**Evaluation Questions:**
|
||||
- Is the paper easy to understand?
|
||||
- Is the structure logical?
|
||||
- Are figures/tables clear?
|
||||
- Is the writing professional?
|
||||
|
||||
---
|
||||
|
||||
### 5. Relevance & Impact (10%)
|
||||
|
||||
**Score 5 - High Impact:**
|
||||
- Solves important problem
|
||||
- Broad applicability
|
||||
- Expected wide influence
|
||||
- Addresses fundamental challenge
|
||||
|
||||
**Score 4 - Domain Important:**
|
||||
- Important problem in field
|
||||
- Significant potential impact
|
||||
- Relevant to many researchers
|
||||
|
||||
**Score 3 - Meaningful:**
|
||||
- Meaningful contribution
|
||||
- Moderate impact expected
|
||||
- Relevant to subset of field
|
||||
|
||||
**Score 2 - Niche:**
|
||||
- Specialized problem
|
||||
- Limited applicability
|
||||
- Narrow impact
|
||||
|
||||
**Score 1 - Limited:**
|
||||
- Very narrow problem
|
||||
- Minimal impact expected
|
||||
- Limited relevance
|
||||
|
||||
**Evaluation Questions:**
|
||||
- Is this an important problem?
|
||||
- Will this influence future work?
|
||||
- Is it relevant to current research needs?
|
||||
- Does it address a significant challenge?
|
||||
|
||||
---
|
||||
|
||||
## Scoring Calculation
|
||||
|
||||
**Weighted Total:**
|
||||
```
|
||||
Total = (Innovation × 0.30) + (Method × 0.25) + (Experiments × 0.25) + (Writing × 0.10) + (Impact × 0.10)
|
||||
```
|
||||
|
||||
**Example Calculation:**
|
||||
- Innovation: 4/5
|
||||
- Method: 3/5
|
||||
- Experiments: 4/5
|
||||
- Writing: 3/5
|
||||
- Impact: 4/5
|
||||
|
||||
```
|
||||
Total = (4 × 0.30) + (3 × 0.25) + (4 × 0.25) + (3 × 0.10) + (4 × 0.10)
|
||||
= 1.20 + 0.75 + 1.00 + 0.30 + 0.40
|
||||
= 3.65 / 5.0
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Selection Process
|
||||
|
||||
### For Literature Reviews
|
||||
|
||||
1. **Screen papers** by title/abstract for relevance
|
||||
2. **Full review** of potentially relevant papers
|
||||
3. **Score each paper** using all 5 dimensions
|
||||
4. **Rank by total score**
|
||||
5. **Select top papers** for detailed review
|
||||
|
||||
### Quality Thresholds
|
||||
|
||||
- **Excellent**: 4.0+ (include definitely)
|
||||
- **Good**: 3.5-3.9 (include if relevant)
|
||||
- **Fair**: 3.0-3.4 (include if highly relevant)
|
||||
- **Poor**: <3.0 (exclude unless essential)
|
||||
|
||||
---
|
||||
|
||||
## Quick Screening Indicators
|
||||
|
||||
Before detailed review, check:
|
||||
|
||||
**Positive Indicators:**
|
||||
- Published at top venue (NeurIPS, ICML, ICLR)
|
||||
- Citations in top papers
|
||||
- Code available with stars
|
||||
- Authors from top labs
|
||||
- Clear novelty in abstract
|
||||
|
||||
**Negative Indicators:**
|
||||
- Vague abstract
|
||||
- Limited experiments mentioned
|
||||
- No baselines mentioned
|
||||
- Poor writing in abstract
|
||||
- incremental claims only
|
||||
|
||||
---
|
||||
|
||||
## Integration with Paper Discovery
|
||||
|
||||
When using arXiv search (`arxiv-search-guide.md`):
|
||||
|
||||
1. **Search** for relevant papers
|
||||
2. **Extract metadata** from arXiv pages
|
||||
3. **Quick screen** by abstract/relevance
|
||||
4. **Detailed review** of promising papers
|
||||
5. **Score using** these criteria
|
||||
6. **Rank and select** top candidates
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
- These criteria are designed for ML papers specifically
|
||||
- Adjust weights based on your specific needs
|
||||
- Use scores as relative comparisons, not absolute judgments
|
||||
- Consider venue reputation as additional signal
|
||||
- Code availability is increasingly important for reproducibility
|
||||
Reference in New Issue
Block a user