# "Rethinking" Papers: Challenging Conventional Wisdom **Source**: Kaiming He et al., "Autoregressive Image Generation without Vector Quantization" (NeurIPS 2024 Spotlight) **Paper Type**: Paradigm-challenging / "Rethinking" paper **Core Pattern**: Question deeply-held assumptions → Propose alternative → Demonstrate effectiveness --- ## 1. Abstract Structure: Challenging Conventional Wisdom ### Pattern: The "Conventional Wisdom" Opening **Template**: ```markdown Abstract: 1. [Hook] Conventional wisdom holds that [widely-believed assumption]. 2. [Observation] We observe that [counter-point or nuance]. 3. [Question] Is it necessary for [core assumption]? 4. [Proposal] In this work, we propose [alternative approach]. 5. [Solution] Rather than [traditional method], we [novel method]. 6. [Results] This approach [eliminates/enables] X, achieving [strong results]. 7. [Vision] We hope this work will [broader impact statement]. ``` ### MAR Abstract Example (annotated): ```latex Conventional wisdom holds that [autoregressive models for image generation are typically accompanied by vector-quantized tokens]. We observe that while [discrete-valued space can facilitate representing a categorical distribution], it is [not a necessity for autoregressive modeling]. In this work, we propose to [model the per-token probability distribution using a diffusion procedure], which allows us to [apply autoregressive models in a continuous-valued space]. Rather than using [categorical cross-entropy loss], we define a [Diffusion Loss function] to model the per-token probability. This approach [eliminates the need for discrete-valued tokenizers]. We evaluate its effectiveness across [a wide range of cases], including [standard autoregressive models and generalized masked autoregressive (MAR) variants]. By removing vector quantization, our image generator achieves [strong results] while enjoying [the speed advantage of sequence modeling]. We hope this work will motivate [the use of autoregressive generation in other continuous-valued domains and applications]. ``` ### Key Techniques: 1. **"Conventional wisdom holds that..."** - Strong, respectful opening that acknowledges prevailing beliefs 2. **"We observe that..."** - Signals empirical insight rather than mere speculation 3. **"it is not a necessity for..."** - Gentle but direct challenge to the assumption 4. **"Rather than..."** - Clear alternative positioning 5. **"This approach eliminates the need for..."** - Practical benefit statement 6. **"We hope this work will..."** - Forward-looking vision --- ## 2. Introduction: The Question-Driven Framework ### Pattern: Start with the Question **Traditional Introduction** (less effective): - "Image generation is an important problem..." - "Existing methods use VQ tokenizers..." - "We propose a new method..." **"Rethinking" Introduction** (Kaiming He style): ```markdown 1. [Establish Context] Domain context + prevailing approach 2. [Identify Assumption] The widely-held belief 3. [Formulate Question] "Is it necessary for..." 4. [Analyze Requirements] What is truly essential? 5. [Propose Alternative] If alternative model exists... 6. [Present Solution] Our specific proposal 7. [Broader Implications] What this enables ``` ### MAR Introduction Structure (lines 20-79): #### Part 1: Context and Prevailing Belief (lines 20-29) ```latex Autoregressive models are currently the de facto solution to generative models in natural language processing [38, 39, 3]. These models predict the next word or token in a sequence based on the previous words as input. Given the discrete nature of languages, the inputs and outputs of these models are in a categorical, discrete-valued space. This prevailing approach has led to a widespread belief that autoregressive models are inherently linked to discrete representations. ``` **Technique**: Start from uncontroversial facts (NLP success) → Show how they led to a belief (AR ↔ discrete) #### Part 2: Consequences of the Belief (lines 25-29) ```latex As a result, research on generalizing autoregressive models to continuous- valued domains—most notably, image generation—has intensely focused on discretizing the data [6, 13, 40]. A commonly adopted strategy is to train a discrete-valued tokenizer on images, which involves a finite vocabulary obtained by vector quantization (VQ) [51, 41, 41]. ``` **Technique**: Show concrete consequences of the belief (VQ tokenizers everywhere) #### Part 3: The Research Question (lines 30-33) ```latex In this work, we aim to address the following question: Is it necessary for autoregressive models to be coupled with vector- quantized representations? ``` **Technique**: Explicit question format. Bold, direct, centered. #### Part 4: Analyzing What's Essential (lines 31-37) ```latex We note that the autoregressive nature, i.e., "predicting next tokens based on previous ones", is independent of whether the values are discrete or continuous. What is needed is to model the per-token probability distribution, which can be measured by a loss function and used to draw samples from. Discrete-valued representations can be conveniently modeled by a categorical distribution, but it is not conceptually necessary. ``` **Technique**: Distinguish mechanism (autoregression) from implementation (discrete tokens) #### Part 5: The "If..." Condition (lines 36-37) ```latex If alternative models for per-token probability distributions are presented, autoregressive models can be approached without vector quantization. ``` **Technique**: Logical bridge - "If X, then Y" sets up your contribution #### Part 6: Your Solution (lines 38-58) ```latex With this observation, we propose to model the per-token probability distribution by a diffusion procedure operating on continuous-valued domains. [Technical description of Diffusion Loss...] ``` #### Part 7: Benefits and Implications (lines 59-79) ```latex Our approach eliminates the need for discrete-valued tokenizers. Vector- quantized tokenizers are difficult to train and are sensitive to gradient approximation strategies [51, 41, 40, 27]. Their reconstruction quality often falls short compared to continuous-valued counterparts [42]. Our approach allows autoregressive models to enjoy the benefits of higher- quality, non-quantized tokenizers. [...] The effectiveness of our method reveals a largely uncharted realm of image generation: modeling the interdependence of tokens by autoregression, jointly with the per-token distribution by diffusion. ``` **Technique**: Multiple benefit levels: - **Eliminates pain points**: VQ is hard to train - **Enables benefits**: Higher-quality tokenizers - **Reveals new territory**: "uncharted realm" --- ## 3. Rethinking Section: Dissecting the Status Quo ### Pattern: "Rethinking X" Chapter When challenging conventional wisdom, dedicate a section to re-examining the assumptions. ### MAR Section 3.1: "Rethinking Discrete-Valued Tokens" **Structure**: ```markdown 1. [Setup] Define the conventional approach mathematically 2. [Analysis] Identify the essential properties (not the implementation) 3. [Insight] Show that the implementation is not necessary 4. [Conclusion] State what's actually needed ``` ### MAR Example (lines 110-126): #### Step 1: Define the Conventional Approach ```latex To begin with, we revisit the roles of discrete-valued tokens in autoregressive generation models. Denote as x the ground-truth token to be predicted at the next position. With a discrete tokenizer, x can be represented as an integer: 0 ≤ x < K, with a vocabulary size K. The autoregressive model produces a continuous-valued D-dim vector z ∈ R^D, which is then projected by a K-way classifier matrix W ∈ R^(K×D). Conceptually, this formulation models a categorical probability distribution in the form of p(x|z) = softmax(Wz). ``` **Technique**: Mathematically precise setup. Shows you deeply understand the status quo. #### Step 2: Identify Essential Properties ```latex In the context of generative modeling, this probability distribution must exhibit two essential properties. (i) A loss function that can measure the difference between the estimated and true distributions. (ii) A sampler that can draw samples from the distribution x ~ p(x|z) at inference time. ``` **Technique**: Abstract away from implementation to functional requirements #### Step 3: Show Convention is Not Necessary ```latex In the case of categorical distribution, this can be simply done by the cross-entropy loss. [...] This analysis suggests that discrete-valued tokens are not necessary for autoregressive models. ``` **Technique**: "This analysis suggests" - evidence-based conclusion #### Step 4: What's Actually Needed ```latex Instead, it is the requirement of modeling a distribution that is essential. A discrete-valued token space implies a categorical distribution, whose loss function and sampler are simple to define. What we actually need are a loss function and its corresponding sampler for distribution modeling. ``` **Technique**: Reposition the problem - it's about distribution modeling, not discrete tokens --- ## 4. Unifying Different Methods: The "Generalized Framework" Pattern ### Pattern: Show Two Seemingly Different Methods Are Special Cases **MAR Section 3.4**: "Unifying Autoregressive and Masked Generative Models" **Structure**: ```markdown 1. [Observation] These methods seem different but share a core principle 2. [Insight] The core principle is X, not the surface differences 3. [Demonstration] Show how Method A fits the principle 4. [Demonstration] Show how Method B fits the principle 5. [Unification] Both are special cases of a general framework 6. [Benefit] This understanding enables new combinations/insights ``` ### MAR Example (lines 181-259): #### Opening Observation ```latex We show that masked generative models, e.g., MaskGIT [4] and MAGE [29], can be generalized under the broad concept of autoregression, i.e., next token prediction. ``` #### The Insight ```latex Bidirectional attention can perform autoregression. The concept of autoregression is orthogonal to network architectures: autoregression can be done by RNNs [50], CNNs [49, 7], and Transformers [38, 36, 6]. When using Transformers, although autoregressive models are popularly implemented by causal attention, we show that they can also be done by bidirectional attention. ``` **Technique**: "Orthogonal to" - shows independence of concepts #### Visual Demonstration (Figure 2) ```latex Note that the goal of autoregression is to predict the next token given the previous tokens; it does not constrain how the previous tokens communicate with the next token. We can adopt the bidirectional attention implementation as done in Masked Autoencoder (MAE) [21]. See Figure 2(b). ``` **Technique**: Use visualization to show "unexpected" equivalence #### Generalization (Figure 3) ```latex Autoregressive models in random orders... Masked autoregressive models: In masked generative modeling [4, 29], the models predict a random subset of tokens based on known/predicted tokens. This can be formulated as permuting the token sequence by a random order, and then predicting multiple tokens based on previous tokens. Conceptually, this is an autoregressive procedure... ``` **Technique**: Mathematical unification - same formula, different instantiations #### Naming the Generalization ```latex We refer to this variant as Masked Autoregressive (MAR) models. MAR is a random-order autoregressive model that can predict multiple tokens simultaneously. ``` **Technique**: Define new terminology to capture the unified concept --- ## 5. Flexibility Arguments: Demonstrating Generality ### Pattern: Show Your Method Works in Many Configurations **MAR Section 5.1**: "Flexibility of Diffusion Loss" **Structure**: ```markdown 1. [Claim] One significant advantage is flexibility with various X 2. [Demonstration 1] Works with configuration A 3. [Demonstration 2] Works with configuration B (even surprising case) 4. [Demonstration 3] Works with configuration C 5. [Insight] This flexibility enables Y ``` ### MAR Example (lines 349-366, Table 2): #### Opening Claim ```latex One significant advantage of Diffusion Loss is its flexibility with various tokenizers. ``` #### Surprising Case 1: VQ Tokenizers ```latex Diffusion Loss can be easily used even given a VQ tokenizer. We simply treat the continuous-valued latent before the VQ layer as the tokens. This variant gives us 7.82 FID (w/o CFG), compared favorably with 8.79 FID (Table 1) of cross-entropy loss using the same VQ tokenizer. ``` **Technique**: Show it works even with the "wrong" tokenizer type (VQ) #### Surprising Case 2: Mismatched Stride ```latex Interestingly, Diffusion Loss also enables us to use tokenizers with mismatched strides. In Table 2, we study a KL-8 tokenizer whose stride is 8 and output sequence length is 32×32. Without increasing the sequence length of the generator, we group 2×2 tokens into a new token. Despite the mismatch, we are able to obtain decent results... ``` **Technique**: "Interestingly" + "Despite the mismatch" - emphasizes robustness #### Surprising Case 3: Different Architecture ```latex Further, this property allows us to investigate other tokenizers, e.g., Consistency Decoder [35], a non-VQ tokenizer of a different architecture/stride designed for different goals. ``` **Technique**: Show applicability beyond original design space #### Comprehiveness Check ```latex For comprehensiveness, we also train a KL-16 tokenizer on ImageNet using the code of [42]... ``` **Technique**: "For comprehensiveness" - signals thoroughness --- ## 6. Speed/Accuracy Trade-offs: Plotting the Pareto Frontier ### Pattern: Show Your Method Dominates or Expands the Frontier **MAR Section 5.2**: "Speed/accuracy Trade-off" (Figure 6) **Structure**: ```markdown 1. [Setup] We enjoy flexibility of X 2. [Visualization] Figure/plot showing trade-off curves 3. [Comparison 1] Compare within your variants 4. [Comparison 2] Compare to other methods 5. [Highlight] Notable point on the curve ``` ### MAR Example (lines 425-474): #### Setup ```latex Following MaskGIT [4], our MAR enjoys the flexibility of predicting multiple tokens at a time. This is controlled by the number of autoregressive steps at inference time. ``` #### Comparison Within Family ```latex Figure 6 plots the speed/accuracy trade-off. MAR has a better trade-off than its AR counterpart, noting that AR is with the efficient kv-cache. ``` **Technique**: "noting that..." - acknowledge AR's advantage, still win #### Comparison to Other Methods ```latex With Diffusion Loss, MAR also shows a favorable trade-off in comparison with the recently popular Diffusion Transformer (DiT) [37]. As a latent diffusion model, DiT models the interdependence of all tokens by the diffusion process. The speed/accuracy trade-off of DiT is mainly controlled by its diffusion steps. Unlike our diffusion process on a small MLP, the diffusion process of DiT involves the entire Transformer architecture. Our method is more accurate and faster. ``` **Technique**: Explain *why* you win (architectural difference) #### Highlight Specific Point ```latex Notably, our method can generate at a rate of <0.3 second per image with a strong FID of <2.0. ``` **Technique**: "Notably" + impressive numbers - memorable takeaway --- ## 7. System-Level Comparison Tables: Organizing by Categories ### Pattern: Group Related Work, Highlight Your Position **MAR Table 4**: "System-level comparison on ImageNet 256×256" **Structure**: ```markdown Table 4: ┌─────────────────────────────────┐ │ Category 1: pixel-based │ │ - Method A │ │ - Method B │ ├─────────────────────────────────┤ │ Category 2: vector-quantized │ │ - Method C │ │ - Method D │ ├─────────────────────────────────┤ │ Category 3: continuous-valued │ │ - Method E │ │ - Method F │ ├─────────────────────────────────┤ │ **Your Methods** (highlighted) │ │ - Your Method B │ │ - Your Method L │ │ - Your Method H │ └─────────────────────────────────┘ ``` ### MAR Example (lines 448-470): ```latex Table 4: System-level comparison on ImageNet 256×256 conditional generation. Diffusion Loss enables Masked Autoregression to achieve leading results in comparison with previous systems. †: LDM operates on continuous-valued tokens, though this result uses a quantized tokenizer. w/o CFG w/ CFG #params FID↓ IS↑ Pre.↑ Rec.↑ FID↓ IS↑ Pre.↑ Rec.↑ ──────────────────────────────────────────────────────────── pixel-based ADM [10] 554M 10.94 101.0 0.69 0.63 4.59 186.7 0.82 0.52 VDM++ [26] 2B 2.40 225.3 - - 2.12 267.7 - - vector-quantized tokens Autoreg. w/ VQGAN [13] 1.4B 15.78 78.3 - - - - - - MaskGIT [4] 227M 6.18 182.1 0.80 0.51 - - - - MAGE [29] 230M 6.93 195.8 - - - - - - MAGVIT-v2 [55] 307M 3.65 200.5 - - 1.78 319.4 - - continuous-valued tokens LDM-4† [42] 400M 10.56 103.5 0.71 0.62 3.60 247.7 0.87 0.48 U-ViT-H/2-G [2] 501M - - - - 2.29 263.9 0.82 0.57 DiT-XL/2 [37] 675M 9.62 121.5 0.67 0.67 2.27 278.2 0.83 0.57 [... other methods ...] MAR-B, Diff Loss 208M 3.48 192.4 0.78 0.58 2.31 281.7 0.82 0.57 MAR-L, Diff Loss 479M 2.60 221.4 0.79 0.60 1.78 296.0 0.81 0.60 MAR-H, Diff Loss 943M 2.35 227.8 0.79 0.62 1.55 303.7 0.81 0.62 ``` **Key Techniques**: 1. **Group by paradigm**: pixel-based → VQ → continuous-valued 2. **Show progression**: Earlier methods → later methods → yours 3. **Multiple metrics**: FID, IS, Precision, Recall (comprehensive) 4. **Multiple configurations**: w/o CFG and w/ CFG (fair comparison) 5. **Model sizes**: Show efficiency (your 479M vs their 675M) 6. **Dagger (†)**: Clarify edge cases (LDM uses quantized despite continuous) 7. **Bold your methods**: Visual separation (here shown as last group) --- ## 8. Admitting Limitations: Honest and Specific ### Pattern: Acknowledge Weaknesses Proactively **MAR Appendix A**: "Limitations and Broader Impacts" **Structure**: ```markdown Limitations: 1. [Specific artifact] Show example (Figure 8) 2. [Dependency] Acknowledge reliance on external components 3. [Scope] Be honest about what you haven't tested Broader Impacts: 1. [Positive] How this advances the field 2. [Positive] Practical applications 3. [Negative] Potential misuse considerations ``` ### MAR Limitations Example (lines 629-642): #### Limitation 1: Visual Artifacts ```latex First of all, our image generation system can produce images with noticeable artifacts (Figure 8). This limitation is commonly observed in existing methods, especially when trained on controlled, academic data (e.g., ImageNet). Research-driven models trained on ImageNet still have a noticeable gap in visual quality in comparison with commercial models trained on massive data. ``` **Techniques**: - Show example (Figure 8) - transparency - "commonly observed" - you're not uniquely bad - Contextualize: academic vs. commercial data #### Limitation 2: External Dependency ```latex Second, our image generation system relies on existing pre-trained tokenizers. The quality of our system can be limited by the quality of these tokenizers. Pre-training better tokenizers is beyond the scope of this paper. ``` **Techniques**: - "relies on" - direct acknowledgment - "can be limited by" - honest constraint - "beyond the scope" - boundary setting #### Limitation 3: Unexplored Territory ```latex Last, we note that given the limited computational resources, we have primarily tested our method on the ImageNet benchmark. Further validation is needed to assess the scalability and robustness of our approach in more diverse and real-world scenarios. ``` **Techniques**: - "given limited computational resources" - context, not excuse - "primarily tested" - honesty about scope - "Further validation is needed" - explicit call for future work --- ## 9. Visionary Conclusion: The "We Hope" Pattern ### Pattern: End with Forward-Looking Impact Statement **MAR Conclusion** (lines 482-488): ```latex The effectiveness of Diffusion Loss on various autoregressive models suggests new opportunities: modeling the interdependence of tokens by autoregression, jointly with the per-token distribution by diffusion. This is unlike the common usage of diffusion that models the joint distribution of all tokens. Our strong results on image generation suggest that autoregressive models or their extensions are powerful tools beyond language modeling. These models do not need to be constrained by vector-quantized representations. We hope our work will motivate the research community to explore sequence models with continuous-valued representations in other domains. ``` ### Structure: ```markdown 1. [Synthesis] What your results suggest (new opportunities) 2. [Contrast] How this differs from common usage 3. [Generalization] Broader lesson (powerful tools beyond X) 4. [Freedom] Removing constraints (not limited by Y) 5. [Call to action] "We hope" - inspire future work ``` ### Key Phrases: - "suggests new opportunities" - "unlike the common usage" - "powerful tools beyond [original domain]" - "do not need to be constrained by" - "We hope our work will motivate..." --- ## 10. Writing Style: Kaiming He's Signature Techniques ### Tone Characteristics: 1. **Respectful but Firm Challenge** - "Conventional wisdom holds that..." (respect) - "...is not a necessity" (firm) - Never dismissive of prior work 2. **Empirically Grounded** - "We observe that..." (not "We believe that...") - "This analysis suggests..." (evidence-based) - Show, don't just tell 3. **Logical Precision** - Distinguish mechanism from implementation - "If X, then Y" formulations - Mathematical clarity in conceptual arguments 4. **Measured Claims** - "can be" not "is always" - "suggests" not "proves" - "enables" not "guarantees" 5. **Generous Citation** - Cite the methods you're challenging - Credit relevant prior work - "following [X]" for techniques you adopt ### Sentence Patterns: **Challenge Patterns:** - "Conventional wisdom holds that [X]. We observe that [Y]." - "This prevailing approach has led to a widespread belief that [X]." - "[X] is not a necessity for [Y]." - "What is needed is [X], not [Y]." **Insight Patterns:** - "The concept of [X] is orthogonal to [Y]." - "This is unlike the common usage of [X]." - "These models do not need to be constrained by [X]." - "This reveals a largely uncharted realm of [X]." **Demonstration Patterns:** - "Interestingly, [X] also enables us to..." - "Despite [challenge], we are able to..." - "For comprehensiveness, we also..." - "Notably, [impressive result]." **Vision Patterns:** - "We hope this work will motivate..." - "This suggests new opportunities for..." - "We believe this will be beneficial to..." - "Further validation is needed to assess..." --- ## 11. Figures: Making the Invisible Visible ### Pattern: Use Visualization to Show Conceptual Equivalence **MAR Figure 2**: "Bidirectional attention can do autoregression" **Purpose**: Challenge the assumption that autoregression requires causal attention **Technique**: Side-by-side comparison - (a) Causal attention (conventional) - (b) Bidirectional attention (surprising but equivalent) **Annotation Key Points**: - "next token prediction" (highlight both do this) - "loss on [all/unknown] tokens" (show the difference) - Arrows showing attention flow **MAR Figure 3**: "Generalized Autoregressive Models" **Purpose**: Show AR, random-order AR, and MAR are special cases **Technique**: Progressive generalization - (a) AR, raster order (standard) - (b) AR, random order (small variation) - (c) Masked AR (generalization) **Visual Language**: - "unknown" vs "known/predicted" tokens - Color coding for what's predicted at each step - Consistent notation across panels **Key**: Figures should make conceptual contributions visually obvious --- ## 12. Related Work: Positioning Within Traditions ### Pattern: Acknowledge Lineage, Show Departure **MAR Section 2**: "Related Work" (lines 80-103) **Structure**: ```markdown 1. [Main tradition] Sequence Models for Image Generation - Pioneering work [citations] - Your position: "Related to our work, [X] also focuses on..." - Contrast: "In [X], [limitation]. In contrast, our method..." 2. [Adjacent tradition] Diffusion for Representation Learning - Prior work [citations] - Clarify difference: "These efforts have been focused on [X], rather than [Y]" - "In their scenarios, [goal] is not a goal" 3. [Conceptual connection] Diffusion for Policy Learning - "Our work is conceptually related to [X]" - Analogy: "In image generation, we can think of [X] as [Y]" - Difference: "Despite this conceptual connection, [key difference]" ``` ### Key Techniques: 1. **"Related to our work..."** - Acknowledge contemporaries generously 2. **"In contrast..."** - Sharp differentiation 3. **"These efforts have been focused on..."** - Respectful boundary setting 4. **"In their scenarios..."** - Contextualize differences 5. **"Conceptually related to..."** - Find connections across domains 6. **"We can think of X as Y"** - Analogies for insight 7. **"Despite this conceptual connection..."** - Honest differences --- ## 13. Experiments: Ablation as Storytelling ### Pattern: Experiments Should Validate the "Rethinking" **MAR Section 5**: Experiments organized to prove the rethinking **Flow**: ```markdown 1. Properties of Diffusion Loss (prove the alternative works) - vs Cross-entropy (Table 1) - Flexibility (Table 2) - Ablations (Table 3, Figures 4-5) 2. Properties of Generalized AR (prove the unification) - From AR to MAR (Table 1) - Speed/accuracy trade-off (Figure 6) 3. Benchmarking (prove the results are strong) - System-level comparison (Table 4) - Qualitative results (Figure 7) ``` **Key**: Every experiment should map to a conceptual claim from the introduction ### Table Design Patterns: **Comparison Tables** (Table 1): - Group by variants (AR → MAR) - Show progression (Cross-ent → Diff Loss) - Multiple configurations (w/ and w/o CFG) - Clear metrics (FID↓, IS↑) **Flexibility Tables** (Table 2): - "Flexibility of X" in caption - Diverse configurations - Architecture variations - Stride mismatches - rFID column (reconstruction quality) **Ablation Tables** (Table 3): - One dimension varied (MLP width) - Show params and inference time - Demonstrate efficiency - Default row marked (implicit) --- ## 14. Summary: The "Rethinking" Paper Template ### When to Use This Pattern: **Your paper is a "Rethinking" paper if:** - You challenge a widely-held assumption in your field - You show two seemingly different methods are equivalent - You propose removing a "necessary" component - You unify multiple approaches under one framework **Not a "Rethinking" paper if:** - You propose an incremental improvement - You combine existing techniques without conceptual insight - You're optimizing within an established paradigm ### Abstract Template: ``` Conventional wisdom holds that [assumption]. We observe that [nuance]. Is it necessary for [assumption]? In this work, we propose [alternative]. Rather than [traditional method], we [novel method]. This approach [benefits]. We hope this work will [vision]. ``` ### Introduction Template: ``` [Context] Domain X uses method Y widely. [Belief] This has led to belief that Z. [Question] Is Z necessary? [Analysis] What's actually needed is W. [Proposal] We propose alternative that achieves W. [Results] Strong performance + [specific benefit]. [Vision] This opens new direction. ``` ### Section Template ("Rethinking X"): ``` 1. Define the conventional approach mathematically 2. Identify essential properties (2-3 items) 3. Show conventional is just one way to achieve these 4. State what's actually needed (abstract) 5. Position your method as satisfying these needs ``` ### Unification Template: ``` [Observation] Methods A and B seem different. [Insight] But they share core principle X. [Demonstration] Show A = X + implementation Y [Demonstration] Show B = X + implementation Z [Generalization] Both are special cases of framework F [Benefit] This understanding enables [new possibilities] ``` ### Conclusion Template: ``` Our results suggest [new opportunity]. Unlike common usage that [old approach], we show [new insight]. This reveals [uncharted realm]. We hope this will motivate [future direction]. ``` --- ## 15. Common Pitfalls to Avoid ### ❌ Don't: - Dismiss prior work as "wrong" - it led to the belief for good reasons - Overstate your case - "completely eliminates" vs "reduces the need for" - Ignore implementation realities - VQ was solving real problems - Be vague about alternatives - show concrete alternatives - Forget to cite the methods you're challenging ### ✅ Do: - Acknowledge why the conventional wisdom emerged - Show limitations respectfully - Provide rigorous comparisons - Admit when your method doesn't help - Credit relevant work extensively - Use precise language ("can be" not "is always") --- ## 16. Citation Integration Examples ### How MAR Cites Challenges: **Citing the method you're improving:** ```latex Discrete tokenizers are difficult to train [51, 41, 40, 27]. ``` - Multiple citations show it's a known problem - Recent citations ([27]) show it's still active **Citing contemporaries:** ```latex Related to our work, the recent work on GIVT [48] also focuses on continuous-valued tokens in sequence models. GIVT and our work both reveal the significance and potential of this direction. In GIVT, [limitation]. In contrast, our method [advantage]. ``` - "Related to our work" - generous positioning - "both reveal" - shared progress - "In contrast" - clear differentiation **Citing the technique you adapt:** ```latex We can adopt the bidirectional attention implementation as done in Masked Autoencoder (MAE) [21]. See Figure 2(b). ``` - "as done in" - giving credit - Reference to figure for transparency --- ## 17. Revision Checklist for "Rethinking" Papers **Before Submission, Verify:** - [ ] Abstract clearly states the challenged assumption - [ ] Introduction asks an explicit question ("Is it necessary...") - [ ] "Rethinking" section dissects the status quo rigorously - [ ] Unification section shows mathematical/conceptual equivalence - [ ] Experiments directly test the rethinking claims - [ ] Flexibility/robustness experiments included - [ ] Limitations acknowledge remaining challenges - [ ] Related work cites challenged methods generously - [ ] Conclusion frames work as opening new territory - [ ] Figures make conceptual contributions visible - [ ] Tone is respectful but firm - [ ] Claims are measured ("can be", "suggests", not "proves") - [ ] Vision statement inspires without overpromising --- ## 18. Example: Applying This Pattern ### Original Idea (Not "Rethinking"): "We propose a new tokenizer that improves image quality by 10%." ### Rethinking Version: "Conventional wisdom holds that autoregressive models require vector-quantized tokenizers. We observe that VQ introduces artifacts and training difficulties. Is quantization necessary for autoregressive image generation? We propose Diffusion Loss that enables continuous-valued tokens, eliminating VQ while achieving SOTA results. We hope this will motivate continuous-valued modeling in other domains." **The Rethinking Frame:** - Assumption: AR ↔ VQ (challenged) - Insight: What's needed is distribution modeling, not discreteness - Alternative: Diffusion Loss for continuous tokens - Benefit: Eliminates VQ artifacts + enables better tokenizers - Vision: Continuous-valued AR beyond images --- ## Paper Metadata **Title**: Autoregressive Image Generation without Vector Quantization **Authors**: Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, Kaiming He **Venue**: NeurIPS 2024 (Spotlight) **arXiv**: 2406.11838 **Code**: https://github.com/LTH14/mar **Key Citations**: - VQ-VAE [51]: Original VQ paper - MaskGIT [4], MAGE [29]: Masked generative models - MAE [21]: Bidirectional attention implementation - DiT [37]: Speed/accuracy comparison - LDM [42]: Tokenizer source --- ## Extracted by **Date**: 2025-01-26 **Source**: Full PDF analysis of arXiv:2406.11838v3 **Extraction Focus**: Writing patterns for challenging conventional wisdom, unifying methods, and structuring "rethinking" papers **For Integration**: ml-paper-writing skill knowledge base