first commit
This commit is contained in:
@@ -0,0 +1,199 @@
|
||||
# pubfig API map (source-driven)
|
||||
|
||||
This guide maps the public `pubfig` API from `pubfig/src/pubfig/__init__.py` to the underlying modules.
|
||||
|
||||
## 1. Stable entrypoint
|
||||
|
||||
The public contract is defined by re-exports in `pubfig.__init__`.
|
||||
|
||||
For agents, this means:
|
||||
|
||||
- if a symbol is re-exported there, it is a good default public entrypoint;
|
||||
- if a helper only exists in deep internal modules, treat it as implementation detail unless there is a strong reason not to.
|
||||
|
||||
## 2. Public API groups
|
||||
|
||||
### Themes
|
||||
|
||||
Public re-exports:
|
||||
|
||||
- `set_default_theme`
|
||||
- `get_default_theme`
|
||||
- `get_theme`
|
||||
- `register_theme`
|
||||
|
||||
Use these when the task is really about reusable visual policy.
|
||||
Do not hardcode theme assumptions if a registry call is more appropriate.
|
||||
|
||||
### Colors and palettes
|
||||
|
||||
Public re-exports include:
|
||||
|
||||
- `DEFAULT`, `NATURE`, `SCIENCE`, `LANCET`, `JAMA`
|
||||
- `get_palette`
|
||||
- `register_palette`
|
||||
- `color_to_rgba`
|
||||
- `darken_color`
|
||||
- `show_palette`
|
||||
|
||||
Source fact:
|
||||
|
||||
- palette registration and palette inspection are exposed from the package root.
|
||||
|
||||
Operational implication:
|
||||
|
||||
- treat palette selection and palette registration as public API usage, not as deep internal customization.
|
||||
|
||||
### Export
|
||||
|
||||
Public re-exports:
|
||||
|
||||
- `save_figure`
|
||||
- `batch_export`
|
||||
- `PanelExportRecord`
|
||||
- `export_panel`
|
||||
- `export_panels`
|
||||
- `package_figma_bundle`
|
||||
- `validate_figma_bundle`
|
||||
- `inspect_figma_bundle`
|
||||
|
||||
Skill implication:
|
||||
|
||||
- normal paper figures should usually stop at `save_figure` or `batch_export`
|
||||
- panel workflows should use `export_panel(s)`
|
||||
- bundle helpers are for bridge/Figma handoff, not the default answer
|
||||
|
||||
### Figure specs
|
||||
|
||||
Public re-exports:
|
||||
|
||||
- `FigureSpec`
|
||||
- `get_figure_spec`
|
||||
- `register_figure_spec`
|
||||
- `list_figure_specs`
|
||||
|
||||
Use this layer whenever the user asks for venue-aware width, journal defaults, or a custom export profile.
|
||||
|
||||
### Plot families
|
||||
|
||||
The public plot surface is broad, but the source still clusters naturally.
|
||||
|
||||
#### Comparison / summary figures
|
||||
|
||||
Representative public calls:
|
||||
|
||||
- `bar`
|
||||
- `bar_scatter`
|
||||
- `stacked_bar`
|
||||
- `stacked_ratio_barh`
|
||||
- `donut`
|
||||
- `dumbbell`
|
||||
- `forest_plot`
|
||||
- `grouped_scatter`
|
||||
- `upset`
|
||||
|
||||
Use these for benchmark, ablation, summary, composition, and set-overlap tasks.
|
||||
|
||||
#### Distribution figures
|
||||
|
||||
Representative public calls:
|
||||
|
||||
- `ecdf`
|
||||
- `qq`
|
||||
- `box`
|
||||
- `density`
|
||||
- `hexbin`
|
||||
- `histogram`
|
||||
- `raincloud`
|
||||
- `strip`
|
||||
- `ridgeline`
|
||||
- `violin`
|
||||
|
||||
Use these when the scientific claim is about spread, calibration of assumptions, or cohort structure.
|
||||
|
||||
#### Trend / profile figures
|
||||
|
||||
Representative public calls:
|
||||
|
||||
- `line`
|
||||
- `area`
|
||||
- `parallel_coordinates`
|
||||
- `radar`
|
||||
- `radial_hierarchy`
|
||||
- `circular_stacked_bar`
|
||||
- `circular_grouped_bar`
|
||||
|
||||
Not all of these are equally strong for publication use. The skill should still apply chart-selection discipline before calling them.
|
||||
|
||||
#### Relationship / embedding figures
|
||||
|
||||
Representative public calls:
|
||||
|
||||
- `scatter`
|
||||
- `volcano`
|
||||
- `bubble`
|
||||
- `contour2d`
|
||||
- `paired`
|
||||
- `heatmap`
|
||||
- `corr_matrix`
|
||||
- `clustermap`
|
||||
- `dimreduce`
|
||||
- `pca_biplot`
|
||||
|
||||
Use these for association, error structure, feature layout, and representation views.
|
||||
|
||||
#### Evaluation figures
|
||||
|
||||
Representative public calls:
|
||||
|
||||
- `roc`
|
||||
- `pr_curve`
|
||||
- `calibration`
|
||||
- `bland_altman`
|
||||
|
||||
This cluster matters because the source gives them dedicated implementation in `plots/evaluation.py`, which is a sign that evaluation charts are a first-class use case.
|
||||
|
||||
## 3. Return-value contract
|
||||
|
||||
The source in `export/io.py` makes a subtle but important contract explicit:
|
||||
|
||||
- export functions accept a Matplotlib `Figure`,
|
||||
- or an `Axes`,
|
||||
- or any object exposing a `.figure` attribute that resolves to a `Figure`.
|
||||
|
||||
Source fact:
|
||||
|
||||
- the export layer accepts standard Matplotlib figure objects or figure-bearing wrappers.
|
||||
|
||||
Operational implication:
|
||||
|
||||
- keep the `Figure` handle available and route export through the standard Matplotlib-facing export path.
|
||||
|
||||
For the skill, the safest phrasing is:
|
||||
|
||||
- create the figure,
|
||||
- keep a handle to the `Figure`,
|
||||
- then export explicitly.
|
||||
|
||||
## 4. What is not the main stable plotting interface
|
||||
|
||||
From the source tree, the CLI is not where normal chart creation happens.
|
||||
It is mostly an operational layer for Figma bundle and bridge actions.
|
||||
|
||||
So if a user says “generate a paper-ready figure,” the skill should not default to a CLI answer.
|
||||
|
||||
## 5. Safe public usage pattern
|
||||
|
||||
The most source-faithful pattern is:
|
||||
|
||||
1. choose a public plot function from `pubfig`
|
||||
2. generate a `Figure`
|
||||
3. export via `save_figure(...)` or `batch_export(...)`
|
||||
4. only use panel/bundle helpers when composition is actually required
|
||||
|
||||
## 6. Source-guided caution points
|
||||
|
||||
- Do not mix up plot-time design sizing with export-time publication sizing.
|
||||
- Do not use `save_figure(...)` as a multi-format exporter; the source now pushes that role to `batch_export(...)`.
|
||||
- Do not route ordinary figure-generation tasks through the Figma bridge CLI.
|
||||
- Do not assume all public plot families are equally appropriate; the skill must still filter by scientific communication quality.
|
||||
@@ -0,0 +1,190 @@
|
||||
# pubfig architecture (source-driven)
|
||||
|
||||
This guide reads `pubfig` from the source tree rather than from high-level overview material.
|
||||
|
||||
Source root:
|
||||
|
||||
- `pubfig/src/pubfig/__init__.py`
|
||||
- `pubfig/src/pubfig/plots/`
|
||||
- `pubfig/src/pubfig/export/`
|
||||
- `pubfig/src/pubfig/specs.py`
|
||||
- `pubfig/src/pubfig/themes/`
|
||||
- `pubfig/src/pubfig/colors/`
|
||||
- `pubfig/src/pubfig/cli.py`
|
||||
|
||||
## 1. Start at `pubfig.__init__`
|
||||
|
||||
The stable user-facing surface is re-exported from `__init__.py`.
|
||||
|
||||
That file tells you the package is organized into five main layers:
|
||||
|
||||
1. **plot constructors** from `plots/`
|
||||
2. **export helpers** from `export/`
|
||||
3. **publication sizing** from `specs.py`
|
||||
4. **theme and palette registries** from `themes/` and `colors/`
|
||||
5. **Figma/bridge helpers** and related CLI support
|
||||
|
||||
For skill design, this is the most important architectural fact:
|
||||
|
||||
- figure generation lives in `plots/`
|
||||
- figure export lives in `export/`
|
||||
- venue sizing lives in `specs.py`
|
||||
- multi-panel/Figma handoff is a separate downstream layer
|
||||
|
||||
The default mental model is **plot first, export second, compose third**.
|
||||
|
||||
## 2. Package boundaries
|
||||
|
||||
### `plots/`
|
||||
|
||||
This is the core figure-construction layer.
|
||||
|
||||
Representative files:
|
||||
|
||||
- `plots/line.py`
|
||||
- `plots/comparison.py`
|
||||
- `plots/evaluation.py`
|
||||
- `plots/_grouped_scatter.py`
|
||||
|
||||
From the source, plot functions usually do the same sequence:
|
||||
|
||||
1. normalize/coerce input data,
|
||||
2. enter `theme_context(theme)`,
|
||||
3. resolve design-time size via `resolve_design_dpi(...)`,
|
||||
4. allocate figure/axes via `get_fig_ax(...)`,
|
||||
5. style axes/legends through helpers in `_style.py`,
|
||||
6. return a Matplotlib `Figure`.
|
||||
|
||||
Interpretation from source:
|
||||
|
||||
- `pubfig` behaves as a **Matplotlib-first figure factory layer**, not as a separate scene-graph runtime.
|
||||
|
||||
### `export/`
|
||||
|
||||
This is intentionally separated from plotting.
|
||||
|
||||
Important files:
|
||||
|
||||
- `export/io.py`
|
||||
- `export/panels.py`
|
||||
|
||||
`export/io.py` handles normal figure export:
|
||||
|
||||
- coerce Figure/Axes into a real `Figure`
|
||||
- enforce explicit suffixes
|
||||
- apply publication width/height rules
|
||||
- write vector or raster output
|
||||
|
||||
Current source implication:
|
||||
|
||||
- `batch_export(...)` now belongs to the same publication-aware export layer, rather than to a simple multi-format `savefig` wrapper.
|
||||
|
||||
`export/panels.py` handles panel-level export for composite or Figma-oriented workflows:
|
||||
|
||||
- one panel at a time or many panels together
|
||||
- optional publication-aware sizing
|
||||
- optional title stripping
|
||||
- metadata index generation (`panel-index.json`)
|
||||
|
||||
### `specs.py`
|
||||
|
||||
This file is the publication-sizing contract.
|
||||
|
||||
`FigureSpec` defines:
|
||||
|
||||
- `font_family`
|
||||
- `design_dpi`
|
||||
- `single_column_mm`
|
||||
- `double_column_mm`
|
||||
- `default_raster_dpi`
|
||||
- `background_color`
|
||||
|
||||
Built-in registry entries include:
|
||||
|
||||
- `nature`
|
||||
- `science`
|
||||
- `cell`
|
||||
|
||||
The source shows a strong split between:
|
||||
|
||||
- **design size** used when constructing interactive figures,
|
||||
- **physical export size** used when saving publication figures.
|
||||
|
||||
That split is why the skill should not treat `width` in plot calls and `width` in export calls as the same semantic layer.
|
||||
|
||||
### `themes/` and `colors/`
|
||||
|
||||
These are registries, not plain constants.
|
||||
|
||||
From `__init__.py`, the public surface includes:
|
||||
|
||||
- `get_theme`, `register_theme`, `set_default_theme`
|
||||
- `get_palette`, `register_palette`, `show_palette`
|
||||
|
||||
Operational implication:
|
||||
|
||||
- treat theme and palette selection as first-class API configuration rather than as hardcoded styling trivia.
|
||||
|
||||
### `cli.py`
|
||||
|
||||
The current CLI is not the main figure-generation interface.
|
||||
|
||||
From the source, `cli.py` is mainly about:
|
||||
|
||||
- Figma bridge serving
|
||||
- bundle packaging/inspection/validation
|
||||
- sync job submission and waiting
|
||||
- local bridge auto-start logic
|
||||
|
||||
So for this skill:
|
||||
|
||||
- **Python API is the primary route for figure generation**
|
||||
- `pubfig.cli` is a secondary operational layer for bridge/Figma workflows
|
||||
|
||||
## 3. Plotting architecture pattern
|
||||
|
||||
From `line.py`, `comparison.py`, and `evaluation.py`, the recurring internal pattern is:
|
||||
|
||||
- input normalization is local to the chart family,
|
||||
- shared visual behavior is delegated to internal helpers,
|
||||
- returned artifact is still a standard Matplotlib figure.
|
||||
|
||||
This is why the skill should map requests to a chart family first, instead of jumping directly to export or panel assembly.
|
||||
|
||||
Examples from source:
|
||||
|
||||
- `line.py` groups time/trend style plots
|
||||
- `comparison.py` groups comparison-style statistical displays like `dumbbell` and `forest_plot`
|
||||
- `evaluation.py` groups metric/evaluation plots like `roc`, `pr_curve`, and `calibration`
|
||||
- `_grouped_scatter.py` contains the more specialized placement/jitter/annotation logic behind grouped scatter layouts
|
||||
|
||||
## 4. Export architecture pattern
|
||||
|
||||
From `export/io.py` and `export/panels.py`, `pubfig` uses three distinct output modes:
|
||||
|
||||
1. **single explicit artifact** via `save_figure(...)`
|
||||
2. **publication-aware multi-format artifact set** via `batch_export(...)`
|
||||
3. **panel bundle workflow** via `export_panel(...)` / `export_panels(...)`
|
||||
|
||||
Those are different contracts, and the skill should keep them separate in its recommendations.
|
||||
|
||||
## 5. Reading order for deep debugging
|
||||
|
||||
When a skill or agent needs source-level certainty, use this order:
|
||||
|
||||
1. `pubfig/src/pubfig/__init__.py`
|
||||
2. relevant chart-family module in `plots/`
|
||||
3. `pubfig/src/pubfig/specs.py`
|
||||
4. `pubfig/src/pubfig/export/io.py`
|
||||
5. `pubfig/src/pubfig/export/panels.py`
|
||||
6. `pubfig/src/pubfig/cli.py` only if the task involves bridge/Figma sync
|
||||
|
||||
## 6. Implications for this skill
|
||||
|
||||
This source layout implies the skill should:
|
||||
|
||||
- default to **Python plot API + explicit export call**,
|
||||
- treat publication sizing as an export concern,
|
||||
- treat panel/Figma work as optional downstream composition,
|
||||
- avoid presenting the CLI as the main path for ordinary figure generation,
|
||||
- keep chart selection logically ahead of export tuning.
|
||||
@@ -0,0 +1,210 @@
|
||||
# pubfig export flow (source-driven)
|
||||
|
||||
This guide explains how `pubfig` moves from a generated Matplotlib figure to paper-ready files.
|
||||
|
||||
Primary source files:
|
||||
|
||||
- `pubfig/src/pubfig/export/io.py`
|
||||
- `pubfig/src/pubfig/export/panels.py`
|
||||
- `pubfig/src/pubfig/specs.py`
|
||||
|
||||
## 1. Core export contract
|
||||
|
||||
`export/io.py` separates two concerns:
|
||||
|
||||
- **coercing a valid figure object**
|
||||
- **writing explicit output files**
|
||||
|
||||
The helper `_coerce_mpl_figure(...)` accepts:
|
||||
|
||||
- a `Figure`
|
||||
- an `Axes`
|
||||
- an object with a `.figure` attribute pointing to a `Figure`
|
||||
|
||||
The export layer is standardized around Matplotlib figures, even if upstream plotting code returns a richer wrapper.
|
||||
|
||||
## 2. `save_figure(...)` is now single-target and suffix-explicit
|
||||
|
||||
A key source-level rule lives in `_resolve_save_figure_target(...)`:
|
||||
|
||||
- `save_figure(...)` now requires an explicit filename suffix,
|
||||
- supported examples include `.pdf`, `.svg`, `.png`, `.jpg`, `.tif`, `.eps`, `.ps`,
|
||||
- if there is no suffix, the function raises an error,
|
||||
- if multiple outputs are wanted, the source tells you to use `batch_export(...)`.
|
||||
|
||||
Skill implication:
|
||||
|
||||
- always write `results/figure1.pdf` rather than `results/figure1`
|
||||
- when you want several formats, recommend `batch_export(...)`, not legacy vector/raster format lists
|
||||
|
||||
## 3. Publication sizing path
|
||||
|
||||
`save_figure(...)` is publication-aware.
|
||||
|
||||
Internally it does the following:
|
||||
|
||||
1. load the chosen `FigureSpec` via `get_figure_spec(...)`
|
||||
2. resolve width through `resolve_width_mm(...)`
|
||||
3. resolve height through `resolve_height_mm(...)`
|
||||
4. set the Matplotlib figure size in inches using mm-to-inch conversion
|
||||
5. choose raster DPI from the spec unless overridden
|
||||
6. save the explicit target file
|
||||
7. restore original caller state afterward
|
||||
|
||||
Interpretation from source:
|
||||
|
||||
- export sizing is more than a file-write step; it can temporarily resize the figure to venue-oriented physical dimensions before output.
|
||||
|
||||
## 4. Width and height semantics
|
||||
|
||||
From `specs.py`:
|
||||
|
||||
- width can be `single`, `double`, or a numeric mm value
|
||||
- the built-in registry contains `nature`, `science`, and `cell`
|
||||
- height can be explicit `height_mm`
|
||||
- otherwise height is derived from `aspect_ratio`
|
||||
|
||||
That yields a clean rule for the skill:
|
||||
|
||||
- if the user asks for publication width, use `save_figure(..., spec=..., width=...)`
|
||||
- if the user only wants quick draft export, keep the recommendation minimal
|
||||
|
||||
## 5. `batch_export(...)`
|
||||
|
||||
`batch_export(...)` is the publication-aware multi-format lane.
|
||||
|
||||
Source behavior:
|
||||
|
||||
- it takes a `base_path`
|
||||
- it accepts publication export controls such as `spec`, `width`, `height_mm`, `aspect_ratio`, and `dpi`
|
||||
- appends each explicit suffix from `formats`
|
||||
- relayouts the figure through `_export_with_publication_layout(...)` for each target format
|
||||
- restores the original in-memory figure size/state after export
|
||||
|
||||
This is the right recommendation when the user needs, for example:
|
||||
|
||||
- `PDF` for manuscript submission
|
||||
- `SVG` for downstream editing
|
||||
- `PNG` for slides or issue threads
|
||||
|
||||
Operational implication:
|
||||
|
||||
- use `batch_export(...)` when the task needs multiple publication-style outputs from the same figure,
|
||||
- do not describe it as a plain `savefig` loop.
|
||||
|
||||
## 6. What `_save_basic_figure(...)` still does
|
||||
|
||||
`_save_basic_figure(...)` is still relevant, but it is no longer the main multi-format path for `batch_export(...)`.
|
||||
|
||||
It remains the lower-level path used for:
|
||||
|
||||
- direct basic export helpers,
|
||||
- size-preserving panel export in `export/panels.py`,
|
||||
- and internal single-target save operations that do not need publication relayout.
|
||||
|
||||
From the source, it also handles:
|
||||
|
||||
- output directory creation
|
||||
- vector-text rcParams (important for editable SVG/PDF text handling)
|
||||
- post-layout legend alignment
|
||||
- post-layout callbacks attached by plot code
|
||||
- trim/tight bbox behavior
|
||||
|
||||
So export quality is partially centralized in the export layer, not only inside plot modules.
|
||||
|
||||
## 7. Panel export lane
|
||||
|
||||
`export/panels.py` defines the multi-panel handoff path.
|
||||
|
||||
Key components:
|
||||
|
||||
- `PanelExportRecord`
|
||||
- `export_panel(...)`
|
||||
- `export_panels(...)`
|
||||
- `_write_panel_index(...)`
|
||||
|
||||
A `PanelExportRecord` stores:
|
||||
|
||||
- `panel_id`
|
||||
- `path`
|
||||
- `format`
|
||||
- `exported_at`
|
||||
- `figma_node_name`
|
||||
- `pubfig_version`
|
||||
- optional `title`
|
||||
- optional `label`
|
||||
|
||||
This shows that panel export is not only file emission. It also preserves minimal sync metadata.
|
||||
|
||||
## 8. Title stripping is intentional
|
||||
|
||||
One subtle but important source behavior:
|
||||
|
||||
- `_temporarily_strip_titles(...)` removes figure/axes titles during panel export by default unless `include_title=True`
|
||||
|
||||
Operational implication:
|
||||
|
||||
- panel-first composite assembly usually wants clean panel artwork,
|
||||
- whole-figure titles and layout labels are often handled later,
|
||||
- prefer exporting clean panel art first and adding whole-figure titles or layout labels downstream when needed.
|
||||
|
||||
## 9. Publication-aware vs size-preserving panel export
|
||||
|
||||
`export_panel(...)` has two modes:
|
||||
|
||||
### Publication-aware mode
|
||||
|
||||
Triggered when any of these are supplied:
|
||||
|
||||
- `spec`
|
||||
- `width`
|
||||
- `height_mm`
|
||||
|
||||
Then it delegates to `save_figure(...)`.
|
||||
|
||||
### Size-preserving mode
|
||||
|
||||
If none of those are supplied, it falls back to `_save_basic_figure(...)` and preserves the figure’s current size.
|
||||
|
||||
Skill implication:
|
||||
|
||||
- for reproducible paper panels, specify publication export parameters
|
||||
- for design review or quick composition, preserving current size may be acceptable
|
||||
|
||||
## 10. Multiple panel export
|
||||
|
||||
`export_panels(...)` does three main things:
|
||||
|
||||
1. normalize and validate panel ids,
|
||||
2. resolve labels for each panel,
|
||||
3. export each panel and optionally write `panel-index.json`.
|
||||
|
||||
Default recommendation:
|
||||
|
||||
- prefer this route when the user wants a structured panel directory rather than a single whole-figure asset.
|
||||
|
||||
## 11. Overwrite and safety behavior
|
||||
|
||||
From `_ensure_writable_target(...)`:
|
||||
|
||||
- an existing panel file raises unless `overwrite=True`
|
||||
|
||||
That is useful for skill guidance because it means refresh-in-place is an explicit decision.
|
||||
|
||||
## 12. Recommended source-faithful export patterns
|
||||
|
||||
### Single paper figure
|
||||
|
||||
- plot with `pubfig.<chart_family>(...)`
|
||||
- save with `save_figure(fig, 'out/figure1.pdf', spec='nature', width='single')`
|
||||
|
||||
### Same figure in several formats
|
||||
|
||||
- plot once
|
||||
- export with `batch_export(fig, 'out/figure1', formats=('pdf', 'svg', 'png'), spec='nature', width='single', dpi=300)`
|
||||
|
||||
### Multi-panel downstream assembly
|
||||
|
||||
- generate each panel as a separate `Figure`
|
||||
- export with `export_panels(...)`
|
||||
- use the index file for composite/Figma-aware downstream handling
|
||||
@@ -0,0 +1,216 @@
|
||||
# pubtab architecture (source-driven)
|
||||
|
||||
This guide explains `pubtab` from the actual package layout under `pubtab/src/pubtab`.
|
||||
|
||||
Core files:
|
||||
|
||||
- `__init__.py`
|
||||
- `cli.py`
|
||||
- `models.py`
|
||||
- `reader.py`
|
||||
- `renderer.py`
|
||||
- `config.py`
|
||||
- `_preview.py`
|
||||
- `tex_reader.py`
|
||||
- `backends/`
|
||||
- `themes/`
|
||||
|
||||
## 1. Start at `pubtab.__init__`
|
||||
|
||||
The public contract is defined in `__init__.py`.
|
||||
|
||||
Public exports are intentionally small:
|
||||
|
||||
- `xlsx2tex`
|
||||
- `preview`
|
||||
- `compile_pdf`
|
||||
- `tex_to_excel`
|
||||
- `SpacingConfig`
|
||||
|
||||
This is the key architectural signal: `pubtab` exposes a compact API, while the real complexity is pushed into reader/renderer/preview internals.
|
||||
|
||||
## 2. Core data model layer
|
||||
|
||||
`models.py` defines the structured table representation.
|
||||
|
||||
Important dataclasses:
|
||||
|
||||
- `CellStyle`
|
||||
- `Cell`
|
||||
- `TableData`
|
||||
- `SpacingConfig`
|
||||
- `ThemeConfig`
|
||||
- `BackendConfig`
|
||||
|
||||
Source fact:
|
||||
|
||||
- the shared structured representation is `TableData`, not raw Excel cells or raw LaTeX text.
|
||||
|
||||
Forward and reverse conversions both pass through this structured table model.
|
||||
|
||||
## 3. Forward pipeline: Excel to LaTeX
|
||||
|
||||
The main forward path is:
|
||||
|
||||
1. `xlsx2tex(...)` in `__init__.py`
|
||||
2. `read_excel(...)` in `reader.py`
|
||||
3. `render(...)` in `renderer.py`
|
||||
4. optional preview through `_preview.py`
|
||||
|
||||
This gives a clean source-driven decomposition:
|
||||
|
||||
- `reader.py` = file ingestion and structure recovery
|
||||
- `renderer.py` = LaTeX generation
|
||||
- `_preview.py` = compile and raster preview
|
||||
|
||||
## 4. Reverse pipeline: LaTeX to Excel
|
||||
|
||||
The reverse path is:
|
||||
|
||||
1. `tex_to_excel(...)` in `__init__.py`
|
||||
2. `read_tex_multi(...)` from `tex_reader.py`
|
||||
3. writer functions to `.xlsx`
|
||||
|
||||
So roundtrip support is not an afterthought. It is a real architecture branch.
|
||||
|
||||
## 5. `reader.py` is richer than a plain spreadsheet loader
|
||||
|
||||
From the source, `reader.py` does much more than “read cells”:
|
||||
|
||||
- supports `.xlsx` and `.xls`
|
||||
- extracts rich text segments
|
||||
- reconstructs merged cells and spans
|
||||
- reads styling information
|
||||
- trims only trailing globally empty columns
|
||||
- reads pubtab metadata sheets
|
||||
- restores group separators, multicolumn alignment hints, and math-script hints
|
||||
|
||||
Interpretation from source:
|
||||
|
||||
- `pubtab` is optimized for **publication table semantics**, rather than for plain tabular text dumping.
|
||||
|
||||
## 6. `renderer.py` is the central logic hub
|
||||
|
||||
`renderer.py` turns `TableData` into backend-specific LaTeX.
|
||||
|
||||
From the source, it handles:
|
||||
|
||||
- style/theme loading
|
||||
- backend template loading
|
||||
- spacing resolution
|
||||
- column spec construction or projection
|
||||
- tabular vs tabularray differences
|
||||
- merged cells, row/column spans, header rules, vertical rules
|
||||
- background colors and grouped separators
|
||||
- final template rendering
|
||||
|
||||
This file is where most of the difficult publication logic lives.
|
||||
|
||||
For skill design, that means:
|
||||
|
||||
- backend choice is not a cosmetic toggle
|
||||
- column spec and rule behavior are structural concerns
|
||||
- preview/render bugs usually require reading `renderer.py`
|
||||
|
||||
## 7. Theme vs backend is a real split
|
||||
|
||||
The codebase separates:
|
||||
|
||||
- **themes** in `themes/`
|
||||
- **LaTeX backends** in `backends/`
|
||||
|
||||
That is reflected in two dataclasses:
|
||||
|
||||
- `ThemeConfig`
|
||||
- `BackendConfig`
|
||||
|
||||
And in two loaders:
|
||||
|
||||
- `load_theme(...)`
|
||||
- `load_backend(...)`
|
||||
|
||||
This is a major architectural point for the skill:
|
||||
|
||||
- theme decides stylistic defaults,
|
||||
- backend decides LaTeX environment/template behavior.
|
||||
|
||||
Do not explain them as if they were the same thing.
|
||||
|
||||
## 8. Config precedence is explicit
|
||||
|
||||
In `xlsx2tex(...)`, the source implements a clear precedence order:
|
||||
|
||||
1. hardcoded defaults
|
||||
2. YAML config loaded by `config.py`
|
||||
3. explicit function kwargs
|
||||
4. in some roundtrip cases, values recovered from `TableData`
|
||||
|
||||
This is why user-facing guidance should say “CLI flags or function kwargs override YAML config.”
|
||||
|
||||
## 9. Multi-file and multi-sheet support are first-class
|
||||
|
||||
From `__init__.py`:
|
||||
|
||||
- directory input is supported for both forward and reverse paths
|
||||
- sheet enumeration is supported when `sheet=None`
|
||||
- multi-sheet export produces `*_sheetNN.tex`
|
||||
|
||||
The default skill guidance can therefore recommend batch/file-driven workflows, not only one-table-at-a-time usage.
|
||||
|
||||
## 10. Preview is a real compilation layer
|
||||
|
||||
`preview(...)` and `compile_pdf(...)` in `__init__.py` delegate into `_preview.py`.
|
||||
|
||||
That layer:
|
||||
|
||||
- finds or installs `pdflatex`
|
||||
- builds a standalone document
|
||||
- compiles the output
|
||||
- retries missing packages through `tlmgr`
|
||||
- converts PDF to PNG when requested
|
||||
|
||||
Preview is not a fake HTML-like snapshot. It is a real LaTeX compile pipeline.
|
||||
|
||||
## 11. `tex_reader.py` closes the roundtrip loop
|
||||
|
||||
`tex_reader.py` is substantial, not decorative.
|
||||
|
||||
From the source it supports parsing of:
|
||||
|
||||
- `tabular`
|
||||
- `tblr`
|
||||
- `longtblr`
|
||||
- `talltblr`
|
||||
|
||||
It also handles:
|
||||
|
||||
- color parsing
|
||||
- rule parsing
|
||||
- multirow/multicolumn reconstruction
|
||||
- metadata extraction
|
||||
- grouped rows and placeholder cleanup
|
||||
|
||||
This makes `pubtab` suitable for source-aware roundtrip and migration tasks, rather than only one-way Excel export.
|
||||
|
||||
## 12. Reading order for source debugging
|
||||
|
||||
When you need source-level certainty, use this order:
|
||||
|
||||
1. `pubtab/src/pubtab/__init__.py`
|
||||
2. `pubtab/src/pubtab/models.py`
|
||||
3. `pubtab/src/pubtab/reader.py`
|
||||
4. `pubtab/src/pubtab/renderer.py`
|
||||
5. `pubtab/src/pubtab/_preview.py`
|
||||
6. `pubtab/src/pubtab/tex_reader.py`
|
||||
7. `pubtab/src/pubtab/backends/` and `themes/`
|
||||
8. `pubtab/src/pubtab/cli.py` for flag-to-API mapping only
|
||||
|
||||
## 13. Implications for this skill
|
||||
|
||||
The source says the most faithful default workflow is:
|
||||
|
||||
- use `xlsx2tex(...)` or CLI `xlsx2tex` for forward generation,
|
||||
- use `preview(...)` to verify actual compile output,
|
||||
- use `tex_to_excel(...)` for roundtrip or migration tasks,
|
||||
- explain theme/backend separately,
|
||||
- escalate into renderer/source debugging only when table structure or LaTeX behavior is the real problem.
|
||||
@@ -0,0 +1,201 @@
|
||||
# pubtab backend and preview internals (source-driven)
|
||||
|
||||
This guide focuses on the two most important deeper layers in `pubtab`:
|
||||
|
||||
- backend/theme separation
|
||||
- real preview/compile execution
|
||||
|
||||
Primary source files:
|
||||
|
||||
- `pubtab/src/pubtab/themes/__init__.py`
|
||||
- `pubtab/src/pubtab/backends/__init__.py`
|
||||
- `pubtab/src/pubtab/renderer.py`
|
||||
- `pubtab/src/pubtab/_preview.py`
|
||||
|
||||
## 1. Theme and backend are different contracts
|
||||
|
||||
The source separates them cleanly.
|
||||
|
||||
### Theme layer
|
||||
|
||||
In `themes/__init__.py`, a theme resolves to `ThemeConfig`.
|
||||
|
||||
It carries style defaults such as:
|
||||
|
||||
- `column_sep`
|
||||
- `font_size`
|
||||
- `caption_position`
|
||||
- `SpacingConfig`
|
||||
|
||||
### Backend layer
|
||||
|
||||
In `backends/__init__.py`, a backend resolves to:
|
||||
|
||||
- `BackendConfig`
|
||||
- a Jinja template string
|
||||
|
||||
`BackendConfig` carries things like:
|
||||
|
||||
- package requirements
|
||||
- preamble hints
|
||||
- backend identity
|
||||
|
||||
Operational implication:
|
||||
|
||||
- theme = stylistic defaults
|
||||
- backend = LaTeX environment/template machinery
|
||||
|
||||
## 2. Legacy normalization exists, but it is not the conceptual model
|
||||
|
||||
The source still supports legacy composite theme names like `_tabularray` suffixed themes.
|
||||
|
||||
Normalization happens in places like:
|
||||
|
||||
- `_normalize_theme_backend_choice(...)`
|
||||
- `resolve_theme(...)`
|
||||
|
||||
But that compatibility layer should not define the skill’s main explanation.
|
||||
The current conceptual model is still **separate theme + backend**.
|
||||
|
||||
## 3. What `render(...)` actually does
|
||||
|
||||
`renderer.py` is the core place where theme/backend decisions become concrete LaTeX.
|
||||
|
||||
Inside `render(...)`, the source does roughly this:
|
||||
|
||||
1. normalize theme/backend choice
|
||||
2. load theme config
|
||||
3. load backend config and template
|
||||
4. merge default/theme/user spacing
|
||||
5. compute or project column specs
|
||||
6. branch into backend-specific row/cell rendering
|
||||
7. render through Jinja
|
||||
|
||||
Backend choice changes the internal rendering algorithm, not only the final environment name.
|
||||
|
||||
## 4. `tabular` vs `tabularray` from the source perspective
|
||||
|
||||
The source suggests this practical distinction:
|
||||
|
||||
### `tabular`
|
||||
|
||||
- more classic LaTeX path
|
||||
- column spec and rules are handled in the traditional environment
|
||||
- when source `column_spec` exists, renderer tries to preserve classic rule structure
|
||||
|
||||
### `tabularray`
|
||||
|
||||
- dedicated `tblr`-style rendering path
|
||||
- special handling for promoted vertical lines and grouped header boundaries
|
||||
- extra sanitization in preview compile path
|
||||
|
||||
Interpretation from source:
|
||||
|
||||
- `tabularray` is a distinct renderer path with its own structural handling, not merely a cosmetic wrapper around `tabular`.
|
||||
|
||||
## 5. Why backend choice sometimes matters a lot
|
||||
|
||||
From `renderer.py`, backend differences affect:
|
||||
|
||||
- how colspec is interpreted
|
||||
- how vertical rules are preserved or promoted
|
||||
- how merged cells are encoded
|
||||
- how row coloring and header boundaries are emitted
|
||||
|
||||
So if a table is structurally complex, the skill should not present backend choice as arbitrary.
|
||||
|
||||
## 6. Preview is a real LaTeX toolchain
|
||||
|
||||
`_preview.py` proves preview is a full execution pipeline.
|
||||
|
||||
Major steps:
|
||||
|
||||
1. locate `pdflatex`
|
||||
2. install TinyTeX if missing
|
||||
3. build a standalone document around the table
|
||||
4. compile with `pdflatex`
|
||||
5. auto-install missing LaTeX packages when possible
|
||||
6. return PDF or convert to PNG
|
||||
|
||||
Preview is therefore a genuine verification step.
|
||||
|
||||
## 7. `pdflatex` discovery and TinyTeX fallback
|
||||
|
||||
The source checks:
|
||||
|
||||
- system `PATH`
|
||||
- pubtab-managed TinyTeX under `~/.pubtab/TinyTeX`
|
||||
|
||||
If neither exists, it installs TinyTeX automatically.
|
||||
|
||||
This is why the skill can confidently describe preview as relatively self-healing, while still warning that first-run setup may download TeX assets.
|
||||
|
||||
## 8. Missing package retry
|
||||
|
||||
A particularly important source behavior:
|
||||
|
||||
- compile logs are scanned for missing `.sty`
|
||||
- missing style names are mapped to `tlmgr` package names when needed
|
||||
- `tlmgr install <pkg>` is run automatically
|
||||
- compilation is retried
|
||||
|
||||
This is strong evidence that the recommended workflow should include preview, because preview can repair part of the environment on the way.
|
||||
|
||||
## 9. Standalone preview wrapping
|
||||
|
||||
`_build_standalone(...)` does more than wrap text in a document.
|
||||
|
||||
It also:
|
||||
|
||||
- imports backend-required packages
|
||||
- preserves setup commands outside the `resizebox` body
|
||||
- wraps the body in a preview-friendly standalone/minipage layout
|
||||
- converts `\caption{...}` into `\captionof{table}{...}` during float stripping
|
||||
|
||||
This explains why preview output can differ from naive manual compilation if the user simply pastes a table fragment into a document incorrectly.
|
||||
|
||||
## 10. `tabularray` preview sanitization
|
||||
|
||||
Before compilation, `_sanitize_tblr_for_compile(...)` removes some commands that break inside `tblr` preview contexts, including certain row-color and `\cmidrule` forms.
|
||||
|
||||
So when debugging preview-vs-final-document differences, this source behavior matters.
|
||||
|
||||
## 11. PDF to PNG conversion fallback stack
|
||||
|
||||
For PNG previews, the source tries:
|
||||
|
||||
1. `pdf2image`
|
||||
2. fallback to PyMuPDF (`fitz`)
|
||||
|
||||
Preview generation remains resilient even after PDF compilation succeeds.
|
||||
|
||||
## 12. Practical skill guidance from the source
|
||||
|
||||
### Recommend `tabular` when
|
||||
|
||||
- the table is simple/classic,
|
||||
- the user wants conservative LaTeX output,
|
||||
- compatibility matters more than modern layout features.
|
||||
|
||||
### Recommend `tabularray` when
|
||||
|
||||
- grouped headers and more complex structural layouts matter,
|
||||
- the user is already targeting a `tblr`-capable workflow,
|
||||
- the roundtrip/source table has rule structure that benefits from the dedicated renderer path.
|
||||
|
||||
### Recommend preview almost always when
|
||||
|
||||
- the table is intended for publication,
|
||||
- the user is changing backend/theme/colspec,
|
||||
- the task involves debugging table layout,
|
||||
- the task depends on compile-time package correctness.
|
||||
|
||||
## 13. Failure triage order
|
||||
|
||||
When the generated table is wrong, debug in this order:
|
||||
|
||||
1. check theme/backend choice,
|
||||
2. inspect `render(...)` inputs (`TableData`, colspec, span settings),
|
||||
3. run preview/compile,
|
||||
4. inspect missing package or backend-specific compile issues,
|
||||
5. only then move into manuscript-level integration debugging.
|
||||
@@ -0,0 +1,178 @@
|
||||
# pubtab CLI and API flow (source-driven)
|
||||
|
||||
This guide follows the actual control flow from `pubtab/src/pubtab/cli.py` into the public API and then into the internal pipeline.
|
||||
|
||||
## 1. Architectural headline
|
||||
|
||||
`cli.py` is a **thin Click wrapper** over the public API in `pubtab.__init__`.
|
||||
|
||||
Operational implication:
|
||||
|
||||
- command-line behavior should usually match Python API behavior,
|
||||
- when docs disagree, the source of truth is `pubtab.__init__`, not CLI help text alone.
|
||||
|
||||
## 2. CLI commands exposed in `cli.py`
|
||||
|
||||
The main commands are:
|
||||
|
||||
- `pubtab xlsx2tex`
|
||||
- `pubtab themes`
|
||||
- `pubtab tex2xlsx`
|
||||
- `pubtab preview`
|
||||
|
||||
There is also a hidden backward-compatible alias:
|
||||
|
||||
- `convert` -> `xlsx2tex`
|
||||
|
||||
## 3. `xlsx2tex` command flow
|
||||
|
||||
CLI entrypoint:
|
||||
|
||||
- `xlsx2tex_cmd(...)` in `cli.py`
|
||||
|
||||
Control flow:
|
||||
|
||||
1. validate input/output shape
|
||||
2. coerce `--sheet` into int when possible
|
||||
3. build kwargs only for explicitly provided options
|
||||
4. call `pubtab.xlsx2tex(input_file, output, **kwargs)`
|
||||
5. print output summary based on sheet count and preview mode
|
||||
|
||||
This thin-wrapper design matters because the CLI does **not** reimplement conversion logic.
|
||||
|
||||
## 4. `xlsx2tex(...)` API flow
|
||||
|
||||
The public API in `__init__.py` adds the real orchestration:
|
||||
|
||||
### Input modes
|
||||
|
||||
- single Excel file
|
||||
- directory of Excel files
|
||||
- single sheet
|
||||
- all sheets (`sheet=None`)
|
||||
|
||||
### Output path behavior
|
||||
|
||||
- single-sheet export can target a direct `.tex` path
|
||||
- directory input must target a directory
|
||||
- multi-sheet export uses `*_sheetNN.tex`
|
||||
|
||||
This behavior is implemented by `_build_sheet_output_paths(...)` and directory iteration helpers.
|
||||
|
||||
## 5. Config precedence in the real API
|
||||
|
||||
Inside `xlsx2tex(...)`, the source builds parameters in this order:
|
||||
|
||||
1. defaults
|
||||
2. YAML config via `load_config(...)`
|
||||
3. explicit kwargs passed from CLI or Python
|
||||
4. roundtrip-restored values where relevant
|
||||
|
||||
Operational rule:
|
||||
|
||||
- YAML config sets baseline behavior,
|
||||
- CLI flags / Python kwargs override it.
|
||||
|
||||
## 6. Sheet expansion behavior
|
||||
|
||||
When `sheet is None`, the source does not simply choose the first sheet.
|
||||
It calls `list_excel_sheets(...)` and expands all sheet names into separate outputs.
|
||||
|
||||
That is why a single workbook can generate:
|
||||
|
||||
- `table_sheet01.tex`
|
||||
- `table_sheet02.tex`
|
||||
- ...
|
||||
|
||||
The skill should explicitly mention this when users want appendix exports or workbook-wide conversion.
|
||||
|
||||
## 7. Read -> render -> write flow
|
||||
|
||||
For each selected sheet, `xlsx2tex(...)` does:
|
||||
|
||||
1. `read_excel(...)`
|
||||
2. optional header or group-separator reconstruction
|
||||
3. `render(...)`
|
||||
4. write `.tex`
|
||||
5. optional preview generation to `.png`
|
||||
|
||||
Preview is downstream of actual `.tex` generation, not an alternate renderer.
|
||||
|
||||
## 8. `preview` command flow
|
||||
|
||||
CLI entrypoint:
|
||||
|
||||
- `preview_cmd(...)`
|
||||
|
||||
The CLI again mostly validates paths and forwards to `pubtab.preview(...)`.
|
||||
|
||||
The public `preview(...)` API supports:
|
||||
|
||||
- raw LaTeX content
|
||||
- a single `.tex` file
|
||||
- a directory of `.tex` files
|
||||
- `png` or `pdf` output
|
||||
|
||||
A key source detail: when backend is omitted, `preview(...)` may infer it from the LaTeX content using `_resolve_preview_inputs(...)`.
|
||||
|
||||
## 9. Backend inference path
|
||||
|
||||
`_infer_latex_backend(...)` checks for environments like:
|
||||
|
||||
- `tblr`
|
||||
- `longtblr`
|
||||
- `talltblr`
|
||||
|
||||
If found, backend becomes `tabularray`; otherwise `tabular`.
|
||||
|
||||
Operational implication:
|
||||
|
||||
- a preview or compile call can often resolve the correct backend without requiring an explicit `--latex-backend` flag.
|
||||
|
||||
## 10. `compile_pdf(...)` API flow
|
||||
|
||||
Public `compile_pdf(...)` in `__init__.py` does:
|
||||
|
||||
1. detect whether input is raw LaTeX or a file path,
|
||||
2. infer theme/backend if needed,
|
||||
3. delegate to `_preview.compile_pdf(...)`.
|
||||
|
||||
The compile path is still part of the public API, even though the heavy lifting is in `_preview.py`.
|
||||
|
||||
## 11. `tex2xlsx` command flow
|
||||
|
||||
CLI entrypoint:
|
||||
|
||||
- `tex2xlsx(...)` in `cli.py`
|
||||
|
||||
It forwards to `pubtab.tex_to_excel(...)`.
|
||||
|
||||
The public API then handles:
|
||||
|
||||
- single `.tex` file -> one `.xlsx`
|
||||
- multi-table `.tex` -> one workbook with multiple sheets
|
||||
- directory of `.tex` files -> one `.xlsx` per file
|
||||
|
||||
This keeps the reverse path operationally symmetric with the forward path.
|
||||
|
||||
## 12. Why the CLI should stay thin in this skill
|
||||
|
||||
Because the real logic is centralized in `pubtab.__init__`, the skill should:
|
||||
|
||||
- use CLI examples for file-driven shell workflows,
|
||||
- use Python API examples for notebooks or scripted pipelines,
|
||||
- avoid duplicating pseudo-logic that already exists in the library.
|
||||
|
||||
## 13. Recommended source-faithful routing
|
||||
|
||||
### Use CLI when
|
||||
|
||||
- the user already has Excel or `.tex` files on disk,
|
||||
- the task is batch conversion,
|
||||
- the user wants a terminal-first workflow.
|
||||
|
||||
### Use Python API when
|
||||
|
||||
- the user is in a notebook or script,
|
||||
- the table needs custom preprocessing before render,
|
||||
- the agent is composing a larger Python pipeline.
|
||||
Reference in New Issue
Block a user