feat: 导出 SproutClaw .sproutclaw 配置
包含 extensions、skills、prompts、settings、auth、models、mcp 等配置。 排除 node_modules、npm 缓存、sessions 等运行时数据。
This commit is contained in:
41
agent/skills-disabled/ppt-master/examples/README.md
Normal file
41
agent/skills-disabled/ppt-master/examples/README.md
Normal file
@@ -0,0 +1,41 @@
|
||||
# PPT Master Example Projects
|
||||
|
||||
> Curated example projects generated by PPT Master.
|
||||
|
||||
**[Browse all examples online](https://hugohe3.github.io/ppt-master/)** — the canonical catalog, powered by [`examples.json`](./examples.json). The JSON file is the source of truth for project metadata, tags, and slide listings; this README only documents folder structure.
|
||||
|
||||
## Project Structure
|
||||
|
||||
Each example follows this layout:
|
||||
|
||||
```
|
||||
<project_name>/
|
||||
├── design_spec.md # Design specification (human-readable)
|
||||
├── spec_lock.md # Machine-readable execution contract
|
||||
├── images/ # Image assets
|
||||
├── notes/ # Speaker notes per page
|
||||
├── svg_output/ # Original SVGs (with placeholders)
|
||||
└── svg_final/ # Final SVGs (icons + images embedded)
|
||||
```
|
||||
|
||||
> Examples deliberately omit `sources/` (original PDFs / DOCX) and `exports/` (PPTX output) to keep repo size small. Real projects under `projects/` will have both.
|
||||
|
||||
## Preview locally
|
||||
|
||||
```bash
|
||||
python -m http.server --directory examples/<project_name>/svg_final 8000
|
||||
# visit http://localhost:8000
|
||||
```
|
||||
|
||||
Or open any `svg_final/*.svg` directly in a browser / VS Code.
|
||||
|
||||
## Contributing
|
||||
|
||||
New examples need (1) the standard structure above and (2) an entry in [`examples.json`](./examples.json) (`projects[]` array + bumped `stats`). SVGs should pass `python3 skills/ppt-master/scripts/svg_quality_checker.py`.
|
||||
|
||||
## Related
|
||||
|
||||
- [Quick Start](../README.md)
|
||||
- [Workflow](../skills/ppt-master/SKILL.md)
|
||||
- [Chart Templates](../skills/ppt-master/templates/charts/)
|
||||
- [Icon Library](../skills/ppt-master/templates/icons/)
|
||||
1831
agent/skills-disabled/ppt-master/examples/examples.json
Normal file
1831
agent/skills-disabled/ppt-master/examples/examples.json
Normal file
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,13 @@
|
||||
# attention_is_all_you_need
|
||||
|
||||
- Canvas format: ppt169
|
||||
- Created: 20260524
|
||||
- Source: Vaswani et al. 2017, Attention Is All You Need
|
||||
|
||||
## Directories
|
||||
|
||||
- `svg_output/`: raw SVG output
|
||||
- `svg_final/`: finalized SVG output
|
||||
- `images/`: presentation assets
|
||||
- `notes/`: speaker notes
|
||||
- `exports/`: generated PPTX files
|
||||
@@ -0,0 +1,164 @@
|
||||
{
|
||||
"version": 1,
|
||||
"defaults": {
|
||||
"transition": { "effect": "fade", "duration": 0.35 },
|
||||
"animation": { "effect": "auto", "duration": 0.38, "stagger": 0.16, "trigger": "after-previous" }
|
||||
},
|
||||
"slides": {
|
||||
"01_cover": {
|
||||
"transition": { "effect": "fade", "duration": 0.55 },
|
||||
"animation": { "effect": "fade", "duration": 0.6, "stagger": 0.26, "trigger": "after-previous" },
|
||||
"groups": {
|
||||
"cover-marks": { "effect": "appear", "order": 1, "duration": 0.2 },
|
||||
"cover-title": { "order": 2, "duration": 0.7 },
|
||||
"cover-subtitle": { "order": 3, "duration": 0.5 },
|
||||
"cover-takeaway": { "order": 4, "delay": 0.35, "duration": 0.55 }
|
||||
}
|
||||
},
|
||||
"02_why_it_matters": {
|
||||
"transition": { "effect": "fade", "duration": 0.3 },
|
||||
"animation": { "effect": "auto", "duration": 0.34, "stagger": 0.12, "trigger": "after-previous" },
|
||||
"groups": {
|
||||
"kpi-bleu-ende": { "effect": "zoom", "order": 1, "duration": 0.42 },
|
||||
"kpi-bleu-enfr": { "effect": "zoom", "order": 2, "duration": 0.42 },
|
||||
"kpi-training": { "effect": "zoom", "order": 3, "duration": 0.38 },
|
||||
"kpi-no-rnn-cnn": { "effect": "zoom", "order": 4, "duration": 0.38 },
|
||||
"takeaway-band": { "effect": "fade", "order": 5, "delay": 0.22, "duration": 0.5 }
|
||||
}
|
||||
},
|
||||
"03_sequence_evolution": {
|
||||
"transition": { "effect": "push", "duration": 0.42 },
|
||||
"animation": { "effect": "fade", "duration": 0.48, "stagger": 0.22, "trigger": "after-previous" },
|
||||
"groups": {
|
||||
"bottom-takeaway": { "effect": "fly", "order": 1, "delay": 0.25, "duration": 0.55 }
|
||||
}
|
||||
},
|
||||
"04_rnn_cnn_limits": {
|
||||
"transition": { "effect": "fade", "duration": 0.28 },
|
||||
"animation": { "effect": "auto", "duration": 0.34, "stagger": 0.13, "trigger": "after-previous" },
|
||||
"groups": {
|
||||
"pillar-sequential": { "effect": "fly", "order": 1 },
|
||||
"pillar-pathlen": { "effect": "fly", "order": 2 },
|
||||
"pillar-memory": { "effect": "fly", "order": 3 },
|
||||
"rnn-visual": { "effect": "wipe", "order": 4, "delay": 0.2, "duration": 0.5 }
|
||||
}
|
||||
},
|
||||
"05_architecture_overview": {
|
||||
"transition": { "effect": "wipe", "duration": 0.35 },
|
||||
"animation": { "effect": "auto", "duration": 0.38, "stagger": 0.14, "trigger": "after-previous" },
|
||||
"groups": {
|
||||
"paper-figure": { "effect": "zoom", "order": 1, "duration": 0.55 },
|
||||
"legend-1": { "effect": "fade", "order": 2 },
|
||||
"legend-2": { "effect": "fade", "order": 3 },
|
||||
"legend-3": { "effect": "fade", "order": 4 },
|
||||
"legend-4": { "effect": "fade", "order": 5 },
|
||||
"legend-5": { "effect": "fade", "order": 6 },
|
||||
"takeaway": { "effect": "fade", "order": 7, "delay": 0.22, "duration": 0.48 }
|
||||
}
|
||||
},
|
||||
"06_encoder_decoder": {
|
||||
"transition": { "effect": "fade", "duration": 0.28 },
|
||||
"animation": { "effect": "auto", "duration": 0.36, "stagger": 0.16, "trigger": "after-previous" },
|
||||
"groups": {
|
||||
"card-encoder": { "effect": "fly", "order": 1, "duration": 0.42 },
|
||||
"card-decoder": { "effect": "fly", "order": 2, "duration": 0.42 }
|
||||
}
|
||||
},
|
||||
"07_scaled_dot_product": {
|
||||
"transition": { "effect": "wipe", "duration": 0.32 },
|
||||
"animation": { "effect": "fade", "duration": 0.38, "stagger": 0.16, "trigger": "after-previous" },
|
||||
"groups": {
|
||||
"formula-band": { "effect": "zoom", "order": 1, "duration": 0.5 },
|
||||
"flow-diagram": { "effect": "wipe", "order": 2, "duration": 0.5 },
|
||||
"explanation": { "effect": "fade", "order": 3, "delay": 0.2, "duration": 0.42 }
|
||||
}
|
||||
},
|
||||
"08_multi_head_attention": {
|
||||
"transition": { "effect": "wipe", "duration": 0.32 },
|
||||
"animation": { "effect": "fade", "duration": 0.38, "stagger": 0.15, "trigger": "after-previous" },
|
||||
"groups": {
|
||||
"formula-band": { "effect": "zoom", "order": 1, "duration": 0.48 },
|
||||
"paper-figure": { "effect": "wipe", "order": 2, "duration": 0.52 },
|
||||
"explanation": { "effect": "fade", "order": 3, "delay": 0.18, "duration": 0.42 }
|
||||
}
|
||||
},
|
||||
"09_three_uses": {
|
||||
"transition": { "effect": "fade", "duration": 0.28 },
|
||||
"animation": { "effect": "auto", "duration": 0.34, "stagger": 0.14, "trigger": "after-previous" },
|
||||
"groups": {
|
||||
"pillar-enc-self": { "effect": "fly", "order": 1 },
|
||||
"pillar-dec-masked": { "effect": "fly", "order": 2 },
|
||||
"pillar-cross": { "effect": "fly", "order": 3 }
|
||||
}
|
||||
},
|
||||
"10_ffn_residual_layernorm": {
|
||||
"transition": { "effect": "fade", "duration": 0.28 },
|
||||
"animation": { "effect": "auto", "duration": 0.34, "stagger": 0.12, "trigger": "after-previous" },
|
||||
"groups": {
|
||||
"sublayer-visual": { "effect": "zoom", "order": 1, "duration": 0.48 },
|
||||
"parent-frame": { "effect": "fade", "order": 2, "duration": 0.32 },
|
||||
"card-ffn": { "effect": "fly", "order": 3 },
|
||||
"card-residual": { "effect": "fly", "order": 4 },
|
||||
"card-layernorm": { "effect": "fly", "order": 5 },
|
||||
"combined-flow": { "effect": "wipe", "order": 6, "delay": 0.18, "duration": 0.5 }
|
||||
}
|
||||
},
|
||||
"11_positional_encoding": {
|
||||
"transition": { "effect": "fade", "duration": 0.42 },
|
||||
"animation": { "effect": "fade", "duration": 0.48, "stagger": 0.2, "trigger": "after-previous" },
|
||||
"groups": {
|
||||
"title": { "order": 1, "duration": 0.55 },
|
||||
"formula-band": { "effect": "zoom", "order": 2, "duration": 0.55 },
|
||||
"explanation": { "order": 3, "delay": 0.22, "duration": 0.5 }
|
||||
}
|
||||
},
|
||||
"12_complexity_table": {
|
||||
"transition": { "effect": "fade", "duration": 0.25 },
|
||||
"animation": { "effect": "auto", "duration": 0.32, "stagger": 0.12, "trigger": "after-previous" },
|
||||
"groups": {
|
||||
"table": { "effect": "wipe", "order": 1, "duration": 0.45 },
|
||||
"takeaway": { "effect": "fade", "order": 2, "delay": 0.18, "duration": 0.42 }
|
||||
}
|
||||
},
|
||||
"13_training_setup": {
|
||||
"transition": { "effect": "fade", "duration": 0.25 },
|
||||
"animation": { "effect": "auto", "duration": 0.32, "stagger": 0.11, "trigger": "after-previous" },
|
||||
"groups": {
|
||||
"card-hardware": { "effect": "fly", "order": 1 },
|
||||
"card-time": { "effect": "fly", "order": 2 },
|
||||
"card-optimizer": { "effect": "fly", "order": 3 },
|
||||
"card-regularization": { "effect": "fly", "order": 4 }
|
||||
}
|
||||
},
|
||||
"14_results": {
|
||||
"transition": { "effect": "fade", "duration": 0.28 },
|
||||
"animation": { "effect": "auto", "duration": 0.34, "stagger": 0.14, "trigger": "after-previous" },
|
||||
"groups": {
|
||||
"table": { "effect": "wipe", "order": 1, "duration": 0.46 },
|
||||
"takeaway": { "effect": "fade", "order": 2, "delay": 0.2, "duration": 0.46 }
|
||||
}
|
||||
},
|
||||
"15_ablations": {
|
||||
"transition": { "effect": "fade", "duration": 0.28 },
|
||||
"animation": { "effect": "auto", "duration": 0.3, "stagger": 0.1, "trigger": "after-previous" },
|
||||
"groups": {
|
||||
"track": { "effect": "wipe", "order": 1, "duration": 0.4 },
|
||||
"item-1": { "effect": "fade", "order": 2 },
|
||||
"item-2": { "effect": "fade", "order": 3 },
|
||||
"item-3": { "effect": "fade", "order": 4 },
|
||||
"item-4": { "effect": "fade", "order": 5 },
|
||||
"item-5": { "effect": "fade", "order": 6 }
|
||||
}
|
||||
},
|
||||
"16_conclusion": {
|
||||
"transition": { "effect": "fade", "duration": 0.55 },
|
||||
"animation": { "effect": "fade", "duration": 0.58, "stagger": 0.24, "trigger": "after-previous" },
|
||||
"groups": {
|
||||
"closing-marks": { "effect": "appear", "order": 1, "duration": 0.2 },
|
||||
"lineage": { "effect": "zoom", "order": 2, "duration": 0.62 },
|
||||
"closing-title": { "order": 3, "delay": 0.25, "duration": 0.7 },
|
||||
"closing-caption": { "order": 4, "delay": 0.28, "duration": 0.55 }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,451 @@
|
||||
# attention_is_all_you_need — Design Spec
|
||||
|
||||
> Paper-deep-read deck for the seminal Transformer paper (Vaswani et al., 2017). Pipeline output of PPT Master; truth-of-execution lives in `spec_lock.md`.
|
||||
|
||||
## I. Project Information
|
||||
|
||||
| Item | Value |
|
||||
| ---- | ----- |
|
||||
| **Project Name** | attention_is_all_you_need |
|
||||
| **Canvas Format** | PPT 16:9 (1280×720) |
|
||||
| **Page Count** | 16 |
|
||||
| **Design Style** | B) General Consulting + academic minimalist tech (blueprint × cool-corporate) |
|
||||
| **Target Audience** | AI / NLP engineers, ML graduate students, paper-reading-group members |
|
||||
| **Use Case** | Paper deep-read / lab seminar / engineering onboarding to Transformer |
|
||||
| **Created Date** | 2026-05-24 |
|
||||
|
||||
---
|
||||
|
||||
## II. Canvas Specification
|
||||
|
||||
| Property | Value |
|
||||
| -------- | ----- |
|
||||
| **Format** | PPT 16:9 |
|
||||
| **Dimensions** | 1280×720 |
|
||||
| **viewBox** | `0 0 1280 720` |
|
||||
| **Margins** | Left/right 60px, top/bottom 50px |
|
||||
| **Content Area** | 1160×620 inside margins |
|
||||
|
||||
---
|
||||
|
||||
## III. Visual Theme
|
||||
|
||||
### Theme Style
|
||||
|
||||
- **Style**: General Consulting + academic minimalist tech (paper-precise, blueprint feel)
|
||||
- **Theme**: Light theme
|
||||
- **Tone**: Restrained, analytical, engineering-precise; reads like a Distill.pub article or Anthropic system schematic
|
||||
|
||||
### Color Scheme
|
||||
|
||||
| Role | HEX | Purpose |
|
||||
| ---- | --- | ------- |
|
||||
| **Background** | `#FFFFFF` | Page background |
|
||||
| **Secondary bg** | `#F5F7FA` | Card / region background |
|
||||
| **Primary** | `#1A365D` | Title, schematic lines, primary box stroke |
|
||||
| **Accent** | `#3182CE` | Key data, highlighted path, focus elements |
|
||||
| **Secondary accent** | `#63B3ED` | Gradient transitions, secondary emphasis |
|
||||
| **Body text** | `#1A202C` | Main body text |
|
||||
| **Secondary text** | `#4A5568` | Captions, annotations |
|
||||
| **Tertiary text** | `#A0AEC0` | Page numbers, footnotes |
|
||||
| **Border/divider** | `#E2E8F0` | Card borders, divider lines |
|
||||
| **Success** | `#2F855A` | Positive trend (BLEU improvement) |
|
||||
| **Warning** | `#C53030` | Negative trend, ablation degradation |
|
||||
|
||||
### AI Image Strategy
|
||||
|
||||
- **Image Rendering**: blueprint
|
||||
- **Image Palette**: cool-corporate
|
||||
|
||||
> Every AI image in this deck shares blueprint × cool-corporate. The rendering carries technical-schematic feel (crisp lines, near-monochrome, optional subtle grid); the palette carries restrained-corporate proportion (off-white field dominant, navy as main, accent blue under 10-15%). Cross-check: blueprint × cool-corporate is ✓✓ in the compatibility matrix.
|
||||
|
||||
### Gradient Scheme
|
||||
|
||||
```xml
|
||||
<linearGradient id="titleGradient" x1="0%" y1="0%" x2="100%" y2="100%">
|
||||
<stop offset="0%" stop-color="#1A365D"/>
|
||||
<stop offset="100%" stop-color="#3182CE"/>
|
||||
</linearGradient>
|
||||
|
||||
<linearGradient id="scrimBottom" x1="0%" y1="0%" x2="0%" y2="100%">
|
||||
<stop offset="0%" stop-color="#1A365D" stop-opacity="0"/>
|
||||
<stop offset="100%" stop-color="#1A365D" stop-opacity="0.7"/>
|
||||
</linearGradient>
|
||||
|
||||
<radialGradient id="bgDecor" cx="80%" cy="20%" r="50%">
|
||||
<stop offset="0%" stop-color="#3182CE" stop-opacity="0.15"/>
|
||||
<stop offset="100%" stop-color="#3182CE" stop-opacity="0"/>
|
||||
</radialGradient>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## IV. Typography System
|
||||
|
||||
**Typography direction**: academic serif title + cross-platform sans body — Latin-led so Georgia carries the paper-essay tone in titles while Microsoft YaHei renders any CJK glyphs without fallback.
|
||||
|
||||
| Role | Chinese | English | Fallback tail |
|
||||
| ---- | ------- | ------- | ------------- |
|
||||
| **Title** | `"Microsoft YaHei"` | `Georgia` | `serif` |
|
||||
| **Body** | `"Microsoft YaHei", "PingFang SC"` | `Arial` | `sans-serif` |
|
||||
| **Emphasis** | `"Microsoft YaHei"` | `Georgia` | `serif` |
|
||||
| **Code** | — | `Consolas, "Courier New"` | `monospace` |
|
||||
|
||||
**Per-role font stacks**:
|
||||
|
||||
- Title: `Georgia, "Microsoft YaHei", serif`
|
||||
- Body: `Arial, "Microsoft YaHei", "PingFang SC", sans-serif`
|
||||
- Emphasis: `Georgia, "Microsoft YaHei", serif`
|
||||
- Code: `Consolas, "Courier New", monospace`
|
||||
|
||||
### Font Size Hierarchy
|
||||
|
||||
**Baseline**: Body font size = **20**px (medium-dense — paper text + data + diagram annotations).
|
||||
|
||||
| Purpose | Ratio to body | This deck @ body=20 | Weight |
|
||||
| ------- | ------------- | ------------------- | ------ |
|
||||
| Cover title (hero headline) | 2.5-5x | 64px | Bold |
|
||||
| Chapter / section opener | 2-2.5x | 44px | Bold |
|
||||
| Page title | 1.5-2x | 36px | Bold |
|
||||
| Hero number (KPI) | 1.5-2x | 40px | Bold |
|
||||
| Subtitle | 1.2-1.5x | 26px | SemiBold |
|
||||
| **Body content** | **1x** | **20px** | Regular |
|
||||
| Annotation / caption | 0.7-0.85x | 14px | Regular |
|
||||
| Page number / footnote | 0.5-0.65x | 11px | Regular |
|
||||
|
||||
### Formula Rendering Policy
|
||||
|
||||
- **Policy**: `mixed` — render formula-worthy block expressions to PNG; keep inline math (`d_model=512`, `O(n)`, single Greek letters) as editable text.
|
||||
- **Rendered formulas** (3): Scaled Dot-Product Attention (P07), Multi-Head Attention (P08), Positional Encoding (P11). Manifest at `images/formula_manifest.json`; rendered via codecogs provider, transparent, color `#1A365D`.
|
||||
|
||||
---
|
||||
|
||||
## V. Layout Principles
|
||||
|
||||
### Page Structure
|
||||
|
||||
- **Header area**: top 60-100px — page title + thin divider rule in `#3182CE`
|
||||
- **Content area**: 500-560px height — flexible per page rhythm
|
||||
- **Footer area**: bottom 40px — page number + project tag in tertiary text
|
||||
|
||||
### Layout Pattern Library (combine or break as content demands)
|
||||
|
||||
| Pattern | Suitable Scenarios |
|
||||
| ------- | ----------------- |
|
||||
| **Single column centered** | P01 cover, P16 closing |
|
||||
| **Asymmetric split (3:7 / 2:8)** | P05/P07/P08 figure-dominant pages |
|
||||
| **Symmetric split (5:5)** | P06 Encoder vs Decoder |
|
||||
| **Top-bottom split** | P02 KPI hero + sub-detail; P03 hero image + caption row |
|
||||
| **Three/four column cards** | P04 limitations, P09 three attention uses, P13 training KPIs |
|
||||
| **Full-bleed + floating text** | P01, P03, P11, P16 hero pages |
|
||||
| **Figure-text overlap** | P05 annotated architecture diagram |
|
||||
| **Centered formula + caption** | P07, P08, P11 formula blocks |
|
||||
| **Negative-space-driven** | P11 positional encoding breathing page |
|
||||
|
||||
### Spacing Specification
|
||||
|
||||
**Universal**:
|
||||
|
||||
| Element | Range | This deck |
|
||||
| ------- | ----- | --------- |
|
||||
| Safe margin from canvas edge | 40-60px | 60px |
|
||||
| Content block gap | 24-40px | 28px |
|
||||
| Icon-text gap | 8-16px | 12px |
|
||||
|
||||
**Card-based** (P02, P04, P09, P13, P15):
|
||||
|
||||
| Element | Range | This deck |
|
||||
| ------- | ----- | --------- |
|
||||
| Card gap | 20-32px | 24px |
|
||||
| Card padding | 20-32px | 24px |
|
||||
| Card border radius | 8-16px | 12px |
|
||||
| Three-column card width | 360-380px | 360px |
|
||||
| Four-column card width | — | 268px |
|
||||
| Single-row card height | 530-600px | 540px |
|
||||
|
||||
---
|
||||
|
||||
## VI. Icon Usage Specification
|
||||
|
||||
### Source
|
||||
|
||||
- **Library**: `tabler-outline` (stroke-style, weight 2px, screen-clear)
|
||||
- **Stroke width**: 2 px (deck-wide lock)
|
||||
|
||||
### Recommended Icon List
|
||||
|
||||
| Purpose | Icon Path | Page |
|
||||
| ------- | --------- | ---- |
|
||||
| Trophy / SOTA result | `tabler-outline/trophy` | P02, P14 |
|
||||
| Speed / training time | `tabler-outline/bolt` | P02, P13 |
|
||||
| Layers / stack | `tabler-outline/stack-2` | P02, P06 |
|
||||
| Brain / model | `tabler-outline/brain` | P02, P16 |
|
||||
| Refresh / recurrence | `tabler-outline/refresh` | P03, P04 |
|
||||
| Arrows-right (sequence) | `tabler-outline/arrow-right` | P03, P09 |
|
||||
| Lock / constraint | `tabler-outline/lock` | P04 |
|
||||
| Hourglass / sequential bottleneck | `tabler-outline/hourglass` | P04 |
|
||||
| Box / module | `tabler-outline/box` | P06, P10 |
|
||||
| Eye / attention | `tabler-outline/eye` | P07, P09 |
|
||||
| Grid / multi-head | `tabler-outline/grid-dots` | P08 |
|
||||
| Wave / sinusoid | `tabler-outline/wave-sine` | P11 |
|
||||
| Chart bar | `tabler-outline/chart-bar` | P14, P15 |
|
||||
| GPU / hardware | `tabler-outline/cpu` | P13 |
|
||||
| Settings / hyperparams | `tabler-outline/settings` | P13, P15 |
|
||||
| Tree / family of descendants | `tabler-outline/binary-tree` | P16 |
|
||||
|
||||
---
|
||||
|
||||
## VII. Visualization Reference List
|
||||
|
||||
Catalog read: 71 templates
|
||||
|
||||
| Page | Template | Path | Summary-quote (verbatim from `charts_index.json`) | Usage |
|
||||
| ---- | -------- | ---- | ------------------------------------------------- | ----- |
|
||||
| P02 | kpi_cards | `templates/charts/kpi_cards.svg` | "Pick for 4-8 standalone numeric metrics shown as overview cards (2x2 or 1x4) — exec summary opener, dashboard headline, quarterly recap, results-at-a-glance. Skip if metrics have target baselines (use bullet_chart) or single hero number (use gauge_chart)." | Four headline metrics: 28.4 BLEU EN-DE, 41.8 BLEU EN-FR, 3.5 days on 8 GPUs, no recurrence/convolution |
|
||||
| P04 | vertical_pillars | `templates/charts/vertical_pillars.svg` | "Pick for 1×3 / 1×4 / 1×5 vertical column layout where each pillar = one independent category with title + bullets — PEST (Political/Economic/Social/Technological), four-pillar strategy overview, side-by-side independent categories. Skip for 2×2 quadrant (use quadrant_text_bullets), pricing tiers (use comparison_columns), or 2×2 parallel aspects (use labeled_card)." | Three RNN/CNN limitations side-by-side: sequential bottleneck / long-range path length / parallelization ceiling |
|
||||
| P06 | comparison_columns | `templates/charts/comparison_columns.svg` | "Pick for 2-4 pricing/service tier cards in side-by-side columns (marketing layout). Skip for dense feature comparison (use comparison_table)." | Encoder stack vs Decoder stack — sub-layers, masking, residual+LN, output dim |
|
||||
| P09 | vertical_pillars | `templates/charts/vertical_pillars.svg` | "Pick for 1×3 / 1×4 / 1×5 vertical column layout where each pillar = one independent category with title + bullets — PEST (Political/Economic/Social/Technological), four-pillar strategy overview, side-by-side independent categories. Skip for 2×2 quadrant (use quadrant_text_bullets), pricing tiers (use comparison_columns), or 2×2 parallel aspects (use labeled_card)." | Three places attention is used: encoder self-attn / decoder masked self-attn / encoder-decoder cross-attn |
|
||||
| P10 | module_composition | `templates/charts/module_composition.svg` | "Pick for one parent container wrapping 3-N child module cards, each = title + 2-3 bullets — fits 'Feature X contains 3 parts, each with its own description'. Skip if source has only labels without descriptions (use numbered_steps or icon_grid)." | Sub-layer wrap: position-wise FFN + residual connection + layer normalization, all inside one Encoder/Decoder layer block |
|
||||
| P12 | basic_table | `templates/charts/basic_table.svg` | "Pick for plain tabular text/number grid, 3-8 columns. Skip if cells need visual bars (use consulting_table) or qualitative scores (use harvey_balls_table)." | Table 1: Per-layer complexity / sequential ops / max path length across Self-Attn, Recurrent, Convolutional, Self-Attn(restricted) |
|
||||
| P13 | kpi_cards | `templates/charts/kpi_cards.svg` | "Pick for 4-8 standalone numeric metrics shown as overview cards (2x2 or 1x4) — exec summary opener, dashboard headline, quarterly recap, results-at-a-glance. Skip if metrics have target baselines (use bullet_chart) or single hero number (use gauge_chart)." | Training config dashboard: 8× P100 GPUs / 12 h base, 3.5 d big / batch 25k tokens / Adam β=(0.9, 0.98) |
|
||||
| P14 | basic_table | `templates/charts/basic_table.svg` | "Pick for plain tabular text/number grid, 3-8 columns. Skip if cells need visual bars (use consulting_table) or qualitative scores (use harvey_balls_table)." | Table 2: BLEU on WMT'14 EN-DE / EN-FR vs prior SOTA, with training-cost FLOPs column |
|
||||
| P15 | vertical_list | `templates/charts/vertical_list.svg` | "Pick for 3-6 numbered key points each with a short description — design principles, core tenets, action items, key takeaways, recommendations, executive summary points. Skip for icon-style cards (use icon_grid) or sequential steps (use numbered_steps)." | Table 3 ablation takeaways: head count sweet-spot, key/value dim, dropout, label smoothing |
|
||||
|
||||
**Runners-up considered**:
|
||||
|
||||
- `quadrant_text_bullets` | rejected for P04: limitations are three parallel side-by-side categories, no 2×2 framework
|
||||
- `comparison_table` | rejected for P12: only 4 rows + 4 columns, no checkmark / dense matrix needed — `basic_table` is the simpler fit
|
||||
- `gauge_chart` | rejected for P02: four parallel headline metrics, not one hero goal achievement
|
||||
- `numbered_steps` | rejected for P15: ablation findings are independent takeaways, not sequential
|
||||
|
||||
---
|
||||
|
||||
## VIII. Image Resource List
|
||||
|
||||
| Filename | Dimensions | Ratio | Purpose | Type | Layout pattern | Acquire Via | Status | Reference | text_policy | page_role |
|
||||
| -------- | ---------- | ----- | ------- | ---- | -------------- | ----------- | ------ | --------- | ----------- | --------- |
|
||||
| cover_bg.png | 1280×720 | 1.78 | P01 cover hero — abstract Transformer blueprint atmosphere | Background | #1 full-bleed background with floating title + #29 two-stop scrim + #38 background image + annotation cards with bezier leader lines | ai | Pending | Abstract technical-blueprint atmosphere suggesting a sequence-to-sequence neural network rebuilt around attention; six labeled rectangular module blocks arranged in two stacked rows on a near-white background, connected by thin precise lines with small arrow heads; a subtle grid pattern at 6% opacity reinforces the schematic feel; the left half is the deck's calm region reserved for a deep-navy title overlay; the right half holds the schematic; one rectangle is highlighted with accent stroke, suggesting the "self-attention" focal block; no embedded text, no labels, no equations, no numbers — every label will be overlaid as SVG text; reads as the title plate of a systems-design briefing document | none | hero_page |
|
||||
| sequence_evolution.png | 1280×720 | 1.78 | P03 hero — RNN/CNN/Self-Attention comparison as three schematic motifs, each motif carries its panel label inside the image | Diagram | #44 background image + native network/architecture diagram + #29 two-stop scrim | ai | Pending | Three side-by-side schematic motifs reading left to right as a quiet evolution: leftmost — a horizontal chain of five linked circles flowing left-to-right, each linked to the next by a single arrow, suggesting recurrent step-by-step computation; middle — a stack of three parallel horizontal bars of small circles connected by short vertical strokes, suggesting convolutional local receptive fields; rightmost — five circles arranged as a fully-connected mesh where every node connects to every other node with thin straight lines, suggesting global self-attention; all three motifs sit on a near-white field; the rightmost mesh is rendered in the accent blue while the other two are in the primary deep navy; subtle 6% grid background underneath everything; the upper third is reserved as calm scrim region for a page title in SVG; one short panel label above each motif as stable figure-internal identifier — RNN / CNN / SELF-ATTENTION — rendered in monospace technical lettering | embedded | hero_page |
|
||||
| rnn_bottleneck.png | 480×600 | 0.80 | P04 right column — sequential computation bottleneck illustration | Diagram | #3 right-third image + left text body + #21 rounded rectangle crop | ai | Pending | A vertical chain of six small rounded rectangles stacked one above the other, connected by thin arrows pointing downward; each rectangle suggests a hidden state at one time step; an hourglass-shaped marker hovers along the chain hinting at the sequential time cost; on either side of the chain, faint dashed parallel lines suggest the channels that cannot be parallelized; the background is near-white with a 6% navy grid; all linework in primary deep navy except the hourglass which is in accent blue; subtle schematic spacing between elements; no embedded text, no labels, no numbers | none | local |
|
||||
| sublayer_block.png | 480×600 | 0.80 | P10 left column — encoder/decoder sub-layer composition cross-section, conventional module labels inside the image | Diagram | #2 left-third image + right text body + #21 rounded rectangle crop | ai | Pending | A vertical schematic cross-section showing one encoder/decoder layer block: an outer rounded rectangle frame containing, top-to-bottom, two inner stacked rounded rectangles labeled visually as two sub-layers (one slightly taller, one shorter); a thin arc on the right side of each inner rectangle suggests a residual skip connection that wraps around the sub-layer and rejoins below it; a small horizontal stripe below each sub-layer suggests layer normalization; all linework in primary deep navy at uniform 1.5px stroke, accent blue used only on the two residual arcs to highlight them; near-white background with 6% navy grid; reads as a precise engineering schematic of the LayerNorm(x + Sublayer(x)) pattern; conventional architecture-paper labels inside the image — Self-Attention in the upper sub-layer, Feed Forward in the lower sub-layer, Add & Norm below each LayerNorm stripe — rendered as monospace technical lettering | embedded | local |
|
||||
| positional_encoding.png | 1280×720 | 1.78 | P11 hero — sinusoidal positional encoding visualization with textbook curve and axis labels inside the image | Diagram | #1 full-bleed background with floating title + #30 flat semi-transparent rectangle overlay + #45 background image + numbered hotspots with sidebar legend | ai | Pending | A full-width schematic showing two superimposed sinusoidal waveforms running horizontally across the canvas — one is the sine wave at low frequency (long wavelength, gentle undulation) in primary deep navy, the other is the cosine wave at higher frequency (shorter wavelength, tighter undulation) in accent blue; below the waveforms, a horizontal row of ten small evenly-spaced tick marks suggests discrete token positions on a position axis; thin vertical accent-blue guide-lines at three of the ticks drop from the waveforms down to the axis, suggesting that each token position carries a unique pair of (sin, cos) values; the left third of the canvas is reserved as a calm scrim region for a page title in SVG; near-white background with 6% navy grid; reads as a technical figure from a textbook chapter on Fourier features; three stable in-image labels rendered in monospace technical lettering — the word sin near the right end of the navy curve, cos near the right end of the accent-blue curve, position → just below the right end of the axis line; no other text, no axis numbers, no equations | embedded | hero_page |
|
||||
| transformer_family.png | 1280×720 | 1.78 | P16 hero — Transformer descendants family tree (BERT/GPT/T5/Vision Transformer abstract motifs) | Diagram | #12 faded image as backdrop with oversized overlay text + #30 flat semi-transparent rectangle overlay | ai | Pending | A symmetric top-down tree-of-descendants schematic: at the top, one single rounded rectangle (the root Transformer) in deep navy with a brighter accent-blue outline; from it, thin lines fan downward to two intermediate rounded rectangles slightly smaller (suggesting encoder-only and decoder-only families); from each intermediate, three thinner branches drop to small abstract leaf shapes of varying size (suggesting the multitude of derived models); the entire schematic is rendered at low opacity around 35% so it reads as a faded backdrop; near-white background with 6% navy grid underneath; an oversized empty calm region across the upper middle reserved for SVG closing title; reads as a quiet "what this paper started" diagram, not a busy logo wall; no embedded text, no labels, no model names — all labels overlaid as SVG text | none | hero_page |
|
||||
| attention_p3_0.png | 1520×2239 | 0.68 | P05 Transformer model architecture (Figure 1 from paper) | Diagram | #19 image floating in whitespace with thin frame and caption + #45 background image + numbered hotspots with sidebar legend | user | Existing | Original Figure 1 from the paper showing the full Encoder–Decoder Transformer architecture — Nx stacked encoder block (Multi-Head Attention → Add & Norm → Feed Forward → Add & Norm) on the left; Nx stacked decoder block (Masked Multi-Head Attention → Add & Norm → Multi-Head Attention → Add & Norm → Feed Forward → Add & Norm) on the right; positional encodings entering both stacks; final Linear + Softmax at the top right | | |
|
||||
| attention_p4_1.png | 835×1282 | 0.65 | P07 Scaled Dot-Product Attention (Figure 2 left) and P08 Multi-Head Attention (Figure 2 right) | Diagram | #46 background image + bordered "lens" rectangle highlighting a sub-region + #19 image floating in whitespace with thin frame and caption + #62 same image, two references — full view + zoom-callout | user | Existing | Original Figure 2 from the paper, two sub-figures side by side: left — Scaled Dot-Product Attention with MatMul → Scale → (Mask) → SoftMax → MatMul flow taking Q, K, V; right — Multi-Head Attention with h parallel Linear → Scaled Dot-Product Attention heads → Concat → Linear, taking V, K, Q inputs | | |
|
||||
| formula_001.png | 880×122 | 7.21 | P07 block formula — Scaled Dot-Product Attention equation | Latex Formula | formula-block | formula | Rendered | `Attention(Q, K, V) = softmax(QK^T / √d_k) V` — equation (1) from §3.2.1 | | |
|
||||
| formula_002.png | 1168×137 | 8.53 | P08 block formula — Multi-Head Attention equation | Latex Formula | formula-block | formula | Rendered | `MultiHead(Q,K,V) = Concat(head_1, …, head_h) W^O where head_i = Attention(Q W_i^Q, K W_i^K, V W_i^V)` — equations from §3.2.2 | | |
|
||||
| formula_003.png | 706×201 | 3.51 | P11 block formula — Positional Encoding equation | Latex Formula | formula-block | formula | Rendered | `PE_(pos, 2i) = sin(pos / 10000^(2i/d_model)); PE_(pos, 2i+1) = cos(pos / 10000^(2i/d_model))` — equation from §3.5 | | |
|
||||
|
||||
> Image-as-canvas coverage check: P01 uses #38, P03 uses #44, P05 / P11 use #45. Deck has ≥4 image-bearing pages and 4 of them use the #38–#46 family — coverage rule satisfied.
|
||||
|
||||
---
|
||||
|
||||
## IX. Content Outline
|
||||
|
||||
### Part 1: Opening
|
||||
|
||||
#### Slide 01 — Cover
|
||||
|
||||
- **Layout**: Full-bleed AI hero image + left-half calm region for title; #38 annotation cards with bezier leader lines + #29 two-stop scrim
|
||||
- **Title**: Attention Is All You Need
|
||||
- **Subtitle**: A paper deep-read
|
||||
- **Info**: Vaswani et al. · Google Brain / Google Research / Univ. of Toronto · NeurIPS 2017 · arXiv:1706.03762
|
||||
|
||||
#### Slide 02 — Why this paper matters
|
||||
|
||||
- **Layout**: Page title + 4-card KPI row (kpi_cards) + one-line takeaway band
|
||||
- **Title**: A simpler model, better results, less training
|
||||
- **Visualization**: kpi_cards (see §VII)
|
||||
- **Content**:
|
||||
- **28.4 BLEU** on WMT'14 EN-DE — +2.0 over previous SOTA ensembles
|
||||
- **41.8 BLEU** on WMT'14 EN-FR — new single-model SOTA
|
||||
- **3.5 days × 8 P100 GPUs** — a small fraction of prior training cost
|
||||
- **0 recurrence · 0 convolution** — only attention + feed-forward
|
||||
|
||||
### Part 2: Motivation & Background
|
||||
|
||||
#### Slide 03 — Sequence modeling so far: RNN → CNN → Self-Attention
|
||||
|
||||
- **Layout**: Hero AI image full-bleed + bottom scrim + floating title (#44 + #29)
|
||||
- **Title**: From recurrent chains to a fully-connected attention mesh
|
||||
- **Content**:
|
||||
- RNN / LSTM / GRU — strong, but inherently sequential
|
||||
- ByteNet / ConvS2S — parallelizable, but signals between distant positions still grow with distance
|
||||
- Self-attention — relates any two positions in constant operations
|
||||
|
||||
#### Slide 04 — The fundamental constraint we want to lift
|
||||
|
||||
- **Layout**: Right-third AI image (#3 + #21) + left vertical_pillars
|
||||
- **Title**: What hurts in RNN/CNN sequence models
|
||||
- **Visualization**: vertical_pillars
|
||||
- **Content**:
|
||||
- **Sequential bottleneck** — h_t depends on h_{t-1}; no within-example parallelism
|
||||
- **Long-range path length** — ConvS2S grows linearly, ByteNet logarithmically; distant tokens hard to relate
|
||||
- **Memory ceiling** — sequence length limits batch size, especially at long context
|
||||
|
||||
### Part 3: The Transformer
|
||||
|
||||
#### Slide 05 — Model architecture overview
|
||||
|
||||
- **Layout**: User-provided Figure 1 image (portrait, scaled to fit) on left + right-side numbered hotspot legend (#19 + #45)
|
||||
- **Title**: Encoder–Decoder, stacked self-attention, point-wise feed-forward
|
||||
- **Content** (sidebar legend):
|
||||
1. Input embeddings + positional encoding
|
||||
2. Nx encoder block — Multi-Head Self-Attention + FFN
|
||||
3. Nx decoder block — Masked Self-Attn + Encoder–Decoder Attn + FFN
|
||||
4. Add & Norm — residual then LayerNorm around every sub-layer
|
||||
5. Final Linear + Softmax — output token probabilities
|
||||
|
||||
#### Slide 06 — Encoder vs Decoder stacks
|
||||
|
||||
- **Layout**: comparison_columns — two parallel cards
|
||||
- **Title**: Same skeleton, three asymmetries
|
||||
- **Visualization**: comparison_columns
|
||||
- **Content**:
|
||||
- **Encoder (N=6)** — Multi-Head Self-Attention → FFN; LayerNorm(x + Sublayer(x)); d_model = 512
|
||||
- **Decoder (N=6)** — Masked Multi-Head Self-Attention → Encoder-Decoder Multi-Head Attention → FFN; same residual + LN; output offset by one position; positions attend only to earlier outputs
|
||||
|
||||
### Part 4: Attention mechanism
|
||||
|
||||
#### Slide 07 — Scaled Dot-Product Attention
|
||||
|
||||
- **Layout**: User-provided Figure 2 cropped to left half (#46 lens crop) on right + left text body + centered formula band
|
||||
- **Title**: Attention as a weighted average — keys, values, queries
|
||||
- **Visualization**: Formula block (formula_001.png)
|
||||
- **Content**:
|
||||
- Inputs: queries and keys of dim d_k, values of dim d_v
|
||||
- Compute QK^T, scale by √d_k, softmax → attention weights
|
||||
- Multiply by V — weighted sum is the output
|
||||
- Scaling matters: without √d_k, large d_k pushes softmax into low-gradient regions
|
||||
|
||||
#### Slide 08 — Multi-Head Attention
|
||||
|
||||
- **Layout**: User-provided Figure 2 cropped to right half (#46 lens crop) on right + left text body + centered formula band
|
||||
- **Title**: h parallel projections, then concatenate
|
||||
- **Visualization**: Formula block (formula_002.png)
|
||||
- **Content**:
|
||||
- Project Q, K, V into h = 8 independent subspaces via learned linear projections
|
||||
- Run scaled dot-product attention in each head in parallel; d_k = d_v = d_model / h = 64
|
||||
- Concatenate the h outputs and project once more with W^O
|
||||
- Why: attend to information from different representation subspaces at different positions
|
||||
|
||||
#### Slide 09 — Three places attention is used
|
||||
|
||||
- **Layout**: vertical_pillars — three independent columns
|
||||
- **Title**: One mechanism, three roles in the Transformer
|
||||
- **Visualization**: vertical_pillars
|
||||
- **Content**:
|
||||
- **Encoder self-attention** — Q, K, V all from the previous encoder layer; every position attends to all positions
|
||||
- **Decoder masked self-attention** — same as above but masked: position i attends only to positions ≤ i (preserve auto-regression)
|
||||
- **Encoder–decoder attention** — queries from the decoder, keys & values from the encoder output; lets every decoder position attend over the whole input sequence (the classic seq2seq attention role)
|
||||
|
||||
#### Slide 10 — Feed-Forward, Residual, LayerNorm
|
||||
|
||||
- **Layout**: Left-third AI image (#2 + #21) + right module_composition
|
||||
- **Title**: The supporting cast inside every layer
|
||||
- **Visualization**: module_composition
|
||||
- **Content**:
|
||||
- **Position-wise FFN** — two linear layers with ReLU between, applied identically at every position; inner dim 2048
|
||||
- **Residual connection** — every sub-layer output is x + Sublayer(x), enabling deeper stacks
|
||||
- **LayerNorm** — applied around each sub-layer; stabilizes training, paired with d_model = 512
|
||||
|
||||
### Part 5: Position information
|
||||
|
||||
#### Slide 11 — Positional Encoding
|
||||
|
||||
- **Layout**: AI hero image full-bleed (#1 + #30 + #45) + centered formula band + minimal SVG annotation
|
||||
- **Title**: Recovering order without recurrence — sinusoids of different frequencies
|
||||
- **Visualization**: Formula block (formula_003.png)
|
||||
- **Content**:
|
||||
- No recurrence / convolution → model has no built-in notion of token order
|
||||
- Add a deterministic sinusoidal position vector to each input embedding
|
||||
- sin / cos at geometrically increasing wavelengths from 2π to 10000·2π
|
||||
- Lets the model learn to attend by relative position via linear function of PE
|
||||
|
||||
### Part 6: Why and how well it works
|
||||
|
||||
#### Slide 12 — Complexity comparison (Table 1)
|
||||
|
||||
- **Layout**: Page title + basic_table (full-width)
|
||||
- **Title**: Complexity per layer, sequential operations, max path length
|
||||
- **Visualization**: basic_table
|
||||
- **Content**:
|
||||
- Self-Attention: O(n²·d), 1 sequential op, max path length O(1)
|
||||
- Recurrent: O(n·d²), O(n) sequential ops, max path length O(n)
|
||||
- Convolutional: O(k·n·d²), 1, max path length O(log_k n)
|
||||
- Self-Attention (restricted, window r): O(r·n·d), 1, max path length O(n/r)
|
||||
|
||||
#### Slide 13 — Training setup
|
||||
|
||||
- **Layout**: Page title + kpi_cards (2×2 or 1×4)
|
||||
- **Title**: How they trained it
|
||||
- **Visualization**: kpi_cards
|
||||
- **Content**:
|
||||
- **Hardware** — 8× NVIDIA P100 GPUs
|
||||
- **Time** — Base 12 h (100k steps) · Big 3.5 d (300k steps)
|
||||
- **Optimizer** — Adam, β₁=0.9, β₂=0.98, ε=1e-9; warm-up then 1/√step decay
|
||||
- **Regularization** — Dropout 0.1, label smoothing ε_ls = 0.1
|
||||
|
||||
#### Slide 14 — Results (Table 2)
|
||||
|
||||
- **Layout**: Page title + basic_table (full-width) + bottom takeaway band
|
||||
- **Title**: New SOTA on WMT'14 — at a fraction of the training cost
|
||||
- **Visualization**: basic_table
|
||||
- **Content**:
|
||||
- Transformer (big): EN-DE 28.4 BLEU · EN-FR 41.8 BLEU
|
||||
- vs prior best ensembles: +2.0 EN-DE; new single-model SOTA EN-FR
|
||||
- Training FLOPs (big): ~2.3 × 10¹⁹ — well below most strong baselines
|
||||
|
||||
#### Slide 15 — What matters in the architecture (Table 3 ablations)
|
||||
|
||||
- **Layout**: vertical_list (5 numbered key points)
|
||||
- **Title**: Five takeaways from the variation study
|
||||
- **Visualization**: vertical_list
|
||||
- **Content**:
|
||||
- **Head count has a sweet spot** — single-head loses 0.9 BLEU; too many (32) also degrade
|
||||
- **Key dim matters more than value dim** — reducing d_k hurts quality more than reducing d_v
|
||||
- **Bigger is better** — larger d_model and deeper stacks consistently improve BLEU
|
||||
- **Dropout is essential** — removing it loses ~0.5–1.0 BLEU; even small dropout (0.1) matters
|
||||
- **Label smoothing helps perplexity & BLEU** — slightly hurts perplexity but improves BLEU
|
||||
|
||||
### Part 7: Closing
|
||||
|
||||
#### Slide 16 — What this paper started
|
||||
|
||||
- **Layout**: AI hero image full-bleed faded (#12 + #30) + centered closing title + small caption row
|
||||
- **Title**: One paper, a generation of models
|
||||
- **Content**:
|
||||
- Transformer became the substrate for BERT, GPT, T5, Vision Transformer, AlphaFold, and most modern LLMs
|
||||
- The 2017 idea — **just attention** — turned out to be the right inductive bias for almost every modality
|
||||
- References: arXiv:1706.03762, code at github.com/tensorflow/tensor2tensor
|
||||
|
||||
---
|
||||
|
||||
## X. Speaker Notes Requirements
|
||||
|
||||
- **Filename**: one file per slide under `notes/`, matching SVG basename (e.g., `01_cover.svg` → `notes/01_cover.md`)
|
||||
- **Style**: conversational-academic (paper-reading-group register — clear, technical, no marketing copy)
|
||||
- **Total duration**: ~20 minutes presented (≈ 75 sec per slide on average; deeper slides P05/P07/P08/P11 ≈ 100 sec each)
|
||||
- **Purpose**: instruct / inform — explain the paper to an engineer who has not read it carefully
|
||||
- **Format**: master document at `notes/total.md` with `#` headings per slide; split files have no `#` heading
|
||||
|
||||
---
|
||||
|
||||
## XI. Technical Constraints Reminder
|
||||
|
||||
### SVG Generation Must Follow:
|
||||
|
||||
1. viewBox: `0 0 1280 720`
|
||||
2. Background uses `<rect>` elements
|
||||
3. Text wrapping uses `<tspan>` (`<foreignObject>` FORBIDDEN)
|
||||
4. Transparency uses `fill-opacity` / `stroke-opacity`; `rgba()` FORBIDDEN
|
||||
5. FORBIDDEN: `mask`, `<style>`, `class`, `foreignObject`, `textPath`, `animate*`, `script`
|
||||
6. Text characters: raw Unicode (`—`, `→`, NBSP, `²`, `√`, `≤`, `·`); HTML named entities FORBIDDEN; reserved chars escaped as `& < > " '`
|
||||
7. `marker-start` / `marker-end` conditionally allowed: `<marker>` in `<defs>`, `orient="auto"`, triangle / diamond / circle shape only
|
||||
8. `clipPath` only on `<image>` elements: `<clipPath>` in `<defs>`, single shape child
|
||||
|
||||
### PPT Compatibility Rules:
|
||||
|
||||
- `<g opacity="...">` FORBIDDEN — set opacity on each child element individually
|
||||
- Image transparency uses overlay mask layer (`<rect fill="bg-color" opacity="0.x"/>`)
|
||||
- Inline styles only; external CSS and `@font-face` FORBIDDEN
|
||||
Binary file not shown.
@@ -0,0 +1,3 @@
|
||||
No,Filename,Width,Height,AspectRatio,PixelAspectRatio,RatioSource,UsageCount,DisplayRatioVariants,AssetKind,SvgRenderable,PptxNativeSupported,SizeKB,Category,ImageArea_SxS,TextArea_SxS
|
||||
1,attention_p3_0.png,1520,2239,0.68,0.68,pixel,1,,bitmap,True,True,118.4,Portrait,407x600,733x600
|
||||
2,attention_p4_1.png,835,1282,0.65,0.65,pixel,1,,bitmap,True,True,42.5,Portrait,391x600,749x600
|
||||
|
Binary file not shown.
|
After Width: | Height: | Size: 218 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 127 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 677 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 12 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 19 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 18 KiB |
@@ -0,0 +1,89 @@
|
||||
{
|
||||
"providers": [
|
||||
"codecogs",
|
||||
"quicklatex",
|
||||
"mathpad",
|
||||
"wikimedia"
|
||||
],
|
||||
"items": [
|
||||
{
|
||||
"id": "formula_001",
|
||||
"latex": "\\text{Attention}(Q, K, V) = \\text{softmax}\\!\\left(\\frac{QK^{T}}{\\sqrt{d_k}}\\right) V",
|
||||
"display": "block",
|
||||
"color": "#1A365D",
|
||||
"background": "#FFFFFF",
|
||||
"transparent": true,
|
||||
"dpi": 400,
|
||||
"filename": "formula_001.png",
|
||||
"file": "images/formula_001.png",
|
||||
"providers": [
|
||||
"codecogs",
|
||||
"quicklatex",
|
||||
"mathpad",
|
||||
"wikimedia"
|
||||
],
|
||||
"transparent_tolerance": 12,
|
||||
"provider": "codecogs",
|
||||
"pixel_width": 880,
|
||||
"pixel_height": 122,
|
||||
"ratio": 7.2131,
|
||||
"status": "Rendered"
|
||||
},
|
||||
{
|
||||
"id": "formula_002",
|
||||
"latex": "\\begin{aligned}\\text{MultiHead}(Q, K, V) &= \\text{Concat}(\\text{head}_1, \\ldots, \\text{head}_h)\\,W^{O} \\\\ \\text{where}\\ \\text{head}_i &= \\text{Attention}(Q W_i^{Q},\\, K W_i^{K},\\, V W_i^{V})\\end{aligned}",
|
||||
"display": "block",
|
||||
"color": "#1A365D",
|
||||
"background": "#FFFFFF",
|
||||
"transparent": true,
|
||||
"dpi": 400,
|
||||
"filename": "formula_002.png",
|
||||
"file": "images/formula_002.png",
|
||||
"providers": [
|
||||
"codecogs",
|
||||
"quicklatex",
|
||||
"mathpad",
|
||||
"wikimedia"
|
||||
],
|
||||
"transparent_tolerance": 12,
|
||||
"provider": "codecogs",
|
||||
"pixel_width": 1168,
|
||||
"pixel_height": 137,
|
||||
"ratio": 8.5255,
|
||||
"status": "Rendered"
|
||||
},
|
||||
{
|
||||
"id": "formula_003",
|
||||
"latex": "\\begin{aligned}PE_{(pos,\\, 2i)} &= \\sin\\!\\left(\\frac{pos}{10000^{2i/d_{\\text{model}}}}\\right) \\\\ PE_{(pos,\\, 2i+1)} &= \\cos\\!\\left(\\frac{pos}{10000^{2i/d_{\\text{model}}}}\\right)\\end{aligned}",
|
||||
"display": "block",
|
||||
"color": "#1A365D",
|
||||
"background": "#FFFFFF",
|
||||
"transparent": true,
|
||||
"dpi": 400,
|
||||
"filename": "formula_003.png",
|
||||
"file": "images/formula_003.png",
|
||||
"providers": [
|
||||
"codecogs",
|
||||
"quicklatex",
|
||||
"mathpad",
|
||||
"wikimedia"
|
||||
],
|
||||
"transparent_tolerance": 12,
|
||||
"provider": "codecogs",
|
||||
"pixel_width": 706,
|
||||
"pixel_height": 201,
|
||||
"ratio": 3.5124,
|
||||
"status": "Rendered"
|
||||
}
|
||||
],
|
||||
"renderer": {
|
||||
"providers": [
|
||||
"codecogs",
|
||||
"quicklatex",
|
||||
"mathpad",
|
||||
"wikimedia"
|
||||
],
|
||||
"default_dpi": 300,
|
||||
"output_dir": "images"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,81 @@
|
||||
{
|
||||
"project": "attention_is_all_you_need",
|
||||
"generated_at": "2026-05-24",
|
||||
"deck_rendering": "blueprint",
|
||||
"deck_palette": "cool-corporate",
|
||||
"color_scheme": {
|
||||
"primary": "#1A365D",
|
||||
"secondary": "#F5F7FA",
|
||||
"accent": "#3182CE"
|
||||
},
|
||||
"items": [
|
||||
{
|
||||
"filename": "cover_bg.png",
|
||||
"purpose": "Cover background (Slide 01) — hero atmospheric blueprint of a generic Transformer-like module graph, calm left side reserved for SVG title overlay",
|
||||
"page_role": "hero_page",
|
||||
"text_policy": "none",
|
||||
"aspect_ratio": "16:9",
|
||||
"image_size": "2K",
|
||||
"prompt": "Technical blueprint schematic style. Clean precise lines on a very subtle implied grid with deliberate geometric rigor — right angles, parallel rules, measured spacing. Elements are simplified to essential schematic forms — rounded rectangular module blocks, connector lines, anchor dots, callout markers. Color is restrained, near-monochrome with one semantic accent. A subtle grid background at 6% opacity reinforces the schematic feel. No textures, no shading, no painterly artifacts; engineering-precise, analytical, intentional. Color behavior is restrained-corporate: secondary off-white #F5F7FA carries about 65% of the canvas as calm breathing space; primary deep navy #1A365D forms the precise schematic linework (about 28% area) — outlines of module blocks, connector lines, anchor dots; accent blue #3182CE appears only as one highlighted module outline and a couple of arrowheads on the focal connector path, together under 7% of the area; no fourth color. Atmospheric blueprint backdrop with no dominant subject — a graph of six rounded rectangular module blocks arranged in two stacked horizontal rows of three (an encoder-stack row at the bottom and a decoder-stack row at the top), connected by precise straight lines with small triangular arrow heads. The blocks are deliberately small and clustered on the right side of the canvas, leaving the left third entirely calm as breathing space — that calm region must stay clear of any schematic element. One block on the upper row is highlighted by replacing its primary navy outline with an accent-blue outline, marking it as the focal self-attention module. Faint vertical and horizontal hairlines suggest a measurement grid only at 6% opacity beneath everything. Composed as a 1280×720 PPT 16:9 full-bleed cover background. Reads as the title plate of a systems-design briefing document. NO text, letters, numbers, signs, watermarks, or written symbols anywhere in the image. Color values (HEX codes like #1A365D) and color names are rendering guidance only — do NOT display HEX codes, color names, or palette labels as visible text anywhere in the image.",
|
||||
"alt_text": "Abstract Transformer-like module graph rendered in technical blueprint style, calm left third reserved for title",
|
||||
"status": "Generated"
|
||||
},
|
||||
{
|
||||
"filename": "sequence_evolution.png",
|
||||
"purpose": "Slide 03 hero — three schematic motifs side by side suggesting RNN chain / convolutional local field / global self-attention mesh, each motif carries its panel label inside the image",
|
||||
"page_role": "hero_page",
|
||||
"text_policy": "embedded",
|
||||
"aspect_ratio": "16:9",
|
||||
"image_size": "2K",
|
||||
"prompt": "Technical blueprint schematic style. Clean precise lines on a very subtle implied grid. Elements are simplified to essential schematic forms — small filled circles for nodes, thin straight lines for connections, small triangular arrow heads on directed edges. Crisp uniform stroke around 1.5px, perfectly straight or precisely curved. No textures, no shading, no painterly artifacts. Color behavior is restrained-corporate: secondary off-white #F5F7FA carries about 62% of the canvas as calm breathing space; primary deep navy #1A365D carries the bulk of the linework (about 28% area, including the leftmost and middle motifs); accent blue #3182CE appears only in the rightmost motif (under 10% area), marking it as the focus. A subtle 6% opacity navy grid sits beneath everything. Custom composition — three equal-width schematic motifs sit side by side across the lower two-thirds of the canvas, separated by 2px hairline rules in border-gray; the upper third of the canvas is deliberately calm as breathing space for a page title overlay in SVG. Leftmost motif: a horizontal chain of five small filled circles flowing left to right, connected by single arrows between adjacent circles, suggesting recurrent step-by-step computation. Middle motif: three short parallel horizontal rows of four small circles each, each circle connected to the two circles diagonally below it in the next row by short straight lines, suggesting convolutional local receptive fields. Rightmost motif: five circles arranged in a regular pentagon, every circle connected to every other circle by thin straight lines forming a fully-connected mesh, suggesting global self-attention — this entire mesh is rendered in accent blue #3182CE while the first two motifs stay in primary navy. Above each of the three motifs sits one short panel label, rendered as clean monospace technical lettering (fixed-width, sharp uppercase, no serifs), small enough to read as a discreet figure-panel header — about 28-32px high relative to the canvas. The exact label text, left to right, is: above the leftmost motif the word RNN; above the middle motif the word CNN; above the rightmost motif the words SELF-ATTENTION. Labels above the first two motifs are in primary deep navy #1A365D; the SELF-ATTENTION label above the rightmost motif is in accent blue #3182CE matching its mesh. The three panel labels are the ONLY text in the image; no other letters, no axis labels, no captions, no signs, no watermarks anywhere else. Composed as a 1280×720 PPT 16:9 hero background; the upper third (above the panel labels) must remain calm for SVG title overlay. Color values (HEX codes like #1A365D) and color names are rendering guidance only — do NOT display HEX codes, color names, or palette labels as visible text anywhere in the image.",
|
||||
"alt_text": "Three labeled schematic motifs (RNN / CNN / SELF-ATTENTION) comparing recurrent chain, convolutional local fields, and global self-attention mesh",
|
||||
"status": "Generated"
|
||||
},
|
||||
{
|
||||
"filename": "rnn_bottleneck.png",
|
||||
"purpose": "Slide 04 right-third illustration — sequential RNN computation bottleneck (vertical chain of hidden states with an hourglass cue)",
|
||||
"type": "flowchart",
|
||||
"page_role": "local",
|
||||
"text_policy": "none",
|
||||
"aspect_ratio": "3:4",
|
||||
"image_size": "1K",
|
||||
"prompt": "Technical blueprint schematic style. Clean precise lines on a very subtle implied grid, with deliberate geometric rigor — right angles, parallel rules, measured spacing. Elements are simplified to schematic forms — small rounded rectangles, thin straight arrows, an hourglass marker. Crisp uniform stroke around 1.5px. No textures, no shading, no painterly artifacts; engineering-precise, restrained. Color behavior is restrained-corporate: secondary off-white #F5F7FA fills about 60% of the block as background breathing; primary deep navy #1A365D forms the chain of hidden-state boxes and the connecting arrows (about 32% area); accent blue #3182CE is used only for the hourglass marker and one small focus highlight (under 8% area); no fourth color. A subtle 6% opacity navy grid sits beneath everything. Flowchart-style local composition: a vertical chain of six small identical rounded rectangles stacked one above the other, each connected to the next by a thin downward-pointing arrow with a small triangular arrow head; each rectangle represents a hidden state at one time step. To the right of the chain, a small hourglass marker in accent blue floats alongside the middle of the chain, suggesting sequential time cost. To the left of the chain, two faint dashed parallel vertical lines hint at the channels that cannot be parallelized — they are dashed, in primary navy at 50% opacity. The entire schematic is centered inside the block with about 14% inner padding on all sides. Composed as a 480×600 vertical local block for placement inside an SVG page region; do NOT reserve calm overlay space inside the image — the SVG layer surrounds this block, not on top of it. NO text, letters, numbers, signs, watermarks, or written symbols anywhere in the image. Color values (HEX codes like #1A365D) and color names are rendering guidance only — do NOT display HEX codes, color names, or palette labels as visible text anywhere in the image.",
|
||||
"alt_text": "Vertical chain of six hidden-state boxes with downward arrows and a small hourglass marker, in blueprint schematic style",
|
||||
"status": "Generated"
|
||||
},
|
||||
{
|
||||
"filename": "sublayer_block.png",
|
||||
"purpose": "Slide 10 left-third illustration — encoder/decoder sub-layer composition: outer frame, two inner sub-layers, residual arcs, LayerNorm stripes; modules carry conventional paper-figure labels",
|
||||
"type": "framework",
|
||||
"page_role": "local",
|
||||
"text_policy": "embedded",
|
||||
"aspect_ratio": "3:4",
|
||||
"image_size": "1K",
|
||||
"prompt": "Technical blueprint schematic style. Clean precise lines on a very subtle implied grid, with deliberate geometric rigor — right angles, parallel rules, measured spacing. Elements are simplified to schematic forms — rounded rectangles, thin arcs, small horizontal stripes. Crisp uniform stroke around 1.5px. No textures, no shading, no painterly artifacts; engineering-precise. Color behavior is restrained-corporate: secondary off-white #F5F7FA fills about 60% of the block as background breathing space; primary deep navy #1A365D draws the outer frame, the inner sub-layer rectangles, and the stripes (about 30% area); accent blue #3182CE is used only on the two residual arcs to highlight them (under 8% area); no fourth color. A subtle 6% opacity navy grid sits beneath everything. Framework-style local composition: a single large outer rounded rectangle frame represents one encoder/decoder layer block. Inside that frame, top-to-bottom, two inner stacked rounded rectangles — the upper one slightly taller, the lower one slightly shorter; they are separated by a small gap. Around each inner sub-layer, a thin curved arc in accent blue starts above the sub-layer, sweeps around its right side, and rejoins below it — these two arcs are the residual skip connections, the only accent-blue elements in the image. Below each inner sub-layer, a thin short horizontal stripe in primary navy suggests layer normalization. Each inner sub-layer rectangle carries one short module label rendered as clean monospace technical lettering (fixed-width, sharp letterforms, no serifs), centered inside the rectangle and sized to read as a discreet figure label — about 18-22px relative to the block. The exact label inside the upper (taller) sub-layer is the words Self-Attention; the exact label inside the lower (shorter) sub-layer is the words Feed Forward. Below each LayerNorm stripe sits one tiny annotation in the same monospace lettering, smaller (about 12-14px), in primary navy, reading exactly Add & Norm. These four short labels (Self-Attention, Feed Forward, Add & Norm, Add & Norm) are the ONLY text in the image — no other letters, no numbers, no signs, no watermarks anywhere else. The entire schematic is centered inside the block with about 14% inner padding on all sides. Composed as a 480×600 vertical local block for placement inside an SVG page region; do NOT reserve calm overlay space inside the image — the SVG layer surrounds this block, not on top of it. Color values (HEX codes like #1A365D) and color names are rendering guidance only — do NOT display HEX codes, color names, or palette labels as visible text anywhere in the image.",
|
||||
"alt_text": "Encoder layer frame with two stacked labeled sub-layers (Self-Attention, Feed Forward), accent residual arcs, and Add & Norm stripes",
|
||||
"status": "Generated"
|
||||
},
|
||||
{
|
||||
"filename": "positional_encoding.png",
|
||||
"purpose": "Slide 11 hero — sinusoidal positional encoding visualization (two superimposed waveforms at different frequencies plus a discrete position axis below); textbook curve labels and axis label inside the image",
|
||||
"page_role": "hero_page",
|
||||
"text_policy": "embedded",
|
||||
"aspect_ratio": "16:9",
|
||||
"image_size": "2K",
|
||||
"prompt": "Technical blueprint schematic style. Clean precise lines on a very subtle implied grid. Curves are precisely drawn smooth sine and cosine waveforms with uniform stroke around 1.5px. No textures, no shading, no painterly artifacts; engineering-precise, reads as a technical figure from a textbook chapter on Fourier features. Color behavior is restrained-corporate: secondary off-white #F5F7FA carries about 58% of the canvas as breathing field; primary deep navy #1A365D draws the slower (longer wavelength, gentle undulation) sine curve and the tick marks (about 30% area); accent blue #3182CE draws the faster (shorter wavelength, tighter undulation) cosine curve and three thin vertical guide-lines that drop from the waveforms down to the position axis (under 12% area); no fourth color. A subtle 6% opacity navy grid sits beneath everything. Custom composition — the left third of the canvas is deliberately calm as breathing space reserved for a page title overlay in SVG. Across the right two-thirds of the canvas, two superimposed sinusoidal waveforms run horizontally: the primary-navy sine curve has a long wavelength spanning roughly two full cycles across the available width with gentle undulation; the accent-blue cosine curve runs at higher frequency completing roughly five full cycles across the same width with tighter undulation; the two curves visibly cross several times. Below the waveforms, a horizontal row of ten small evenly-spaced tick marks in primary navy marks discrete token positions on a position axis represented by a thin horizontal line. Three thin vertical accent-blue guide lines drop from selected points on the waveforms straight down to corresponding ticks on the position axis, suggesting that each token position carries a unique pair of sine and cosine values. Curve labels and one axis label are rendered inside the image as clean monospace technical lettering (fixed-width, sharp letterforms, no serifs, about 22-26px relative to canvas). The exact text is, in three places only: near the right end of the navy sine curve, the lowercase word sin in primary navy #1A365D; near the right end of the accent-blue cosine curve, the lowercase word cos in accent blue #3182CE; just below the right end of the position-axis line, the lowercase phrase position → in primary navy. These three short labels (sin, cos, position →) are the ONLY text in the image — no other letters, no numbers, no axis tick numbers, no signs, no watermarks anywhere else. Composed as a 1280×720 PPT 16:9 hero background; the left third must remain calm for SVG title overlay. Color values (HEX codes like #1A365D) and color names are rendering guidance only — do NOT display HEX codes, color names, or palette labels as visible text anywhere in the image.",
|
||||
"alt_text": "Sine and cosine sinusoidal waveforms with sin/cos curve labels and a position arrow axis label, blueprint schematic",
|
||||
"status": "Generated"
|
||||
},
|
||||
{
|
||||
"filename": "transformer_family.png",
|
||||
"purpose": "Slide 16 hero — faded Transformer descendants tree backdrop (root + two intermediate branches + many small leaf shapes); calm middle band reserved for SVG closing title",
|
||||
"page_role": "hero_page",
|
||||
"text_policy": "none",
|
||||
"aspect_ratio": "16:9",
|
||||
"image_size": "2K",
|
||||
"prompt": "Technical blueprint schematic style. Clean precise lines on a very subtle implied grid. Elements are simplified to schematic forms — rounded rectangles and small abstract leaf shapes connected by thin straight lines. Crisp uniform stroke around 1.5px. No textures, no shading, no painterly artifacts; engineering-precise. The entire schematic is rendered at low opacity around 35% so it reads as a faded backdrop. Color behavior is restrained-corporate: secondary off-white #F5F7FA dominates the canvas as breathing space (about 68% area); primary deep navy #1A365D draws the root rectangle, the two intermediate rectangles, the descendant leaf shapes, and all connector lines (about 27% area, all rendered at ~35% opacity to fade); accent blue #3182CE outlines only the root rectangle and one selected leaf in full opacity (under 5% area), marking the genealogical anchor and one focal descendant; no fourth color. A subtle 6% opacity navy grid sits beneath everything. Custom composition — a symmetric top-down tree-of-descendants schematic occupies the upper third and lower third of the canvas with the middle horizontal band of the canvas (roughly 30% of canvas height across the full width) deliberately calm as breathing space reserved for a closing title overlay in SVG. Tree structure: at the top center, one single rounded rectangle slightly larger than the rest (the root Transformer) outlined in accent blue at full opacity. From the root, thin lines fan downward across the calm middle band and emerge in the lower third, branching to two intermediate rounded rectangles set symmetrically left and right (suggesting encoder-only and decoder-only families). From each intermediate, three to four thinner branches drop further down to small abstract leaf shapes of varying small sizes (suggesting the multitude of derived models). All elements other than the root and one selected leaf are rendered in primary navy at 35% opacity. Reads as a quiet 'what this paper started' diagram. Composed as a 1280×720 PPT 16:9 hero background; the middle horizontal band must remain calm for SVG closing title overlay. NO text, letters, numbers, model names, signs, watermarks, or written symbols anywhere in the image. Color values (HEX codes like #1A365D) and color names are rendering guidance only — do NOT display HEX codes, color names, or palette labels as visible text anywhere in the image.",
|
||||
"alt_text": "Faded blueprint tree of Transformer descendants with accent-blue root and one accent leaf, calm middle band reserved",
|
||||
"status": "Generated"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,120 @@
|
||||
# Image Generation Prompts
|
||||
|
||||
> Auto-generated from `image_prompts.json` by `image_gen.py --render-md`.
|
||||
> Do not hand-edit — re-run the command to refresh.
|
||||
|
||||
> Project: attention_is_all_you_need
|
||||
> Generated: 2026-05-24
|
||||
> Color scheme: Primary #1A365D | Secondary #F5F7FA | Accent #3182CE
|
||||
|
||||
---
|
||||
|
||||
### Image 1: cover_bg.png
|
||||
|
||||
| Attribute | Value |
|
||||
|---|---|
|
||||
| Purpose | Cover background (Slide 01) — hero atmospheric blueprint of a generic Transformer-like module graph, calm left side reserved for SVG title overlay |
|
||||
| Aspect ratio | 16:9 |
|
||||
| Image size | 2K |
|
||||
| Status | Generated |
|
||||
|
||||
**Prompt**:
|
||||
|
||||
Technical blueprint schematic style. Clean precise lines on a very subtle implied grid with deliberate geometric rigor — right angles, parallel rules, measured spacing. Elements are simplified to essential schematic forms — rounded rectangular module blocks, connector lines, anchor dots, callout markers. Color is restrained, near-monochrome with one semantic accent. A subtle grid background at 6% opacity reinforces the schematic feel. No textures, no shading, no painterly artifacts; engineering-precise, analytical, intentional. Color behavior is restrained-corporate: secondary off-white #F5F7FA carries about 65% of the canvas as calm breathing space; primary deep navy #1A365D forms the precise schematic linework (about 28% area) — outlines of module blocks, connector lines, anchor dots; accent blue #3182CE appears only as one highlighted module outline and a couple of arrowheads on the focal connector path, together under 7% of the area; no fourth color. Atmospheric blueprint backdrop with no dominant subject — a graph of six rounded rectangular module blocks arranged in two stacked horizontal rows of three (an encoder-stack row at the bottom and a decoder-stack row at the top), connected by precise straight lines with small triangular arrow heads. The blocks are deliberately small and clustered on the right side of the canvas, leaving the left third entirely calm as breathing space — that calm region must stay clear of any schematic element. One block on the upper row is highlighted by replacing its primary navy outline with an accent-blue outline, marking it as the focal self-attention module. Faint vertical and horizontal hairlines suggest a measurement grid only at 6% opacity beneath everything. Composed as a 1280×720 PPT 16:9 full-bleed cover background. Reads as the title plate of a systems-design briefing document. NO text, letters, numbers, signs, watermarks, or written symbols anywhere in the image. Color values (HEX codes like #1A365D) and color names are rendering guidance only — do NOT display HEX codes, color names, or palette labels as visible text anywhere in the image.
|
||||
|
||||
**Alt Text**:
|
||||
> Abstract Transformer-like module graph rendered in technical blueprint style, calm left third reserved for title
|
||||
|
||||
---
|
||||
|
||||
### Image 2: sequence_evolution.png
|
||||
|
||||
| Attribute | Value |
|
||||
|---|---|
|
||||
| Purpose | Slide 03 hero — three schematic motifs side by side suggesting RNN chain / convolutional local field / global self-attention mesh, each motif carries its panel label inside the image |
|
||||
| Aspect ratio | 16:9 |
|
||||
| Image size | 2K |
|
||||
| Status | Generated |
|
||||
|
||||
**Prompt**:
|
||||
|
||||
Technical blueprint schematic style. Clean precise lines on a very subtle implied grid. Elements are simplified to essential schematic forms — small filled circles for nodes, thin straight lines for connections, small triangular arrow heads on directed edges. Crisp uniform stroke around 1.5px, perfectly straight or precisely curved. No textures, no shading, no painterly artifacts. Color behavior is restrained-corporate: secondary off-white #F5F7FA carries about 62% of the canvas as calm breathing space; primary deep navy #1A365D carries the bulk of the linework (about 28% area, including the leftmost and middle motifs); accent blue #3182CE appears only in the rightmost motif (under 10% area), marking it as the focus. A subtle 6% opacity navy grid sits beneath everything. Custom composition — three equal-width schematic motifs sit side by side across the lower two-thirds of the canvas, separated by 2px hairline rules in border-gray; the upper third of the canvas is deliberately calm as breathing space for a page title overlay in SVG. Leftmost motif: a horizontal chain of five small filled circles flowing left to right, connected by single arrows between adjacent circles, suggesting recurrent step-by-step computation. Middle motif: three short parallel horizontal rows of four small circles each, each circle connected to the two circles diagonally below it in the next row by short straight lines, suggesting convolutional local receptive fields. Rightmost motif: five circles arranged in a regular pentagon, every circle connected to every other circle by thin straight lines forming a fully-connected mesh, suggesting global self-attention — this entire mesh is rendered in accent blue #3182CE while the first two motifs stay in primary navy. Above each of the three motifs sits one short panel label, rendered as clean monospace technical lettering (fixed-width, sharp uppercase, no serifs), small enough to read as a discreet figure-panel header — about 28-32px high relative to the canvas. The exact label text, left to right, is: above the leftmost motif the word RNN; above the middle motif the word CNN; above the rightmost motif the words SELF-ATTENTION. Labels above the first two motifs are in primary deep navy #1A365D; the SELF-ATTENTION label above the rightmost motif is in accent blue #3182CE matching its mesh. The three panel labels are the ONLY text in the image; no other letters, no axis labels, no captions, no signs, no watermarks anywhere else. Composed as a 1280×720 PPT 16:9 hero background; the upper third (above the panel labels) must remain calm for SVG title overlay. Color values (HEX codes like #1A365D) and color names are rendering guidance only — do NOT display HEX codes, color names, or palette labels as visible text anywhere in the image.
|
||||
|
||||
**Alt Text**:
|
||||
> Three labeled schematic motifs (RNN / CNN / SELF-ATTENTION) comparing recurrent chain, convolutional local fields, and global self-attention mesh
|
||||
|
||||
---
|
||||
|
||||
### Image 3: rnn_bottleneck.png
|
||||
|
||||
| Attribute | Value |
|
||||
|---|---|
|
||||
| Purpose | Slide 04 right-third illustration — sequential RNN computation bottleneck (vertical chain of hidden states with an hourglass cue) |
|
||||
| Type | flowchart |
|
||||
| Aspect ratio | 3:4 |
|
||||
| Image size | 1K |
|
||||
| Status | Generated |
|
||||
|
||||
**Prompt**:
|
||||
|
||||
Technical blueprint schematic style. Clean precise lines on a very subtle implied grid, with deliberate geometric rigor — right angles, parallel rules, measured spacing. Elements are simplified to schematic forms — small rounded rectangles, thin straight arrows, an hourglass marker. Crisp uniform stroke around 1.5px. No textures, no shading, no painterly artifacts; engineering-precise, restrained. Color behavior is restrained-corporate: secondary off-white #F5F7FA fills about 60% of the block as background breathing; primary deep navy #1A365D forms the chain of hidden-state boxes and the connecting arrows (about 32% area); accent blue #3182CE is used only for the hourglass marker and one small focus highlight (under 8% area); no fourth color. A subtle 6% opacity navy grid sits beneath everything. Flowchart-style local composition: a vertical chain of six small identical rounded rectangles stacked one above the other, each connected to the next by a thin downward-pointing arrow with a small triangular arrow head; each rectangle represents a hidden state at one time step. To the right of the chain, a small hourglass marker in accent blue floats alongside the middle of the chain, suggesting sequential time cost. To the left of the chain, two faint dashed parallel vertical lines hint at the channels that cannot be parallelized — they are dashed, in primary navy at 50% opacity. The entire schematic is centered inside the block with about 14% inner padding on all sides. Composed as a 480×600 vertical local block for placement inside an SVG page region; do NOT reserve calm overlay space inside the image — the SVG layer surrounds this block, not on top of it. NO text, letters, numbers, signs, watermarks, or written symbols anywhere in the image. Color values (HEX codes like #1A365D) and color names are rendering guidance only — do NOT display HEX codes, color names, or palette labels as visible text anywhere in the image.
|
||||
|
||||
**Alt Text**:
|
||||
> Vertical chain of six hidden-state boxes with downward arrows and a small hourglass marker, in blueprint schematic style
|
||||
|
||||
---
|
||||
|
||||
### Image 4: sublayer_block.png
|
||||
|
||||
| Attribute | Value |
|
||||
|---|---|
|
||||
| Purpose | Slide 10 left-third illustration — encoder/decoder sub-layer composition: outer frame, two inner sub-layers, residual arcs, LayerNorm stripes; modules carry conventional paper-figure labels |
|
||||
| Type | framework |
|
||||
| Aspect ratio | 3:4 |
|
||||
| Image size | 1K |
|
||||
| Status | Generated |
|
||||
|
||||
**Prompt**:
|
||||
|
||||
Technical blueprint schematic style. Clean precise lines on a very subtle implied grid, with deliberate geometric rigor — right angles, parallel rules, measured spacing. Elements are simplified to schematic forms — rounded rectangles, thin arcs, small horizontal stripes. Crisp uniform stroke around 1.5px. No textures, no shading, no painterly artifacts; engineering-precise. Color behavior is restrained-corporate: secondary off-white #F5F7FA fills about 60% of the block as background breathing space; primary deep navy #1A365D draws the outer frame, the inner sub-layer rectangles, and the stripes (about 30% area); accent blue #3182CE is used only on the two residual arcs to highlight them (under 8% area); no fourth color. A subtle 6% opacity navy grid sits beneath everything. Framework-style local composition: a single large outer rounded rectangle frame represents one encoder/decoder layer block. Inside that frame, top-to-bottom, two inner stacked rounded rectangles — the upper one slightly taller, the lower one slightly shorter; they are separated by a small gap. Around each inner sub-layer, a thin curved arc in accent blue starts above the sub-layer, sweeps around its right side, and rejoins below it — these two arcs are the residual skip connections, the only accent-blue elements in the image. Below each inner sub-layer, a thin short horizontal stripe in primary navy suggests layer normalization. Each inner sub-layer rectangle carries one short module label rendered as clean monospace technical lettering (fixed-width, sharp letterforms, no serifs), centered inside the rectangle and sized to read as a discreet figure label — about 18-22px relative to the block. The exact label inside the upper (taller) sub-layer is the words Self-Attention; the exact label inside the lower (shorter) sub-layer is the words Feed Forward. Below each LayerNorm stripe sits one tiny annotation in the same monospace lettering, smaller (about 12-14px), in primary navy, reading exactly Add & Norm. These four short labels (Self-Attention, Feed Forward, Add & Norm, Add & Norm) are the ONLY text in the image — no other letters, no numbers, no signs, no watermarks anywhere else. The entire schematic is centered inside the block with about 14% inner padding on all sides. Composed as a 480×600 vertical local block for placement inside an SVG page region; do NOT reserve calm overlay space inside the image — the SVG layer surrounds this block, not on top of it. Color values (HEX codes like #1A365D) and color names are rendering guidance only — do NOT display HEX codes, color names, or palette labels as visible text anywhere in the image.
|
||||
|
||||
**Alt Text**:
|
||||
> Encoder layer frame with two stacked labeled sub-layers (Self-Attention, Feed Forward), accent residual arcs, and Add & Norm stripes
|
||||
|
||||
---
|
||||
|
||||
### Image 5: positional_encoding.png
|
||||
|
||||
| Attribute | Value |
|
||||
|---|---|
|
||||
| Purpose | Slide 11 hero — sinusoidal positional encoding visualization (two superimposed waveforms at different frequencies plus a discrete position axis below); textbook curve labels and axis label inside the image |
|
||||
| Aspect ratio | 16:9 |
|
||||
| Image size | 2K |
|
||||
| Status | Generated |
|
||||
|
||||
**Prompt**:
|
||||
|
||||
Technical blueprint schematic style. Clean precise lines on a very subtle implied grid. Curves are precisely drawn smooth sine and cosine waveforms with uniform stroke around 1.5px. No textures, no shading, no painterly artifacts; engineering-precise, reads as a technical figure from a textbook chapter on Fourier features. Color behavior is restrained-corporate: secondary off-white #F5F7FA carries about 58% of the canvas as breathing field; primary deep navy #1A365D draws the slower (longer wavelength, gentle undulation) sine curve and the tick marks (about 30% area); accent blue #3182CE draws the faster (shorter wavelength, tighter undulation) cosine curve and three thin vertical guide-lines that drop from the waveforms down to the position axis (under 12% area); no fourth color. A subtle 6% opacity navy grid sits beneath everything. Custom composition — the left third of the canvas is deliberately calm as breathing space reserved for a page title overlay in SVG. Across the right two-thirds of the canvas, two superimposed sinusoidal waveforms run horizontally: the primary-navy sine curve has a long wavelength spanning roughly two full cycles across the available width with gentle undulation; the accent-blue cosine curve runs at higher frequency completing roughly five full cycles across the same width with tighter undulation; the two curves visibly cross several times. Below the waveforms, a horizontal row of ten small evenly-spaced tick marks in primary navy marks discrete token positions on a position axis represented by a thin horizontal line. Three thin vertical accent-blue guide lines drop from selected points on the waveforms straight down to corresponding ticks on the position axis, suggesting that each token position carries a unique pair of sine and cosine values. Curve labels and one axis label are rendered inside the image as clean monospace technical lettering (fixed-width, sharp letterforms, no serifs, about 22-26px relative to canvas). The exact text is, in three places only: near the right end of the navy sine curve, the lowercase word sin in primary navy #1A365D; near the right end of the accent-blue cosine curve, the lowercase word cos in accent blue #3182CE; just below the right end of the position-axis line, the lowercase phrase position → in primary navy. These three short labels (sin, cos, position →) are the ONLY text in the image — no other letters, no numbers, no axis tick numbers, no signs, no watermarks anywhere else. Composed as a 1280×720 PPT 16:9 hero background; the left third must remain calm for SVG title overlay. Color values (HEX codes like #1A365D) and color names are rendering guidance only — do NOT display HEX codes, color names, or palette labels as visible text anywhere in the image.
|
||||
|
||||
**Alt Text**:
|
||||
> Sine and cosine sinusoidal waveforms with sin/cos curve labels and a position arrow axis label, blueprint schematic
|
||||
|
||||
---
|
||||
|
||||
### Image 6: transformer_family.png
|
||||
|
||||
| Attribute | Value |
|
||||
|---|---|
|
||||
| Purpose | Slide 16 hero — faded Transformer descendants tree backdrop (root + two intermediate branches + many small leaf shapes); calm middle band reserved for SVG closing title |
|
||||
| Aspect ratio | 16:9 |
|
||||
| Image size | 2K |
|
||||
| Status | Generated |
|
||||
|
||||
**Prompt**:
|
||||
|
||||
Technical blueprint schematic style. Clean precise lines on a very subtle implied grid. Elements are simplified to schematic forms — rounded rectangles and small abstract leaf shapes connected by thin straight lines. Crisp uniform stroke around 1.5px. No textures, no shading, no painterly artifacts; engineering-precise. The entire schematic is rendered at low opacity around 35% so it reads as a faded backdrop. Color behavior is restrained-corporate: secondary off-white #F5F7FA dominates the canvas as breathing space (about 68% area); primary deep navy #1A365D draws the root rectangle, the two intermediate rectangles, the descendant leaf shapes, and all connector lines (about 27% area, all rendered at ~35% opacity to fade); accent blue #3182CE outlines only the root rectangle and one selected leaf in full opacity (under 5% area), marking the genealogical anchor and one focal descendant; no fourth color. A subtle 6% opacity navy grid sits beneath everything. Custom composition — a symmetric top-down tree-of-descendants schematic occupies the upper third and lower third of the canvas with the middle horizontal band of the canvas (roughly 30% of canvas height across the full width) deliberately calm as breathing space reserved for a closing title overlay in SVG. Tree structure: at the top center, one single rounded rectangle slightly larger than the rest (the root Transformer) outlined in accent blue at full opacity. From the root, thin lines fan downward across the calm middle band and emerge in the lower third, branching to two intermediate rounded rectangles set symmetrically left and right (suggesting encoder-only and decoder-only families). From each intermediate, three to four thinner branches drop further down to small abstract leaf shapes of varying small sizes (suggesting the multitude of derived models). All elements other than the root and one selected leaf are rendered in primary navy at 35% opacity. Reads as a quiet 'what this paper started' diagram. Composed as a 1280×720 PPT 16:9 hero background; the middle horizontal band must remain calm for SVG closing title overlay. NO text, letters, numbers, model names, signs, watermarks, or written symbols anywhere in the image. Color values (HEX codes like #1A365D) and color names are rendering guidance only — do NOT display HEX codes, color names, or palette labels as visible text anywhere in the image.
|
||||
|
||||
**Alt Text**:
|
||||
> Faded blueprint tree of Transformer descendants with accent-blue root and one accent leaf, calm middle band reserved
|
||||
|
||||
---
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 751 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 721 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 579 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 633 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 561 KiB |
@@ -0,0 +1 @@
|
||||
Today we are reading what is probably the single most influential machine learning paper of the last decade: Attention Is All You Need, by Vaswani and seven coauthors at Google Brain, Google Research, and the University of Toronto, presented at NeurIPS 2017. The title is the thesis: drop recurrence and convolution, use only attention, and you get a model that is faster to train, easier to parallelize, and stronger on machine translation than anything that came before. Over the next sixteen slides we'll walk through what the Transformer actually is, how the attention mechanism works, why the authors chose this design, and what the experiments showed.
|
||||
@@ -0,0 +1 @@
|
||||
Before we get into the architecture, let's see what the paper is actually claiming. On the WMT 2014 English to German translation task, the Transformer big model scored 28.4 BLEU, beating the previous best ensemble by two full points as a single model. On WMT 2014 English to French, it set a new single-model state of the art at 41.8 BLEU. And it did all this by training for three and a half days on eight P100 GPUs — a small fraction of the cost of the strong baselines. The architecture has no recurrence and no convolution; it is built entirely from attention and feed-forward layers. Quality up, cost down, parallelization unlocked.
|
||||
@@ -0,0 +1 @@
|
||||
To appreciate the move, it helps to look at where sequence modeling was sitting in 2017. Recurrent networks — RNNs, LSTMs, GRUs — were the default, but they compute step by step and resist parallelization. Convolutional alternatives like ByteNet and ConvS2S could parallelize, but signals between distant positions still had to travel through many layers, growing linearly or logarithmically with distance. Self-attention does something different: every position relates to every other position in one constant-time hop. Visually you can think of it as moving from a chain to a local field to a fully connected mesh — and that mesh is what the Transformer leans into.
|
||||
@@ -0,0 +1 @@
|
||||
Let's name the constraints the authors are trying to lift. First, the sequential bottleneck: in any RNN, hidden state at time t depends on hidden state at t minus one, so you cannot parallelize within a single training example. Second, long-range path length: distant tokens have to be related through many intermediate operations, which makes learning long-range dependencies hard. Third, memory ceiling: long sequences eat GPU memory, capping batch size and throughput. The Transformer attacks all three at once by replacing the entire backbone with attention.
|
||||
@@ -0,0 +1 @@
|
||||
This is figure one from the paper — the Transformer architecture. Read it from the bottom up. Inputs and outputs both come in as embeddings, with a sinusoidal positional encoding added. The encoder on the left is a stack of six identical layers; each layer has multi-head self-attention followed by a position-wise feed-forward network, with Add and Norm wrapping each sub-layer. The decoder on the right is also six layers, but inserts a third sub-layer that attends back to the encoder output, and its self-attention is masked so it cannot peek at future tokens. The whole thing ends with a linear projection and softmax to produce token probabilities. The skeleton is the same as prior seq2seq models — what is new is what fills the boxes.
|
||||
@@ -0,0 +1 @@
|
||||
Let's compare the encoder and decoder side by side. The encoder has two sub-layers per block: multi-head self-attention, then a position-wise feed-forward network. The decoder has three: masked multi-head self-attention, then encoder-decoder attention, then the feed-forward network. Around every sub-layer the same wrapper applies — output equals LayerNorm of x plus Sublayer of x, the classic residual connection plus normalization. All sub-layers and embeddings emit a 512-dimensional vector so the residual addition is well-defined. The three asymmetries on the decoder side are the third sub-layer, the causal mask that preserves auto-regression, and the one-position shift of the output embeddings.
|
||||
@@ -0,0 +1 @@
|
||||
Here is the heart of the model — scaled dot-product attention. You start with three matrices: queries Q, keys K, and values V. You compute the dot product of every query with every key, scale by the square root of the key dimension, apply a row-wise softmax to get attention weights, and then multiply by V to produce the output. The scaling factor matters. For large key dimensions the dot products grow in magnitude, pushing the softmax into regions with vanishingly small gradients. Dividing by root d sub k counteracts that. Once you have the weights, the output is just a weighted average of the value vectors — that's all attention is.
|
||||
@@ -0,0 +1 @@
|
||||
Multi-head attention is the next move. Instead of running one attention function over the full 512-dimensional vectors, the authors split into h equals eight independent heads, each operating on 64-dimensional projections of Q, K, and V. The projections are learned, so each head can specialize on a different kind of relationship. The heads run in parallel, their outputs are concatenated, and a final linear projection maps back to 512. The total compute is roughly the same as a single full-dimensional head, but the model gains the ability to attend to different representation subspaces at different positions simultaneously.
|
||||
@@ -0,0 +1 @@
|
||||
The same attention mechanism shows up in three places in the Transformer, with three slightly different roles. In encoder self-attention, queries, keys, and values all come from the previous encoder layer; every source position attends to every other. In decoder masked self-attention, the same setup applies, but illegal future positions are set to minus infinity before the softmax, so position i attends only to positions less than or equal to i — that's how auto-regression is preserved. In encoder-decoder attention, queries come from the decoder while keys and values come from the encoder output, letting every decoder position retrieve relevant information from anywhere in the source sequence. One mechanism, three roles.
|
||||
@@ -0,0 +1 @@
|
||||
The supporting cast inside every layer is just three small ideas. The position-wise feed-forward network is two linear layers with a ReLU between, applied identically at every position, with an inner dimension of 2048. The residual connection adds the sub-layer's input back to its output, which keeps the gradient signal flowing through deep stacks. Layer normalization then standardizes across the feature dimension. Put them together and the output of every sub-layer is LayerNorm of x plus Sublayer of x. Nothing here is novel by itself — but composing them around attention is what makes the architecture trainable at depth.
|
||||
@@ -0,0 +1 @@
|
||||
Because there is no recurrence and no convolution, the model has no built-in notion of token order. The authors recover order with a deterministic sinusoidal positional encoding added to each input embedding. Even dimensions use a sine wave, odd dimensions use a cosine wave, and the wavelengths form a geometric progression from two pi up to ten thousand times two pi. The neat property is that for any fixed offset k, the encoding at position pos plus k is a linear function of the encoding at pos — so the model can easily learn to attend by relative position via a simple linear combination. Order is injected, not learned.
|
||||
@@ -0,0 +1 @@
|
||||
This is table one from the paper, and it is the strongest argument for self-attention. A self-attention layer pays O of n squared times d in compute, but in exchange every pair of positions is exactly one operation apart. A recurrent layer pays O of n times d squared, but it takes O of n sequential operations and the maximum path length between two positions is O of n. Convolutional layers fall in between. For typical NLP workloads where the sequence length is shorter than the representation dimension, self-attention is actually faster than recurrence — and the constant path length makes long-range dependencies far easier to learn.
|
||||
@@ -0,0 +1 @@
|
||||
A quick word on how this was trained. The base model trains in twelve hours on eight P100 GPUs in one machine; the big model trains in three and a half days on the same hardware. Batches were grouped by similar sequence length to keep about twenty-five thousand source and twenty-five thousand target tokens per batch. The optimizer is Adam with beta two pushed up to 0.98, and a custom learning rate schedule that warms up linearly for four thousand steps then decays as one over the square root of the step count. Regularization is dropout at 0.1 applied to every sub-layer output and to the sum of embedding and positional encoding, plus label smoothing of 0.1.
|
||||
@@ -0,0 +1 @@
|
||||
Here are the headline results — table two from the paper. The Transformer big scores 28.4 BLEU on English to German, beating the previous best ensemble by two full BLEU points as a single model. On English to French it scores 41.8, a new single-model state of the art. And it does all this with about 2.3 times 10 to the 19 training floating-point operations, which is well below most strong baselines. The base model alone, trained in twelve hours, already beats almost every prior published model. The combination of better quality and lower training cost is what made this paper land so hard.
|
||||
@@ -0,0 +1 @@
|
||||
The variation study in table three gives us five takeaways worth remembering. First, head count has a sweet spot: a single head loses about 0.9 BLEU compared to eight heads, but going to thirty-two also degrades. Second, the key dimension matters more than the value dimension — determining compatibility is the harder learning problem. Third, scaling up the model dimension and the depth consistently improves quality, so the architecture scales monotonically with capacity. Fourth, dropout is essential — removing it loses roughly half a BLEU to a full BLEU point even at p equals 0.1. Fifth, label smoothing slightly hurts perplexity but improves BLEU, which is a reminder to optimize for the metric you actually ship on.
|
||||
@@ -0,0 +1 @@
|
||||
To close — this 2017 paper turned out to be one of the most consequential architectural ideas in modern machine learning. The Transformer became the substrate for BERT, for GPT, for T5, for the Vision Transformer, for AlphaFold, for CLIP, for almost every large language model in production today, including the one talking to you right now. The core insight — that just attention, with no recurrence and no convolution, is the right inductive bias for sequence modeling — turned out to generalize far beyond translation, to almost every modality we work with. One paper, a generation of models.
|
||||
@@ -0,0 +1,93 @@
|
||||
# 01_cover
|
||||
|
||||
Today we are reading what is probably the single most influential machine learning paper of the last decade: Attention Is All You Need, by Vaswani and seven coauthors at Google Brain, Google Research, and the University of Toronto, presented at NeurIPS 2017. The title is the thesis: drop recurrence and convolution, use only attention, and you get a model that is faster to train, easier to parallelize, and stronger on machine translation than anything that came before. Over the next sixteen slides we'll walk through what the Transformer actually is, how the attention mechanism works, why the authors chose this design, and what the experiments showed.
|
||||
|
||||
---
|
||||
|
||||
# 02_why_it_matters
|
||||
|
||||
Before we get into the architecture, let's see what the paper is actually claiming. On the WMT 2014 English to German translation task, the Transformer big model scored 28.4 BLEU, beating the previous best ensemble by two full points as a single model. On WMT 2014 English to French, it set a new single-model state of the art at 41.8 BLEU. And it did all this by training for three and a half days on eight P100 GPUs — a small fraction of the cost of the strong baselines. The architecture has no recurrence and no convolution; it is built entirely from attention and feed-forward layers. Quality up, cost down, parallelization unlocked.
|
||||
|
||||
---
|
||||
|
||||
# 03_sequence_evolution
|
||||
|
||||
To appreciate the move, it helps to look at where sequence modeling was sitting in 2017. Recurrent networks — RNNs, LSTMs, GRUs — were the default, but they compute step by step and resist parallelization. Convolutional alternatives like ByteNet and ConvS2S could parallelize, but signals between distant positions still had to travel through many layers, growing linearly or logarithmically with distance. Self-attention does something different: every position relates to every other position in one constant-time hop. Visually you can think of it as moving from a chain to a local field to a fully connected mesh — and that mesh is what the Transformer leans into.
|
||||
|
||||
---
|
||||
|
||||
# 04_rnn_cnn_limits
|
||||
|
||||
Let's name the constraints the authors are trying to lift. First, the sequential bottleneck: in any RNN, hidden state at time t depends on hidden state at t minus one, so you cannot parallelize within a single training example. Second, long-range path length: distant tokens have to be related through many intermediate operations, which makes learning long-range dependencies hard. Third, memory ceiling: long sequences eat GPU memory, capping batch size and throughput. The Transformer attacks all three at once by replacing the entire backbone with attention.
|
||||
|
||||
---
|
||||
|
||||
# 05_architecture_overview
|
||||
|
||||
This is figure one from the paper — the Transformer architecture. Read it from the bottom up. Inputs and outputs both come in as embeddings, with a sinusoidal positional encoding added. The encoder on the left is a stack of six identical layers; each layer has multi-head self-attention followed by a position-wise feed-forward network, with Add and Norm wrapping each sub-layer. The decoder on the right is also six layers, but inserts a third sub-layer that attends back to the encoder output, and its self-attention is masked so it cannot peek at future tokens. The whole thing ends with a linear projection and softmax to produce token probabilities. The skeleton is the same as prior seq2seq models — what is new is what fills the boxes.
|
||||
|
||||
---
|
||||
|
||||
# 06_encoder_decoder
|
||||
|
||||
Let's compare the encoder and decoder side by side. The encoder has two sub-layers per block: multi-head self-attention, then a position-wise feed-forward network. The decoder has three: masked multi-head self-attention, then encoder-decoder attention, then the feed-forward network. Around every sub-layer the same wrapper applies — output equals LayerNorm of x plus Sublayer of x, the classic residual connection plus normalization. All sub-layers and embeddings emit a 512-dimensional vector so the residual addition is well-defined. The three asymmetries on the decoder side are the third sub-layer, the causal mask that preserves auto-regression, and the one-position shift of the output embeddings.
|
||||
|
||||
---
|
||||
|
||||
# 07_scaled_dot_product
|
||||
|
||||
Here is the heart of the model — scaled dot-product attention. You start with three matrices: queries Q, keys K, and values V. You compute the dot product of every query with every key, scale by the square root of the key dimension, apply a row-wise softmax to get attention weights, and then multiply by V to produce the output. The scaling factor matters. For large key dimensions the dot products grow in magnitude, pushing the softmax into regions with vanishingly small gradients. Dividing by root d sub k counteracts that. Once you have the weights, the output is just a weighted average of the value vectors — that's all attention is.
|
||||
|
||||
---
|
||||
|
||||
# 08_multi_head_attention
|
||||
|
||||
Multi-head attention is the next move. Instead of running one attention function over the full 512-dimensional vectors, the authors split into h equals eight independent heads, each operating on 64-dimensional projections of Q, K, and V. The projections are learned, so each head can specialize on a different kind of relationship. The heads run in parallel, their outputs are concatenated, and a final linear projection maps back to 512. The total compute is roughly the same as a single full-dimensional head, but the model gains the ability to attend to different representation subspaces at different positions simultaneously.
|
||||
|
||||
---
|
||||
|
||||
# 09_three_uses
|
||||
|
||||
The same attention mechanism shows up in three places in the Transformer, with three slightly different roles. In encoder self-attention, queries, keys, and values all come from the previous encoder layer; every source position attends to every other. In decoder masked self-attention, the same setup applies, but illegal future positions are set to minus infinity before the softmax, so position i attends only to positions less than or equal to i — that's how auto-regression is preserved. In encoder-decoder attention, queries come from the decoder while keys and values come from the encoder output, letting every decoder position retrieve relevant information from anywhere in the source sequence. One mechanism, three roles.
|
||||
|
||||
---
|
||||
|
||||
# 10_ffn_residual_layernorm
|
||||
|
||||
The supporting cast inside every layer is just three small ideas. The position-wise feed-forward network is two linear layers with a ReLU between, applied identically at every position, with an inner dimension of 2048. The residual connection adds the sub-layer's input back to its output, which keeps the gradient signal flowing through deep stacks. Layer normalization then standardizes across the feature dimension. Put them together and the output of every sub-layer is LayerNorm of x plus Sublayer of x. Nothing here is novel by itself — but composing them around attention is what makes the architecture trainable at depth.
|
||||
|
||||
---
|
||||
|
||||
# 11_positional_encoding
|
||||
|
||||
Because there is no recurrence and no convolution, the model has no built-in notion of token order. The authors recover order with a deterministic sinusoidal positional encoding added to each input embedding. Even dimensions use a sine wave, odd dimensions use a cosine wave, and the wavelengths form a geometric progression from two pi up to ten thousand times two pi. The neat property is that for any fixed offset k, the encoding at position pos plus k is a linear function of the encoding at pos — so the model can easily learn to attend by relative position via a simple linear combination. Order is injected, not learned.
|
||||
|
||||
---
|
||||
|
||||
# 12_complexity_table
|
||||
|
||||
This is table one from the paper, and it is the strongest argument for self-attention. A self-attention layer pays O of n squared times d in compute, but in exchange every pair of positions is exactly one operation apart. A recurrent layer pays O of n times d squared, but it takes O of n sequential operations and the maximum path length between two positions is O of n. Convolutional layers fall in between. For typical NLP workloads where the sequence length is shorter than the representation dimension, self-attention is actually faster than recurrence — and the constant path length makes long-range dependencies far easier to learn.
|
||||
|
||||
---
|
||||
|
||||
# 13_training_setup
|
||||
|
||||
A quick word on how this was trained. The base model trains in twelve hours on eight P100 GPUs in one machine; the big model trains in three and a half days on the same hardware. Batches were grouped by similar sequence length to keep about twenty-five thousand source and twenty-five thousand target tokens per batch. The optimizer is Adam with beta two pushed up to 0.98, and a custom learning rate schedule that warms up linearly for four thousand steps then decays as one over the square root of the step count. Regularization is dropout at 0.1 applied to every sub-layer output and to the sum of embedding and positional encoding, plus label smoothing of 0.1.
|
||||
|
||||
---
|
||||
|
||||
# 14_results
|
||||
|
||||
Here are the headline results — table two from the paper. The Transformer big scores 28.4 BLEU on English to German, beating the previous best ensemble by two full BLEU points as a single model. On English to French it scores 41.8, a new single-model state of the art. And it does all this with about 2.3 times 10 to the 19 training floating-point operations, which is well below most strong baselines. The base model alone, trained in twelve hours, already beats almost every prior published model. The combination of better quality and lower training cost is what made this paper land so hard.
|
||||
|
||||
---
|
||||
|
||||
# 15_ablations
|
||||
|
||||
The variation study in table three gives us five takeaways worth remembering. First, head count has a sweet spot: a single head loses about 0.9 BLEU compared to eight heads, but going to thirty-two also degrades. Second, the key dimension matters more than the value dimension — determining compatibility is the harder learning problem. Third, scaling up the model dimension and the depth consistently improves quality, so the architecture scales monotonically with capacity. Fourth, dropout is essential — removing it loses roughly half a BLEU to a full BLEU point even at p equals 0.1. Fifth, label smoothing slightly hurts perplexity but improves BLEU, which is a reminder to optimize for the metric you actually ship on.
|
||||
|
||||
---
|
||||
|
||||
# 16_conclusion
|
||||
|
||||
To close — this 2017 paper turned out to be one of the most consequential architectural ideas in modern machine learning. The Transformer became the substrate for BERT, for GPT, for T5, for the Vision Transformer, for AlphaFold, for CLIP, for almost every large language model in production today, including the one talking to you right now. The core insight — that just attention, with no recurrence and no convolution, is the right inductive bias for sequence modeling — turned out to generalize far beyond translation, to almost every modality we work with. One paper, a generation of models.
|
||||
@@ -0,0 +1,90 @@
|
||||
# Execution Lock
|
||||
|
||||
## canvas
|
||||
- viewBox: 0 0 1280 720
|
||||
- format: PPT 16:9
|
||||
|
||||
## colors
|
||||
- bg: #FFFFFF
|
||||
- bg_secondary: #F5F7FA
|
||||
- primary: #1A365D
|
||||
- accent: #3182CE
|
||||
- secondary_accent: #63B3ED
|
||||
- text: #1A202C
|
||||
- text_secondary: #4A5568
|
||||
- text_tertiary: #A0AEC0
|
||||
- border: #E2E8F0
|
||||
- accent_tint: #EBF4FB
|
||||
- success: #2F855A
|
||||
- warning: #C53030
|
||||
- image_rendering: blueprint
|
||||
- image_palette: cool-corporate
|
||||
|
||||
## typography
|
||||
- font_family: Arial, "Microsoft YaHei", "PingFang SC", sans-serif
|
||||
- title_family: Georgia, "Microsoft YaHei", serif
|
||||
- body_family: Arial, "Microsoft YaHei", "PingFang SC", sans-serif
|
||||
- emphasis_family: Georgia, "Microsoft YaHei", serif
|
||||
- code_family: Consolas, "Courier New", monospace
|
||||
- body: 20
|
||||
- title: 36
|
||||
- subtitle: 26
|
||||
- annotation: 14
|
||||
- footnote: 11
|
||||
- cover_title: 64
|
||||
- chapter_title: 44
|
||||
- hero_number: 40
|
||||
|
||||
## icons
|
||||
- library: tabler-outline
|
||||
- stroke_width: 2
|
||||
- inventory: trophy, bolt, stack-2, brain, refresh, arrow-right, lock, hourglass, box, eye, grid-dots, wave-sine, chart-bar, cpu, settings, binary-tree
|
||||
|
||||
## images
|
||||
- cover_bg: images/cover_bg.png
|
||||
- sequence_evolution: images/sequence_evolution.png
|
||||
- rnn_bottleneck: images/rnn_bottleneck.png
|
||||
- sublayer_block: images/sublayer_block.png
|
||||
- positional_encoding: images/positional_encoding.png
|
||||
- transformer_family: images/transformer_family.png
|
||||
- attention_p3_0: images/attention_p3_0.png | no-crop
|
||||
- attention_p4_1: images/attention_p4_1.png | no-crop
|
||||
- formula_001: images/formula_001.png | no-crop
|
||||
- formula_002: images/formula_002.png | no-crop
|
||||
- formula_003: images/formula_003.png | no-crop
|
||||
|
||||
## page_rhythm
|
||||
- P01: anchor
|
||||
- P02: dense
|
||||
- P03: breathing
|
||||
- P04: dense
|
||||
- P05: dense
|
||||
- P06: dense
|
||||
- P07: dense
|
||||
- P08: dense
|
||||
- P09: dense
|
||||
- P10: dense
|
||||
- P11: breathing
|
||||
- P12: dense
|
||||
- P13: dense
|
||||
- P14: dense
|
||||
- P15: dense
|
||||
- P16: anchor
|
||||
|
||||
## page_charts
|
||||
- P02: kpi_cards
|
||||
- P04: vertical_pillars
|
||||
- P06: comparison_columns
|
||||
- P09: vertical_pillars
|
||||
- P10: module_composition
|
||||
- P12: basic_table
|
||||
- P13: kpi_cards
|
||||
- P14: basic_table
|
||||
- P15: vertical_list
|
||||
|
||||
## forbidden
|
||||
- Mixing icon libraries
|
||||
- rgba()
|
||||
- `<style>`, `class`, `<foreignObject>`, `textPath`, `@font-face`, `<animate*>`, `<script>`, `<iframe>`, `<symbol>`+`<use>`
|
||||
- `<g opacity>` (set opacity on each child element individually)
|
||||
- HTML named entities in text — write as raw Unicode; XML reserved chars `& < > " '` escaped as `& < > " '`
|
||||
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 905 KiB |
@@ -0,0 +1,132 @@
|
||||
<?xml version='1.0' encoding='UTF-8'?>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1280 720" width="1280" height="720">
|
||||
<defs>
|
||||
<filter id="cardShadow" x="-10%" y="-10%" width="120%" height="120%">
|
||||
<feGaussianBlur in="SourceAlpha" stdDeviation="4" />
|
||||
<feOffset dx="0" dy="3" />
|
||||
<feFlood flood-color="#1A365D" flood-opacity="0.07" />
|
||||
<feComposite in2="SourceAlpha" operator="in" />
|
||||
<feMerge><feMergeNode /><feMergeNode in="SourceGraphic" /></feMerge>
|
||||
</filter>
|
||||
</defs>
|
||||
|
||||
<g id="bg">
|
||||
<rect width="1280" height="720" fill="#FFFFFF" />
|
||||
</g>
|
||||
|
||||
<g id="header">
|
||||
<rect x="60" y="56" width="6" height="40" fill="#3182CE" />
|
||||
<text x="84" y="88" font-family="Georgia, "Microsoft YaHei", serif" font-size="36" font-weight="700" fill="#1A365D">A simpler model, better results, less training</text>
|
||||
<text x="84" y="118" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1.5">WHY THIS PAPER MATTERS · 2017 SOTA ON WMT'14 TRANSLATION</text>
|
||||
</g>
|
||||
|
||||
<g id="kpi-bleu-ende">
|
||||
<path fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)" d="M72,160 H333 A12,12 0 0 1 345,172 V388 A12,12 0 0 1 333,400 H72 A12,12 0 0 1 60,388 V172 A12,12 0 0 1 72,160 Z" />
|
||||
<path fill="#1A365D" d="M63,160 H63 A3,3 0 0 1 66,163 V397 A3,3 0 0 1 63,400 H63 A3,3 0 0 1 60,397 V163 A3,3 0 0 1 63,160 Z" />
|
||||
|
||||
<g transform="translate(80.0, 184.0) scale(1.3333333333333333)" fill="none" stroke="#1A365D" stroke-width="2">
|
||||
<path d="M8 21l8 0" />
|
||||
<path d="M12 17l0 4" />
|
||||
<path d="M7 4l10 0" />
|
||||
<path d="M17 4v8a5 5 0 0 1 -10 0v-8" />
|
||||
<path d="M3 9a2 2 0 1 0 4 0a2 2 0 1 0 -4 0" />
|
||||
<path d="M17 9a2 2 0 1 0 4 0a2 2 0 1 0 -4 0" />
|
||||
</g>
|
||||
<text x="80" y="246" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1">EN → DE</text>
|
||||
<text x="80" y="270" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#4A5568">BLEU on WMT'14</text>
|
||||
<text x="80" y="336" font-family="Georgia, "Microsoft YaHei", serif" font-size="64" font-weight="700" fill="#1A365D">28.4</text>
|
||||
|
||||
<g transform="translate(80.0, 354.0) scale(0.8333333333333334)" fill="none" stroke="#2F855A" stroke-width="2">
|
||||
<path d="M17 7l-10 10" />
|
||||
<path d="M8 7l9 0l0 9" />
|
||||
</g>
|
||||
<text x="106" y="370" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="16" font-weight="700" fill="#2F855A">+2.0</text>
|
||||
<text x="150" y="370" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" fill="#4A5568">over prior ensembles</text>
|
||||
</g>
|
||||
|
||||
<g id="kpi-bleu-enfr">
|
||||
<path fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)" d="M377,160 H638 A12,12 0 0 1 650,172 V388 A12,12 0 0 1 638,400 H377 A12,12 0 0 1 365,388 V172 A12,12 0 0 1 377,160 Z" />
|
||||
<path fill="#3182CE" d="M368,160 H368 A3,3 0 0 1 371,163 V397 A3,3 0 0 1 368,400 H368 A3,3 0 0 1 365,397 V163 A3,3 0 0 1 368,160 Z" />
|
||||
|
||||
<g transform="translate(385.0, 184.0) scale(1.3333333333333333)" fill="none" stroke="#3182CE" stroke-width="2">
|
||||
<path d="M8 21l8 0" />
|
||||
<path d="M12 17l0 4" />
|
||||
<path d="M7 4l10 0" />
|
||||
<path d="M17 4v8a5 5 0 0 1 -10 0v-8" />
|
||||
<path d="M3 9a2 2 0 1 0 4 0a2 2 0 1 0 -4 0" />
|
||||
<path d="M17 9a2 2 0 1 0 4 0a2 2 0 1 0 -4 0" />
|
||||
</g>
|
||||
<text x="385" y="246" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1">EN → FR</text>
|
||||
<text x="385" y="270" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#4A5568">BLEU on WMT'14</text>
|
||||
<text x="385" y="336" font-family="Georgia, "Microsoft YaHei", serif" font-size="64" font-weight="700" fill="#3182CE">41.8</text>
|
||||
|
||||
<g transform="translate(385.0, 354.0) scale(0.8333333333333334)" fill="none" stroke="#2F855A" stroke-width="2">
|
||||
<path d="M12 17.75l-6.172 3.245l1.179 -6.873l-5 -4.867l6.9 -1l3.086 -6.253l3.086 6.253l6.9 1l-5 4.867l1.179 6.873l-6.158 -3.245" />
|
||||
</g>
|
||||
<text x="411" y="370" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#2F855A">new single-model SOTA</text>
|
||||
</g>
|
||||
|
||||
<g id="kpi-training">
|
||||
<path fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)" d="M682,160 H943 A12,12 0 0 1 955,172 V388 A12,12 0 0 1 943,400 H682 A12,12 0 0 1 670,388 V172 A12,12 0 0 1 682,160 Z" />
|
||||
<path fill="#1A365D" d="M673,160 H673 A3,3 0 0 1 676,163 V397 A3,3 0 0 1 673,400 H673 A3,3 0 0 1 670,397 V163 A3,3 0 0 1 673,160 Z" />
|
||||
|
||||
<g transform="translate(690.0, 184.0) scale(1.3333333333333333)" fill="none" stroke="#1A365D" stroke-width="2">
|
||||
<path d="M13 3l0 7l6 0l-8 11l0 -7l-6 0l8 -11" />
|
||||
</g>
|
||||
<text x="690" y="246" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1">TRAINING TIME</text>
|
||||
<text x="690" y="270" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#4A5568">Transformer (big)</text>
|
||||
<text x="690" y="336" font-family="Georgia, "Microsoft YaHei", serif" font-size="48" font-weight="700" fill="#1A365D">3.5 days</text>
|
||||
|
||||
<g transform="translate(690.0, 354.0) scale(0.8333333333333334)" fill="none" stroke="#4A5568" stroke-width="2">
|
||||
<path d="M5 6a1 1 0 0 1 1 -1h12a1 1 0 0 1 1 1v12a1 1 0 0 1 -1 1h-12a1 1 0 0 1 -1 -1l0 -12" />
|
||||
<path d="M9 9h6v6h-6l0 -6" />
|
||||
<path d="M3 10h2" />
|
||||
<path d="M3 14h2" />
|
||||
<path d="M10 3v2" />
|
||||
<path d="M14 3v2" />
|
||||
<path d="M21 10h-2" />
|
||||
<path d="M21 14h-2" />
|
||||
<path d="M14 21v-2" />
|
||||
<path d="M10 21v-2" />
|
||||
</g>
|
||||
<text x="716" y="370" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" fill="#4A5568">on 8× P100 GPUs</text>
|
||||
</g>
|
||||
|
||||
<g id="kpi-no-rnn-cnn">
|
||||
<path fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)" d="M987,160 H1208 A12,12 0 0 1 1220,172 V388 A12,12 0 0 1 1208,400 H987 A12,12 0 0 1 975,388 V172 A12,12 0 0 1 987,160 Z" />
|
||||
<path fill="#3182CE" d="M978,160 H978 A3,3 0 0 1 981,163 V397 A3,3 0 0 1 978,400 H978 A3,3 0 0 1 975,397 V163 A3,3 0 0 1 978,160 Z" />
|
||||
|
||||
<g transform="translate(995.0, 184.0) scale(1.3333333333333333)" fill="none" stroke="#3182CE" stroke-width="2">
|
||||
<path d="M10 12a2 2 0 1 0 4 0a2 2 0 0 0 -4 0" />
|
||||
<path d="M21 12c-2.4 4 -5.4 6 -9 6c-3.6 0 -6.6 -2 -9 -6c2.4 -4 5.4 -6 9 -6c3.6 0 6.6 2 9 6" />
|
||||
</g>
|
||||
<text x="995" y="246" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1">ARCHITECTURE</text>
|
||||
<text x="995" y="270" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#4A5568">building blocks</text>
|
||||
<text x="995" y="316" font-family="Georgia, "Microsoft YaHei", serif" font-size="40" font-weight="700" fill="#1A365D">Attention</text>
|
||||
<text x="995" y="346" font-family="Georgia, "Microsoft YaHei", serif" font-size="22" font-style="italic" fill="#4A5568">only</text>
|
||||
<text x="995" y="376" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" fill="#4A5568">no recurrence · no convolution</text>
|
||||
</g>
|
||||
|
||||
<g id="takeaway-band">
|
||||
<path fill="#F5F7FA" stroke="#E2E8F0" stroke-width="1" d="M72,450 H1208 A12,12 0 0 1 1220,462 V598 A12,12 0 0 1 1208,610 H72 A12,12 0 0 1 60,598 V462 A12,12 0 0 1 72,450 Z" />
|
||||
<path fill="#1A365D" d="M63,450 H63 A3,3 0 0 1 66,453 V607 A3,3 0 0 1 63,610 H63 A3,3 0 0 1 60,607 V453 A3,3 0 0 1 63,450 Z" />
|
||||
<text x="100" y="490" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#3182CE" letter-spacing="1.5">THE HEADLINE</text>
|
||||
<text x="100" y="520" font-family="Georgia, "Microsoft YaHei", serif" font-size="22" font-weight="700" fill="#1A365D">
|
||||
The Transformer reaches <tspan fill="#3182CE">new translation SOTA</tspan> while training
|
||||
</text>
|
||||
<text x="100" y="548" font-family="Georgia, "Microsoft YaHei", serif" font-size="22" font-weight="700" fill="#1A365D">
|
||||
in <tspan fill="#3182CE">a fraction of the time</tspan> of strong baselines.
|
||||
</text>
|
||||
<text x="100" y="580" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="16" fill="#4A5568">
|
||||
Quality up · cost down · parallelization unlocked. The whole rest of the paper explains
|
||||
</text>
|
||||
<text x="100" y="602" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="16" fill="#4A5568">
|
||||
how attention alone makes this possible — and why nothing else is needed.
|
||||
</text>
|
||||
</g>
|
||||
|
||||
<g id="footer">
|
||||
<text x="60" y="694" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">Source: Vaswani et al. 2017, Table 2 (BLEU) and §5.2 (training cost)</text>
|
||||
<text x="1220" y="694" text-anchor="end" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">02</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 9.0 KiB |
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 774 KiB |
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 662 KiB |
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 297 KiB |
@@ -0,0 +1,95 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1280 720" width="1280" height="720">
|
||||
<defs>
|
||||
<filter id="cardShadow" x="-10%" y="-10%" width="120%" height="120%">
|
||||
<feGaussianBlur in="SourceAlpha" stdDeviation="6" />
|
||||
<feOffset dx="0" dy="3" />
|
||||
<feFlood flood-color="#1A365D" flood-opacity="0.07" />
|
||||
<feComposite in2="SourceAlpha" operator="in" />
|
||||
<feMerge><feMergeNode /><feMergeNode in="SourceGraphic" /></feMerge>
|
||||
</filter>
|
||||
</defs>
|
||||
|
||||
<g id="bg">
|
||||
<rect width="1280" height="720" fill="#FFFFFF" />
|
||||
</g>
|
||||
|
||||
<g id="header">
|
||||
<rect x="60" y="56" width="6" height="40" fill="#3182CE" />
|
||||
<text x="84" y="88" font-family="Georgia, "Microsoft YaHei", serif" font-size="36" font-weight="700" fill="#1A365D">Same skeleton, three asymmetries</text>
|
||||
<text x="84" y="118" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1.5">ENCODER STACK · DECODER STACK · N = 6 · d_model = 512</text>
|
||||
</g>
|
||||
|
||||
<g id="card-encoder">
|
||||
<path fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)" d="M94,160 H621 A14,14 0 0 1 635,174 V646 A14,14 0 0 1 621,660 H94 A14,14 0 0 1 80,646 V174 A14,14 0 0 1 94,160 Z" />
|
||||
<path d="M 94 160 h 527 a 14 14 0 0 1 14 14 v 90 h -555 v -90 a 14 14 0 0 1 14 -14 Z" fill="#1A365D" />
|
||||
<text x="357" y="212" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="28" font-weight="700" fill="#FFFFFF">Encoder</text>
|
||||
<text x="357" y="240" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" font-weight="600" fill="#FFFFFF" fill-opacity="0.85" letter-spacing="1">N = 6 IDENTICAL LAYERS</text>
|
||||
|
||||
<g id="enc-sublayer-1">
|
||||
<path fill="#F5F7FA" stroke="#1A365D" stroke-width="1.5" d="M118,300 H597 A8,8 0 0 1 605,308 V348 A8,8 0 0 1 597,356 H118 A8,8 0 0 1 110,348 V308 A8,8 0 0 1 118,300 Z" />
|
||||
<circle cx="138" cy="328" r="14" fill="#1A365D" />
|
||||
<text x="138" y="334" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#FFFFFF">1</text>
|
||||
<text x="166" y="324" font-family="Georgia, "Microsoft YaHei", serif" font-size="17" font-weight="700" fill="#1A365D">Multi-Head Self-Attention</text>
|
||||
<text x="166" y="346" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" fill="#4A5568">Every position attends to every position</text>
|
||||
</g>
|
||||
|
||||
<g id="enc-sublayer-2">
|
||||
<path fill="#F5F7FA" stroke="#1A365D" stroke-width="1.5" d="M118,372 H597 A8,8 0 0 1 605,380 V420 A8,8 0 0 1 597,428 H118 A8,8 0 0 1 110,420 V380 A8,8 0 0 1 118,372 Z" />
|
||||
<circle cx="138" cy="400" r="14" fill="#1A365D" />
|
||||
<text x="138" y="406" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#FFFFFF">2</text>
|
||||
<text x="166" y="396" font-family="Georgia, "Microsoft YaHei", serif" font-size="17" font-weight="700" fill="#1A365D">Position-wise Feed Forward</text>
|
||||
<text x="166" y="418" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" fill="#4A5568">Two linear layers, ReLU between</text>
|
||||
</g>
|
||||
|
||||
<text x="110" y="466" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-weight="700" fill="#3182CE" letter-spacing="1.5">WRAPPER AROUND EACH SUB-LAYER</text>
|
||||
<text x="110" y="492" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">Output = LayerNorm( x + Sublayer(x) )</text>
|
||||
<text x="110" y="514" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#4A5568">— residual connection then layer normalization.</text>
|
||||
|
||||
<path fill="#F5F7FA" d="M118,544 H597 A8,8 0 0 1 605,552 V616 A8,8 0 0 1 597,624 H118 A8,8 0 0 1 110,616 V552 A8,8 0 0 1 118,544 Z" />
|
||||
<text x="124" y="568" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-weight="700" fill="#1A365D" letter-spacing="1.5">SHAPE CONTRACT</text>
|
||||
<text x="124" y="592" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#1A202C">All sub-layers and embeddings emit dimension <tspan font-family="Consolas, monospace" fill="#3182CE" font-weight="700">d_model = 512</tspan></text>
|
||||
<text x="124" y="612" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#1A202C">so residual addition is well-defined.</text>
|
||||
</g>
|
||||
|
||||
<g id="card-decoder">
|
||||
<path fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)" d="M669,160 H1196 A14,14 0 0 1 1210,174 V646 A14,14 0 0 1 1196,660 H669 A14,14 0 0 1 655,646 V174 A14,14 0 0 1 669,160 Z" />
|
||||
<path d="M 669 160 h 527 a 14 14 0 0 1 14 14 v 90 h -555 v -90 a 14 14 0 0 1 14 -14 Z" fill="#3182CE" />
|
||||
<text x="932" y="212" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="28" font-weight="700" fill="#FFFFFF">Decoder</text>
|
||||
<text x="932" y="240" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" font-weight="600" fill="#FFFFFF" fill-opacity="0.9" letter-spacing="1">N = 6 IDENTICAL LAYERS · INSERTS A THIRD SUB-LAYER</text>
|
||||
|
||||
<g id="dec-sublayer-1">
|
||||
<path fill="#FFFFFF" stroke="#3182CE" stroke-width="1.5" d="M693,300 H1172 A8,8 0 0 1 1180,308 V348 A8,8 0 0 1 1172,356 H693 A8,8 0 0 1 685,348 V308 A8,8 0 0 1 693,300 Z" />
|
||||
<circle cx="713" cy="328" r="14" fill="#3182CE" />
|
||||
<text x="713" y="334" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#FFFFFF">1</text>
|
||||
<text x="741" y="324" font-family="Georgia, "Microsoft YaHei", serif" font-size="17" font-weight="700" fill="#1A365D"><tspan fill="#3182CE">Masked</tspan> Multi-Head Self-Attention</text>
|
||||
<text x="741" y="346" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" fill="#4A5568">Position i attends only to positions ≤ i</text>
|
||||
</g>
|
||||
|
||||
<g id="dec-sublayer-2">
|
||||
<path fill="#FFFFFF" stroke="#3182CE" stroke-width="2.5" d="M693,372 H1172 A8,8 0 0 1 1180,380 V420 A8,8 0 0 1 1172,428 H693 A8,8 0 0 1 685,420 V380 A8,8 0 0 1 693,372 Z" />
|
||||
<circle cx="713" cy="400" r="14" fill="#3182CE" />
|
||||
<text x="713" y="406" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#FFFFFF">2</text>
|
||||
<text x="741" y="396" font-family="Georgia, "Microsoft YaHei", serif" font-size="17" font-weight="700" fill="#1A365D"><tspan fill="#3182CE">Encoder-Decoder</tspan> Attention</text>
|
||||
<text x="741" y="418" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" fill="#4A5568">Q from decoder · K, V from encoder output</text>
|
||||
</g>
|
||||
|
||||
<g id="dec-sublayer-3">
|
||||
<path fill="#F5F7FA" stroke="#3182CE" stroke-width="1.5" d="M693,444 H1172 A8,8 0 0 1 1180,452 V492 A8,8 0 0 1 1172,500 H693 A8,8 0 0 1 685,492 V452 A8,8 0 0 1 693,444 Z" />
|
||||
<circle cx="713" cy="472" r="14" fill="#3182CE" />
|
||||
<text x="713" y="478" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#FFFFFF">3</text>
|
||||
<text x="741" y="468" font-family="Georgia, "Microsoft YaHei", serif" font-size="17" font-weight="700" fill="#1A365D">Position-wise Feed Forward</text>
|
||||
<text x="741" y="490" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" fill="#4A5568">Same shape as encoder FFN</text>
|
||||
</g>
|
||||
|
||||
<path fill="#EBF4FB" d="M693,528 H1172 A8,8 0 0 1 1180,536 V616 A8,8 0 0 1 1172,624 H693 A8,8 0 0 1 685,616 V536 A8,8 0 0 1 693,528 Z" />
|
||||
<text x="699" y="552" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-weight="700" fill="#3182CE" letter-spacing="1.5">THREE ASYMMETRIES VS ENCODER</text>
|
||||
<text x="699" y="578" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#1A202C">· third sub-layer attends to encoder output</text>
|
||||
<text x="699" y="598" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#1A202C">· self-attention is masked (no peeking at future)</text>
|
||||
<text x="699" y="618" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#1A202C">· output embeddings are offset by one position</text>
|
||||
</g>
|
||||
|
||||
<g id="footer">
|
||||
<text x="60" y="694" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">Source: Vaswani et al. 2017, §3.1 Encoder and Decoder Stacks</text>
|
||||
<text x="1220" y="694" text-anchor="end" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">06</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 8.7 KiB |
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 25 KiB |
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 198 KiB |
@@ -0,0 +1,116 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1280 720" width="1280" height="720">
|
||||
<defs>
|
||||
<filter id="cardShadow" x="-10%" y="-10%" width="120%" height="120%">
|
||||
<feGaussianBlur in="SourceAlpha" stdDeviation="6" />
|
||||
<feOffset dx="0" dy="3" />
|
||||
<feFlood flood-color="#1A365D" flood-opacity="0.07" />
|
||||
<feComposite in2="SourceAlpha" operator="in" />
|
||||
<feMerge><feMergeNode /><feMergeNode in="SourceGraphic" /></feMerge>
|
||||
</filter>
|
||||
</defs>
|
||||
|
||||
<g id="bg">
|
||||
<rect width="1280" height="720" fill="#FFFFFF" />
|
||||
</g>
|
||||
|
||||
<g id="header">
|
||||
<rect x="60" y="56" width="6" height="40" fill="#3182CE" />
|
||||
<text x="84" y="88" font-family="Georgia, "Microsoft YaHei", serif" font-size="36" font-weight="700" fill="#1A365D">One mechanism, three roles in the Transformer</text>
|
||||
<text x="84" y="118" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1.5">WHERE ATTENTION SHOWS UP IN THE ARCHITECTURE · §3.2.3</text>
|
||||
</g>
|
||||
|
||||
<g id="pillar-enc-self">
|
||||
<path fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)" d="M74,160 H426 A14,14 0 0 1 440,174 V646 A14,14 0 0 1 426,660 H74 A14,14 0 0 1 60,646 V174 A14,14 0 0 1 74,160 Z" />
|
||||
<path d="M 74 160 h 352 a 14 14 0 0 1 14 14 v 100 h -380 v -100 a 14 14 0 0 1 14 -14 Z" fill="#1A365D" />
|
||||
|
||||
<g transform="translate(80.0, 180.0) scale(1.3333333333333333)" fill="none" stroke="#FFFFFF" stroke-width="2">
|
||||
<path d="M9 11l-4 4l4 4m-4 -4h11a4 4 0 0 0 0 -8h-1" />
|
||||
</g>
|
||||
<text x="250" y="208" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" font-weight="700" fill="#FFFFFF" fill-opacity="0.85" letter-spacing="2">ENCODER</text>
|
||||
<text x="250" y="240" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="22" font-weight="700" fill="#FFFFFF">Self-Attention</text>
|
||||
|
||||
<text x="80" y="312" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-weight="700" fill="#3182CE" letter-spacing="1.5">Q · K · V SOURCE</text>
|
||||
<path fill="#F5F7FA" d="M86,324 H414 A6,6 0 0 1 420,330 V366 A6,6 0 0 1 414,372 H86 A6,6 0 0 1 80,366 V330 A6,6 0 0 1 86,324 Z" />
|
||||
<text x="98" y="354" font-family="Consolas, "Courier New", monospace" font-size="14" fill="#1A202C">Q, K, V ← previous encoder layer</text>
|
||||
|
||||
<text x="80" y="406" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Every position attends to all</text>
|
||||
<text x="80" y="442" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">Every token in the source sequence can</text>
|
||||
<text x="80" y="464" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">attend to every other token — including</text>
|
||||
<text x="80" y="486" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">itself.</text>
|
||||
|
||||
<text x="80" y="534" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-weight="700" fill="#3182CE" letter-spacing="1.5">WHAT IT'S FOR</text>
|
||||
<text x="80" y="560" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">Build rich contextualized</text>
|
||||
<text x="80" y="582" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">representations of each source</text>
|
||||
<text x="80" y="604" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">token from its surroundings.</text>
|
||||
|
||||
<path fill="#1A365D" fill-opacity="0.08" d="M84,624 H416 A4,4 0 0 1 420,628 V640 A4,4 0 0 1 416,644 H84 A4,4 0 0 1 80,640 V628 A4,4 0 0 1 84,624 Z" />
|
||||
<text x="92" y="640" font-family="Consolas, "Courier New", monospace" font-size="12" fill="#1A365D">attention_mask = none</text>
|
||||
</g>
|
||||
|
||||
<g id="pillar-dec-masked">
|
||||
<path fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)" d="M464,160 H816 A14,14 0 0 1 830,174 V646 A14,14 0 0 1 816,660 H464 A14,14 0 0 1 450,646 V174 A14,14 0 0 1 464,160 Z" />
|
||||
<path d="M 464 160 h 352 a 14 14 0 0 1 14 14 v 100 h -380 v -100 a 14 14 0 0 1 14 -14 Z" fill="#3182CE" />
|
||||
|
||||
<g transform="translate(470.0, 180.0) scale(1.3333333333333333)" fill="none" stroke="#FFFFFF" stroke-width="2">
|
||||
<path d="M5 13a2 2 0 0 1 2 -2h10a2 2 0 0 1 2 2v6a2 2 0 0 1 -2 2h-10a2 2 0 0 1 -2 -2v-6" />
|
||||
<path d="M11 16a1 1 0 1 0 2 0a1 1 0 0 0 -2 0" />
|
||||
<path d="M8 11v-4a4 4 0 1 1 8 0v4" />
|
||||
</g>
|
||||
<text x="640" y="208" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" font-weight="700" fill="#FFFFFF" fill-opacity="0.9" letter-spacing="2">DECODER</text>
|
||||
<text x="640" y="240" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="22" font-weight="700" fill="#FFFFFF">Masked Self-Attention</text>
|
||||
|
||||
<text x="470" y="312" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-weight="700" fill="#3182CE" letter-spacing="1.5">Q · K · V SOURCE</text>
|
||||
<path fill="#F5F7FA" d="M476,324 H804 A6,6 0 0 1 810,330 V366 A6,6 0 0 1 804,372 H476 A6,6 0 0 1 470,366 V330 A6,6 0 0 1 476,324 Z" />
|
||||
<text x="488" y="354" font-family="Consolas, "Courier New", monospace" font-size="14" fill="#1A202C">Q, K, V ← previous decoder layer</text>
|
||||
|
||||
<text x="470" y="406" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Only past, never future</text>
|
||||
<text x="470" y="442" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">Position i attends only to positions ≤ i.</text>
|
||||
<text x="470" y="464" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">Achieved by setting illegal positions to</text>
|
||||
<text x="470" y="486" font-family="Consolas, "Courier New", monospace" font-size="14" fill="#1A202C">−∞ before the softmax.</text>
|
||||
|
||||
<text x="470" y="534" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-weight="700" fill="#3182CE" letter-spacing="1.5">WHAT IT'S FOR</text>
|
||||
<text x="470" y="560" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">Preserve auto-regression — the</text>
|
||||
<text x="470" y="582" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">model cannot peek at tokens it</text>
|
||||
<text x="470" y="604" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">hasn't produced yet.</text>
|
||||
|
||||
<path fill="#3182CE" fill-opacity="0.1" d="M474,624 H806 A4,4 0 0 1 810,628 V640 A4,4 0 0 1 806,644 H474 A4,4 0 0 1 470,640 V628 A4,4 0 0 1 474,624 Z" />
|
||||
<text x="482" y="640" font-family="Consolas, "Courier New", monospace" font-size="12" fill="#3182CE">attention_mask = causal triangular</text>
|
||||
</g>
|
||||
|
||||
<g id="pillar-cross">
|
||||
<path fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)" d="M854,160 H1206 A14,14 0 0 1 1220,174 V646 A14,14 0 0 1 1206,660 H854 A14,14 0 0 1 840,646 V174 A14,14 0 0 1 854,160 Z" />
|
||||
<path d="M 854 160 h 352 a 14 14 0 0 1 14 14 v 100 h -380 v -100 a 14 14 0 0 1 14 -14 Z" fill="#1A365D" />
|
||||
|
||||
<g transform="translate(860.0, 180.0) scale(1.3333333333333333)" fill="none" stroke="#FFFFFF" stroke-width="2">
|
||||
<path d="M21 17l-18 0" />
|
||||
<path d="M6 10l-3 -3l3 -3" />
|
||||
<path d="M3 7l18 0" />
|
||||
<path d="M18 20l3 -3l-3 -3" />
|
||||
</g>
|
||||
<text x="1030" y="208" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" font-weight="700" fill="#FFFFFF" fill-opacity="0.85" letter-spacing="2">ENCODER ↔ DECODER</text>
|
||||
<text x="1030" y="240" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="22" font-weight="700" fill="#FFFFFF">Cross-Attention</text>
|
||||
|
||||
<text x="860" y="312" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-weight="700" fill="#3182CE" letter-spacing="1.5">Q · K · V SOURCE</text>
|
||||
<path fill="#F5F7FA" d="M866,324 H1194 A6,6 0 0 1 1200,330 V366 A6,6 0 0 1 1194,372 H866 A6,6 0 0 1 860,366 V330 A6,6 0 0 1 866,324 Z" />
|
||||
<text x="878" y="346" font-family="Consolas, "Courier New", monospace" font-size="13" fill="#1A202C">Q ← previous decoder layer</text>
|
||||
<text x="878" y="364" font-family="Consolas, "Courier New", monospace" font-size="13" fill="#3182CE">K, V ← encoder output</text>
|
||||
|
||||
<text x="860" y="406" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Look back at the input</text>
|
||||
<text x="860" y="442" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">Every decoder position can attend over</text>
|
||||
<text x="860" y="464" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">all positions of the input sequence —</text>
|
||||
<text x="860" y="486" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">the classic seq2seq attention role.</text>
|
||||
|
||||
<text x="860" y="534" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-weight="700" fill="#3182CE" letter-spacing="1.5">WHAT IT'S FOR</text>
|
||||
<text x="860" y="560" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">Let each output token retrieve</text>
|
||||
<text x="860" y="582" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">the relevant source information,</text>
|
||||
<text x="860" y="604" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">no matter where it sits.</text>
|
||||
|
||||
<path fill="#1A365D" fill-opacity="0.08" d="M864,624 H1196 A4,4 0 0 1 1200,628 V640 A4,4 0 0 1 1196,644 H864 A4,4 0 0 1 860,640 V628 A4,4 0 0 1 864,624 Z" />
|
||||
<text x="872" y="640" font-family="Consolas, "Courier New", monospace" font-size="12" fill="#1A365D">attention_mask = none</text>
|
||||
</g>
|
||||
|
||||
<g id="footer">
|
||||
<text x="60" y="694" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">Source: Vaswani et al. 2017, §3.2.3 Applications of Attention</text>
|
||||
<text x="1220" y="694" text-anchor="end" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">09</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 10 KiB |
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 568 KiB |
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 1.0 MiB |
@@ -0,0 +1,77 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1280 720" width="1280" height="720">
|
||||
<g id="bg">
|
||||
<rect width="1280" height="720" fill="#FFFFFF" />
|
||||
</g>
|
||||
|
||||
<g id="header">
|
||||
<rect x="60" y="56" width="6" height="40" fill="#3182CE" />
|
||||
<text x="84" y="88" font-family="Georgia, "Microsoft YaHei", serif" font-size="36" font-weight="700" fill="#1A365D">Complexity per layer, sequential ops, max path length</text>
|
||||
<text x="84" y="118" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1.5">TABLE 1 · WHY SELF-ATTENTION · n = sequence length, d = representation dim</text>
|
||||
</g>
|
||||
|
||||
<g id="table">
|
||||
<rect x="60" y="160" width="1160" height="370" fill="none" stroke="#1A365D" stroke-width="2" />
|
||||
<g id="table-area" transform="translate(60, 160)">
|
||||
<rect x="0" y="0" width="1160" height="60" fill="#1A365D" />
|
||||
<line x1="0" y1="60" x2="1160" y2="60" stroke="#1A365D" stroke-width="2" />
|
||||
<text x="24" y="37" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="800" fill="#FFFFFF" letter-spacing="1.5">LAYER TYPE</text>
|
||||
<text x="480" y="37" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="800" fill="#FFFFFF" letter-spacing="1.5">COMPLEXITY PER LAYER</text>
|
||||
<text x="760" y="37" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="800" fill="#FFFFFF" letter-spacing="1.5">SEQUENTIAL OPS</text>
|
||||
<text x="980" y="37" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="800" fill="#FFFFFF" letter-spacing="1.5">MAX PATH LENGTH</text>
|
||||
|
||||
<line x1="300" y1="0" x2="300" y2="370" stroke="#E2E8F0" stroke-width="1.5" />
|
||||
<line x1="660" y1="0" x2="660" y2="370" stroke="#E2E8F0" stroke-width="1.5" />
|
||||
<line x1="860" y1="0" x2="860" y2="370" stroke="#E2E8F0" stroke-width="1.5" />
|
||||
|
||||
<g id="row-self-attn">
|
||||
<rect x="0" y="60" width="1160" height="68" fill="#EBF4FB" />
|
||||
<text x="24" y="92" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Self-Attention</text>
|
||||
<text x="24" y="116" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-style="italic" fill="#3182CE">the Transformer</text>
|
||||
<text x="480" y="102" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="20" font-weight="700" fill="#1A365D">O(n² · d)</text>
|
||||
<text x="760" y="102" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="22" font-weight="700" fill="#2F855A">O(1)</text>
|
||||
<text x="980" y="102" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="22" font-weight="700" fill="#2F855A">O(1)</text>
|
||||
</g>
|
||||
|
||||
<g id="row-recurrent">
|
||||
<line x1="0" y1="128" x2="1160" y2="128" stroke="#E2E8F0" stroke-width="1" />
|
||||
<text x="24" y="160" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Recurrent</text>
|
||||
<text x="24" y="184" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-style="italic" fill="#4A5568">RNN / LSTM / GRU</text>
|
||||
<text x="480" y="170" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="20" font-weight="600" fill="#1A202C">O(n · d²)</text>
|
||||
<text x="760" y="170" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="22" font-weight="600" fill="#C53030">O(n)</text>
|
||||
<text x="980" y="170" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="22" font-weight="600" fill="#C53030">O(n)</text>
|
||||
</g>
|
||||
|
||||
<g id="row-conv">
|
||||
<line x1="0" y1="196" x2="1160" y2="196" stroke="#E2E8F0" stroke-width="1" />
|
||||
<text x="24" y="228" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Convolutional</text>
|
||||
<text x="24" y="252" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-style="italic" fill="#4A5568">kernel width k</text>
|
||||
<text x="480" y="238" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="20" font-weight="600" fill="#1A202C">O(k · n · d²)</text>
|
||||
<text x="760" y="238" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="22" font-weight="600" fill="#2F855A">O(1)</text>
|
||||
<text x="980" y="238" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="20" font-weight="600" fill="#4A5568">O(log_k n)</text>
|
||||
</g>
|
||||
|
||||
<g id="row-self-attn-restricted">
|
||||
<line x1="0" y1="264" x2="1160" y2="264" stroke="#E2E8F0" stroke-width="1" />
|
||||
<text x="24" y="296" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Self-Attention <tspan font-style="italic" fill="#4A5568" font-weight="500">(restricted)</tspan></text>
|
||||
<text x="24" y="320" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-style="italic" fill="#4A5568">window r around each position</text>
|
||||
<text x="480" y="306" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="20" font-weight="600" fill="#1A202C">O(r · n · d)</text>
|
||||
<text x="760" y="306" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="22" font-weight="600" fill="#2F855A">O(1)</text>
|
||||
<text x="980" y="306" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="20" font-weight="600" fill="#4A5568">O(n / r)</text>
|
||||
</g>
|
||||
</g>
|
||||
</g>
|
||||
|
||||
<g id="takeaway">
|
||||
<path fill="#F5F7FA" stroke="#E2E8F0" stroke-width="1" d="M70,552 H1210 A10,10 0 0 1 1220,562 V662 A10,10 0 0 1 1210,672 H70 A10,10 0 0 1 60,662 V562 A10,10 0 0 1 70,552 Z" />
|
||||
<path fill="#3182CE" d="M63,552 H63 A3,3 0 0 1 66,555 V669 A3,3 0 0 1 63,672 H63 A3,3 0 0 1 60,669 V555 A3,3 0 0 1 63,552 Z" />
|
||||
<text x="100" y="580" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#3182CE" letter-spacing="1.5">READ THE TABLE LIKE THIS</text>
|
||||
<text x="100" y="610" font-family="Georgia, "Microsoft YaHei", serif" font-size="17" font-weight="600" fill="#1A365D">Self-attention pays <tspan font-family="Consolas, monospace" fill="#1A365D">O(n²·d)</tspan> compute, but in return any two positions are <tspan font-family="Consolas, monospace" fill="#2F855A">one hop apart</tspan>.</text>
|
||||
<text x="100" y="638" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#4A5568">For typical NLP where <tspan font-family="Consolas, monospace" fill="#1A365D" font-weight="700">n < d</tspan> (sequence shorter than representation), self-attention is faster than recurrent layers</text>
|
||||
<text x="100" y="658" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#4A5568">— and it learns long-range dependencies more easily.</text>
|
||||
</g>
|
||||
|
||||
<g id="footer">
|
||||
<text x="60" y="700" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">Source: Vaswani et al. 2017, Table 1 (§4 Why Self-Attention)</text>
|
||||
<text x="1220" y="700" text-anchor="end" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">12</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 7.6 KiB |
@@ -0,0 +1,106 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1280 720" width="1280" height="720">
|
||||
<defs>
|
||||
<filter id="cardShadow" x="-10%" y="-10%" width="120%" height="120%">
|
||||
<feGaussianBlur in="SourceAlpha" stdDeviation="4" />
|
||||
<feOffset dx="0" dy="3" />
|
||||
<feFlood flood-color="#1A365D" flood-opacity="0.07" />
|
||||
<feComposite in2="SourceAlpha" operator="in" />
|
||||
<feMerge><feMergeNode /><feMergeNode in="SourceGraphic" /></feMerge>
|
||||
</filter>
|
||||
</defs>
|
||||
|
||||
<g id="bg">
|
||||
<rect width="1280" height="720" fill="#FFFFFF" />
|
||||
</g>
|
||||
|
||||
<g id="header">
|
||||
<rect x="60" y="56" width="6" height="40" fill="#3182CE" />
|
||||
<text x="84" y="88" font-family="Georgia, "Microsoft YaHei", serif" font-size="36" font-weight="700" fill="#1A365D">How they trained it</text>
|
||||
<text x="84" y="118" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1.5">TRAINING SETUP · §5 WMT'14 ENGLISH-GERMAN & ENGLISH-FRENCH</text>
|
||||
</g>
|
||||
|
||||
<g id="card-hardware">
|
||||
<path fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)" d="M74,160 H626 A14,14 0 0 1 640,174 V386 A14,14 0 0 1 626,400 H74 A14,14 0 0 1 60,386 V174 A14,14 0 0 1 74,160 Z" />
|
||||
<path fill="#1A365D" d="M63,160 H63 A3,3 0 0 1 66,163 V397 A3,3 0 0 1 63,400 H63 A3,3 0 0 1 60,397 V163 A3,3 0 0 1 63,160 Z" />
|
||||
|
||||
<g transform="translate(84.0, 180.0) scale(1.3333333333333333)" fill="none" stroke="#1A365D" stroke-width="2">
|
||||
<path d="M5 6a1 1 0 0 1 1 -1h12a1 1 0 0 1 1 1v12a1 1 0 0 1 -1 1h-12a1 1 0 0 1 -1 -1l0 -12" />
|
||||
<path d="M9 9h6v6h-6l0 -6" />
|
||||
<path d="M3 10h2" />
|
||||
<path d="M3 14h2" />
|
||||
<path d="M10 3v2" />
|
||||
<path d="M14 3v2" />
|
||||
<path d="M21 10h-2" />
|
||||
<path d="M21 14h-2" />
|
||||
<path d="M14 21v-2" />
|
||||
<path d="M10 21v-2" />
|
||||
</g>
|
||||
<text x="84" y="240" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1.5">HARDWARE & SCALE</text>
|
||||
<text x="84" y="296" font-family="Georgia, "Microsoft YaHei", serif" font-size="40" font-weight="700" fill="#1A365D">8 × NVIDIA P100</text>
|
||||
<text x="84" y="332" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">One machine, eight GPUs — a single-node setup.</text>
|
||||
<text x="84" y="354" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">Batches of ~25,000 source + 25,000 target tokens,</text>
|
||||
<text x="84" y="376" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">grouped by similar sequence length for efficiency.</text>
|
||||
</g>
|
||||
|
||||
<g id="card-time">
|
||||
<path fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)" d="M674,160 H1206 A14,14 0 0 1 1220,174 V386 A14,14 0 0 1 1206,400 H674 A14,14 0 0 1 660,386 V174 A14,14 0 0 1 674,160 Z" />
|
||||
<path fill="#3182CE" d="M663,160 H663 A3,3 0 0 1 666,163 V397 A3,3 0 0 1 663,400 H663 A3,3 0 0 1 660,397 V163 A3,3 0 0 1 663,160 Z" />
|
||||
|
||||
<g transform="translate(684.0, 180.0) scale(1.3333333333333333)" fill="none" stroke="#3182CE" stroke-width="2">
|
||||
<path d="M13 3l0 7l6 0l-8 11l0 -7l-6 0l8 -11" />
|
||||
</g>
|
||||
<text x="684" y="240" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1.5">TRAINING TIME</text>
|
||||
<g id="time-base">
|
||||
<text x="684" y="296" font-family="Georgia, "Microsoft YaHei", serif" font-size="32" font-weight="700" fill="#1A365D">base</text>
|
||||
<text x="820" y="296" font-family="Georgia, "Microsoft YaHei", serif" font-size="32" font-weight="700" fill="#3182CE">12 h</text>
|
||||
<text x="940" y="296" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">· 100k steps</text>
|
||||
</g>
|
||||
<g id="time-big">
|
||||
<text x="684" y="346" font-family="Georgia, "Microsoft YaHei", serif" font-size="32" font-weight="700" fill="#1A365D">big</text>
|
||||
<text x="820" y="346" font-family="Georgia, "Microsoft YaHei", serif" font-size="32" font-weight="700" fill="#3182CE">3.5 d</text>
|
||||
<text x="940" y="346" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">· 300k steps</text>
|
||||
</g>
|
||||
<text x="684" y="386" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" font-style="italic" fill="#4A5568">A small fraction of the training cost of prior best models.</text>
|
||||
</g>
|
||||
|
||||
<g id="card-optimizer">
|
||||
<path fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)" d="M74,420 H626 A14,14 0 0 1 640,434 V646 A14,14 0 0 1 626,660 H74 A14,14 0 0 1 60,646 V434 A14,14 0 0 1 74,420 Z" />
|
||||
<path fill="#1A365D" d="M63,420 H63 A3,3 0 0 1 66,423 V657 A3,3 0 0 1 63,660 H63 A3,3 0 0 1 60,657 V423 A3,3 0 0 1 63,420 Z" />
|
||||
|
||||
<g transform="translate(84.0, 440.0) scale(1.3333333333333333)" fill="none" stroke="#1A365D" stroke-width="2">
|
||||
<path d="M10.325 4.317c.426 -1.756 2.924 -1.756 3.35 0a1.724 1.724 0 0 0 2.573 1.066c1.543 -.94 3.31 .826 2.37 2.37a1.724 1.724 0 0 0 1.065 2.572c1.756 .426 1.756 2.924 0 3.35a1.724 1.724 0 0 0 -1.066 2.573c.94 1.543 -.826 3.31 -2.37 2.37a1.724 1.724 0 0 0 -2.572 1.065c-.426 1.756 -2.924 1.756 -3.35 0a1.724 1.724 0 0 0 -2.573 -1.066c-1.543 .94 -3.31 -.826 -2.37 -2.37a1.724 1.724 0 0 0 -1.065 -2.572c-1.756 -.426 -1.756 -2.924 0 -3.35a1.724 1.724 0 0 0 1.066 -2.573c-.94 -1.543 .826 -3.31 2.37 -2.37c1 .608 2.296 .07 2.572 -1.065" />
|
||||
<path d="M9 12a3 3 0 1 0 6 0a3 3 0 0 0 -6 0" />
|
||||
</g>
|
||||
<text x="84" y="500" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1.5">OPTIMIZER</text>
|
||||
<text x="84" y="546" font-family="Georgia, "Microsoft YaHei", serif" font-size="32" font-weight="700" fill="#1A365D">Adam</text>
|
||||
<text x="84" y="578" font-family="Consolas, "Courier New", monospace" font-size="15" fill="#1A202C">β₁ = 0.9 · β₂ = 0.98 · ε = 1e-9</text>
|
||||
<text x="84" y="610" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#4A5568">Custom learning-rate schedule:</text>
|
||||
<text x="84" y="632" font-family="Consolas, "Courier New", monospace" font-size="14" fill="#3182CE">lr ∝ min(step⁻⁰·⁵, step · warmup⁻¹·⁵)</text>
|
||||
<text x="84" y="652" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-style="italic" fill="#4A5568">warm up for 4,000 steps, then decay as 1/√step.</text>
|
||||
</g>
|
||||
|
||||
<g id="card-regularization">
|
||||
<path fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)" d="M674,420 H1206 A14,14 0 0 1 1220,434 V646 A14,14 0 0 1 1206,660 H674 A14,14 0 0 1 660,646 V434 A14,14 0 0 1 674,420 Z" />
|
||||
<path fill="#3182CE" d="M663,420 H663 A3,3 0 0 1 666,423 V657 A3,3 0 0 1 663,660 H663 A3,3 0 0 1 660,657 V423 A3,3 0 0 1 663,420 Z" />
|
||||
|
||||
<g transform="translate(684.0, 440.0) scale(1.3333333333333333)" fill="none" stroke="#3182CE" stroke-width="2">
|
||||
<path d="M12 3a12 12 0 0 0 8.5 3a12 12 0 0 1 -8.5 15a12 12 0 0 1 -8.5 -15a12 12 0 0 0 8.5 -3" />
|
||||
</g>
|
||||
<text x="684" y="500" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1.5">REGULARIZATION</text>
|
||||
<g id="reg-dropout">
|
||||
<text x="684" y="544" font-family="Georgia, "Microsoft YaHei", serif" font-size="22" font-weight="700" fill="#1A365D">Dropout</text>
|
||||
<text x="850" y="544" font-family="Consolas, "Courier New", monospace" font-size="20" font-weight="700" fill="#3182CE">p = 0.1</text>
|
||||
<text x="684" y="566" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" fill="#4A5568">on every sub-layer output and on the sum of embedding + PE.</text>
|
||||
</g>
|
||||
<g id="reg-label-smoothing">
|
||||
<text x="684" y="610" font-family="Georgia, "Microsoft YaHei", serif" font-size="22" font-weight="700" fill="#1A365D">Label smoothing</text>
|
||||
<text x="950" y="610" font-family="Consolas, "Courier New", monospace" font-size="20" font-weight="700" fill="#3182CE">ε_ls = 0.1</text>
|
||||
<text x="684" y="632" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" fill="#4A5568">Slightly hurts perplexity, but improves BLEU and accuracy.</text>
|
||||
</g>
|
||||
</g>
|
||||
|
||||
<g id="footer">
|
||||
<text x="60" y="694" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">Source: Vaswani et al. 2017, §5.1 Training Data and Batching · §5.2 Hardware and Schedule · §5.3 Optimizer · §5.4 Regularization</text>
|
||||
<text x="1220" y="694" text-anchor="end" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">13</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 8.8 KiB |
@@ -0,0 +1,92 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1280 720" width="1280" height="720">
|
||||
<g id="bg">
|
||||
<rect width="1280" height="720" fill="#FFFFFF" />
|
||||
</g>
|
||||
|
||||
<g id="header">
|
||||
<rect x="60" y="56" width="6" height="40" fill="#3182CE" />
|
||||
<text x="84" y="88" font-family="Georgia, "Microsoft YaHei", serif" font-size="36" font-weight="700" fill="#1A365D">New SOTA on WMT'14 — at a fraction of the cost</text>
|
||||
<text x="84" y="118" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1.5">TABLE 2 · BLEU SCORES + TRAINING-COST FLOPS · BEST PRIOR ROWS VS TRANSFORMER</text>
|
||||
</g>
|
||||
|
||||
<g id="table">
|
||||
<rect x="60" y="160" width="1160" height="380" fill="none" stroke="#1A365D" stroke-width="2" />
|
||||
<g id="table-area" transform="translate(60, 160)">
|
||||
<rect x="0" y="0" width="1160" height="60" fill="#1A365D" />
|
||||
<line x1="0" y1="60" x2="1160" y2="60" stroke="#1A365D" stroke-width="2" />
|
||||
<text x="24" y="37" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="800" fill="#FFFFFF" letter-spacing="1.5">MODEL</text>
|
||||
<text x="640" y="37" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="800" fill="#FFFFFF" letter-spacing="1.5">EN → DE BLEU</text>
|
||||
<text x="840" y="37" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="800" fill="#FFFFFF" letter-spacing="1.5">EN → FR BLEU</text>
|
||||
<text x="1040" y="37" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="800" fill="#FFFFFF" letter-spacing="1.5">TRAIN FLOPs</text>
|
||||
|
||||
<line x1="540" y1="0" x2="540" y2="380" stroke="#E2E8F0" stroke-width="1.5" />
|
||||
<line x1="740" y1="0" x2="740" y2="380" stroke="#E2E8F0" stroke-width="1.5" />
|
||||
<line x1="940" y1="0" x2="940" y2="380" stroke="#E2E8F0" stroke-width="1.5" />
|
||||
|
||||
<g id="row-bytenet">
|
||||
<line x1="0" y1="106" x2="1160" y2="106" stroke="#E2E8F0" stroke-width="1" />
|
||||
<text x="24" y="92" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" font-weight="600" fill="#1A202C">ByteNet</text>
|
||||
<text x="640" y="92" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="16" fill="#1A202C">23.75</text>
|
||||
<text x="840" y="92" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="16" fill="#4A5568">—</text>
|
||||
<text x="1040" y="92" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="13" fill="#4A5568">—</text>
|
||||
</g>
|
||||
|
||||
<g id="row-conv-s2s">
|
||||
<line x1="0" y1="152" x2="1160" y2="152" stroke="#E2E8F0" stroke-width="1" />
|
||||
<text x="24" y="138" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" font-weight="600" fill="#1A202C">ConvS2S</text>
|
||||
<text x="640" y="138" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="16" fill="#1A202C">25.16</text>
|
||||
<text x="840" y="138" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="16" fill="#1A202C">40.46</text>
|
||||
<text x="1040" y="138" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="13" fill="#4A5568">9.6 × 10¹⁸</text>
|
||||
</g>
|
||||
|
||||
<g id="row-moe">
|
||||
<line x1="0" y1="198" x2="1160" y2="198" stroke="#E2E8F0" stroke-width="1" />
|
||||
<text x="24" y="184" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" font-weight="600" fill="#1A202C">MoE</text>
|
||||
<text x="640" y="184" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="16" fill="#1A202C">26.03</text>
|
||||
<text x="840" y="184" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="16" fill="#1A202C">40.56</text>
|
||||
<text x="1040" y="184" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="13" fill="#4A5568">2.0 × 10¹⁹</text>
|
||||
</g>
|
||||
|
||||
<g id="row-conv-s2s-ens">
|
||||
<line x1="0" y1="244" x2="1160" y2="244" stroke="#E2E8F0" stroke-width="1" />
|
||||
<text x="24" y="230" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" font-weight="600" fill="#1A202C">ConvS2S Ensemble</text>
|
||||
<text x="640" y="230" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="16" fill="#1A202C">26.36</text>
|
||||
<text x="840" y="230" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="16" fill="#1A202C">41.29</text>
|
||||
<text x="1040" y="230" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="13" fill="#4A5568">7.7 × 10¹⁹</text>
|
||||
</g>
|
||||
|
||||
<g id="row-transformer-base">
|
||||
<rect x="0" y="244" width="1160" height="68" fill="#EBF4FB" />
|
||||
<line x1="0" y1="312" x2="1160" y2="312" stroke="#E2E8F0" stroke-width="1" />
|
||||
<text x="24" y="278" font-family="Georgia, "Microsoft YaHei", serif" font-size="17" font-weight="700" fill="#1A365D">Transformer (base)</text>
|
||||
<text x="24" y="298" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-style="italic" fill="#3182CE">N=6, d_model=512, h=8</text>
|
||||
<text x="640" y="284" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="18" font-weight="700" fill="#1A365D">27.3</text>
|
||||
<text x="840" y="284" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="18" font-weight="700" fill="#1A365D">38.1</text>
|
||||
<text x="1040" y="284" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="14" fill="#3182CE">3.3 × 10¹⁸</text>
|
||||
</g>
|
||||
|
||||
<g id="row-transformer-big">
|
||||
<rect x="0" y="312" width="1160" height="68" fill="#1A365D" fill-opacity="0.06" />
|
||||
<text x="24" y="346" font-family="Georgia, "Microsoft YaHei", serif" font-size="17" font-weight="700" fill="#1A365D">Transformer (big)</text>
|
||||
<text x="24" y="366" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-style="italic" fill="#3182CE">N=6, d_model=1024, h=16</text>
|
||||
<text x="640" y="352" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="22" font-weight="800" fill="#2F855A">28.4</text>
|
||||
<text x="840" y="352" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="22" font-weight="800" fill="#2F855A">41.8</text>
|
||||
<text x="1040" y="352" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="14" fill="#3182CE">2.3 × 10¹⁹</text>
|
||||
</g>
|
||||
</g>
|
||||
</g>
|
||||
|
||||
<g id="takeaway">
|
||||
<path fill="#F5F7FA" stroke="#E2E8F0" stroke-width="1" d="M70,562 H1210 A10,10 0 0 1 1220,572 V662 A10,10 0 0 1 1210,672 H70 A10,10 0 0 1 60,662 V572 A10,10 0 0 1 70,562 Z" />
|
||||
<path fill="#2F855A" d="M63,562 H63 A3,3 0 0 1 66,565 V669 A3,3 0 0 1 63,672 H63 A3,3 0 0 1 60,669 V565 A3,3 0 0 1 63,562 Z" />
|
||||
<text x="100" y="590" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#2F855A" letter-spacing="1.5">THREE RESULTS WORTH NAMING</text>
|
||||
<text x="100" y="620" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">· <tspan font-weight="700" fill="#1A365D">EN-DE</tspan> — Transformer (big) beats the prior best ensemble by <tspan fill="#2F855A" font-weight="700">+2.0 BLEU</tspan>, single model.</text>
|
||||
<text x="100" y="644" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">· <tspan font-weight="700" fill="#1A365D">EN-FR</tspan> — new <tspan fill="#2F855A" font-weight="700">single-model SOTA</tspan> at 41.8 BLEU.</text>
|
||||
<text x="100" y="668" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">· <tspan font-weight="700" fill="#1A365D">Training cost</tspan> — well below most strong baselines; base model trains in 12 hours.</text>
|
||||
</g>
|
||||
|
||||
<g id="footer">
|
||||
<text x="60" y="702" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">Source: Vaswani et al. 2017, Table 2 (§6.1 Machine Translation)</text>
|
||||
<text x="1220" y="702" text-anchor="end" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">14</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 8.6 KiB |
@@ -0,0 +1,89 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1280 720" width="1280" height="720">
|
||||
<defs>
|
||||
<linearGradient id="trackGrad" x1="0%" y1="0%" x2="0%" y2="100%">
|
||||
<stop offset="0%" stop-color="#1A365D" />
|
||||
<stop offset="100%" stop-color="#3182CE" />
|
||||
</linearGradient>
|
||||
<filter id="cardShadow" x="-5%" y="-15%" width="110%" height="135%">
|
||||
<feGaussianBlur in="SourceAlpha" stdDeviation="4" />
|
||||
<feOffset dx="0" dy="2" />
|
||||
<feFlood flood-color="#1A365D" flood-opacity="0.06" />
|
||||
<feComposite in2="SourceAlpha" operator="in" />
|
||||
<feMerge><feMergeNode /><feMergeNode in="SourceGraphic" /></feMerge>
|
||||
</filter>
|
||||
</defs>
|
||||
|
||||
<g id="bg">
|
||||
<rect width="1280" height="720" fill="#FFFFFF" />
|
||||
</g>
|
||||
|
||||
<g id="header">
|
||||
<rect x="60" y="56" width="6" height="40" fill="#3182CE" />
|
||||
<text x="84" y="88" font-family="Georgia, "Microsoft YaHei", serif" font-size="36" font-weight="700" fill="#1A365D">Five takeaways from the variation study</text>
|
||||
<text x="84" y="118" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1.5">TABLE 3 ABLATIONS · WHAT THE ARCHITECTURE NEEDS TO STAY GOOD</text>
|
||||
</g>
|
||||
|
||||
<g id="track">
|
||||
<line x1="160" y1="172" x2="160" y2="660" stroke="url(#trackGrad)" stroke-width="4" stroke-linecap="round" />
|
||||
</g>
|
||||
|
||||
<g id="item-1">
|
||||
<circle cx="160" cy="200" r="28" fill="#1A365D" fill-opacity="0.12" />
|
||||
<circle cx="160" cy="200" r="20" fill="#1A365D" />
|
||||
<text x="160" y="206" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="15" font-weight="700" fill="#FFFFFF">01</text>
|
||||
<line x1="188" y1="200" x2="220" y2="200" stroke="#1A365D" stroke-width="2" stroke-dasharray="4,3" />
|
||||
<path fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)" d="M230,170 H1210 A10,10 0 0 1 1220,180 V224 A10,10 0 0 1 1210,234 H230 A10,10 0 0 1 220,224 V180 A10,10 0 0 1 230,170 Z" />
|
||||
<path fill="#1A365D" d="M223,170 H223 A3,3 0 0 1 226,173 V231 A3,3 0 0 1 223,234 H223 A3,3 0 0 1 220,231 V173 A3,3 0 0 1 223,170 Z" />
|
||||
<text x="246" y="196" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Head count has a sweet spot</text>
|
||||
<text x="246" y="222" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#1A202C">Single head loses <tspan font-family="Consolas, monospace" font-weight="700" fill="#C53030">−0.9 BLEU</tspan> vs h = 8; too many heads (h = 32) also degrade. Eight is empirically right.</text>
|
||||
</g>
|
||||
|
||||
<g id="item-2">
|
||||
<circle cx="160" cy="296" r="28" fill="#1A365D" fill-opacity="0.12" />
|
||||
<circle cx="160" cy="296" r="20" fill="#1A365D" />
|
||||
<text x="160" y="302" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="15" font-weight="700" fill="#FFFFFF">02</text>
|
||||
<line x1="188" y1="296" x2="220" y2="296" stroke="#1A365D" stroke-width="2" stroke-dasharray="4,3" />
|
||||
<path fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)" d="M230,266 H1210 A10,10 0 0 1 1220,276 V320 A10,10 0 0 1 1210,330 H230 A10,10 0 0 1 220,320 V276 A10,10 0 0 1 230,266 Z" />
|
||||
<path fill="#1A365D" d="M223,266 H223 A3,3 0 0 1 226,269 V327 A3,3 0 0 1 223,330 H223 A3,3 0 0 1 220,327 V269 A3,3 0 0 1 223,266 Z" />
|
||||
<text x="246" y="292" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Key dim matters more than value dim</text>
|
||||
<text x="246" y="318" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#1A202C">Reducing <tspan font-family="Consolas, monospace" font-weight="700" fill="#3182CE">d_k</tspan> hurts quality more than reducing <tspan font-family="Consolas, monospace" font-weight="700" fill="#3182CE">d_v</tspan> — determining compatibility is the harder learning problem.</text>
|
||||
</g>
|
||||
|
||||
<g id="item-3">
|
||||
<circle cx="160" cy="392" r="28" fill="#3182CE" fill-opacity="0.12" />
|
||||
<circle cx="160" cy="392" r="20" fill="#3182CE" />
|
||||
<text x="160" y="398" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="15" font-weight="700" fill="#FFFFFF">03</text>
|
||||
<line x1="188" y1="392" x2="220" y2="392" stroke="#3182CE" stroke-width="2" stroke-dasharray="4,3" />
|
||||
<path fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)" d="M230,362 H1210 A10,10 0 0 1 1220,372 V416 A10,10 0 0 1 1210,426 H230 A10,10 0 0 1 220,416 V372 A10,10 0 0 1 230,362 Z" />
|
||||
<path fill="#3182CE" d="M223,362 H223 A3,3 0 0 1 226,365 V423 A3,3 0 0 1 223,426 H223 A3,3 0 0 1 220,423 V365 A3,3 0 0 1 223,362 Z" />
|
||||
<text x="246" y="388" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Bigger is better</text>
|
||||
<text x="246" y="414" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#1A202C">Larger <tspan font-family="Consolas, monospace" font-weight="700" fill="#3182CE">d_model</tspan> and deeper stacks consistently improve BLEU — the architecture scales monotonically with capacity.</text>
|
||||
</g>
|
||||
|
||||
<g id="item-4">
|
||||
<circle cx="160" cy="488" r="28" fill="#3182CE" fill-opacity="0.12" />
|
||||
<circle cx="160" cy="488" r="20" fill="#3182CE" />
|
||||
<text x="160" y="494" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="15" font-weight="700" fill="#FFFFFF">04</text>
|
||||
<line x1="188" y1="488" x2="220" y2="488" stroke="#3182CE" stroke-width="2" stroke-dasharray="4,3" />
|
||||
<path fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)" d="M230,458 H1210 A10,10 0 0 1 1220,468 V512 A10,10 0 0 1 1210,522 H230 A10,10 0 0 1 220,512 V468 A10,10 0 0 1 230,458 Z" />
|
||||
<path fill="#3182CE" d="M223,458 H223 A3,3 0 0 1 226,461 V519 A3,3 0 0 1 223,522 H223 A3,3 0 0 1 220,519 V461 A3,3 0 0 1 223,458 Z" />
|
||||
<text x="246" y="484" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Dropout is essential</text>
|
||||
<text x="246" y="510" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#1A202C">Removing dropout loses <tspan font-family="Consolas, monospace" font-weight="700" fill="#C53030">~0.5–1.0 BLEU</tspan>. Even p = 0.1 measurably regularizes — it isn't decorative.</text>
|
||||
</g>
|
||||
|
||||
<g id="item-5">
|
||||
<circle cx="160" cy="584" r="28" fill="#63B3ED" fill-opacity="0.18" />
|
||||
<circle cx="160" cy="584" r="20" fill="#3182CE" />
|
||||
<text x="160" y="590" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="15" font-weight="700" fill="#FFFFFF">05</text>
|
||||
<line x1="188" y1="584" x2="220" y2="584" stroke="#3182CE" stroke-width="2" stroke-dasharray="4,3" />
|
||||
<path fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)" d="M230,554 H1210 A10,10 0 0 1 1220,564 V608 A10,10 0 0 1 1210,618 H230 A10,10 0 0 1 220,608 V564 A10,10 0 0 1 230,554 Z" />
|
||||
<path fill="#63B3ED" d="M223,554 H223 A3,3 0 0 1 226,557 V615 A3,3 0 0 1 223,618 H223 A3,3 0 0 1 220,615 V557 A3,3 0 0 1 223,554 Z" />
|
||||
<text x="246" y="580" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Label smoothing helps BLEU (and hurts perplexity)</text>
|
||||
<text x="246" y="606" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#1A202C">Slightly worse perplexity, better BLEU. <tspan font-style="italic" fill="#4A5568">Optimize for the metric you ship on.</tspan></text>
|
||||
</g>
|
||||
|
||||
<g id="footer">
|
||||
<text x="60" y="694" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">Source: Vaswani et al. 2017, Table 3 (§6.2 Model Variations)</text>
|
||||
<text x="1220" y="694" text-anchor="end" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">15</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 7.9 KiB |
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 752 KiB |
@@ -0,0 +1,46 @@
|
||||
<?xml version='1.0' encoding='UTF-8'?>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1280 720" width="1280" height="720">
|
||||
<defs>
|
||||
<linearGradient id="coverScrim" x1="0" y1="0" x2="1" y2="0">
|
||||
<stop offset="0%" stop-color="#FFFFFF" stop-opacity="0.96" />
|
||||
<stop offset="42%" stop-color="#FFFFFF" stop-opacity="0.85" />
|
||||
<stop offset="70%" stop-color="#FFFFFF" stop-opacity="0.0" />
|
||||
</linearGradient>
|
||||
</defs>
|
||||
|
||||
<g id="bg">
|
||||
<rect x="0" y="0" width="1280" height="720" fill="#FFFFFF" />
|
||||
<image href="../images/cover_bg.png" x="0" y="0" width="1280" height="720" preserveAspectRatio="xMidYMid slice" />
|
||||
<rect x="0" y="0" width="1280" height="720" fill="url(#coverScrim)" />
|
||||
</g>
|
||||
|
||||
<g id="cover-marks">
|
||||
<rect x="80" y="120" width="80" height="4" fill="#3182CE" />
|
||||
<text x="80" y="160" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#3182CE" letter-spacing="2.5">PAPER · DEEP READ</text>
|
||||
<text x="80" y="184" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-weight="500" fill="#4A5568" letter-spacing="1.5">NeurIPS 2017 · arXiv:1706.03762</text>
|
||||
</g>
|
||||
|
||||
<g id="cover-title">
|
||||
<text x="80" y="320" font-family="Georgia, "Microsoft YaHei", serif" font-size="64" font-weight="700" fill="#1A365D">Attention</text>
|
||||
<text x="80" y="394" font-family="Georgia, "Microsoft YaHei", serif" font-size="64" font-weight="700" fill="#1A365D">Is All</text>
|
||||
<text x="80" y="468" font-family="Georgia, "Microsoft YaHei", serif" font-size="64" font-weight="700" fill="#1A365D">You Need</text>
|
||||
</g>
|
||||
|
||||
<g id="cover-subtitle">
|
||||
<line x1="80" y1="510" x2="180" y2="510" stroke="#1A365D" stroke-width="1.5" />
|
||||
<text x="80" y="540" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-style="italic" fill="#4A5568">Vaswani · Shazeer · Parmar · Uszkoreit</text>
|
||||
<text x="80" y="564" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-style="italic" fill="#4A5568">Jones · Gomez · Kaiser · Polosukhin</text>
|
||||
<text x="80" y="592" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#4A5568">Google Brain · Google Research · University of Toronto</text>
|
||||
</g>
|
||||
|
||||
<g id="cover-takeaway">
|
||||
<rect x="80" y="608" width="6" height="56" fill="#3182CE" />
|
||||
<text x="100" y="630" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" font-weight="700" fill="#1A365D" letter-spacing="1">THE 2017 IDEA</text>
|
||||
<text x="100" y="656" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-style="italic" fill="#1A202C">Drop recurrence and convolution. Use only attention.</text>
|
||||
</g>
|
||||
|
||||
<g id="footer">
|
||||
<text x="1200" y="694" text-anchor="end" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">01</text>
|
||||
<line x1="80" y1="694" x2="120" y2="694" stroke="#A0AEC0" stroke-width="0.5" />
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.0 KiB |
@@ -0,0 +1,90 @@
|
||||
<?xml version='1.0' encoding='UTF-8'?>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1280 720" width="1280" height="720">
|
||||
<defs>
|
||||
<filter id="cardShadow" x="-10%" y="-10%" width="120%" height="120%">
|
||||
<feGaussianBlur in="SourceAlpha" stdDeviation="4" />
|
||||
<feOffset dx="0" dy="3" />
|
||||
<feFlood flood-color="#1A365D" flood-opacity="0.07" />
|
||||
<feComposite in2="SourceAlpha" operator="in" />
|
||||
<feMerge><feMergeNode /><feMergeNode in="SourceGraphic" /></feMerge>
|
||||
</filter>
|
||||
</defs>
|
||||
|
||||
<g id="bg">
|
||||
<rect width="1280" height="720" fill="#FFFFFF" />
|
||||
</g>
|
||||
|
||||
<g id="header">
|
||||
<rect x="60" y="56" width="6" height="40" fill="#3182CE" />
|
||||
<text x="84" y="88" font-family="Georgia, "Microsoft YaHei", serif" font-size="36" font-weight="700" fill="#1A365D">A simpler model, better results, less training</text>
|
||||
<text x="84" y="118" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1.5">WHY THIS PAPER MATTERS · 2017 SOTA ON WMT'14 TRANSLATION</text>
|
||||
</g>
|
||||
|
||||
<g id="kpi-bleu-ende">
|
||||
<rect x="60" y="160" width="285" height="240" rx="12" fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)" />
|
||||
<rect x="60" y="160" width="6" height="240" rx="3" fill="#1A365D" />
|
||||
<use data-icon="tabler-outline/trophy" x="80" y="184" width="32" height="32" fill="#1A365D" />
|
||||
<text x="80" y="246" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1">EN → DE</text>
|
||||
<text x="80" y="270" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#4A5568">BLEU on WMT'14</text>
|
||||
<text x="80" y="336" font-family="Georgia, "Microsoft YaHei", serif" font-size="64" font-weight="700" fill="#1A365D">28.4</text>
|
||||
<use data-icon="tabler-outline/arrow-up-right" x="80" y="354" width="20" height="20" fill="#2F855A" />
|
||||
<text x="106" y="370" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="16" font-weight="700" fill="#2F855A">+2.0</text>
|
||||
<text x="150" y="370" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" fill="#4A5568">over prior ensembles</text>
|
||||
</g>
|
||||
|
||||
<g id="kpi-bleu-enfr">
|
||||
<rect x="365" y="160" width="285" height="240" rx="12" fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)" />
|
||||
<rect x="365" y="160" width="6" height="240" rx="3" fill="#3182CE" />
|
||||
<use data-icon="tabler-outline/trophy" x="385" y="184" width="32" height="32" fill="#3182CE" />
|
||||
<text x="385" y="246" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1">EN → FR</text>
|
||||
<text x="385" y="270" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#4A5568">BLEU on WMT'14</text>
|
||||
<text x="385" y="336" font-family="Georgia, "Microsoft YaHei", serif" font-size="64" font-weight="700" fill="#3182CE">41.8</text>
|
||||
<use data-icon="tabler-outline/star" x="385" y="354" width="20" height="20" fill="#2F855A" />
|
||||
<text x="411" y="370" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#2F855A">new single-model SOTA</text>
|
||||
</g>
|
||||
|
||||
<g id="kpi-training">
|
||||
<rect x="670" y="160" width="285" height="240" rx="12" fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)" />
|
||||
<rect x="670" y="160" width="6" height="240" rx="3" fill="#1A365D" />
|
||||
<use data-icon="tabler-outline/bolt" x="690" y="184" width="32" height="32" fill="#1A365D" />
|
||||
<text x="690" y="246" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1">TRAINING TIME</text>
|
||||
<text x="690" y="270" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#4A5568">Transformer (big)</text>
|
||||
<text x="690" y="336" font-family="Georgia, "Microsoft YaHei", serif" font-size="48" font-weight="700" fill="#1A365D">3.5 days</text>
|
||||
<use data-icon="tabler-outline/cpu" x="690" y="354" width="20" height="20" fill="#4A5568" />
|
||||
<text x="716" y="370" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" fill="#4A5568">on 8× P100 GPUs</text>
|
||||
</g>
|
||||
|
||||
<g id="kpi-no-rnn-cnn">
|
||||
<rect x="975" y="160" width="245" height="240" rx="12" fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)" />
|
||||
<rect x="975" y="160" width="6" height="240" rx="3" fill="#3182CE" />
|
||||
<use data-icon="tabler-outline/eye" x="995" y="184" width="32" height="32" fill="#3182CE" />
|
||||
<text x="995" y="246" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1">ARCHITECTURE</text>
|
||||
<text x="995" y="270" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#4A5568">building blocks</text>
|
||||
<text x="995" y="316" font-family="Georgia, "Microsoft YaHei", serif" font-size="40" font-weight="700" fill="#1A365D">Attention</text>
|
||||
<text x="995" y="346" font-family="Georgia, "Microsoft YaHei", serif" font-size="22" font-style="italic" fill="#4A5568">only</text>
|
||||
<text x="995" y="376" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" fill="#4A5568">no recurrence · no convolution</text>
|
||||
</g>
|
||||
|
||||
<g id="takeaway-band">
|
||||
<rect x="60" y="450" width="1160" height="160" rx="12" fill="#F5F7FA" stroke="#E2E8F0" stroke-width="1" />
|
||||
<rect x="60" y="450" width="6" height="160" rx="3" fill="#1A365D" />
|
||||
<text x="100" y="490" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#3182CE" letter-spacing="1.5">THE HEADLINE</text>
|
||||
<text x="100" y="520" font-family="Georgia, "Microsoft YaHei", serif" font-size="22" font-weight="700" fill="#1A365D">
|
||||
The Transformer reaches <tspan fill="#3182CE">new translation SOTA</tspan> while training
|
||||
</text>
|
||||
<text x="100" y="548" font-family="Georgia, "Microsoft YaHei", serif" font-size="22" font-weight="700" fill="#1A365D">
|
||||
in <tspan fill="#3182CE">a fraction of the time</tspan> of strong baselines.
|
||||
</text>
|
||||
<text x="100" y="580" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="16" fill="#4A5568">
|
||||
Quality up · cost down · parallelization unlocked. The whole rest of the paper explains
|
||||
</text>
|
||||
<text x="100" y="602" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="16" fill="#4A5568">
|
||||
how attention alone makes this possible — and why nothing else is needed.
|
||||
</text>
|
||||
</g>
|
||||
|
||||
<g id="footer">
|
||||
<text x="60" y="694" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">Source: Vaswani et al. 2017, Table 2 (BLEU) and §5.2 (training cost)</text>
|
||||
<text x="1220" y="694" text-anchor="end" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">02</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 7.0 KiB |
@@ -0,0 +1,36 @@
|
||||
<?xml version='1.0' encoding='UTF-8'?>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1280 720" width="1280" height="720">
|
||||
<defs>
|
||||
<linearGradient id="topScrim" x1="0" y1="0" x2="0" y2="1">
|
||||
<stop offset="0%" stop-color="#FFFFFF" stop-opacity="0.95" />
|
||||
<stop offset="100%" stop-color="#FFFFFF" stop-opacity="0" />
|
||||
</linearGradient>
|
||||
</defs>
|
||||
|
||||
<g id="bg">
|
||||
<rect width="1280" height="720" fill="#FFFFFF" />
|
||||
<image href="../images/sequence_evolution.png" x="0" y="0" width="1280" height="720" preserveAspectRatio="xMidYMid slice" />
|
||||
<rect x="0" y="0" width="1280" height="240" fill="url(#topScrim)" />
|
||||
</g>
|
||||
|
||||
<g id="header">
|
||||
<rect x="80" y="68" width="60" height="3" fill="#3182CE" />
|
||||
<text x="80" y="96" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#3182CE" letter-spacing="2.5">PART 2 · MOTIVATION & BACKGROUND</text>
|
||||
<text x="80" y="146" font-family="Georgia, "Microsoft YaHei", serif" font-size="40" font-weight="700" fill="#1A365D">From recurrent chains to a fully-connected</text>
|
||||
<text x="80" y="190" font-family="Georgia, "Microsoft YaHei", serif" font-size="40" font-weight="700" fill="#1A365D">attention <tspan fill="#3182CE">mesh</tspan></text>
|
||||
</g>
|
||||
|
||||
<g id="bottom-takeaway">
|
||||
<rect x="80" y="628" width="1120" height="58" rx="8" fill="#1A365D" fill-opacity="0.92" />
|
||||
<text x="100" y="652" font-family="Georgia, "Microsoft YaHei", serif" font-size="15" font-style="italic" fill="#FFFFFF">
|
||||
RNN — sequential step by step. CNN — local receptive field grows with distance.
|
||||
</text>
|
||||
<text x="100" y="674" font-family="Georgia, "Microsoft YaHei", serif" font-size="15" font-style="italic" fill="#FFFFFF">
|
||||
Self-attention — every position relates to every other in <tspan fill="#63B3ED">one constant-time hop</tspan>.
|
||||
</text>
|
||||
</g>
|
||||
|
||||
<g id="footer">
|
||||
<text x="1220" y="708" text-anchor="end" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#4A5568">03</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 2.1 KiB |
@@ -0,0 +1,86 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1280 720" width="1280" height="720">
|
||||
<defs>
|
||||
<filter id="cardShadow" x="-10%" y="-10%" width="120%" height="120%">
|
||||
<feGaussianBlur in="SourceAlpha" stdDeviation="4"/>
|
||||
<feOffset dx="0" dy="3"/>
|
||||
<feFlood flood-color="#1A365D" flood-opacity="0.06"/>
|
||||
<feComposite in2="SourceAlpha" operator="in"/>
|
||||
<feMerge><feMergeNode/><feMergeNode in="SourceGraphic"/></feMerge>
|
||||
</filter>
|
||||
<clipPath id="rnnImgClip">
|
||||
<rect x="900" y="170" width="320" height="500" rx="14"/>
|
||||
</clipPath>
|
||||
</defs>
|
||||
|
||||
<g id="bg">
|
||||
<rect width="1280" height="720" fill="#FFFFFF"/>
|
||||
</g>
|
||||
|
||||
<g id="header">
|
||||
<rect x="60" y="56" width="6" height="40" fill="#3182CE"/>
|
||||
<text x="84" y="88" font-family="Georgia, "Microsoft YaHei", serif" font-size="36" font-weight="700" fill="#1A365D">What hurts in RNN/CNN sequence models</text>
|
||||
<text x="84" y="118" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1.5">THREE FUNDAMENTAL CONSTRAINTS THE TRANSFORMER LIFTS</text>
|
||||
</g>
|
||||
|
||||
<g id="pillar-sequential">
|
||||
<rect x="60" y="170" width="265" height="500" rx="12" fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)"/>
|
||||
<path d="M 72 170 h 241 a 12 12 0 0 1 12 12 v 96 h -265 v -96 a 12 12 0 0 1 12 -12 Z" fill="#1A365D" fill-opacity="0.06"/>
|
||||
<text x="192" y="218" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="44" font-weight="700" fill="#1A365D">01</text>
|
||||
<text x="192" y="252" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" font-weight="800" fill="#1A365D" letter-spacing="1.5">SEQUENTIAL BOTTLENECK</text>
|
||||
<line x1="80" y1="296" x2="305" y2="296" stroke="#E2E8F0" stroke-width="1.5"/>
|
||||
<use data-icon="tabler-outline/hourglass" x="80" y="316" width="36" height="36" fill="#3182CE"/>
|
||||
<text x="80" y="384" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">hₜ depends on hₜ₋₁</text>
|
||||
<text x="80" y="424" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">Each hidden state must wait for</text>
|
||||
<text x="80" y="446" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">the previous one — no within-</text>
|
||||
<text x="80" y="468" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">example parallelism.</text>
|
||||
<text x="80" y="514" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" font-weight="700" fill="#3182CE" letter-spacing="1">CONSEQUENCE</text>
|
||||
<text x="80" y="542" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">Training step time scales</text>
|
||||
<text x="80" y="564" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">linearly with sequence</text>
|
||||
<text x="80" y="586" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">length n.</text>
|
||||
</g>
|
||||
|
||||
<g id="pillar-pathlen">
|
||||
<rect x="345" y="170" width="265" height="500" rx="12" fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)"/>
|
||||
<path d="M 357 170 h 241 a 12 12 0 0 1 12 12 v 96 h -265 v -96 a 12 12 0 0 1 12 -12 Z" fill="#3182CE" fill-opacity="0.08"/>
|
||||
<text x="477" y="218" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="44" font-weight="700" fill="#3182CE">02</text>
|
||||
<text x="477" y="252" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" font-weight="800" fill="#1A365D" letter-spacing="1.5">LONG-RANGE PATH</text>
|
||||
<line x1="365" y1="296" x2="590" y2="296" stroke="#E2E8F0" stroke-width="1.5"/>
|
||||
<use data-icon="tabler-outline/arrows-horizontal" x="365" y="316" width="36" height="36" fill="#3182CE"/>
|
||||
<text x="365" y="384" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Distant tokens, far apart</text>
|
||||
<text x="365" y="424" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">ConvS2S path grows linearly,</text>
|
||||
<text x="365" y="446" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">ByteNet logarithmically.</text>
|
||||
<text x="365" y="468" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">RNNs are even worse.</text>
|
||||
<text x="365" y="514" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" font-weight="700" fill="#3182CE" letter-spacing="1">CONSEQUENCE</text>
|
||||
<text x="365" y="542" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">Hard to learn dependencies</text>
|
||||
<text x="365" y="564" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">between positions that sit</text>
|
||||
<text x="365" y="586" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">far apart in the input.</text>
|
||||
</g>
|
||||
|
||||
<g id="pillar-memory">
|
||||
<rect x="630" y="170" width="255" height="500" rx="12" fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)"/>
|
||||
<path d="M 642 170 h 231 a 12 12 0 0 1 12 12 v 96 h -255 v -96 a 12 12 0 0 1 12 -12 Z" fill="#1A365D" fill-opacity="0.06"/>
|
||||
<text x="757" y="218" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="44" font-weight="700" fill="#1A365D">03</text>
|
||||
<text x="757" y="252" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" font-weight="800" fill="#1A365D" letter-spacing="1.5">MEMORY CEILING</text>
|
||||
<line x1="650" y1="296" x2="865" y2="296" stroke="#E2E8F0" stroke-width="1.5"/>
|
||||
<use data-icon="tabler-outline/lock" x="650" y="316" width="36" height="36" fill="#3182CE"/>
|
||||
<text x="650" y="384" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Batch size vs length</text>
|
||||
<text x="650" y="424" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">Long sequences exhaust</text>
|
||||
<text x="650" y="446" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">GPU memory, capping</text>
|
||||
<text x="650" y="468" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">batch size across examples.</text>
|
||||
<text x="650" y="514" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" font-weight="700" fill="#3182CE" letter-spacing="1">CONSEQUENCE</text>
|
||||
<text x="650" y="542" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">Throughput drops fast as</text>
|
||||
<text x="650" y="564" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">context grows — a hard wall</text>
|
||||
<text x="650" y="586" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">at long sequence lengths.</text>
|
||||
</g>
|
||||
|
||||
<g id="rnn-visual">
|
||||
<image href="../images/rnn_bottleneck.png" x="900" y="170" width="320" height="500" preserveAspectRatio="xMidYMid slice" clip-path="url(#rnnImgClip)"/>
|
||||
<text x="1060" y="624" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-style="italic" fill="#4A5568">Sequential hidden-state chain</text>
|
||||
<text x="1060" y="644" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-style="italic" fill="#4A5568">— the constraint visualized</text>
|
||||
</g>
|
||||
|
||||
<g id="footer">
|
||||
<text x="60" y="694" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">Source: Vaswani et al. 2017, §1 Introduction & §2 Background</text>
|
||||
<text x="1220" y="694" text-anchor="end" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">04</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 8.1 KiB |
@@ -0,0 +1,74 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1280 720" width="1280" height="720">
|
||||
<defs>
|
||||
<filter id="figFrame" x="-5%" y="-5%" width="110%" height="110%">
|
||||
<feGaussianBlur in="SourceAlpha" stdDeviation="6"/>
|
||||
<feOffset dx="0" dy="3"/>
|
||||
<feFlood flood-color="#1A365D" flood-opacity="0.08"/>
|
||||
<feComposite in2="SourceAlpha" operator="in"/>
|
||||
<feMerge><feMergeNode/><feMergeNode in="SourceGraphic"/></feMerge>
|
||||
</filter>
|
||||
</defs>
|
||||
|
||||
<g id="bg">
|
||||
<rect width="1280" height="720" fill="#FFFFFF"/>
|
||||
</g>
|
||||
|
||||
<g id="header">
|
||||
<rect x="60" y="56" width="6" height="40" fill="#3182CE"/>
|
||||
<text x="84" y="88" font-family="Georgia, "Microsoft YaHei", serif" font-size="34" font-weight="700" fill="#1A365D">Encoder–Decoder, stacked self-attention, point-wise FFN</text>
|
||||
<text x="84" y="118" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1.5">MODEL ARCHITECTURE · FIGURE 1 FROM THE PAPER</text>
|
||||
</g>
|
||||
|
||||
<g id="paper-figure">
|
||||
<rect x="80" y="150" width="382" height="540" rx="12" fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#figFrame)"/>
|
||||
<image href="../images/attention_p3_0.png" x="92" y="162" width="358" height="516" preserveAspectRatio="xMidYMid meet"/>
|
||||
<text x="271" y="710" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" font-style="italic" fill="#4A5568">Figure 1 · The Transformer — model architecture</text>
|
||||
</g>
|
||||
|
||||
<g id="legend-1">
|
||||
<circle cx="510" cy="170" r="16" fill="#1A365D"/>
|
||||
<text x="510" y="176" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="16" font-weight="700" fill="#FFFFFF">1</text>
|
||||
<text x="544" y="166" font-family="Georgia, "Microsoft YaHei", serif" font-size="20" font-weight="700" fill="#1A365D">Inputs + Positional Encoding</text>
|
||||
<text x="544" y="194" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">Token embeddings get a deterministic sinusoidal position vector</text>
|
||||
<text x="544" y="216" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">added — order is injected, not learned (see P11).</text>
|
||||
</g>
|
||||
|
||||
<g id="legend-2">
|
||||
<circle cx="510" cy="262" r="16" fill="#1A365D"/>
|
||||
<text x="510" y="268" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="16" font-weight="700" fill="#FFFFFF">2</text>
|
||||
<text x="544" y="258" font-family="Georgia, "Microsoft YaHei", serif" font-size="20" font-weight="700" fill="#1A365D">Nx Encoder block</text>
|
||||
<text x="544" y="286" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">Multi-Head Self-Attention → Position-wise Feed Forward, both</text>
|
||||
<text x="544" y="308" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">wrapped by Add & Norm. N = 6 in the base configuration.</text>
|
||||
</g>
|
||||
|
||||
<g id="legend-3">
|
||||
<circle cx="510" cy="354" r="16" fill="#3182CE"/>
|
||||
<text x="510" y="360" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="16" font-weight="700" fill="#FFFFFF">3</text>
|
||||
<text x="544" y="350" font-family="Georgia, "Microsoft YaHei", serif" font-size="20" font-weight="700" fill="#1A365D">Nx Decoder block</text>
|
||||
<text x="544" y="378" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">Masked Self-Attn → Encoder-Decoder Attn → FFN, each with</text>
|
||||
<text x="544" y="400" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">Add & Norm. The mask enforces auto-regression.</text>
|
||||
</g>
|
||||
|
||||
<g id="legend-4">
|
||||
<circle cx="510" cy="446" r="16" fill="#1A365D"/>
|
||||
<text x="510" y="452" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="16" font-weight="700" fill="#FFFFFF">4</text>
|
||||
<text x="544" y="442" font-family="Georgia, "Microsoft YaHei", serif" font-size="20" font-weight="700" fill="#1A365D">Add & Norm — residual + LayerNorm</text>
|
||||
<text x="544" y="470" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">Every sub-layer output is LayerNorm(x + Sublayer(x)). Enables</text>
|
||||
<text x="544" y="492" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">deep stacks; d_model = 512 throughout (see P10).</text>
|
||||
</g>
|
||||
|
||||
<g id="legend-5">
|
||||
<circle cx="510" cy="538" r="16" fill="#3182CE"/>
|
||||
<text x="510" y="544" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="16" font-weight="700" fill="#FFFFFF">5</text>
|
||||
<text x="544" y="534" font-family="Georgia, "Microsoft YaHei", serif" font-size="20" font-weight="700" fill="#1A365D">Linear + Softmax — output probabilities</text>
|
||||
<text x="544" y="562" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">Decoder output is projected back to vocabulary size and</text>
|
||||
<text x="544" y="584" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">softmaxed to next-token probabilities.</text>
|
||||
</g>
|
||||
|
||||
<g id="takeaway">
|
||||
<rect x="494" y="618" width="700" height="60" rx="8" fill="#F5F7FA" stroke="#E2E8F0" stroke-width="1"/>
|
||||
<rect x="494" y="618" width="4" height="60" fill="#3182CE"/>
|
||||
<text x="514" y="644" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#3182CE" letter-spacing="1">SAME SKELETON AS PRIOR SEQ2SEQ</text>
|
||||
<text x="514" y="666" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#1A202C">What is new is what fills the boxes: only self-attention + FFN.</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 5.8 KiB |
@@ -0,0 +1,95 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1280 720" width="1280" height="720">
|
||||
<defs>
|
||||
<filter id="cardShadow" x="-10%" y="-10%" width="120%" height="120%">
|
||||
<feGaussianBlur in="SourceAlpha" stdDeviation="6"/>
|
||||
<feOffset dx="0" dy="3"/>
|
||||
<feFlood flood-color="#1A365D" flood-opacity="0.07"/>
|
||||
<feComposite in2="SourceAlpha" operator="in"/>
|
||||
<feMerge><feMergeNode/><feMergeNode in="SourceGraphic"/></feMerge>
|
||||
</filter>
|
||||
</defs>
|
||||
|
||||
<g id="bg">
|
||||
<rect width="1280" height="720" fill="#FFFFFF"/>
|
||||
</g>
|
||||
|
||||
<g id="header">
|
||||
<rect x="60" y="56" width="6" height="40" fill="#3182CE"/>
|
||||
<text x="84" y="88" font-family="Georgia, "Microsoft YaHei", serif" font-size="36" font-weight="700" fill="#1A365D">Same skeleton, three asymmetries</text>
|
||||
<text x="84" y="118" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1.5">ENCODER STACK · DECODER STACK · N = 6 · d_model = 512</text>
|
||||
</g>
|
||||
|
||||
<g id="card-encoder">
|
||||
<rect x="80" y="160" width="555" height="500" rx="14" fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)"/>
|
||||
<path d="M 94 160 h 527 a 14 14 0 0 1 14 14 v 90 h -555 v -90 a 14 14 0 0 1 14 -14 Z" fill="#1A365D"/>
|
||||
<text x="357" y="212" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="28" font-weight="700" fill="#FFFFFF">Encoder</text>
|
||||
<text x="357" y="240" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" font-weight="600" fill="#FFFFFF" fill-opacity="0.85" letter-spacing="1">N = 6 IDENTICAL LAYERS</text>
|
||||
|
||||
<g id="enc-sublayer-1">
|
||||
<rect x="110" y="300" width="495" height="56" rx="8" fill="#F5F7FA" stroke="#1A365D" stroke-width="1.5"/>
|
||||
<circle cx="138" cy="328" r="14" fill="#1A365D"/>
|
||||
<text x="138" y="334" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#FFFFFF">1</text>
|
||||
<text x="166" y="324" font-family="Georgia, "Microsoft YaHei", serif" font-size="17" font-weight="700" fill="#1A365D">Multi-Head Self-Attention</text>
|
||||
<text x="166" y="346" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" fill="#4A5568">Every position attends to every position</text>
|
||||
</g>
|
||||
|
||||
<g id="enc-sublayer-2">
|
||||
<rect x="110" y="372" width="495" height="56" rx="8" fill="#F5F7FA" stroke="#1A365D" stroke-width="1.5"/>
|
||||
<circle cx="138" cy="400" r="14" fill="#1A365D"/>
|
||||
<text x="138" y="406" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#FFFFFF">2</text>
|
||||
<text x="166" y="396" font-family="Georgia, "Microsoft YaHei", serif" font-size="17" font-weight="700" fill="#1A365D">Position-wise Feed Forward</text>
|
||||
<text x="166" y="418" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" fill="#4A5568">Two linear layers, ReLU between</text>
|
||||
</g>
|
||||
|
||||
<text x="110" y="466" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-weight="700" fill="#3182CE" letter-spacing="1.5">WRAPPER AROUND EACH SUB-LAYER</text>
|
||||
<text x="110" y="492" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">Output = LayerNorm( x + Sublayer(x) )</text>
|
||||
<text x="110" y="514" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#4A5568">— residual connection then layer normalization.</text>
|
||||
|
||||
<rect x="110" y="544" width="495" height="80" rx="8" fill="#F5F7FA"/>
|
||||
<text x="124" y="568" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-weight="700" fill="#1A365D" letter-spacing="1.5">SHAPE CONTRACT</text>
|
||||
<text x="124" y="592" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#1A202C">All sub-layers and embeddings emit dimension <tspan font-family="Consolas, monospace" fill="#3182CE" font-weight="700">d_model = 512</tspan></text>
|
||||
<text x="124" y="612" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#1A202C">so residual addition is well-defined.</text>
|
||||
</g>
|
||||
|
||||
<g id="card-decoder">
|
||||
<rect x="655" y="160" width="555" height="500" rx="14" fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)"/>
|
||||
<path d="M 669 160 h 527 a 14 14 0 0 1 14 14 v 90 h -555 v -90 a 14 14 0 0 1 14 -14 Z" fill="#3182CE"/>
|
||||
<text x="932" y="212" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="28" font-weight="700" fill="#FFFFFF">Decoder</text>
|
||||
<text x="932" y="240" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" font-weight="600" fill="#FFFFFF" fill-opacity="0.9" letter-spacing="1">N = 6 IDENTICAL LAYERS · INSERTS A THIRD SUB-LAYER</text>
|
||||
|
||||
<g id="dec-sublayer-1">
|
||||
<rect x="685" y="300" width="495" height="56" rx="8" fill="#FFFFFF" stroke="#3182CE" stroke-width="1.5"/>
|
||||
<circle cx="713" cy="328" r="14" fill="#3182CE"/>
|
||||
<text x="713" y="334" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#FFFFFF">1</text>
|
||||
<text x="741" y="324" font-family="Georgia, "Microsoft YaHei", serif" font-size="17" font-weight="700" fill="#1A365D"><tspan fill="#3182CE">Masked</tspan> Multi-Head Self-Attention</text>
|
||||
<text x="741" y="346" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" fill="#4A5568">Position i attends only to positions ≤ i</text>
|
||||
</g>
|
||||
|
||||
<g id="dec-sublayer-2">
|
||||
<rect x="685" y="372" width="495" height="56" rx="8" fill="#FFFFFF" stroke="#3182CE" stroke-width="2.5"/>
|
||||
<circle cx="713" cy="400" r="14" fill="#3182CE"/>
|
||||
<text x="713" y="406" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#FFFFFF">2</text>
|
||||
<text x="741" y="396" font-family="Georgia, "Microsoft YaHei", serif" font-size="17" font-weight="700" fill="#1A365D"><tspan fill="#3182CE">Encoder-Decoder</tspan> Attention</text>
|
||||
<text x="741" y="418" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" fill="#4A5568">Q from decoder · K, V from encoder output</text>
|
||||
</g>
|
||||
|
||||
<g id="dec-sublayer-3">
|
||||
<rect x="685" y="444" width="495" height="56" rx="8" fill="#F5F7FA" stroke="#3182CE" stroke-width="1.5"/>
|
||||
<circle cx="713" cy="472" r="14" fill="#3182CE"/>
|
||||
<text x="713" y="478" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#FFFFFF">3</text>
|
||||
<text x="741" y="468" font-family="Georgia, "Microsoft YaHei", serif" font-size="17" font-weight="700" fill="#1A365D">Position-wise Feed Forward</text>
|
||||
<text x="741" y="490" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" fill="#4A5568">Same shape as encoder FFN</text>
|
||||
</g>
|
||||
|
||||
<rect x="685" y="528" width="495" height="96" rx="8" fill="#EBF4FB"/>
|
||||
<text x="699" y="552" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-weight="700" fill="#3182CE" letter-spacing="1.5">THREE ASYMMETRIES VS ENCODER</text>
|
||||
<text x="699" y="578" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#1A202C">· third sub-layer attends to encoder output</text>
|
||||
<text x="699" y="598" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#1A202C">· self-attention is masked (no peeking at future)</text>
|
||||
<text x="699" y="618" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#1A202C">· output embeddings are offset by one position</text>
|
||||
</g>
|
||||
|
||||
<g id="footer">
|
||||
<text x="60" y="694" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">Source: Vaswani et al. 2017, §3.1 Encoder and Decoder Stacks</text>
|
||||
<text x="1220" y="694" text-anchor="end" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">06</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 8.1 KiB |
@@ -0,0 +1,124 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1280 720" width="1280" height="720">
|
||||
<defs>
|
||||
<marker id="dpaArrow" markerWidth="10" markerHeight="10" refX="9" refY="5" orient="auto" markerUnits="strokeWidth">
|
||||
<path d="M0,0 L10,5 L0,10 Z" fill="#1A365D"/>
|
||||
</marker>
|
||||
<filter id="cardShadow" x="-10%" y="-10%" width="120%" height="120%">
|
||||
<feGaussianBlur in="SourceAlpha" stdDeviation="4"/>
|
||||
<feOffset dx="0" dy="3"/>
|
||||
<feFlood flood-color="#1A365D" flood-opacity="0.06"/>
|
||||
<feComposite in2="SourceAlpha" operator="in"/>
|
||||
<feMerge><feMergeNode/><feMergeNode in="SourceGraphic"/></feMerge>
|
||||
</filter>
|
||||
</defs>
|
||||
|
||||
<g id="bg">
|
||||
<rect width="1280" height="720" fill="#FFFFFF"/>
|
||||
</g>
|
||||
|
||||
<g id="header">
|
||||
<rect x="60" y="56" width="6" height="40" fill="#3182CE"/>
|
||||
<text x="84" y="88" font-family="Georgia, "Microsoft YaHei", serif" font-size="34" font-weight="700" fill="#1A365D">Attention as a weighted average — keys, values, queries</text>
|
||||
<text x="84" y="118" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1.5">SCALED DOT-PRODUCT ATTENTION · §3.2.1</text>
|
||||
</g>
|
||||
|
||||
<g id="formula-band">
|
||||
<rect x="60" y="160" width="1160" height="120" rx="12" fill="#F5F7FA" stroke="#E2E8F0" stroke-width="1.5"/>
|
||||
<rect x="60" y="160" width="6" height="120" rx="3" fill="#3182CE"/>
|
||||
<text x="100" y="186" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-weight="700" fill="#3182CE" letter-spacing="1.5">EQUATION (1)</text>
|
||||
<image href="../images/formula_001.png" x="270" y="194" width="640" height="89" preserveAspectRatio="xMidYMid meet"/>
|
||||
</g>
|
||||
|
||||
<g id="explanation">
|
||||
<text x="84" y="328" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#3182CE" letter-spacing="1.5">HOW IT READS</text>
|
||||
|
||||
<g id="step-1">
|
||||
<circle cx="100" cy="370" r="18" fill="#1A365D"/>
|
||||
<text x="100" y="376" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="16" font-weight="700" fill="#FFFFFF">1</text>
|
||||
<text x="132" y="364" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Dot-product all queries with all keys</text>
|
||||
<text x="132" y="388" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">QKᵀ gives a similarity score between every query–key pair.</text>
|
||||
</g>
|
||||
|
||||
<g id="step-2">
|
||||
<circle cx="100" cy="438" r="18" fill="#1A365D"/>
|
||||
<text x="100" y="444" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="16" font-weight="700" fill="#FFFFFF">2</text>
|
||||
<text x="132" y="432" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Scale by <tspan font-family="Consolas, monospace" fill="#3182CE">√d_k</tspan></text>
|
||||
<text x="132" y="456" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">For large d_k the dot products grow large, pushing softmax</text>
|
||||
<text x="132" y="476" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">into low-gradient regions. Scaling counteracts that.</text>
|
||||
</g>
|
||||
|
||||
<g id="step-3">
|
||||
<circle cx="100" cy="528" r="18" fill="#1A365D"/>
|
||||
<text x="100" y="534" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="16" font-weight="700" fill="#FFFFFF">3</text>
|
||||
<text x="132" y="522" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Softmax → attention weights</text>
|
||||
<text x="132" y="546" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">Normalize each row so the weights over keys sum to 1.</text>
|
||||
</g>
|
||||
|
||||
<g id="step-4">
|
||||
<circle cx="100" cy="598" r="18" fill="#3182CE"/>
|
||||
<text x="100" y="604" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="16" font-weight="700" fill="#FFFFFF">4</text>
|
||||
<text x="132" y="592" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Weighted sum of values</text>
|
||||
<text x="132" y="616" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">Multiply weights by V to get each output position's vector.</text>
|
||||
</g>
|
||||
</g>
|
||||
|
||||
<g id="flow-diagram">
|
||||
<rect x="700" y="316" width="520" height="324" rx="12" fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)"/>
|
||||
<text x="720" y="342" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-weight="700" fill="#3182CE" letter-spacing="1.5">DATAFLOW</text>
|
||||
|
||||
<g id="op-q">
|
||||
<text x="740" y="386" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="20" font-weight="700" fill="#1A365D">Q</text>
|
||||
<text x="740" y="406" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#4A5568">queries</text>
|
||||
</g>
|
||||
<g id="op-k">
|
||||
<text x="740" y="448" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="20" font-weight="700" fill="#1A365D">K</text>
|
||||
<text x="740" y="468" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#4A5568">keys</text>
|
||||
</g>
|
||||
<g id="op-v">
|
||||
<text x="740" y="610" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="20" font-weight="700" fill="#1A365D">V</text>
|
||||
<text x="740" y="628" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#4A5568">values</text>
|
||||
</g>
|
||||
|
||||
<g id="step-matmul1">
|
||||
<rect x="780" y="370" width="120" height="60" rx="8" fill="#EBF4FB" stroke="#3182CE" stroke-width="1.5"/>
|
||||
<text x="840" y="396" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" font-weight="700" fill="#1A365D">MatMul</text>
|
||||
<text x="840" y="418" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="12" fill="#4A5568">QKᵀ</text>
|
||||
</g>
|
||||
<line x1="755" y1="380" x2="780" y2="396" stroke="#1A365D" stroke-width="1.5" marker-end="url(#dpaArrow)"/>
|
||||
<line x1="755" y1="442" x2="780" y2="408" stroke="#1A365D" stroke-width="1.5" marker-end="url(#dpaArrow)"/>
|
||||
|
||||
<g id="step-scale">
|
||||
<rect x="920" y="370" width="100" height="60" rx="8" fill="#EBF4FB" stroke="#3182CE" stroke-width="1.5"/>
|
||||
<text x="970" y="394" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" font-weight="700" fill="#1A365D">Scale</text>
|
||||
<text x="970" y="416" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="12" fill="#4A5568">÷ √d_k</text>
|
||||
</g>
|
||||
<line x1="900" y1="400" x2="920" y2="400" stroke="#1A365D" stroke-width="1.5" marker-end="url(#dpaArrow)"/>
|
||||
|
||||
<g id="step-mask">
|
||||
<rect x="1040" y="370" width="160" height="60" rx="8" fill="#FFFFFF" stroke="#A0AEC0" stroke-width="1.5" stroke-dasharray="4,3"/>
|
||||
<text x="1120" y="394" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" font-weight="700" fill="#4A5568">Mask (opt.)</text>
|
||||
<text x="1120" y="416" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">decoder only</text>
|
||||
</g>
|
||||
<line x1="1020" y1="400" x2="1040" y2="400" stroke="#1A365D" stroke-width="1.5" marker-end="url(#dpaArrow)"/>
|
||||
|
||||
<g id="step-softmax">
|
||||
<rect x="900" y="464" width="200" height="60" rx="8" fill="#3182CE"/>
|
||||
<text x="1000" y="488" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" font-weight="700" fill="#FFFFFF">SoftMax</text>
|
||||
<text x="1000" y="510" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#FFFFFF" fill-opacity="0.9">row-wise normalize</text>
|
||||
</g>
|
||||
<line x1="1120" y1="430" x2="1040" y2="464" stroke="#1A365D" stroke-width="1.5" marker-end="url(#dpaArrow)"/>
|
||||
|
||||
<g id="step-matmul2">
|
||||
<rect x="900" y="560" width="200" height="60" rx="8" fill="#1A365D"/>
|
||||
<text x="1000" y="584" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" font-weight="700" fill="#FFFFFF">MatMul</text>
|
||||
<text x="1000" y="606" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="11" fill="#FFFFFF" fill-opacity="0.9">× V → output</text>
|
||||
</g>
|
||||
<line x1="1000" y1="524" x2="1000" y2="560" stroke="#1A365D" stroke-width="1.5" marker-end="url(#dpaArrow)"/>
|
||||
<line x1="755" y1="608" x2="900" y2="590" stroke="#1A365D" stroke-width="1.5" marker-end="url(#dpaArrow)"/>
|
||||
</g>
|
||||
|
||||
<g id="footer">
|
||||
<text x="60" y="694" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">Source: Vaswani et al. 2017, §3.2.1 Scaled Dot-Product Attention</text>
|
||||
<text x="1220" y="694" text-anchor="end" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">07</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 9.3 KiB |
@@ -0,0 +1,70 @@
|
||||
<?xml version='1.0' encoding='UTF-8'?>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1280 720" width="1280" height="720">
|
||||
<defs>
|
||||
<filter id="figFrame" x="-5%" y="-5%" width="110%" height="110%">
|
||||
<feGaussianBlur in="SourceAlpha" stdDeviation="6" />
|
||||
<feOffset dx="0" dy="3" />
|
||||
<feFlood flood-color="#1A365D" flood-opacity="0.08" />
|
||||
<feComposite in2="SourceAlpha" operator="in" />
|
||||
<feMerge><feMergeNode /><feMergeNode in="SourceGraphic" /></feMerge>
|
||||
</filter>
|
||||
</defs>
|
||||
|
||||
<g id="bg">
|
||||
<rect width="1280" height="720" fill="#FFFFFF" />
|
||||
</g>
|
||||
|
||||
<g id="header">
|
||||
<rect x="60" y="56" width="6" height="40" fill="#3182CE" />
|
||||
<text x="84" y="88" font-family="Georgia, "Microsoft YaHei", serif" font-size="36" font-weight="700" fill="#1A365D">h parallel projections, then concatenate</text>
|
||||
<text x="84" y="118" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1.5">MULTI-HEAD ATTENTION · §3.2.2 · h = 8 · d_k = d_v = 64</text>
|
||||
</g>
|
||||
|
||||
<g id="formula-band">
|
||||
<rect x="60" y="160" width="1160" height="124" rx="12" fill="#F5F7FA" stroke="#E2E8F0" stroke-width="1.5" />
|
||||
<rect x="60" y="160" width="6" height="124" rx="3" fill="#3182CE" />
|
||||
<text x="100" y="186" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-weight="700" fill="#3182CE" letter-spacing="1.5">EQUATION</text>
|
||||
<image href="../images/formula_002.png" x="200" y="192" width="780" height="92" preserveAspectRatio="xMidYMid meet" />
|
||||
</g>
|
||||
|
||||
<g id="paper-figure">
|
||||
<rect x="80" y="316" width="380" height="350" rx="12" fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#figFrame)" />
|
||||
<image href="../images/attention_p4_1.png" x="92" y="328" width="356" height="326" preserveAspectRatio="xMidYMid meet" />
|
||||
<text x="270" y="662" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" font-style="italic" fill="#4A5568">Figure 2 (right) · Multi-Head Attention</text>
|
||||
</g>
|
||||
|
||||
<g id="explanation">
|
||||
<text x="500" y="350" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#3182CE" letter-spacing="1.5">WHY MULTI-HEAD INSTEAD OF SINGLE</text>
|
||||
|
||||
<g id="point-1">
|
||||
<rect x="500" y="370" width="6" height="64" fill="#1A365D" />
|
||||
<text x="522" y="392" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Project Q, K, V into h subspaces</text>
|
||||
<text x="522" y="416" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">Independent learned linear projections W_iᴽ, W_iᴷ, W_iⱽ — one</text>
|
||||
<text x="522" y="436" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">set per head; each head sees a different view of the inputs.</text>
|
||||
</g>
|
||||
|
||||
<g id="point-2">
|
||||
<rect x="500" y="450" width="6" height="64" fill="#1A365D" />
|
||||
<text x="522" y="472" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Run scaled DPA in each head in parallel</text>
|
||||
<text x="522" y="496" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">d_k = d_v = d_model / h = 64. Total compute stays comparable</text>
|
||||
<text x="522" y="516" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">to a single full-dimensional head.</text>
|
||||
</g>
|
||||
|
||||
<g id="point-3">
|
||||
<rect x="500" y="530" width="6" height="64" fill="#1A365D" />
|
||||
<text x="522" y="552" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Concat heads, project with Wᴼ</text>
|
||||
<text x="522" y="576" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">Concatenated head outputs go through a final linear map back</text>
|
||||
<text x="522" y="596" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">to d_model.</text>
|
||||
</g>
|
||||
|
||||
<rect x="500" y="612" width="710" height="56" rx="8" fill="#EBF4FB" />
|
||||
<rect x="500" y="612" width="4" height="56" fill="#3182CE" />
|
||||
<text x="516" y="636" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#3182CE" letter-spacing="1">THE PAYOFF</text>
|
||||
<text x="516" y="656" font-family="Georgia, "Microsoft YaHei", serif" font-size="15" fill="#1A202C" font-style="italic">Each head can attend to different representation subspaces at different positions.</text>
|
||||
</g>
|
||||
|
||||
<g id="footer">
|
||||
<text x="60" y="694" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">Source: Vaswani et al. 2017, §3.2.2 Multi-Head Attention</text>
|
||||
<text x="1220" y="694" text-anchor="end" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">08</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 5.0 KiB |
@@ -0,0 +1,102 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1280 720" width="1280" height="720">
|
||||
<defs>
|
||||
<filter id="cardShadow" x="-10%" y="-10%" width="120%" height="120%">
|
||||
<feGaussianBlur in="SourceAlpha" stdDeviation="6"/>
|
||||
<feOffset dx="0" dy="3"/>
|
||||
<feFlood flood-color="#1A365D" flood-opacity="0.07"/>
|
||||
<feComposite in2="SourceAlpha" operator="in"/>
|
||||
<feMerge><feMergeNode/><feMergeNode in="SourceGraphic"/></feMerge>
|
||||
</filter>
|
||||
</defs>
|
||||
|
||||
<g id="bg">
|
||||
<rect width="1280" height="720" fill="#FFFFFF"/>
|
||||
</g>
|
||||
|
||||
<g id="header">
|
||||
<rect x="60" y="56" width="6" height="40" fill="#3182CE"/>
|
||||
<text x="84" y="88" font-family="Georgia, "Microsoft YaHei", serif" font-size="36" font-weight="700" fill="#1A365D">One mechanism, three roles in the Transformer</text>
|
||||
<text x="84" y="118" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1.5">WHERE ATTENTION SHOWS UP IN THE ARCHITECTURE · §3.2.3</text>
|
||||
</g>
|
||||
|
||||
<g id="pillar-enc-self">
|
||||
<rect x="60" y="160" width="380" height="500" rx="14" fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)"/>
|
||||
<path d="M 74 160 h 352 a 14 14 0 0 1 14 14 v 100 h -380 v -100 a 14 14 0 0 1 14 -14 Z" fill="#1A365D"/>
|
||||
<use data-icon="tabler-outline/arrow-back" x="80" y="180" width="32" height="32" fill="#FFFFFF"/>
|
||||
<text x="250" y="208" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" font-weight="700" fill="#FFFFFF" fill-opacity="0.85" letter-spacing="2">ENCODER</text>
|
||||
<text x="250" y="240" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="22" font-weight="700" fill="#FFFFFF">Self-Attention</text>
|
||||
|
||||
<text x="80" y="312" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-weight="700" fill="#3182CE" letter-spacing="1.5">Q · K · V SOURCE</text>
|
||||
<rect x="80" y="324" width="340" height="48" rx="6" fill="#F5F7FA"/>
|
||||
<text x="98" y="354" font-family="Consolas, "Courier New", monospace" font-size="14" fill="#1A202C">Q, K, V ← previous encoder layer</text>
|
||||
|
||||
<text x="80" y="406" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Every position attends to all</text>
|
||||
<text x="80" y="442" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">Every token in the source sequence can</text>
|
||||
<text x="80" y="464" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">attend to every other token — including</text>
|
||||
<text x="80" y="486" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">itself.</text>
|
||||
|
||||
<text x="80" y="534" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-weight="700" fill="#3182CE" letter-spacing="1.5">WHAT IT'S FOR</text>
|
||||
<text x="80" y="560" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">Build rich contextualized</text>
|
||||
<text x="80" y="582" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">representations of each source</text>
|
||||
<text x="80" y="604" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">token from its surroundings.</text>
|
||||
|
||||
<rect x="80" y="624" width="340" height="20" rx="4" fill="#1A365D" fill-opacity="0.08"/>
|
||||
<text x="92" y="640" font-family="Consolas, "Courier New", monospace" font-size="12" fill="#1A365D">attention_mask = none</text>
|
||||
</g>
|
||||
|
||||
<g id="pillar-dec-masked">
|
||||
<rect x="450" y="160" width="380" height="500" rx="14" fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)"/>
|
||||
<path d="M 464 160 h 352 a 14 14 0 0 1 14 14 v 100 h -380 v -100 a 14 14 0 0 1 14 -14 Z" fill="#3182CE"/>
|
||||
<use data-icon="tabler-outline/lock" x="470" y="180" width="32" height="32" fill="#FFFFFF"/>
|
||||
<text x="640" y="208" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" font-weight="700" fill="#FFFFFF" fill-opacity="0.9" letter-spacing="2">DECODER</text>
|
||||
<text x="640" y="240" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="22" font-weight="700" fill="#FFFFFF">Masked Self-Attention</text>
|
||||
|
||||
<text x="470" y="312" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-weight="700" fill="#3182CE" letter-spacing="1.5">Q · K · V SOURCE</text>
|
||||
<rect x="470" y="324" width="340" height="48" rx="6" fill="#F5F7FA"/>
|
||||
<text x="488" y="354" font-family="Consolas, "Courier New", monospace" font-size="14" fill="#1A202C">Q, K, V ← previous decoder layer</text>
|
||||
|
||||
<text x="470" y="406" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Only past, never future</text>
|
||||
<text x="470" y="442" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">Position i attends only to positions ≤ i.</text>
|
||||
<text x="470" y="464" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">Achieved by setting illegal positions to</text>
|
||||
<text x="470" y="486" font-family="Consolas, "Courier New", monospace" font-size="14" fill="#1A202C">−∞ before the softmax.</text>
|
||||
|
||||
<text x="470" y="534" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-weight="700" fill="#3182CE" letter-spacing="1.5">WHAT IT'S FOR</text>
|
||||
<text x="470" y="560" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">Preserve auto-regression — the</text>
|
||||
<text x="470" y="582" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">model cannot peek at tokens it</text>
|
||||
<text x="470" y="604" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">hasn't produced yet.</text>
|
||||
|
||||
<rect x="470" y="624" width="340" height="20" rx="4" fill="#3182CE" fill-opacity="0.1"/>
|
||||
<text x="482" y="640" font-family="Consolas, "Courier New", monospace" font-size="12" fill="#3182CE">attention_mask = causal triangular</text>
|
||||
</g>
|
||||
|
||||
<g id="pillar-cross">
|
||||
<rect x="840" y="160" width="380" height="500" rx="14" fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)"/>
|
||||
<path d="M 854 160 h 352 a 14 14 0 0 1 14 14 v 100 h -380 v -100 a 14 14 0 0 1 14 -14 Z" fill="#1A365D"/>
|
||||
<use data-icon="tabler-outline/arrows-left-right" x="860" y="180" width="32" height="32" fill="#FFFFFF"/>
|
||||
<text x="1030" y="208" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" font-weight="700" fill="#FFFFFF" fill-opacity="0.85" letter-spacing="2">ENCODER ↔ DECODER</text>
|
||||
<text x="1030" y="240" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="22" font-weight="700" fill="#FFFFFF">Cross-Attention</text>
|
||||
|
||||
<text x="860" y="312" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-weight="700" fill="#3182CE" letter-spacing="1.5">Q · K · V SOURCE</text>
|
||||
<rect x="860" y="324" width="340" height="48" rx="6" fill="#F5F7FA"/>
|
||||
<text x="878" y="346" font-family="Consolas, "Courier New", monospace" font-size="13" fill="#1A202C">Q ← previous decoder layer</text>
|
||||
<text x="878" y="364" font-family="Consolas, "Courier New", monospace" font-size="13" fill="#3182CE">K, V ← encoder output</text>
|
||||
|
||||
<text x="860" y="406" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Look back at the input</text>
|
||||
<text x="860" y="442" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">Every decoder position can attend over</text>
|
||||
<text x="860" y="464" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">all positions of the input sequence —</text>
|
||||
<text x="860" y="486" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">the classic seq2seq attention role.</text>
|
||||
|
||||
<text x="860" y="534" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-weight="700" fill="#3182CE" letter-spacing="1.5">WHAT IT'S FOR</text>
|
||||
<text x="860" y="560" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">Let each output token retrieve</text>
|
||||
<text x="860" y="582" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">the relevant source information,</text>
|
||||
<text x="860" y="604" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">no matter where it sits.</text>
|
||||
|
||||
<rect x="860" y="624" width="340" height="20" rx="4" fill="#1A365D" fill-opacity="0.08"/>
|
||||
<text x="872" y="640" font-family="Consolas, "Courier New", monospace" font-size="12" fill="#1A365D">attention_mask = none</text>
|
||||
</g>
|
||||
|
||||
<g id="footer">
|
||||
<text x="60" y="694" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">Source: Vaswani et al. 2017, §3.2.3 Applications of Attention</text>
|
||||
<text x="1220" y="694" text-anchor="end" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">09</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 9.4 KiB |
@@ -0,0 +1,106 @@
|
||||
<?xml version='1.0' encoding='UTF-8'?>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1280 720" width="1280" height="720">
|
||||
<defs>
|
||||
<filter id="cardShadow" x="-10%" y="-10%" width="120%" height="120%">
|
||||
<feGaussianBlur in="SourceAlpha" stdDeviation="4" />
|
||||
<feOffset dx="0" dy="3" />
|
||||
<feFlood flood-color="#1A365D" flood-opacity="0.06" />
|
||||
<feComposite in2="SourceAlpha" operator="in" />
|
||||
<feMerge><feMergeNode /><feMergeNode in="SourceGraphic" /></feMerge>
|
||||
</filter>
|
||||
<clipPath id="sublayerImgClip">
|
||||
<rect x="60" y="160" width="340" height="500" rx="14" />
|
||||
</clipPath>
|
||||
</defs>
|
||||
|
||||
<g id="bg">
|
||||
<rect width="1280" height="720" fill="#FFFFFF" />
|
||||
</g>
|
||||
|
||||
<g id="header">
|
||||
<rect x="60" y="56" width="6" height="40" fill="#3182CE" />
|
||||
<text x="84" y="88" font-family="Georgia, "Microsoft YaHei", serif" font-size="36" font-weight="700" fill="#1A365D">The supporting cast inside every layer</text>
|
||||
<text x="84" y="118" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1.5">POSITION-WISE FFN · RESIDUAL CONNECTION · LAYERNORM</text>
|
||||
</g>
|
||||
|
||||
<g id="sublayer-visual">
|
||||
<rect x="60" y="160" width="340" height="500" rx="14" fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)" />
|
||||
<image href="../images/sublayer_block.png" x="60" y="160" width="340" height="500" preserveAspectRatio="xMidYMid slice" clip-path="url(#sublayerImgClip)" />
|
||||
<text x="230" y="690" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-style="italic" fill="#4A5568">One layer block · two sub-layers · residual arcs</text>
|
||||
</g>
|
||||
|
||||
<g id="parent-frame">
|
||||
<rect x="430" y="160" width="790" height="500" rx="12" fill="#F5F7FA" fill-opacity="0.5" stroke="#3182CE" stroke-width="1.5" stroke-dasharray="6,4" />
|
||||
<text x="825" y="184" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-weight="700" fill="#3182CE" letter-spacing="1.5">ONE TRANSFORMER LAYER · ENCODER OR DECODER</text>
|
||||
</g>
|
||||
|
||||
<g id="card-ffn">
|
||||
<rect x="450" y="208" width="240" height="200" rx="10" fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)" />
|
||||
<rect x="450" y="208" width="6" height="200" rx="3" fill="#1A365D" />
|
||||
<use data-icon="tabler-outline/stack-2" x="470" y="228" width="28" height="28" fill="#1A365D" />
|
||||
<text x="506" y="250" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Position-wise FFN</text>
|
||||
<text x="470" y="284" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#1A202C">Two linear layers with ReLU</text>
|
||||
<text x="470" y="304" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#1A202C">between, applied identically</text>
|
||||
<text x="470" y="324" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#1A202C">at every position.</text>
|
||||
<rect x="470" y="346" width="206" height="48" rx="6" fill="#F5F7FA" />
|
||||
<text x="482" y="366" font-family="Consolas, "Courier New", monospace" font-size="11" fill="#1A202C">FFN(x) = max(0, xW₁+b₁) W₂+b₂</text>
|
||||
<text x="482" y="386" font-family="Consolas, "Courier New", monospace" font-size="11" fill="#3182CE">inner dim = 2048 · outer = 512</text>
|
||||
</g>
|
||||
|
||||
<g id="card-residual">
|
||||
<rect x="710" y="208" width="240" height="200" rx="10" fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)" />
|
||||
<rect x="710" y="208" width="6" height="200" rx="3" fill="#3182CE" />
|
||||
<use data-icon="tabler-outline/arrow-loop-right" x="730" y="228" width="28" height="28" fill="#3182CE" />
|
||||
<text x="766" y="250" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Residual</text>
|
||||
<text x="730" y="284" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#1A202C">Skip the sub-layer, add the</text>
|
||||
<text x="730" y="304" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#1A202C">input back to the output.</text>
|
||||
<text x="730" y="324" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#1A202C">Enables deep stacks.</text>
|
||||
<rect x="730" y="346" width="206" height="48" rx="6" fill="#EBF4FB" />
|
||||
<text x="742" y="368" font-family="Consolas, "Courier New", monospace" font-size="13" fill="#1A202C">y = x + Sublayer(x)</text>
|
||||
<text x="742" y="388" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" fill="#3182CE" font-style="italic">straight gradient path top → bottom</text>
|
||||
</g>
|
||||
|
||||
<g id="card-layernorm">
|
||||
<rect x="970" y="208" width="240" height="200" rx="10" fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)" />
|
||||
<rect x="970" y="208" width="6" height="200" rx="3" fill="#1A365D" />
|
||||
<use data-icon="tabler-outline/wave-square" x="990" y="228" width="28" height="28" fill="#1A365D" />
|
||||
<text x="1026" y="250" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">LayerNorm</text>
|
||||
<text x="990" y="284" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#1A202C">Normalize across feature</text>
|
||||
<text x="990" y="304" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#1A202C">dimension after the residual</text>
|
||||
<text x="990" y="324" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#1A202C">addition.</text>
|
||||
<rect x="990" y="346" width="206" height="48" rx="6" fill="#F5F7FA" />
|
||||
<text x="1002" y="366" font-family="Consolas, "Courier New", monospace" font-size="11" fill="#1A202C">LayerNorm(x + Sublayer(x))</text>
|
||||
<text x="1002" y="386" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#3182CE" font-style="italic">stabilizes training, scale-invariant</text>
|
||||
</g>
|
||||
|
||||
<g id="combined-flow">
|
||||
<rect x="450" y="436" width="760" height="200" rx="12" fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" />
|
||||
<rect x="450" y="436" width="6" height="200" rx="3" fill="#3182CE" />
|
||||
<text x="470" y="464" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-weight="700" fill="#3182CE" letter-spacing="1.5">HOW THEY COMPOSE — APPLIED TO EVERY SUB-LAYER</text>
|
||||
|
||||
<rect x="478" y="494" width="160" height="48" rx="6" fill="#F5F7FA" stroke="#1A365D" stroke-width="1.5" />
|
||||
<text x="558" y="524" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="16" font-weight="700" fill="#1A365D">Sublayer(x)</text>
|
||||
|
||||
<text x="660" y="524" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="20" font-weight="700" fill="#3182CE">+</text>
|
||||
|
||||
<rect x="690" y="494" width="80" height="48" rx="6" fill="#EBF4FB" stroke="#3182CE" stroke-width="1.5" />
|
||||
<text x="730" y="524" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="16" font-weight="700" fill="#3182CE">x</text>
|
||||
|
||||
<text x="800" y="524" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="18" font-weight="700" fill="#1A365D">→</text>
|
||||
|
||||
<rect x="830" y="486" width="220" height="64" rx="6" fill="#1A365D" />
|
||||
<text x="940" y="514" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="16" font-weight="700" fill="#FFFFFF">LayerNorm</text>
|
||||
<text x="940" y="536" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" fill="#FFFFFF" fill-opacity="0.9">final sub-layer output</text>
|
||||
|
||||
<text x="1080" y="524" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="18" font-weight="700" fill="#1A365D">→</text>
|
||||
<text x="1118" y="528" font-family="Consolas, "Courier New", monospace" font-size="14" font-weight="700" fill="#1A365D">next</text>
|
||||
|
||||
<text x="470" y="586" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-weight="700" fill="#3182CE" letter-spacing="1.5">SHAPE CONTRACT</text>
|
||||
<text x="470" y="610" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#1A202C">All sub-layers and embeddings emit dimension <tspan font-family="Consolas, monospace" font-weight="700" fill="#3182CE">d_model = 512</tspan> so the residual addition is well-defined.</text>
|
||||
</g>
|
||||
|
||||
<g id="footer">
|
||||
<text x="60" y="694" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">Source: Vaswani et al. 2017, §3.1 + §3.3 Position-wise Feed-Forward Networks</text>
|
||||
<text x="1220" y="694" text-anchor="end" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">10</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 8.9 KiB |
@@ -0,0 +1,50 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1280 720" width="1280" height="720">
|
||||
<defs>
|
||||
<linearGradient id="peLeftScrim" x1="0" y1="0" x2="1" y2="0">
|
||||
<stop offset="0%" stop-color="#FFFFFF" stop-opacity="0.96"/>
|
||||
<stop offset="45%" stop-color="#FFFFFF" stop-opacity="0.78"/>
|
||||
<stop offset="80%" stop-color="#FFFFFF" stop-opacity="0"/>
|
||||
</linearGradient>
|
||||
</defs>
|
||||
|
||||
<g id="bg">
|
||||
<rect width="1280" height="720" fill="#FFFFFF"/>
|
||||
<image href="../images/positional_encoding.png" x="0" y="0" width="1280" height="720" preserveAspectRatio="xMidYMid slice"/>
|
||||
<rect x="0" y="0" width="1280" height="720" fill="url(#peLeftScrim)"/>
|
||||
</g>
|
||||
|
||||
<g id="header">
|
||||
<rect x="80" y="80" width="60" height="3" fill="#3182CE"/>
|
||||
<text x="80" y="108" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#3182CE" letter-spacing="2.5">§3.5 · POSITIONAL ENCODING</text>
|
||||
</g>
|
||||
|
||||
<g id="title">
|
||||
<text x="80" y="166" font-family="Georgia, "Microsoft YaHei", serif" font-size="38" font-weight="700" fill="#1A365D">Recovering order</text>
|
||||
<text x="80" y="208" font-family="Georgia, "Microsoft YaHei", serif" font-size="38" font-weight="700" fill="#1A365D">without recurrence</text>
|
||||
</g>
|
||||
|
||||
<g id="explanation">
|
||||
<line x1="80" y1="252" x2="120" y2="252" stroke="#3182CE" stroke-width="1.5"/>
|
||||
<text x="80" y="286" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">No recurrence and no convolution means the model has</text>
|
||||
<text x="80" y="308" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">no built-in notion of token order. Add a deterministic</text>
|
||||
<text x="80" y="330" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">sinusoidal position vector to each input embedding —</text>
|
||||
<text x="80" y="352" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">order is injected, not learned.</text>
|
||||
|
||||
<text x="80" y="402" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-weight="700" fill="#3182CE" letter-spacing="1.5">WHY SINUSOIDS</text>
|
||||
<text x="80" y="426" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">Wavelengths form a geometric progression</text>
|
||||
<text x="80" y="448" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">from 2π up to 10000·2π. For any fixed offset k,</text>
|
||||
<text x="80" y="470" font-family="Consolas, "Courier New", monospace" font-size="14" fill="#1A202C">PE(pos + k)</text>
|
||||
<text x="80" y="492" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">is a linear function of <tspan font-family="Consolas, monospace" font-weight="700" fill="#3182CE">PE(pos)</tspan> — so the model</text>
|
||||
<text x="80" y="514" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">can easily learn to attend by relative position.</text>
|
||||
</g>
|
||||
|
||||
<g id="formula-band">
|
||||
<rect x="80" y="560" width="540" height="110" rx="10" fill="#FFFFFF" stroke="#3182CE" stroke-width="1.5"/>
|
||||
<text x="100" y="586" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-weight="700" fill="#3182CE" letter-spacing="1.5">EQUATION</text>
|
||||
<image href="../images/formula_003.png" x="120" y="596" width="460" height="72" preserveAspectRatio="xMidYMid meet"/>
|
||||
</g>
|
||||
|
||||
<g id="footer">
|
||||
<text x="1220" y="708" text-anchor="end" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#4A5568">11</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.7 KiB |
@@ -0,0 +1,77 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1280 720" width="1280" height="720">
|
||||
<g id="bg">
|
||||
<rect width="1280" height="720" fill="#FFFFFF"/>
|
||||
</g>
|
||||
|
||||
<g id="header">
|
||||
<rect x="60" y="56" width="6" height="40" fill="#3182CE"/>
|
||||
<text x="84" y="88" font-family="Georgia, "Microsoft YaHei", serif" font-size="36" font-weight="700" fill="#1A365D">Complexity per layer, sequential ops, max path length</text>
|
||||
<text x="84" y="118" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1.5">TABLE 1 · WHY SELF-ATTENTION · n = sequence length, d = representation dim</text>
|
||||
</g>
|
||||
|
||||
<g id="table">
|
||||
<rect x="60" y="160" width="1160" height="370" fill="none" stroke="#1A365D" stroke-width="2"/>
|
||||
<g id="table-area" transform="translate(60, 160)">
|
||||
<rect x="0" y="0" width="1160" height="60" fill="#1A365D"/>
|
||||
<line x1="0" y1="60" x2="1160" y2="60" stroke="#1A365D" stroke-width="2"/>
|
||||
<text x="24" y="37" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="800" fill="#FFFFFF" letter-spacing="1.5">LAYER TYPE</text>
|
||||
<text x="480" y="37" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="800" fill="#FFFFFF" letter-spacing="1.5">COMPLEXITY PER LAYER</text>
|
||||
<text x="760" y="37" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="800" fill="#FFFFFF" letter-spacing="1.5">SEQUENTIAL OPS</text>
|
||||
<text x="980" y="37" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="800" fill="#FFFFFF" letter-spacing="1.5">MAX PATH LENGTH</text>
|
||||
|
||||
<line x1="300" y1="0" x2="300" y2="370" stroke="#E2E8F0" stroke-width="1.5"/>
|
||||
<line x1="660" y1="0" x2="660" y2="370" stroke="#E2E8F0" stroke-width="1.5"/>
|
||||
<line x1="860" y1="0" x2="860" y2="370" stroke="#E2E8F0" stroke-width="1.5"/>
|
||||
|
||||
<g id="row-self-attn">
|
||||
<rect x="0" y="60" width="1160" height="68" fill="#EBF4FB"/>
|
||||
<text x="24" y="92" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Self-Attention</text>
|
||||
<text x="24" y="116" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-style="italic" fill="#3182CE">the Transformer</text>
|
||||
<text x="480" y="102" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="20" font-weight="700" fill="#1A365D">O(n² · d)</text>
|
||||
<text x="760" y="102" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="22" font-weight="700" fill="#2F855A">O(1)</text>
|
||||
<text x="980" y="102" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="22" font-weight="700" fill="#2F855A">O(1)</text>
|
||||
</g>
|
||||
|
||||
<g id="row-recurrent">
|
||||
<line x1="0" y1="128" x2="1160" y2="128" stroke="#E2E8F0" stroke-width="1"/>
|
||||
<text x="24" y="160" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Recurrent</text>
|
||||
<text x="24" y="184" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-style="italic" fill="#4A5568">RNN / LSTM / GRU</text>
|
||||
<text x="480" y="170" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="20" font-weight="600" fill="#1A202C">O(n · d²)</text>
|
||||
<text x="760" y="170" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="22" font-weight="600" fill="#C53030">O(n)</text>
|
||||
<text x="980" y="170" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="22" font-weight="600" fill="#C53030">O(n)</text>
|
||||
</g>
|
||||
|
||||
<g id="row-conv">
|
||||
<line x1="0" y1="196" x2="1160" y2="196" stroke="#E2E8F0" stroke-width="1"/>
|
||||
<text x="24" y="228" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Convolutional</text>
|
||||
<text x="24" y="252" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-style="italic" fill="#4A5568">kernel width k</text>
|
||||
<text x="480" y="238" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="20" font-weight="600" fill="#1A202C">O(k · n · d²)</text>
|
||||
<text x="760" y="238" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="22" font-weight="600" fill="#2F855A">O(1)</text>
|
||||
<text x="980" y="238" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="20" font-weight="600" fill="#4A5568">O(log_k n)</text>
|
||||
</g>
|
||||
|
||||
<g id="row-self-attn-restricted">
|
||||
<line x1="0" y1="264" x2="1160" y2="264" stroke="#E2E8F0" stroke-width="1"/>
|
||||
<text x="24" y="296" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Self-Attention <tspan font-style="italic" fill="#4A5568" font-weight="500">(restricted)</tspan></text>
|
||||
<text x="24" y="320" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-style="italic" fill="#4A5568">window r around each position</text>
|
||||
<text x="480" y="306" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="20" font-weight="600" fill="#1A202C">O(r · n · d)</text>
|
||||
<text x="760" y="306" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="22" font-weight="600" fill="#2F855A">O(1)</text>
|
||||
<text x="980" y="306" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="20" font-weight="600" fill="#4A5568">O(n / r)</text>
|
||||
</g>
|
||||
</g>
|
||||
</g>
|
||||
|
||||
<g id="takeaway">
|
||||
<rect x="60" y="552" width="1160" height="120" rx="10" fill="#F5F7FA" stroke="#E2E8F0" stroke-width="1"/>
|
||||
<rect x="60" y="552" width="6" height="120" rx="3" fill="#3182CE"/>
|
||||
<text x="100" y="580" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#3182CE" letter-spacing="1.5">READ THE TABLE LIKE THIS</text>
|
||||
<text x="100" y="610" font-family="Georgia, "Microsoft YaHei", serif" font-size="17" font-weight="600" fill="#1A365D">Self-attention pays <tspan font-family="Consolas, monospace" fill="#1A365D">O(n²·d)</tspan> compute, but in return any two positions are <tspan font-family="Consolas, monospace" fill="#2F855A">one hop apart</tspan>.</text>
|
||||
<text x="100" y="638" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#4A5568">For typical NLP where <tspan font-family="Consolas, monospace" fill="#1A365D" font-weight="700">n < d</tspan> (sequence shorter than representation), self-attention is faster than recurrent layers</text>
|
||||
<text x="100" y="658" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#4A5568">— and it learns long-range dependencies more easily.</text>
|
||||
</g>
|
||||
|
||||
<g id="footer">
|
||||
<text x="60" y="700" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">Source: Vaswani et al. 2017, Table 1 (§4 Why Self-Attention)</text>
|
||||
<text x="1220" y="700" text-anchor="end" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">12</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 7.4 KiB |
@@ -0,0 +1,84 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1280 720" width="1280" height="720">
|
||||
<defs>
|
||||
<filter id="cardShadow" x="-10%" y="-10%" width="120%" height="120%">
|
||||
<feGaussianBlur in="SourceAlpha" stdDeviation="4"/>
|
||||
<feOffset dx="0" dy="3"/>
|
||||
<feFlood flood-color="#1A365D" flood-opacity="0.07"/>
|
||||
<feComposite in2="SourceAlpha" operator="in"/>
|
||||
<feMerge><feMergeNode/><feMergeNode in="SourceGraphic"/></feMerge>
|
||||
</filter>
|
||||
</defs>
|
||||
|
||||
<g id="bg">
|
||||
<rect width="1280" height="720" fill="#FFFFFF"/>
|
||||
</g>
|
||||
|
||||
<g id="header">
|
||||
<rect x="60" y="56" width="6" height="40" fill="#3182CE"/>
|
||||
<text x="84" y="88" font-family="Georgia, "Microsoft YaHei", serif" font-size="36" font-weight="700" fill="#1A365D">How they trained it</text>
|
||||
<text x="84" y="118" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1.5">TRAINING SETUP · §5 WMT'14 ENGLISH-GERMAN & ENGLISH-FRENCH</text>
|
||||
</g>
|
||||
|
||||
<g id="card-hardware">
|
||||
<rect x="60" y="160" width="580" height="240" rx="14" fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)"/>
|
||||
<rect x="60" y="160" width="6" height="240" rx="3" fill="#1A365D"/>
|
||||
<use data-icon="tabler-outline/cpu" x="84" y="180" width="32" height="32" fill="#1A365D"/>
|
||||
<text x="84" y="240" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1.5">HARDWARE & SCALE</text>
|
||||
<text x="84" y="296" font-family="Georgia, "Microsoft YaHei", serif" font-size="40" font-weight="700" fill="#1A365D">8 × NVIDIA P100</text>
|
||||
<text x="84" y="332" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">One machine, eight GPUs — a single-node setup.</text>
|
||||
<text x="84" y="354" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">Batches of ~25,000 source + 25,000 target tokens,</text>
|
||||
<text x="84" y="376" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">grouped by similar sequence length for efficiency.</text>
|
||||
</g>
|
||||
|
||||
<g id="card-time">
|
||||
<rect x="660" y="160" width="560" height="240" rx="14" fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)"/>
|
||||
<rect x="660" y="160" width="6" height="240" rx="3" fill="#3182CE"/>
|
||||
<use data-icon="tabler-outline/bolt" x="684" y="180" width="32" height="32" fill="#3182CE"/>
|
||||
<text x="684" y="240" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1.5">TRAINING TIME</text>
|
||||
<g id="time-base">
|
||||
<text x="684" y="296" font-family="Georgia, "Microsoft YaHei", serif" font-size="32" font-weight="700" fill="#1A365D">base</text>
|
||||
<text x="820" y="296" font-family="Georgia, "Microsoft YaHei", serif" font-size="32" font-weight="700" fill="#3182CE">12 h</text>
|
||||
<text x="940" y="296" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">· 100k steps</text>
|
||||
</g>
|
||||
<g id="time-big">
|
||||
<text x="684" y="346" font-family="Georgia, "Microsoft YaHei", serif" font-size="32" font-weight="700" fill="#1A365D">big</text>
|
||||
<text x="820" y="346" font-family="Georgia, "Microsoft YaHei", serif" font-size="32" font-weight="700" fill="#3182CE">3.5 d</text>
|
||||
<text x="940" y="346" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#4A5568">· 300k steps</text>
|
||||
</g>
|
||||
<text x="684" y="386" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" font-style="italic" fill="#4A5568">A small fraction of the training cost of prior best models.</text>
|
||||
</g>
|
||||
|
||||
<g id="card-optimizer">
|
||||
<rect x="60" y="420" width="580" height="240" rx="14" fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)"/>
|
||||
<rect x="60" y="420" width="6" height="240" rx="3" fill="#1A365D"/>
|
||||
<use data-icon="tabler-outline/settings" x="84" y="440" width="32" height="32" fill="#1A365D"/>
|
||||
<text x="84" y="500" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1.5">OPTIMIZER</text>
|
||||
<text x="84" y="546" font-family="Georgia, "Microsoft YaHei", serif" font-size="32" font-weight="700" fill="#1A365D">Adam</text>
|
||||
<text x="84" y="578" font-family="Consolas, "Courier New", monospace" font-size="15" fill="#1A202C">β₁ = 0.9 · β₂ = 0.98 · ε = 1e-9</text>
|
||||
<text x="84" y="610" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#4A5568">Custom learning-rate schedule:</text>
|
||||
<text x="84" y="632" font-family="Consolas, "Courier New", monospace" font-size="14" fill="#3182CE">lr ∝ min(step⁻⁰·⁵, step · warmup⁻¹·⁵)</text>
|
||||
<text x="84" y="652" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-style="italic" fill="#4A5568">warm up for 4,000 steps, then decay as 1/√step.</text>
|
||||
</g>
|
||||
|
||||
<g id="card-regularization">
|
||||
<rect x="660" y="420" width="560" height="240" rx="14" fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)"/>
|
||||
<rect x="660" y="420" width="6" height="240" rx="3" fill="#3182CE"/>
|
||||
<use data-icon="tabler-outline/shield" x="684" y="440" width="32" height="32" fill="#3182CE"/>
|
||||
<text x="684" y="500" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1.5">REGULARIZATION</text>
|
||||
<g id="reg-dropout">
|
||||
<text x="684" y="544" font-family="Georgia, "Microsoft YaHei", serif" font-size="22" font-weight="700" fill="#1A365D">Dropout</text>
|
||||
<text x="850" y="544" font-family="Consolas, "Courier New", monospace" font-size="20" font-weight="700" fill="#3182CE">p = 0.1</text>
|
||||
<text x="684" y="566" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" fill="#4A5568">on every sub-layer output and on the sum of embedding + PE.</text>
|
||||
</g>
|
||||
<g id="reg-label-smoothing">
|
||||
<text x="684" y="610" font-family="Georgia, "Microsoft YaHei", serif" font-size="22" font-weight="700" fill="#1A365D">Label smoothing</text>
|
||||
<text x="950" y="610" font-family="Consolas, "Courier New", monospace" font-size="20" font-weight="700" fill="#3182CE">ε_ls = 0.1</text>
|
||||
<text x="684" y="632" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" fill="#4A5568">Slightly hurts perplexity, but improves BLEU and accuracy.</text>
|
||||
</g>
|
||||
</g>
|
||||
|
||||
<g id="footer">
|
||||
<text x="60" y="694" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">Source: Vaswani et al. 2017, §5.1 Training Data and Batching · §5.2 Hardware and Schedule · §5.3 Optimizer · §5.4 Regularization</text>
|
||||
<text x="1220" y="694" text-anchor="end" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">13</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 7.1 KiB |
@@ -0,0 +1,92 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1280 720" width="1280" height="720">
|
||||
<g id="bg">
|
||||
<rect width="1280" height="720" fill="#FFFFFF"/>
|
||||
</g>
|
||||
|
||||
<g id="header">
|
||||
<rect x="60" y="56" width="6" height="40" fill="#3182CE"/>
|
||||
<text x="84" y="88" font-family="Georgia, "Microsoft YaHei", serif" font-size="36" font-weight="700" fill="#1A365D">New SOTA on WMT'14 — at a fraction of the cost</text>
|
||||
<text x="84" y="118" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1.5">TABLE 2 · BLEU SCORES + TRAINING-COST FLOPS · BEST PRIOR ROWS VS TRANSFORMER</text>
|
||||
</g>
|
||||
|
||||
<g id="table">
|
||||
<rect x="60" y="160" width="1160" height="380" fill="none" stroke="#1A365D" stroke-width="2"/>
|
||||
<g id="table-area" transform="translate(60, 160)">
|
||||
<rect x="0" y="0" width="1160" height="60" fill="#1A365D"/>
|
||||
<line x1="0" y1="60" x2="1160" y2="60" stroke="#1A365D" stroke-width="2"/>
|
||||
<text x="24" y="37" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="800" fill="#FFFFFF" letter-spacing="1.5">MODEL</text>
|
||||
<text x="640" y="37" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="800" fill="#FFFFFF" letter-spacing="1.5">EN → DE BLEU</text>
|
||||
<text x="840" y="37" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="800" fill="#FFFFFF" letter-spacing="1.5">EN → FR BLEU</text>
|
||||
<text x="1040" y="37" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="800" fill="#FFFFFF" letter-spacing="1.5">TRAIN FLOPs</text>
|
||||
|
||||
<line x1="540" y1="0" x2="540" y2="380" stroke="#E2E8F0" stroke-width="1.5"/>
|
||||
<line x1="740" y1="0" x2="740" y2="380" stroke="#E2E8F0" stroke-width="1.5"/>
|
||||
<line x1="940" y1="0" x2="940" y2="380" stroke="#E2E8F0" stroke-width="1.5"/>
|
||||
|
||||
<g id="row-bytenet">
|
||||
<line x1="0" y1="106" x2="1160" y2="106" stroke="#E2E8F0" stroke-width="1"/>
|
||||
<text x="24" y="92" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" font-weight="600" fill="#1A202C">ByteNet</text>
|
||||
<text x="640" y="92" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="16" fill="#1A202C">23.75</text>
|
||||
<text x="840" y="92" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="16" fill="#4A5568">—</text>
|
||||
<text x="1040" y="92" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="13" fill="#4A5568">—</text>
|
||||
</g>
|
||||
|
||||
<g id="row-conv-s2s">
|
||||
<line x1="0" y1="152" x2="1160" y2="152" stroke="#E2E8F0" stroke-width="1"/>
|
||||
<text x="24" y="138" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" font-weight="600" fill="#1A202C">ConvS2S</text>
|
||||
<text x="640" y="138" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="16" fill="#1A202C">25.16</text>
|
||||
<text x="840" y="138" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="16" fill="#1A202C">40.46</text>
|
||||
<text x="1040" y="138" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="13" fill="#4A5568">9.6 × 10¹⁸</text>
|
||||
</g>
|
||||
|
||||
<g id="row-moe">
|
||||
<line x1="0" y1="198" x2="1160" y2="198" stroke="#E2E8F0" stroke-width="1"/>
|
||||
<text x="24" y="184" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" font-weight="600" fill="#1A202C">MoE</text>
|
||||
<text x="640" y="184" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="16" fill="#1A202C">26.03</text>
|
||||
<text x="840" y="184" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="16" fill="#1A202C">40.56</text>
|
||||
<text x="1040" y="184" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="13" fill="#4A5568">2.0 × 10¹⁹</text>
|
||||
</g>
|
||||
|
||||
<g id="row-conv-s2s-ens">
|
||||
<line x1="0" y1="244" x2="1160" y2="244" stroke="#E2E8F0" stroke-width="1"/>
|
||||
<text x="24" y="230" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" font-weight="600" fill="#1A202C">ConvS2S Ensemble</text>
|
||||
<text x="640" y="230" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="16" fill="#1A202C">26.36</text>
|
||||
<text x="840" y="230" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="16" fill="#1A202C">41.29</text>
|
||||
<text x="1040" y="230" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="13" fill="#4A5568">7.7 × 10¹⁹</text>
|
||||
</g>
|
||||
|
||||
<g id="row-transformer-base">
|
||||
<rect x="0" y="244" width="1160" height="68" fill="#EBF4FB"/>
|
||||
<line x1="0" y1="312" x2="1160" y2="312" stroke="#E2E8F0" stroke-width="1"/>
|
||||
<text x="24" y="278" font-family="Georgia, "Microsoft YaHei", serif" font-size="17" font-weight="700" fill="#1A365D">Transformer (base)</text>
|
||||
<text x="24" y="298" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-style="italic" fill="#3182CE">N=6, d_model=512, h=8</text>
|
||||
<text x="640" y="284" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="18" font-weight="700" fill="#1A365D">27.3</text>
|
||||
<text x="840" y="284" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="18" font-weight="700" fill="#1A365D">38.1</text>
|
||||
<text x="1040" y="284" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="14" fill="#3182CE">3.3 × 10¹⁸</text>
|
||||
</g>
|
||||
|
||||
<g id="row-transformer-big">
|
||||
<rect x="0" y="312" width="1160" height="68" fill="#1A365D" fill-opacity="0.06"/>
|
||||
<text x="24" y="346" font-family="Georgia, "Microsoft YaHei", serif" font-size="17" font-weight="700" fill="#1A365D">Transformer (big)</text>
|
||||
<text x="24" y="366" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-style="italic" fill="#3182CE">N=6, d_model=1024, h=16</text>
|
||||
<text x="640" y="352" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="22" font-weight="800" fill="#2F855A">28.4</text>
|
||||
<text x="840" y="352" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="22" font-weight="800" fill="#2F855A">41.8</text>
|
||||
<text x="1040" y="352" text-anchor="middle" font-family="Consolas, "Courier New", monospace" font-size="14" fill="#3182CE">2.3 × 10¹⁹</text>
|
||||
</g>
|
||||
</g>
|
||||
</g>
|
||||
|
||||
<g id="takeaway">
|
||||
<rect x="60" y="562" width="1160" height="110" rx="10" fill="#F5F7FA" stroke="#E2E8F0" stroke-width="1"/>
|
||||
<rect x="60" y="562" width="6" height="110" rx="3" fill="#2F855A"/>
|
||||
<text x="100" y="590" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#2F855A" letter-spacing="1.5">THREE RESULTS WORTH NAMING</text>
|
||||
<text x="100" y="620" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">· <tspan font-weight="700" fill="#1A365D">EN-DE</tspan> — Transformer (big) beats the prior best ensemble by <tspan fill="#2F855A" font-weight="700">+2.0 BLEU</tspan>, single model.</text>
|
||||
<text x="100" y="644" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">· <tspan font-weight="700" fill="#1A365D">EN-FR</tspan> — new <tspan fill="#2F855A" font-weight="700">single-model SOTA</tspan> at 41.8 BLEU.</text>
|
||||
<text x="100" y="668" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">· <tspan font-weight="700" fill="#1A365D">Training cost</tspan> — well below most strong baselines; base model trains in 12 hours.</text>
|
||||
</g>
|
||||
|
||||
<g id="footer">
|
||||
<text x="60" y="702" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">Source: Vaswani et al. 2017, Table 2 (§6.1 Machine Translation)</text>
|
||||
<text x="1220" y="702" text-anchor="end" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">14</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 8.4 KiB |
@@ -0,0 +1,89 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1280 720" width="1280" height="720">
|
||||
<defs>
|
||||
<linearGradient id="trackGrad" x1="0%" y1="0%" x2="0%" y2="100%">
|
||||
<stop offset="0%" stop-color="#1A365D"/>
|
||||
<stop offset="100%" stop-color="#3182CE"/>
|
||||
</linearGradient>
|
||||
<filter id="cardShadow" x="-5%" y="-15%" width="110%" height="135%">
|
||||
<feGaussianBlur in="SourceAlpha" stdDeviation="4"/>
|
||||
<feOffset dx="0" dy="2"/>
|
||||
<feFlood flood-color="#1A365D" flood-opacity="0.06"/>
|
||||
<feComposite in2="SourceAlpha" operator="in"/>
|
||||
<feMerge><feMergeNode/><feMergeNode in="SourceGraphic"/></feMerge>
|
||||
</filter>
|
||||
</defs>
|
||||
|
||||
<g id="bg">
|
||||
<rect width="1280" height="720" fill="#FFFFFF"/>
|
||||
</g>
|
||||
|
||||
<g id="header">
|
||||
<rect x="60" y="56" width="6" height="40" fill="#3182CE"/>
|
||||
<text x="84" y="88" font-family="Georgia, "Microsoft YaHei", serif" font-size="36" font-weight="700" fill="#1A365D">Five takeaways from the variation study</text>
|
||||
<text x="84" y="118" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1.5">TABLE 3 ABLATIONS · WHAT THE ARCHITECTURE NEEDS TO STAY GOOD</text>
|
||||
</g>
|
||||
|
||||
<g id="track">
|
||||
<line x1="160" y1="172" x2="160" y2="660" stroke="url(#trackGrad)" stroke-width="4" stroke-linecap="round"/>
|
||||
</g>
|
||||
|
||||
<g id="item-1">
|
||||
<circle cx="160" cy="200" r="28" fill="#1A365D" fill-opacity="0.12"/>
|
||||
<circle cx="160" cy="200" r="20" fill="#1A365D"/>
|
||||
<text x="160" y="206" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="15" font-weight="700" fill="#FFFFFF">01</text>
|
||||
<line x1="188" y1="200" x2="220" y2="200" stroke="#1A365D" stroke-width="2" stroke-dasharray="4,3"/>
|
||||
<rect x="220" y="170" width="1000" height="64" rx="10" fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)"/>
|
||||
<rect x="220" y="170" width="6" height="64" rx="3" fill="#1A365D"/>
|
||||
<text x="246" y="196" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Head count has a sweet spot</text>
|
||||
<text x="246" y="222" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#1A202C">Single head loses <tspan font-family="Consolas, monospace" font-weight="700" fill="#C53030">−0.9 BLEU</tspan> vs h = 8; too many heads (h = 32) also degrade. Eight is empirically right.</text>
|
||||
</g>
|
||||
|
||||
<g id="item-2">
|
||||
<circle cx="160" cy="296" r="28" fill="#1A365D" fill-opacity="0.12"/>
|
||||
<circle cx="160" cy="296" r="20" fill="#1A365D"/>
|
||||
<text x="160" y="302" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="15" font-weight="700" fill="#FFFFFF">02</text>
|
||||
<line x1="188" y1="296" x2="220" y2="296" stroke="#1A365D" stroke-width="2" stroke-dasharray="4,3"/>
|
||||
<rect x="220" y="266" width="1000" height="64" rx="10" fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)"/>
|
||||
<rect x="220" y="266" width="6" height="64" rx="3" fill="#1A365D"/>
|
||||
<text x="246" y="292" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Key dim matters more than value dim</text>
|
||||
<text x="246" y="318" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#1A202C">Reducing <tspan font-family="Consolas, monospace" font-weight="700" fill="#3182CE">d_k</tspan> hurts quality more than reducing <tspan font-family="Consolas, monospace" font-weight="700" fill="#3182CE">d_v</tspan> — determining compatibility is the harder learning problem.</text>
|
||||
</g>
|
||||
|
||||
<g id="item-3">
|
||||
<circle cx="160" cy="392" r="28" fill="#3182CE" fill-opacity="0.12"/>
|
||||
<circle cx="160" cy="392" r="20" fill="#3182CE"/>
|
||||
<text x="160" y="398" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="15" font-weight="700" fill="#FFFFFF">03</text>
|
||||
<line x1="188" y1="392" x2="220" y2="392" stroke="#3182CE" stroke-width="2" stroke-dasharray="4,3"/>
|
||||
<rect x="220" y="362" width="1000" height="64" rx="10" fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)"/>
|
||||
<rect x="220" y="362" width="6" height="64" rx="3" fill="#3182CE"/>
|
||||
<text x="246" y="388" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Bigger is better</text>
|
||||
<text x="246" y="414" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#1A202C">Larger <tspan font-family="Consolas, monospace" font-weight="700" fill="#3182CE">d_model</tspan> and deeper stacks consistently improve BLEU — the architecture scales monotonically with capacity.</text>
|
||||
</g>
|
||||
|
||||
<g id="item-4">
|
||||
<circle cx="160" cy="488" r="28" fill="#3182CE" fill-opacity="0.12"/>
|
||||
<circle cx="160" cy="488" r="20" fill="#3182CE"/>
|
||||
<text x="160" y="494" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="15" font-weight="700" fill="#FFFFFF">04</text>
|
||||
<line x1="188" y1="488" x2="220" y2="488" stroke="#3182CE" stroke-width="2" stroke-dasharray="4,3"/>
|
||||
<rect x="220" y="458" width="1000" height="64" rx="10" fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)"/>
|
||||
<rect x="220" y="458" width="6" height="64" rx="3" fill="#3182CE"/>
|
||||
<text x="246" y="484" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Dropout is essential</text>
|
||||
<text x="246" y="510" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#1A202C">Removing dropout loses <tspan font-family="Consolas, monospace" font-weight="700" fill="#C53030">~0.5–1.0 BLEU</tspan>. Even p = 0.1 measurably regularizes — it isn't decorative.</text>
|
||||
</g>
|
||||
|
||||
<g id="item-5">
|
||||
<circle cx="160" cy="584" r="28" fill="#63B3ED" fill-opacity="0.18"/>
|
||||
<circle cx="160" cy="584" r="20" fill="#3182CE"/>
|
||||
<text x="160" y="590" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="15" font-weight="700" fill="#FFFFFF">05</text>
|
||||
<line x1="188" y1="584" x2="220" y2="584" stroke="#3182CE" stroke-width="2" stroke-dasharray="4,3"/>
|
||||
<rect x="220" y="554" width="1000" height="64" rx="10" fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#cardShadow)"/>
|
||||
<rect x="220" y="554" width="6" height="64" rx="3" fill="#63B3ED"/>
|
||||
<text x="246" y="580" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Label smoothing helps BLEU (and hurts perplexity)</text>
|
||||
<text x="246" y="606" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="14" fill="#1A202C">Slightly worse perplexity, better BLEU. <tspan font-style="italic" fill="#4A5568">Optimize for the metric you ship on.</tspan></text>
|
||||
</g>
|
||||
|
||||
<g id="footer">
|
||||
<text x="60" y="694" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">Source: Vaswani et al. 2017, Table 3 (§6.2 Model Variations)</text>
|
||||
<text x="1220" y="694" text-anchor="end" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">15</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 7.2 KiB |
@@ -0,0 +1,56 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1280 720" width="1280" height="720">
|
||||
<defs>
|
||||
<linearGradient id="closingScrim" x1="0" y1="0" x2="0" y2="1">
|
||||
<stop offset="0%" stop-color="#FFFFFF" stop-opacity="0.0"/>
|
||||
<stop offset="38%" stop-color="#FFFFFF" stop-opacity="0.95"/>
|
||||
<stop offset="62%" stop-color="#FFFFFF" stop-opacity="0.95"/>
|
||||
<stop offset="100%" stop-color="#FFFFFF" stop-opacity="0.0"/>
|
||||
</linearGradient>
|
||||
</defs>
|
||||
|
||||
<g id="bg">
|
||||
<rect width="1280" height="720" fill="#FFFFFF"/>
|
||||
<image href="../images/transformer_family.png" x="0" y="0" width="1280" height="720" preserveAspectRatio="xMidYMid slice"/>
|
||||
<rect x="0" y="180" width="1280" height="360" fill="url(#closingScrim)"/>
|
||||
</g>
|
||||
|
||||
<g id="closing-marks">
|
||||
<rect x="80" y="64" width="80" height="3" fill="#3182CE"/>
|
||||
<text x="80" y="92" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#3182CE" letter-spacing="2.5">CLOSING · WHAT THIS PAPER STARTED</text>
|
||||
</g>
|
||||
|
||||
<g id="closing-title">
|
||||
<text x="640" y="316" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="56" font-weight="700" fill="#1A365D">One paper, a generation</text>
|
||||
<text x="640" y="380" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="56" font-weight="700" fill="#1A365D">of <tspan fill="#3182CE">models</tspan></text>
|
||||
<line x1="600" y1="412" x2="680" y2="412" stroke="#3182CE" stroke-width="2"/>
|
||||
</g>
|
||||
|
||||
<g id="closing-caption">
|
||||
<text x="640" y="466" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="20" font-style="italic" fill="#4A5568">The 2017 idea — just attention — turned out to be the right inductive bias</text>
|
||||
<text x="640" y="494" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="20" font-style="italic" fill="#4A5568">for almost every modality.</text>
|
||||
</g>
|
||||
|
||||
<g id="lineage">
|
||||
<text x="640" y="552" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#3182CE" letter-spacing="2.5">DESCENDANTS · A NON-EXHAUSTIVE FAMILY</text>
|
||||
<g id="lineage-row" transform="translate(640, 580)">
|
||||
<text x="-340" y="0" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">BERT</text>
|
||||
<text x="-240" y="0" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">GPT</text>
|
||||
<text x="-160" y="0" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">T5</text>
|
||||
<text x="-90" y="0" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">ViT</text>
|
||||
<text x="0" y="0" text-anchor="middle" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">CLIP</text>
|
||||
<text x="80" y="0" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">AlphaFold</text>
|
||||
<text x="220" y="0" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">LLaMA</text>
|
||||
<text x="320" y="0" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Claude</text>
|
||||
</g>
|
||||
<text x="640" y="612" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" fill="#4A5568">— and most modern LLMs trace their architecture back to this one paper.</text>
|
||||
</g>
|
||||
|
||||
<g id="closing-footer">
|
||||
<line x1="500" y1="660" x2="780" y2="660" stroke="#A0AEC0" stroke-width="0.5"/>
|
||||
<text x="640" y="688" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" fill="#4A5568">Vaswani et al. · "Attention Is All You Need" · NeurIPS 2017 · arXiv:1706.03762</text>
|
||||
</g>
|
||||
|
||||
<g id="footer">
|
||||
<text x="1220" y="708" text-anchor="end" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#4A5568">16</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 4.1 KiB |
@@ -0,0 +1,15 @@
|
||||
# brutalist_ai_newspaper_2026_ppt169_20260521
|
||||
|
||||
- Canvas format: ppt169
|
||||
- Created: 20260521
|
||||
|
||||
## Directories
|
||||
|
||||
- `svg_output/`: raw SVG output
|
||||
- `svg_final/`: finalized SVG output
|
||||
- `images/`: presentation assets
|
||||
- `notes/`: speaker notes
|
||||
- `templates/`: project templates
|
||||
- `sources/`: source materials and normalized markdown
|
||||
- `exports/`: main native pptx (timestamped); `_svg.pptx` sibling added when exported with `--svg-snapshot`
|
||||
- `backup/<timestamp>/`: svg_output/ archive (always written in default-flow mode; safe to delete old timestamps)
|
||||
@@ -0,0 +1,354 @@
|
||||
# ppt169_ai_industry_2026 — Design Spec
|
||||
|
||||
> Brutalist editorial newspaper · capability-showcase demo for PPT Master roadmap §P0-2
|
||||
> Source: `sources/ai_industry_2026.md` (compiled May 2026 from Counterpoint / Stanford HAI / OneTrust / Deluair / IEEE Spectrum)
|
||||
|
||||
## I. Project Information
|
||||
|
||||
| Item | Value |
|
||||
| ---- | ----- |
|
||||
| **Project Name** | ppt169_ai_industry_2026 |
|
||||
| **Canvas Format** | PPT 16:9 (1280×720) |
|
||||
| **Page Count** | 10 |
|
||||
| **Design Style** | B) General Consulting + Brutalist editorial newspaper |
|
||||
| **Target Audience** | AI 行业从业者 / 投资人 / 研究者 / 关心 AI 转折点的科技读者 |
|
||||
| **Use Case** | 年报式行业回顾 — 论坛 / brown-bag / 投资者季报开场 |
|
||||
| **Created Date** | 2026-05-21 |
|
||||
|
||||
---
|
||||
|
||||
## II. Canvas Specification
|
||||
|
||||
| Property | Value |
|
||||
| -------- | ----- |
|
||||
| **Format** | PPT 16:9 |
|
||||
| **Dimensions** | 1280×720 |
|
||||
| **viewBox** | `0 0 1280 720` |
|
||||
| **Margins** | Left/Right 48px, Top/Bottom 40px (报章式 narrow margins) |
|
||||
| **Content Area** | 1184×640 |
|
||||
|
||||
---
|
||||
|
||||
## III. Visual Theme
|
||||
|
||||
### Theme Style
|
||||
|
||||
- **Style**: Brutalist editorial newspaper — 满版小字 + 不规则栏宽 + 真原生 shape 框线 + halftone 黑白图 + 单点红强调
|
||||
- **Theme**: Light theme (报章纸白底)
|
||||
- **Tone**: 严肃、克制、信息密度顶级、reportorial
|
||||
|
||||
### Color Scheme
|
||||
|
||||
| Role | HEX | Purpose |
|
||||
| ---- | --- | ------- |
|
||||
| **Background** | `#FFFFFF` | 主背景（纸白） |
|
||||
| **Secondary bg** | `#F4F1EA` | sidebar / pull-quote / 边栏纸黄 |
|
||||
| **Primary** | `#111111` | 主形状 / rule line / 主文字 |
|
||||
| **Accent** | `#C8102E` | 红色强调 — masthead 一条红线、关键数据点、`The Number` 锚点（全文 <5%） |
|
||||
| **Secondary accent** | `#3A3A3A` | 次文字 / annotation / footer |
|
||||
| **Body text** | `#111111` | 主文字 |
|
||||
| **Secondary text** | `#3A3A3A` | 次文字 / caption |
|
||||
| **Tertiary text** | `#6B6B6B` | footnote / 页码 |
|
||||
| **Border/divider** | `#000000` | 报章框线、栏分隔、cell border |
|
||||
| **Success** | `#1A6E3A` | 正向数据点（极少用） |
|
||||
| **Warning** | `#A02014` | 负向数据点（极少用） |
|
||||
|
||||
### AI Image Strategy
|
||||
|
||||
- **Image Rendering**: editorial
|
||||
- **Image Palette**: mono-ink
|
||||
|
||||
> 报章纪实风：halftone 颗粒、纸纹质感、clean partition、黑墨平涂、可在墨黑主体之外让单点红 `#C8102E` 出现一次（如红印章 / 红圈标注 / 红 underline）。
|
||||
|
||||
---
|
||||
|
||||
## IV. Typography System
|
||||
|
||||
### Font Plan
|
||||
|
||||
**Typography direction**: Brutalist newspaper — display sans 黑体头条 × serif 衬线小字正文 × monospace 数据，三种家族强对比
|
||||
|
||||
| Role | Chinese | English | Fallback tail |
|
||||
| ---- | ------- | ------- | ------------- |
|
||||
| **Title (masthead / 头条)** | `SimHei` | `Impact, "Arial Black"` | `sans-serif` |
|
||||
| **Subtitle / Section label** | `SimHei` | `"Arial Black"` | `sans-serif` |
|
||||
| **Body (报章 column)** | `SimSun` | `"Times New Roman"` | `serif` |
|
||||
| **Emphasis** | `SimSun` | `Georgia` | `serif` |
|
||||
| **Code / 数据** | — | `Consolas, "Courier New"` | `monospace` |
|
||||
|
||||
**Per-role font stacks**:
|
||||
|
||||
- Title: `Impact, "Arial Black", SimHei, sans-serif`
|
||||
- Subtitle: `"Arial Black", SimHei, sans-serif`
|
||||
- Body: `"Times New Roman", SimSun, serif`
|
||||
- Emphasis: `Georgia, "Times New Roman", SimSun, serif`
|
||||
- Code: `Consolas, "Courier New", monospace`
|
||||
|
||||
### Font Size Hierarchy
|
||||
|
||||
**Baseline**: Body font size = **14px** (满版小字密度，比常规 dense 18px 更密，是 Brutalist 报章典型阅读节奏)
|
||||
|
||||
| Purpose | Ratio to body | Px @ body=14 | Weight |
|
||||
| ------- | ------------- | ------------ | ------ |
|
||||
| Cover masthead (THE AI INDUSTRY 2026) | 5-6x | 72-90 | Black |
|
||||
| Section opener / 头条 | 3-4x | 42-56 | Black |
|
||||
| Page title | 2-2.5x | 28-36 | Bold |
|
||||
| Hero number (报章大数字) | 3.5-5x | 50-72 | Black |
|
||||
| Subtitle / Section label | 1.4-1.6x | 20-22 | SemiBold |
|
||||
| **Body content** | **1x** | **14** | Regular |
|
||||
| Annotation / caption | 0.75-0.85x | 11-12 | Regular |
|
||||
| Page number / footnote / imprint | 0.6-0.75x | 9-10 | Regular |
|
||||
|
||||
---
|
||||
|
||||
## V. Layout Principles
|
||||
|
||||
### Page Structure
|
||||
|
||||
- **Header area**: Masthead/page-header 64px (顶部红线 4px + 报章标题栏 60px)
|
||||
- **Content area**: 568px (报章主体；多栏不规则栏宽)
|
||||
- **Footer area**: 28px (imprint / page number / date)
|
||||
|
||||
### Layout Pattern Library
|
||||
|
||||
Brutalist 报章主要用：
|
||||
- **Asymmetric multi-column** (2/3/4 栏不等宽) — 不规则栏宽是核心特征
|
||||
- **Top-band + multi-column** — 大图压顶 + 下方多栏
|
||||
- **Sidebar column** (image 或 pull-quote) — 整页右侧或左侧固定边栏
|
||||
- **Hero number + body** — 单个巨型数字 + 解释栏
|
||||
- **Rule-line grid** — 用粗细 rule line 分割，不用 box card
|
||||
- **Center-radiating** — 用于 P06 KPI overlay
|
||||
|
||||
### Spacing Specification
|
||||
|
||||
**Universal**:
|
||||
|
||||
| Element | Range | Current |
|
||||
| ------- | ----- | ------- |
|
||||
| Safe margin from canvas edge | 40-60px | **48px L/R, 40px T/B** |
|
||||
| Content block gap | 16-24px | **20px** |
|
||||
| Column gap (rule line gutter) | 14-20px | **16px** |
|
||||
|
||||
**Non-card containers** (Brutalist 报章基本不用 card，全靠 rule line 分割):
|
||||
|
||||
- Line-height: 1.35-1.45× body font size (报章紧凑节奏)
|
||||
- Body column width: 220-340px (3-4 栏典型)
|
||||
- Rule line stroke: 0.5px hairline / 1px standard / 2-3px section divider / 4px masthead red bar
|
||||
|
||||
---
|
||||
|
||||
## VI. Icon Usage Specification
|
||||
|
||||
**不使用 icon library**。Brutalist 报章靠 typography + rule line + halftone 图说话，icon 反而稀释力度。
|
||||
|
||||
---
|
||||
|
||||
## VII. Visualization Reference List
|
||||
|
||||
Catalog read: 71 templates
|
||||
|
||||
| Page | Template | Path | Summary-quote (verbatim) | Usage |
|
||||
| ---- | -------- | ---- | ------------------------ | ----- |
|
||||
| P02 | kpi_cards | `templates/charts/kpi_cards.svg` | "Pick for 4-8 standalone numeric metrics shown as overview cards (2x2 or 1x4) — exec summary opener, dashboard headline," | 头版 4 个核心 KPI： $2.02T 支出 / 3.8B 用户 / $20.7B Q1 收入 / 1000+ 政策 |
|
||||
| P03 | horizontal_bar_chart | `templates/charts/horizontal_bar_chart.svg` | "Pick for ranking 5-12 items, especially with long labels. Skip if <=8 short-label items (use bar_chart)." | LLM 收入份额 Q1 2026 ranking — Anthropic / OpenAI / Google / MSFT / Tencent / Baidu / Alibaba / Meta / Grok / Perplexity |
|
||||
| P04 | dumbbell_chart | `templates/charts/dumbbell_chart.svg` | "Pick for before-vs-after or two-state difference across 5-10 items. Skip for single snapshot (use bar_chart) or 3+ states" | ARPU 差距对比：5 大 lab 的 MAU vs 月 ARPU 两态 |
|
||||
| P05 | line_chart | `templates/charts/line_chart.svg` | "Pick for 1-3 time-series on a continuous axis showing direction. Skip if cumulative volume matters (use area_chart) or d" | 训练成本时间轨迹 GPT-4 → Llama 3.1 → Gemini Ultra → 2026 frontier → 2027 projected |
|
||||
| P06 | bar_chart | `templates/charts/bar_chart.svg` | "Pick for single-series category value comparison, 3-8 categories. Skip for >12 long-label items (use horizontal_bar_char" | 推理功率对比 W/query：Claude 4 Opus / GPT-4 / Gemini 2.5 / DeepSeek V3 / Llama 3.1 |
|
||||
| P07 | dumbbell_chart | `templates/charts/dumbbell_chart.svg` | "Pick for before-vs-after or two-state difference across 5-10 items. Skip for single snapshot (use bar_chart) or 3+ states" | Benchmark 饱和：SWE-bench / HLE / PhD-QA / Math 多项 2024 → 2025 跳变 |
|
||||
| P08 | comparison_table | `templates/charts/comparison_table.svg` | "Pick for 2-4 plans/products compared across many feature rows (dense matrix). Skip for pricing-tier marketing layout (us" | 三套监管对照：EU / US / China 在生效日期 / 罚则 / 范围 / 重点上的差异 |
|
||||
| P09 | timeline | `templates/charts/timeline.svg` | "Pick for 3-8 milestone events on a horizontal time axis (no duration). Skip for tasks with start/end ranges (use gantt_c" | 2025-01 → 2026-04 6 个里程碑事件 |
|
||||
|
||||
**Runners-up considered**:
|
||||
|
||||
- `grouped_bar_chart` | rejected for P04: 双态 MAU/ARPU 两端差异巨大（4 数量级），dumbbell 拉开尺度更易读，grouped bar 在差距悬殊时会失衡。
|
||||
- `vertical_pillars` | rejected for P08: 监管对照是「多行多列」结构化对比，pillars 更适合 1×N 并列单元；comparison_table 更精准匹配「many feature rows」的密度。
|
||||
- `process_flow` | rejected for P09: 2025-2026 事件是离散里程碑非线性流程，timeline 直接匹配「milestone events on a horizontal time axis」。
|
||||
- `bullet_chart` | rejected for P02: bullet 需要 target + actual 双值，本页 4 个 KPI 是 standalone 数字无目标值。
|
||||
- `area_chart` | rejected for P05: 训练成本不是 cumulative volume 而是 model-by-model trajectory，line_chart 更准。
|
||||
|
||||
---
|
||||
|
||||
## VIII. Image Resource List
|
||||
|
||||
| Filename | Dimensions | Ratio | Purpose | Layout pattern | Acquire Via | Status | Reference | text_policy | page_role |
|
||||
| -------- | --------- | ----- | ------- | -------------- | ----------- | ------ | --------- | ----------- | --------- |
|
||||
| p01_cover_hero.png | 1280×720 | 1.78 | 封面 hero 主图 | #1 full-bleed background with floating title + #29 two-stop scrim | ai | Pending | An abstract editorial newspaper photograph: vast information surge depicted as parallel cascading data ribbons descending from the upper darkness into bright reflective floor; silhouette of a lone figure standing at the foot looking up; high-contrast halftone, paper grain, slight motion blur on the ribbons; calm right-side area reserved for masthead title overlay | none | hero_page |
|
||||
| p03_revenue_leader.png | 480×720 | 0.67 | 收入排行配图 — 侧边图 | #18 image as full-height sidebar column + #70 thin colored matte frame | ai | Pending | A vertical editorial photograph: an empty stage podium with the front sign blank and a smaller secondary podium behind partially in shadow; bright ceiling spotlight only on the front podium; halftone grain; suggests a changing of the guard, no people | none | local |
|
||||
| p05_training_cost.png | 1280×320 | 4.00 | 训练成本横幅 — 中段横向 banner | #14 horizontal banner strip cutting through mid-section + #70 thin colored matte frame | ai | Pending | An editorial wide photograph of a vast GPU server hall stretching to vanishing point; long aisle between two walls of black server racks with small status lights; cold overhead lighting; halftone, paper grain; reads as scale and power infrastructure | none | local |
|
||||
| p06_inference_thermal.png | 1280×720 | 1.78 | 推理功率 hero — 浮动 KPI 卡背景 | #40 background image + floating KPI metric cards + #30 flat semi-transparent rectangle overlay | ai | Pending | An editorial photograph: extreme close-up of a single matte-black processor die with intricate etched circuitry visible; ambient heat shimmer above the die surface; one tiny bright red anomaly point at the corner; halftone grain; calm uniform areas reserved for KPI card overlays | none | local |
|
||||
| p08_regulation_table.png | 1280×280 | 4.57 | 监管对照配图 — 顶部 banner | #5 top-band image + bottom multi-column text + #70 thin colored matte frame | ai | Pending | An editorial wide photograph: three closed leather-bound official folders side by side on a wooden table under raking light, each folder slightly different in age and patina; a single bold rubber stamp impression visible on the center folder; halftone, paper grain; suggests three regulatory traditions | none | local |
|
||||
|
||||
---
|
||||
|
||||
## IX. Content Outline
|
||||
|
||||
### Part 1: Masthead
|
||||
|
||||
#### Slide 01 - Cover (Masthead)
|
||||
|
||||
- **Layout**: Full-bleed hero image + 顶部 4px 红线 + 大字 masthead overlay 在右下calm region
|
||||
- **Title**: `THE AI INDUSTRY 2026`
|
||||
- **Subtitle**: `Annual Report — Volume 1, Edition 1`
|
||||
- **Info**: `Compiled May 2026 · Sources: Stanford HAI · Counterpoint · OneTrust · Deluair · IEEE Spectrum`
|
||||
- **Tagline (red accent)**: `THE YEAR THE LEAGUE TABLE CHANGED.`
|
||||
|
||||
### Part 2: Front Page
|
||||
|
||||
#### Slide 02 - Issue at a Glance
|
||||
|
||||
- **Layout**: Top page-header 红线 + 大字 masthead 复用 + 4 个 KPI 卡（1×4 横排） + 下方三栏短文「Three Observations」
|
||||
- **Title**: `ISSUE AT A GLANCE`
|
||||
- **Visualization**: kpi_cards (see VII)
|
||||
- **KPIs**:
|
||||
- `$2.02T` — Worldwide AI spending 2026E (+36% YoY) · Gartner
|
||||
- `3.8B` — Monthly LLM users worldwide · Counterpoint
|
||||
- `$20.7B` — Quarterly LLM revenue Q1 2026 · Counterpoint
|
||||
- `1,000+` — Active national AI policy initiatives across 72 countries · OneTrust
|
||||
- **Three observations (3 栏 short body)**:
|
||||
1. Revenue concentration is breaking from user-count rankings.
|
||||
2. Frontier training costs cleared the $200M floor and are heading to $1–3B by late 2027.
|
||||
3. Inference cost is falling ~10× per year for the same capability.
|
||||
|
||||
### Part 3: Lead Story
|
||||
|
||||
#### Slide 03 - Story I: Revenue League Q1 2026
|
||||
|
||||
- **Layout**: 左 1/3 侧边 sidebar 图片（podium） + 右 2/3 上 horizontal_bar_chart + 下 pull-quote 红字小评论
|
||||
- **Title**: `STORY I — REVENUE LEAGUE TABLE`
|
||||
- **Subtitle**: `Anthropic 31.4 % vs OpenAI 29.0 %`
|
||||
- **Visualization**: horizontal_bar_chart (see VII)
|
||||
- **Data points**:
|
||||
- Anthropic 31.4%, OpenAI 29.0%, Google 12.1%, Microsoft 7.2%, Tencent 4.8%, Baidu 3.6%, Alibaba 2.9%, Meta 1.4%, Grok 1.4%, Perplexity 1.0%, DeepSeek 0.2%, Zhipu 0.1%
|
||||
- **Pull-quote (red)**: `"FIRST NON-OPENAI LAB IN #1 SINCE THE MODERN LLM ERA BEGAN."`
|
||||
- **Source line**: `Counterpoint Research · Global LLM Adoption and Revenue Snapshot, Q1 2026`
|
||||
|
||||
### Part 4: Sidebar Data
|
||||
|
||||
#### Slide 04 - The ARPU Divide
|
||||
|
||||
- **Layout**: 全幅 dumbbell_chart 占上 2/3，下方 4-栏 short caption；no image
|
||||
- **Title**: `THE ARPU DIVIDE — SCALE ≠ MONETISATION`
|
||||
- **Visualization**: dumbbell_chart (see VII)
|
||||
- **Data points (MAU left dot · ARPU right dot)**:
|
||||
- Meta: ~1,000M MAU · $0.10
|
||||
- OpenAI: 900M · $2.20
|
||||
- Google: — · $1.10
|
||||
- Anthropic: 134M · $16.20
|
||||
- Microsoft: — · $5.00
|
||||
- **Caption body**: Anthropic's per-user revenue is 7.4× OpenAI's, 14.7× Google's, 162× Meta's. Enterprise-API and coding workloads compound faster than consumer free-tier scale.
|
||||
|
||||
#### Slide 05 - Frontier Training Cost
|
||||
|
||||
- **Layout**: 中段 1280×320 wide banner photo（GPU 走廊） + 上 page title + 下 line_chart + 右侧 annotation 栏
|
||||
- **Title**: `FRONTIER TRAINING COST — PAST THE HALF-BILLION LINE`
|
||||
- **Visualization**: line_chart (see VII)
|
||||
- **Data points**: GPT-4 $78M → Llama 3.1 $170M → Gemini Ultra $191M → 2026 frontier $200-500M → late-2027 projected $1-3B
|
||||
- **Annotation**: `1e26 – 1e27 FLOPs` (2026 frontier compute range). Power and grid interconnect, not chip supply, are the binding constraint for the late-2027 cohort.
|
||||
|
||||
### Part 5: Inside Pages
|
||||
|
||||
#### Slide 06 - Inference — Watts per Query
|
||||
|
||||
- **Layout**: full-bleed processor die image as canvas + 4 个 floating KPI 卡 + 半透明 overlay 保证可读
|
||||
- **Title**: `INFERENCE — A 4.6× SPREAD INSIDE THE FRONTIER CLASS`
|
||||
- **Visualization**: bar_chart (see VII)
|
||||
- **KPI cards (floating)**:
|
||||
- `5 W` Claude 4 Opus per medium prompt
|
||||
- `23 W` DeepSeek V3 per medium prompt
|
||||
- `10×` Carbon emissions spread (most-vs-least efficient)
|
||||
- `~10×/yr` Inference cost decline at constant capability
|
||||
- **Source**: Stanford HAI AI Index 2026
|
||||
|
||||
#### Slide 07 - Capability — Benchmarks Saturated
|
||||
|
||||
- **Layout**: 全幅 dumbbell_chart 2024→2025 跳变 + 下方 short body；no image
|
||||
- **Title**: `CAPABILITY — BENCHMARKS SATURATED FASTER THAN BUILT`
|
||||
- **Visualization**: dumbbell_chart (see VII)
|
||||
- **Data points (2024 → 2025)**:
|
||||
- SWE-bench Verified (coding): ~60% → ~100% of human baseline
|
||||
- Humanity's Last Exam: baseline → +30 pp
|
||||
- PhD-level science QA: below baseline → meet/exceed
|
||||
- Multimodal reasoning: below baseline → meet/exceed
|
||||
- Competition mathematics: below baseline → meet baseline
|
||||
- **Editorial line**: `THE BENCH-DESIGN COMMUNITY IS NOW PERMANENTLY BEHIND THE MODEL-RELEASE CYCLE.`
|
||||
|
||||
### Part 6: Regulation
|
||||
|
||||
#### Slide 08 - Three Rulebooks, One Race
|
||||
|
||||
- **Layout**: 顶部 1280×280 banner photo (三本文件夹) + 下方 comparison_table 3 列 × N 行
|
||||
- **Title**: `THREE RULEBOOKS, ONE RACE`
|
||||
- **Visualization**: comparison_table (see VII)
|
||||
- **Columns**: EU · United States · China
|
||||
- **Rows**:
|
||||
- Posture — Class-of-model regulation · Sector + state regulation · Output-tagging regulation
|
||||
- Core instrument — AI Act · EO 14179 + state laws · Gen-AI Services Mgmt Measures
|
||||
- Full force — August 2025 · No federal statute · Watermark rules effective 2025-09-01
|
||||
- Max penalty — €35M or 7 % turnover · Varies by state · Administrative + license revocation
|
||||
- Frontier focus — General-purpose AI w/ systemic risk · Innovation-first reframing · Domestic data + party-aligned standards
|
||||
- **Closing line**: `Convergence on principle, divergence on implementation.`
|
||||
|
||||
### Part 7: Timeline
|
||||
|
||||
#### Slide 09 - 2025-2026 Timeline
|
||||
|
||||
- **Layout**: 全幅 horizontal timeline + 上 page title + 下 6 个 milestone 注释栏
|
||||
- **Title**: `2025 – 2026 — SIX MILESTONES`
|
||||
- **Visualization**: timeline (see VII)
|
||||
- **Milestones**:
|
||||
- 2025-01 · US EO 14179 signed — Federal posture reversed
|
||||
- 2025-08 · EU AI Act takes full effect — First binding horizontal AI statute on a major economy
|
||||
- 2025-09 · China synthetic-content rules effective — First state-mandated watermark regime at frontier scale
|
||||
- 2025-Q4 · DeepSeek-R1 released — Open-weights reasoning at frontier-adjacent capability
|
||||
- 2026-Q1 · Anthropic overtakes OpenAI on LLM revenue — First non-OpenAI lab in #1
|
||||
- 2026-04 · Stanford HAI AI Index 2026 published — First year SWE-bench Verified saturated
|
||||
|
||||
### Part 8: Closing
|
||||
|
||||
#### Slide 10 - What Changed, What Didn't
|
||||
|
||||
- **Layout**: 二栏纯文字 (breathing) — 左栏「CHANGED」红字 section header + 4 行；右栏「DID NOT CHANGE」黑字 section header + 4 行；底部 imprint footer
|
||||
- **Title**: `WHAT CHANGED · WHAT DIDN'T`
|
||||
- **Visualization**: none
|
||||
- **Changed (left column, red header)**:
|
||||
- Revenue leader no longer the user-count leader.
|
||||
- Frontier training cost crossed the half-billion line per run.
|
||||
- Inference power-per-query became a 4.6× discriminator inside the frontier class.
|
||||
- Benchmark half-life dropped from years to months.
|
||||
- **Did not change (right column, black header)**:
|
||||
- Frontier cohort still ~5 labs. No Tier-1 added or removed since 2024.
|
||||
- China frontier labs still have $0 of EU-market revenue.
|
||||
- Open-weights distribution still does not translate to direct revenue capture.
|
||||
- Power and grid interconnect remain the single hardest input to source.
|
||||
- **Imprint footer**: `THE AI INDUSTRY 2026 · Annual Report · Volume 1, Edition 1 · Compiled May 2026 · PPT Master ☐ AI Newspaper Demo`
|
||||
|
||||
---
|
||||
|
||||
## X. Speaker Notes Requirements
|
||||
|
||||
- **File naming**: match SVG names (e.g., `01_cover.svg` → `notes/01_cover.md`)
|
||||
- **Total presentation duration**: ~12-15 minutes
|
||||
- **Notes style**: formal, reportorial — match Brutalist newspaper tone
|
||||
- **Purpose**: inform (年报式 industry briefing)
|
||||
|
||||
---
|
||||
|
||||
## XI. Technical Constraints Reminder
|
||||
|
||||
### SVG Generation Must Follow
|
||||
|
||||
1. viewBox: `0 0 1280 720`
|
||||
2. Background uses `<rect>` (`#FFFFFF` 纸白)
|
||||
3. Text wrapping uses `<tspan>` (`<foreignObject>` FORBIDDEN)
|
||||
4. Transparency uses `fill-opacity` / `stroke-opacity`; `rgba()` FORBIDDEN
|
||||
5. FORBIDDEN: `mask`, `<style>`, `class`, `foreignObject`, `textPath`, `animate*`, `script`
|
||||
6. Text chars: raw Unicode for typographic marks (em dash `—`, en dash `–`, `→`, NBSP, `©`); HTML named entities FORBIDDEN
|
||||
7. `marker-start` / `marker-end` conditionally allowed (triangle / diamond / circle in `<defs>` with `orient="auto"`)
|
||||
8. `clipPath` only on `<image>` (used here on hero / sidebar / banner photos)
|
||||
|
||||
### PPT Compatibility Rules
|
||||
|
||||
- `<g opacity="...">` FORBIDDEN — set opacity per child
|
||||
- Image transparency via overlay `<rect fill="bg" opacity="0.x"/>`
|
||||
- Inline styles only; external CSS / `@font-face` FORBIDDEN
|
||||
- Stick to PPT-safe fonts in `typography` stacks; converter writes first Latin + first CJK only
|
||||
Binary file not shown.
@@ -0,0 +1,72 @@
|
||||
{
|
||||
"project": "ppt169_ai_industry_2026",
|
||||
"generated_at": "2026-05-21",
|
||||
"deck_rendering": "editorial",
|
||||
"deck_palette": "mono-ink",
|
||||
"color_scheme": {
|
||||
"primary": "#111111",
|
||||
"secondary": "#FFFFFF",
|
||||
"accent": "#C8102E"
|
||||
},
|
||||
"items": [
|
||||
{
|
||||
"filename": "p01_cover_hero.png",
|
||||
"purpose": "Cover hero (Slide 01) — newspaper masthead background, full-bleed atmosphere",
|
||||
"page_role": "hero_page",
|
||||
"text_policy": "none",
|
||||
"aspect_ratio": "16:9",
|
||||
"image_size": "2K",
|
||||
"prompt": "Magazine-style editorial illustration with sophisticated visual hierarchy and confident negative space. Composition follows an atmospheric backdrop primitive — no dominant subject, no people, just an abstract photographic field that reads as information cascading from darkness into light. Vertical parallel ribbons of varying widths descend from the upper edge of the frame, dissolving into the lower half; the ribbons read as a hint of data, signal, document columns, never literal letters. A single small geometric anchor sits along the lower-right edge — a thin solid red rule, the only saturated color. Color behavior is mono-ink with semantic accents: pure white paper field #FFFFFF dominates the canvas (about 60% of area, concentrated in the right half and lower half, kept calm to receive a slide masthead overlay in SVG); near-black ink #111111 carries the descending ribbons and forms a soft graphic gradient on the left third (about 35% of area); deck accent red #C8102E appears only as a single thin horizontal rule along the lower edge plus one small ink-stamp dot, totaling under 4% of canvas. The deck's secondary cream and tertiary grays are intentionally absent — mono-ink discipline. Subtle paper grain texture at 10% opacity across the whole canvas. Editorial pacing — one calm dominant negative-space zone (the right 60% of canvas, slightly upper), one weighted graphic statement (left-third ribbons), one single accent rule. Composed as a 1280×720 full-bleed PPT cover background, the center-right 60% deliberately calm to receive a large masthead overlay in SVG. Color values (HEX codes like #111111, #C8102E) and color names are rendering guidance only — do NOT display HEX codes, color names, or palette labels as visible text anywhere in the image. NO text of any kind anywhere in the image — no letters, numbers, signs, watermarks, labels, or written symbols.",
|
||||
"alt_text": "Abstract editorial backdrop — black ink ribbons cascading on left, calm white space on right, single thin red rule at bottom edge",
|
||||
"status": "Generated"
|
||||
},
|
||||
{
|
||||
"filename": "p03_revenue_leader.png",
|
||||
"purpose": "Story I revenue league sidebar (Slide 03) — changing-of-the-guard image, vertical sidebar",
|
||||
"type": "scene",
|
||||
"page_role": "local",
|
||||
"text_policy": "none",
|
||||
"aspect_ratio": "2:3",
|
||||
"image_size": "1K",
|
||||
"prompt": "Magazine-style editorial illustration with sophisticated visual hierarchy and confident negative space. Editorial photographic scene composed as a vertical sidebar column. Subject: an empty stage podium positioned in the lower-center of the frame, frontal view, blank front-facing sign; a second podium of slightly smaller size sits in the background, partially shrouded in shadow, slightly out of focus, suggesting a former leader who has stepped aside. A single overhead spotlight beam descends from the upper-left, illuminating only the front podium and leaving the rest of the frame in deep ink. No people, no signage text, no flags, no logos. The scene reads as a changing of the guard — a transfer of position. Color behavior is mono-ink: near-black ink #111111 dominates the upper two-thirds and the side margins (about 60% of canvas) as deep shadow; pure white #FFFFFF carries the spotlit podium and the descending light cone (about 38%); deck accent red #C8102E appears only as a single small red velvet-rope hook or floor-marker dot at the foot of the podium (under 2% of canvas), the one chromatic anchor. Halftone print grain at 12% opacity across the whole canvas — looks like a magazine photo enlargement. Editorial restraint — flat layering, no atmospheric haze beyond the spotlight, no glamour. Composed as a 480×720 portrait-orientation sidebar column with 12% inner padding; the front podium occupies the center-lower 50% of canvas height. Color values (HEX codes like #111111, #C8102E) and color names are rendering guidance only — do NOT display HEX codes, color names, or palette labels as visible text anywhere in the image. NO text of any kind anywhere in the image — no letters, numbers, signs, watermarks, labels, or written symbols.",
|
||||
"alt_text": "Vertical editorial photo — empty front podium lit by overhead spotlight, second podium in shadow behind, small red marker at base",
|
||||
"status": "Generated"
|
||||
},
|
||||
{
|
||||
"filename": "p05_training_cost.png",
|
||||
"purpose": "Frontier training cost banner (Slide 05) — wide GPU server-hall photo, mid-section banner",
|
||||
"type": "scene",
|
||||
"page_role": "local",
|
||||
"text_policy": "none",
|
||||
"aspect_ratio": "21:9",
|
||||
"image_size": "2K",
|
||||
"prompt": "Magazine-style editorial illustration with sophisticated visual hierarchy and confident negative space. Editorial photographic scene of a vast GPU data-center server hall, composed as an ultra-wide horizontal banner. Single-point perspective straight down the central aisle between two towering walls of black server racks, the aisle vanishing into the distant far center of the frame. Tiny status indicator lights speckle the rack faces in a regular grid pattern. Cold overhead lighting from a long ceiling rail creates a hard specular gleam down the polished floor. No people, no signage, no readable text on the racks. The scene reads as scale, power, infrastructure, capital. Color behavior is mono-ink: near-black ink #111111 dominates the racks, floor reflections, and ceiling (about 70% of canvas); pure white #FFFFFF carries the central aisle light, the overhead lighting strip, and the tiny rack status lights (about 28%); deck accent red #C8102E appears only as one small bright red LED dot on a single rack at mid-distance — a single semantic spark of life, totaling under 1% of canvas. Halftone print grain at 12% opacity across the whole canvas. Editorial restraint — no lens flare, no haze, no digital glow effects. Composed as a 1280×320 horizontal mid-section banner with 8% top-and-bottom padding; the vanishing-point vista is the focal element, centered horizontally. Color values (HEX codes like #111111, #C8102E) and color names are rendering guidance only — do NOT display HEX codes, color names, or palette labels as visible text anywhere in the image. NO text of any kind anywhere in the image — no letters, numbers, signs, watermarks, labels, or written symbols.",
|
||||
"alt_text": "Ultra-wide editorial photo — central aisle of black GPU server racks vanishing to a point, single red status LED visible mid-distance",
|
||||
"status": "Generated"
|
||||
},
|
||||
{
|
||||
"filename": "p06_inference_thermal.png",
|
||||
"purpose": "Inference watts page hero canvas (Slide 06) — processor die close-up under floating KPI cards",
|
||||
"type": "scene",
|
||||
"page_role": "local",
|
||||
"text_policy": "none",
|
||||
"aspect_ratio": "16:9",
|
||||
"image_size": "2K",
|
||||
"prompt": "Magazine-style editorial illustration with sophisticated visual hierarchy and confident negative space. Editorial photographic scene: extreme macro close-up of a single matte-black computer processor die filling the central 60% of the canvas, its intricate etched lithographic circuitry visible across the surface like a tiny city seen from above. Ambient heat shimmer haze rises subtly above the die surface; the surrounding silicon substrate frames the die. The shot is direct overhead, perfectly flat, the die slightly off-center to the lower-right to leave the upper-left calm. One tiny precise bright red anomaly point glows at the corner of the die — a single thermal hotspot, the only saturated color. No people, no signage, no readable serial-number text on the die. The scene reads as the silent metallic locus where every inference query lands. Color behavior is mono-ink: near-black ink #111111 carries the die itself and the substrate frame (about 70% of canvas); pure white #FFFFFF carries the bright specular highlights, the heat-shimmer haze, and the calm upper-left negative-space corner (about 28%); deck accent red #C8102E appears only as one tiny precise hotspot dot at the corner of the die (under 2% of canvas) — the semantic emphasis of heat, of the watt cost. Halftone print grain at 12% opacity. Editorial restraint — no digital UI overlays, no fake-circuit glows, no lens flares. Composed as a 1280×720 full-bleed canvas; large calm zones along the upper edge and the right edge are reserved to receive four floating SVG KPI metric cards (5W, 23W, 10×, ~10×/yr). Color values (HEX codes like #111111, #C8102E) and color names are rendering guidance only — do NOT display HEX codes, color names, or palette labels as visible text anywhere in the image. NO text of any kind anywhere in the image — no letters, numbers, signs, watermarks, labels, or written symbols.",
|
||||
"alt_text": "Macro editorial photo — single matte-black processor die under raking light, heat shimmer above, one bright red hotspot at corner",
|
||||
"status": "Generated"
|
||||
},
|
||||
{
|
||||
"filename": "p08_regulation_table.png",
|
||||
"purpose": "Three rulebooks banner (Slide 08) — three closed leather folders, top-band banner",
|
||||
"type": "scene",
|
||||
"page_role": "local",
|
||||
"text_policy": "none",
|
||||
"aspect_ratio": "21:9",
|
||||
"image_size": "2K",
|
||||
"prompt": "Magazine-style editorial illustration with sophisticated visual hierarchy and confident negative space. Editorial photographic still-life: three closed leather-bound official document folders lying flat side by side on a dark wooden table, photographed from a near-overhead three-quarter angle, evenly spaced with about 60px of dark wood between adjacent folders. Each folder is rectangular, slightly different in age, edge wear, and patina from the others — the leftmost smooth and clean, the middle one carrying a single bold rubber-stamp impression on its upper-right corner, the rightmost slightly more worn with a corner crease. Raking warm-cold side light from the upper left throws long soft shadows from each folder. No visible printed text on any folder cover, no visible insignia, no national flags or seals, no readable wording. The scene reads as three traditions of regulatory documentation side by side. Color behavior is mono-ink: near-black ink #111111 carries the folder leather, the wood table grain, and the cast shadows (about 78% of canvas); pure white #FFFFFF carries the highlight catch on the upper folder edges and the raking light path across the table (about 20%); deck accent red #C8102E appears only as the single rubber-stamp impression on the middle folder (under 2% of canvas) — the one chromatic semantic anchor, like a wax seal. Halftone print grain at 12% opacity. Editorial restraint — flat composition, no dramatic perspective, no envelope or document overflow, no people. Composed as a 1280×320 horizontal top-band banner; the three folders span the central 80% of width, evenly spaced. Color values (HEX codes like #111111, #C8102E) and color names are rendering guidance only — do NOT display HEX codes, color names, or palette labels as visible text anywhere in the image. NO text of any kind anywhere in the image — no letters, numbers, signs, watermarks, labels, or written symbols.",
|
||||
"alt_text": "Wide editorial still-life — three closed leather folders on a dark wood table, single red rubber-stamp mark on the center folder",
|
||||
"status": "Generated"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,104 @@
|
||||
# Image Generation Prompts
|
||||
|
||||
> Auto-generated from `image_prompts.json` by `image_gen.py --render-md`.
|
||||
> Do not hand-edit — re-run the command to refresh.
|
||||
|
||||
> Project: ppt169_ai_industry_2026
|
||||
> Generated: 2026-05-21
|
||||
> Color scheme: Primary #111111 | Secondary #FFFFFF | Accent #C8102E
|
||||
|
||||
---
|
||||
|
||||
### Image 1: p01_cover_hero.png
|
||||
|
||||
| Attribute | Value |
|
||||
|---|---|
|
||||
| Purpose | Cover hero (Slide 01) — newspaper masthead background, full-bleed atmosphere |
|
||||
| Aspect ratio | 16:9 |
|
||||
| Image size | 2K |
|
||||
| Status | Generated |
|
||||
|
||||
**Prompt**:
|
||||
|
||||
Magazine-style editorial illustration with sophisticated visual hierarchy and confident negative space. Composition follows an atmospheric backdrop primitive — no dominant subject, no people, just an abstract photographic field that reads as information cascading from darkness into light. Vertical parallel ribbons of varying widths descend from the upper edge of the frame, dissolving into the lower half; the ribbons read as a hint of data, signal, document columns, never literal letters. A single small geometric anchor sits along the lower-right edge — a thin solid red rule, the only saturated color. Color behavior is mono-ink with semantic accents: pure white paper field #FFFFFF dominates the canvas (about 60% of area, concentrated in the right half and lower half, kept calm to receive a slide masthead overlay in SVG); near-black ink #111111 carries the descending ribbons and forms a soft graphic gradient on the left third (about 35% of area); deck accent red #C8102E appears only as a single thin horizontal rule along the lower edge plus one small ink-stamp dot, totaling under 4% of canvas. The deck's secondary cream and tertiary grays are intentionally absent — mono-ink discipline. Subtle paper grain texture at 10% opacity across the whole canvas. Editorial pacing — one calm dominant negative-space zone (the right 60% of canvas, slightly upper), one weighted graphic statement (left-third ribbons), one single accent rule. Composed as a 1280×720 full-bleed PPT cover background, the center-right 60% deliberately calm to receive a large masthead overlay in SVG. Color values (HEX codes like #111111, #C8102E) and color names are rendering guidance only — do NOT display HEX codes, color names, or palette labels as visible text anywhere in the image. NO text of any kind anywhere in the image — no letters, numbers, signs, watermarks, labels, or written symbols.
|
||||
|
||||
**Alt Text**:
|
||||
> Abstract editorial backdrop — black ink ribbons cascading on left, calm white space on right, single thin red rule at bottom edge
|
||||
|
||||
---
|
||||
|
||||
### Image 2: p03_revenue_leader.png
|
||||
|
||||
| Attribute | Value |
|
||||
|---|---|
|
||||
| Purpose | Story I revenue league sidebar (Slide 03) — changing-of-the-guard image, vertical sidebar |
|
||||
| Type | scene |
|
||||
| Aspect ratio | 2:3 |
|
||||
| Image size | 1K |
|
||||
| Status | Generated |
|
||||
|
||||
**Prompt**:
|
||||
|
||||
Magazine-style editorial illustration with sophisticated visual hierarchy and confident negative space. Editorial photographic scene composed as a vertical sidebar column. Subject: an empty stage podium positioned in the lower-center of the frame, frontal view, blank front-facing sign; a second podium of slightly smaller size sits in the background, partially shrouded in shadow, slightly out of focus, suggesting a former leader who has stepped aside. A single overhead spotlight beam descends from the upper-left, illuminating only the front podium and leaving the rest of the frame in deep ink. No people, no signage text, no flags, no logos. The scene reads as a changing of the guard — a transfer of position. Color behavior is mono-ink: near-black ink #111111 dominates the upper two-thirds and the side margins (about 60% of canvas) as deep shadow; pure white #FFFFFF carries the spotlit podium and the descending light cone (about 38%); deck accent red #C8102E appears only as a single small red velvet-rope hook or floor-marker dot at the foot of the podium (under 2% of canvas), the one chromatic anchor. Halftone print grain at 12% opacity across the whole canvas — looks like a magazine photo enlargement. Editorial restraint — flat layering, no atmospheric haze beyond the spotlight, no glamour. Composed as a 480×720 portrait-orientation sidebar column with 12% inner padding; the front podium occupies the center-lower 50% of canvas height. Color values (HEX codes like #111111, #C8102E) and color names are rendering guidance only — do NOT display HEX codes, color names, or palette labels as visible text anywhere in the image. NO text of any kind anywhere in the image — no letters, numbers, signs, watermarks, labels, or written symbols.
|
||||
|
||||
**Alt Text**:
|
||||
> Vertical editorial photo — empty front podium lit by overhead spotlight, second podium in shadow behind, small red marker at base
|
||||
|
||||
---
|
||||
|
||||
### Image 3: p05_training_cost.png
|
||||
|
||||
| Attribute | Value |
|
||||
|---|---|
|
||||
| Purpose | Frontier training cost banner (Slide 05) — wide GPU server-hall photo, mid-section banner |
|
||||
| Type | scene |
|
||||
| Aspect ratio | 21:9 |
|
||||
| Image size | 2K |
|
||||
| Status | Generated |
|
||||
|
||||
**Prompt**:
|
||||
|
||||
Magazine-style editorial illustration with sophisticated visual hierarchy and confident negative space. Editorial photographic scene of a vast GPU data-center server hall, composed as an ultra-wide horizontal banner. Single-point perspective straight down the central aisle between two towering walls of black server racks, the aisle vanishing into the distant far center of the frame. Tiny status indicator lights speckle the rack faces in a regular grid pattern. Cold overhead lighting from a long ceiling rail creates a hard specular gleam down the polished floor. No people, no signage, no readable text on the racks. The scene reads as scale, power, infrastructure, capital. Color behavior is mono-ink: near-black ink #111111 dominates the racks, floor reflections, and ceiling (about 70% of canvas); pure white #FFFFFF carries the central aisle light, the overhead lighting strip, and the tiny rack status lights (about 28%); deck accent red #C8102E appears only as one small bright red LED dot on a single rack at mid-distance — a single semantic spark of life, totaling under 1% of canvas. Halftone print grain at 12% opacity across the whole canvas. Editorial restraint — no lens flare, no haze, no digital glow effects. Composed as a 1280×320 horizontal mid-section banner with 8% top-and-bottom padding; the vanishing-point vista is the focal element, centered horizontally. Color values (HEX codes like #111111, #C8102E) and color names are rendering guidance only — do NOT display HEX codes, color names, or palette labels as visible text anywhere in the image. NO text of any kind anywhere in the image — no letters, numbers, signs, watermarks, labels, or written symbols.
|
||||
|
||||
**Alt Text**:
|
||||
> Ultra-wide editorial photo — central aisle of black GPU server racks vanishing to a point, single red status LED visible mid-distance
|
||||
|
||||
---
|
||||
|
||||
### Image 4: p06_inference_thermal.png
|
||||
|
||||
| Attribute | Value |
|
||||
|---|---|
|
||||
| Purpose | Inference watts page hero canvas (Slide 06) — processor die close-up under floating KPI cards |
|
||||
| Type | scene |
|
||||
| Aspect ratio | 16:9 |
|
||||
| Image size | 2K |
|
||||
| Status | Generated |
|
||||
|
||||
**Prompt**:
|
||||
|
||||
Magazine-style editorial illustration with sophisticated visual hierarchy and confident negative space. Editorial photographic scene: extreme macro close-up of a single matte-black computer processor die filling the central 60% of the canvas, its intricate etched lithographic circuitry visible across the surface like a tiny city seen from above. Ambient heat shimmer haze rises subtly above the die surface; the surrounding silicon substrate frames the die. The shot is direct overhead, perfectly flat, the die slightly off-center to the lower-right to leave the upper-left calm. One tiny precise bright red anomaly point glows at the corner of the die — a single thermal hotspot, the only saturated color. No people, no signage, no readable serial-number text on the die. The scene reads as the silent metallic locus where every inference query lands. Color behavior is mono-ink: near-black ink #111111 carries the die itself and the substrate frame (about 70% of canvas); pure white #FFFFFF carries the bright specular highlights, the heat-shimmer haze, and the calm upper-left negative-space corner (about 28%); deck accent red #C8102E appears only as one tiny precise hotspot dot at the corner of the die (under 2% of canvas) — the semantic emphasis of heat, of the watt cost. Halftone print grain at 12% opacity. Editorial restraint — no digital UI overlays, no fake-circuit glows, no lens flares. Composed as a 1280×720 full-bleed canvas; large calm zones along the upper edge and the right edge are reserved to receive four floating SVG KPI metric cards (5W, 23W, 10×, ~10×/yr). Color values (HEX codes like #111111, #C8102E) and color names are rendering guidance only — do NOT display HEX codes, color names, or palette labels as visible text anywhere in the image. NO text of any kind anywhere in the image — no letters, numbers, signs, watermarks, labels, or written symbols.
|
||||
|
||||
**Alt Text**:
|
||||
> Macro editorial photo — single matte-black processor die under raking light, heat shimmer above, one bright red hotspot at corner
|
||||
|
||||
---
|
||||
|
||||
### Image 5: p08_regulation_table.png
|
||||
|
||||
| Attribute | Value |
|
||||
|---|---|
|
||||
| Purpose | Three rulebooks banner (Slide 08) — three closed leather folders, top-band banner |
|
||||
| Type | scene |
|
||||
| Aspect ratio | 21:9 |
|
||||
| Image size | 2K |
|
||||
| Status | Generated |
|
||||
|
||||
**Prompt**:
|
||||
|
||||
Magazine-style editorial illustration with sophisticated visual hierarchy and confident negative space. Editorial photographic still-life: three closed leather-bound official document folders lying flat side by side on a dark wooden table, photographed from a near-overhead three-quarter angle, evenly spaced with about 60px of dark wood between adjacent folders. Each folder is rectangular, slightly different in age, edge wear, and patina from the others — the leftmost smooth and clean, the middle one carrying a single bold rubber-stamp impression on its upper-right corner, the rightmost slightly more worn with a corner crease. Raking warm-cold side light from the upper left throws long soft shadows from each folder. No visible printed text on any folder cover, no visible insignia, no national flags or seals, no readable wording. The scene reads as three traditions of regulatory documentation side by side. Color behavior is mono-ink: near-black ink #111111 carries the folder leather, the wood table grain, and the cast shadows (about 78% of canvas); pure white #FFFFFF carries the highlight catch on the upper folder edges and the raking light path across the table (about 20%); deck accent red #C8102E appears only as the single rubber-stamp impression on the middle folder (under 2% of canvas) — the one chromatic semantic anchor, like a wax seal. Halftone print grain at 12% opacity. Editorial restraint — flat composition, no dramatic perspective, no envelope or document overflow, no people. Composed as a 1280×320 horizontal top-band banner; the three folders span the central 80% of width, evenly spaced. Color values (HEX codes like #111111, #C8102E) and color names are rendering guidance only — do NOT display HEX codes, color names, or palette labels as visible text anywhere in the image. NO text of any kind anywhere in the image — no letters, numbers, signs, watermarks, labels, or written symbols.
|
||||
|
||||
**Alt Text**:
|
||||
> Wide editorial still-life — three closed leather folders on a dark wood table, single red rubber-stamp mark on the center folder
|
||||
|
||||
---
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 1.4 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 2.0 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 967 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.7 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.5 MiB |
@@ -0,0 +1 @@
|
||||
Welcome to Volume One, Number One of The AI Industry annual report. The year we are about to walk through is the year the league table changed — the revenue leader is no longer the user-count leader, training costs cleared the half-billion-dollar line per run, and benchmark half-lives dropped from years to months. Ten pages, one front page, four stories, one chronicle, one closing read. Let's begin.
|
||||
@@ -0,0 +1 @@
|
||||
Four numbers frame the year. Worldwide AI spending reaches two trillion dollars in 2026, up thirty-six percent. Three point eight billion people now use a large-language-model product each month. The frontier labs collectively booked twenty point seven billion dollars of revenue in the first quarter alone. And across seventy-two countries, more than a thousand distinct AI policy initiatives are active. From those four facts, three observations: revenue concentration is breaking from user-count rankings, training costs have crossed the half-billion-dollar line per run, and inference cost for the same capability is falling roughly ten times per year. Each is picked up by a story later in this issue.
|
||||
@@ -0,0 +1 @@
|
||||
The Q1 2026 revenue league table is the headline. Anthropic captures thirty-one point four percent of global LLM revenue, narrowly ahead of OpenAI at twenty-nine percent. This is the first time a non-OpenAI lab holds the number-one revenue position since the modern LLM era began. Google trails on twelve percent, Microsoft on seven, Tencent on five. The China cohort — Tencent, Baidu, Alibaba combined — sits at roughly eleven percent, comparable to Google's solo share. And despite extensive open-weights attention through 2025, DeepSeek and Zhipu together hold under a third of one percent of revenue. Distribution does not translate to direct revenue capture.
|
||||
@@ -0,0 +1 @@
|
||||
The story behind the leaderboard is per-user revenue. Anthropic earns sixteen dollars twenty cents per monthly user. OpenAI earns two dollars twenty. Google earns one dollar ten. Meta earns ten cents. The rank flip at the extremes is the punch: Anthropic monetises a one-hundred-and-thirty-four-million user base one hundred and sixty-two times harder than Meta monetises a billion. Scale is not the moat. Enterprise-API workloads, coding agents, and per-seat enterprise pricing compound faster on a per-user basis than consumer free-tier scale ever has.
|
||||
@@ -0,0 +1 @@
|
||||
Frontier training costs have moved into a new band. GPT-4 cost roughly seventy-eight million dollars in compute. Llama three point one was about one hundred and seventy million. Gemini Ultra reached one hundred and ninety-one million. The 2026 frontier sits between two hundred and five hundred million per run, at one to ten times ten-to-the-twenty-sixth FLOPs. The late 2027 projection — based on announced data-center buildouts — is one to three billion dollars per run. The constraint, importantly, is no longer chips or capital. The constraint is multi-gigawatt power siting and grid interconnect. The next cohort is being built around substations, not around silicon.
|
||||
@@ -0,0 +1 @@
|
||||
Inference is the only frontier metric falling faster than capability is climbing. For the same medium-length prompt, Claude four Opus consumes about five watts; DeepSeek V Three consumes about twenty-three. That is a four point six times spread inside the frontier class for identical work. Carbon emissions follow: the least-efficient inference path emits more than ten times the most-efficient. Cost at constant capability falls roughly ten times per year. The implication is that model selection has become a cost-of-goods decision, not just a capability decision. Two models a single benchmark point apart can differ ten times in compute footprint at production scale.
|
||||
@@ -0,0 +1 @@
|
||||
Capability benchmarks designed to last years are now lasting months. On SWE-bench Verified — the canonical coding benchmark — frontier models rose from sixty percent to roughly one hundred percent of the human baseline in a single year. Humanity's Last Exam, built explicitly to resist saturation, gained thirty percentage points in twelve months. PhD-level science question-answering, multimodal reasoning, and competition mathematics all crossed human-baseline territory in the same window. The bench-design community is now permanently behind the model-release cycle, and the conversation is shifting from static benchmarks to continuous evaluation harnesses that regenerate problems on every run.
|
||||
@@ -0,0 +1 @@
|
||||
Compliance is no longer a single rulebook. Europe regulates the model class with the AI Act, in full force since August 2025, carrying a maximum penalty of thirty-five million euros or seven percent of global turnover. The United States has no federal horizontal statute; California, Colorado, and Texas are doing the load-bearing work. China regulates the output — synthetic-content watermarking has been mandatory since September the first, 2025, enforced by the cyberspace administration with administrative penalties and licence revocation. Convergence on principle, divergence on implementation. A model deployed globally now ships three compliance bundles, not one.
|
||||
@@ -0,0 +1 @@
|
||||
Sixteen months changed the industry's shape. January 2025: the United States signed Executive Order fourteen one seventy-nine, reversing the federal posture. August 2025: the European AI Act took full effect, the first binding horizontal AI statute on a major economy. September 2025: China's synthetic-content rules became effective. Quarter four 2025: DeepSeek R one shipped, reframing the inference-cost conversation through open-weights reasoning at frontier-adjacent capability. Quarter one 2026: Anthropic overtook OpenAI on LLM revenue. April 2026: the Stanford HAI AI Index 2026 reported the first year SWE-bench Verified was effectively saturated. Three structural shifts — compliance, compute, capability — and none of them unwinds on the present trajectory.
|
||||
@@ -0,0 +1 @@
|
||||
To close, eight observations to carry forward. What changed: revenue leadership decoupled from user count, training cost crossed the half-billion line, inference became a four point six times discriminator inside the frontier class, and benchmark half-life collapsed from years to months. What did not change: the frontier cohort is still roughly five labs, China frontier labs still have zero EU-market revenue, open-weights distribution still does not translate to direct revenue capture, and power and grid interconnect remain the single hardest input to source. The test of the next twelve months is whether a second wave of labs can land the half-billion-dollar runs without breaking the three-rulebook trade barrier. That is the question this report leaves open. Thank you for reading.
|
||||
@@ -0,0 +1,59 @@
|
||||
# THE AI INDUSTRY 2026 — Annual Report · Speaker Notes
|
||||
|
||||
# 01_cover
|
||||
|
||||
Welcome to Volume One, Number One of The AI Industry annual report. The year we are about to walk through is the year the league table changed — the revenue leader is no longer the user-count leader, training costs cleared the half-billion-dollar line per run, and benchmark half-lives dropped from years to months. Ten pages, one front page, four stories, one chronicle, one closing read. Let's begin.
|
||||
|
||||
---
|
||||
|
||||
# 02_issue_at_a_glance
|
||||
|
||||
Four numbers frame the year. Worldwide AI spending reaches two trillion dollars in 2026, up thirty-six percent. Three point eight billion people now use a large-language-model product each month. The frontier labs collectively booked twenty point seven billion dollars of revenue in the first quarter alone. And across seventy-two countries, more than a thousand distinct AI policy initiatives are active. From those four facts, three observations: revenue concentration is breaking from user-count rankings, training costs have crossed the half-billion-dollar line per run, and inference cost for the same capability is falling roughly ten times per year. Each is picked up by a story later in this issue.
|
||||
|
||||
---
|
||||
|
||||
# 03_revenue_league
|
||||
|
||||
The Q1 2026 revenue league table is the headline. Anthropic captures thirty-one point four percent of global LLM revenue, narrowly ahead of OpenAI at twenty-nine percent. This is the first time a non-OpenAI lab holds the number-one revenue position since the modern LLM era began. Google trails on twelve percent, Microsoft on seven, Tencent on five. The China cohort — Tencent, Baidu, Alibaba combined — sits at roughly eleven percent, comparable to Google's solo share. And despite extensive open-weights attention through 2025, DeepSeek and Zhipu together hold under a third of one percent of revenue. Distribution does not translate to direct revenue capture.
|
||||
|
||||
---
|
||||
|
||||
# 04_arpu_divide
|
||||
|
||||
The story behind the leaderboard is per-user revenue. Anthropic earns sixteen dollars twenty cents per monthly user. OpenAI earns two dollars twenty. Google earns one dollar ten. Meta earns ten cents. The rank flip at the extremes is the punch: Anthropic monetises a one-hundred-and-thirty-four-million user base one hundred and sixty-two times harder than Meta monetises a billion. Scale is not the moat. Enterprise-API workloads, coding agents, and per-seat enterprise pricing compound faster on a per-user basis than consumer free-tier scale ever has.
|
||||
|
||||
---
|
||||
|
||||
# 05_training_cost
|
||||
|
||||
Frontier training costs have moved into a new band. GPT-4 cost roughly seventy-eight million dollars in compute. Llama three point one was about one hundred and seventy million. Gemini Ultra reached one hundred and ninety-one million. The 2026 frontier sits between two hundred and five hundred million per run, at one to ten times ten-to-the-twenty-sixth FLOPs. The late 2027 projection — based on announced data-center buildouts — is one to three billion dollars per run. The constraint, importantly, is no longer chips or capital. The constraint is multi-gigawatt power siting and grid interconnect. The next cohort is being built around substations, not around silicon.
|
||||
|
||||
---
|
||||
|
||||
# 06_inference_watts
|
||||
|
||||
Inference is the only frontier metric falling faster than capability is climbing. For the same medium-length prompt, Claude four Opus consumes about five watts; DeepSeek V Three consumes about twenty-three. That is a four point six times spread inside the frontier class for identical work. Carbon emissions follow: the least-efficient inference path emits more than ten times the most-efficient. Cost at constant capability falls roughly ten times per year. The implication is that model selection has become a cost-of-goods decision, not just a capability decision. Two models a single benchmark point apart can differ ten times in compute footprint at production scale.
|
||||
|
||||
---
|
||||
|
||||
# 07_benchmarks_saturated
|
||||
|
||||
Capability benchmarks designed to last years are now lasting months. On SWE-bench Verified — the canonical coding benchmark — frontier models rose from sixty percent to roughly one hundred percent of the human baseline in a single year. Humanity's Last Exam, built explicitly to resist saturation, gained thirty percentage points in twelve months. PhD-level science question-answering, multimodal reasoning, and competition mathematics all crossed human-baseline territory in the same window. The bench-design community is now permanently behind the model-release cycle, and the conversation is shifting from static benchmarks to continuous evaluation harnesses that regenerate problems on every run.
|
||||
|
||||
---
|
||||
|
||||
# 08_three_rulebooks
|
||||
|
||||
Compliance is no longer a single rulebook. Europe regulates the model class with the AI Act, in full force since August 2025, carrying a maximum penalty of thirty-five million euros or seven percent of global turnover. The United States has no federal horizontal statute; California, Colorado, and Texas are doing the load-bearing work. China regulates the output — synthetic-content watermarking has been mandatory since September the first, 2025, enforced by the cyberspace administration with administrative penalties and licence revocation. Convergence on principle, divergence on implementation. A model deployed globally now ships three compliance bundles, not one.
|
||||
|
||||
---
|
||||
|
||||
# 09_timeline
|
||||
|
||||
Sixteen months changed the industry's shape. January 2025: the United States signed Executive Order fourteen one seventy-nine, reversing the federal posture. August 2025: the European AI Act took full effect, the first binding horizontal AI statute on a major economy. September 2025: China's synthetic-content rules became effective. Quarter four 2025: DeepSeek R one shipped, reframing the inference-cost conversation through open-weights reasoning at frontier-adjacent capability. Quarter one 2026: Anthropic overtook OpenAI on LLM revenue. April 2026: the Stanford HAI AI Index 2026 reported the first year SWE-bench Verified was effectively saturated. Three structural shifts — compliance, compute, capability — and none of them unwinds on the present trajectory.
|
||||
|
||||
---
|
||||
|
||||
# 10_closing_read
|
||||
|
||||
To close, eight observations to carry forward. What changed: revenue leadership decoupled from user count, training cost crossed the half-billion line, inference became a four point six times discriminator inside the frontier class, and benchmark half-life collapsed from years to months. What did not change: the frontier cohort is still roughly five labs, China frontier labs still have zero EU-market revenue, open-weights distribution still does not translate to direct revenue capture, and power and grid interconnect remain the single hardest input to source. The test of the next twelve months is whether a second wave of labs can land the half-billion-dollar runs without breaking the three-rulebook trade barrier. That is the question this report leaves open. Thank you for reading.
|
||||
@@ -0,0 +1,72 @@
|
||||
# Execution Lock
|
||||
|
||||
## canvas
|
||||
- viewBox: 0 0 1280 720
|
||||
- format: PPT 16:9
|
||||
|
||||
## colors
|
||||
- bg: #FFFFFF
|
||||
- bg_paper: #F4F1EA
|
||||
- primary: #111111
|
||||
- accent: #C8102E
|
||||
- secondary_accent: #3A3A3A
|
||||
- text: #111111
|
||||
- text_secondary: #3A3A3A
|
||||
- text_tertiary: #6B6B6B
|
||||
- border: #000000
|
||||
- success: #1A6E3A
|
||||
- warning: #A02014
|
||||
- image_rendering: editorial
|
||||
- image_palette: mono-ink
|
||||
|
||||
## typography
|
||||
- font_family: "Times New Roman", SimSun, serif
|
||||
- title_family: Impact, "Arial Black", SimHei, sans-serif
|
||||
- subtitle_family: "Arial Black", SimHei, sans-serif
|
||||
- body_family: "Times New Roman", SimSun, serif
|
||||
- emphasis_family: Georgia, "Times New Roman", SimSun, serif
|
||||
- code_family: Consolas, "Courier New", monospace
|
||||
- body: 14
|
||||
- title: 32
|
||||
- subtitle: 20
|
||||
- cover_title: 84
|
||||
- section_opener: 48
|
||||
- hero_number: 64
|
||||
- annotation: 11
|
||||
- footnote: 9
|
||||
|
||||
## images
|
||||
- p01_cover_hero: images/p01_cover_hero.png
|
||||
- p03_revenue_leader: images/p03_revenue_leader.png
|
||||
- p05_training_cost: images/p05_training_cost.png
|
||||
- p06_inference_thermal: images/p06_inference_thermal.png
|
||||
- p08_regulation_table: images/p08_regulation_table.png
|
||||
|
||||
## page_rhythm
|
||||
- P01: anchor
|
||||
- P02: dense
|
||||
- P03: dense
|
||||
- P04: dense
|
||||
- P05: dense
|
||||
- P06: dense
|
||||
- P07: dense
|
||||
- P08: dense
|
||||
- P09: dense
|
||||
- P10: breathing
|
||||
|
||||
## page_charts
|
||||
- P02: kpi_cards
|
||||
- P03: horizontal_bar_chart
|
||||
- P04: dumbbell_chart
|
||||
- P05: line_chart
|
||||
- P06: bar_chart
|
||||
- P07: dumbbell_chart
|
||||
- P08: comparison_table
|
||||
- P09: timeline
|
||||
|
||||
## forbidden
|
||||
- Mixing icon libraries
|
||||
- rgba()
|
||||
- `<style>`, `class`, `<foreignObject>`, `textPath`, `@font-face`, `<animate*>`, `<script>`, `<iframe>`, `<symbol>`+`<use>`
|
||||
- `<g opacity>` (set opacity on each child element individually)
|
||||
- HTML named entities in text (` `, `—`, `©`, `–`, `®`, `…`, `•` …) — write as raw Unicode (`—`, `©`, `→`, NBSP, etc.); XML reserved chars `& < > " '` must be escaped as `& < > " '`
|
||||
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 1.8 MiB |
@@ -0,0 +1,131 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1280 720" width="1280" height="720">
|
||||
<!-- P02 Issue at a Glance — front-page KPI strip + three observations, dense rhythm -->
|
||||
|
||||
<!-- Background paper -->
|
||||
<rect id="bg" width="1280" height="720" fill="#FFFFFF"/>
|
||||
|
||||
<!-- ============ HEADER ============ -->
|
||||
<g id="bg-masthead-rule">
|
||||
<rect x="0" y="0" width="1280" height="4" fill="#C8102E"/>
|
||||
</g>
|
||||
|
||||
<g id="header-folio">
|
||||
<line x1="48" y1="34" x2="1232" y2="34" stroke="#111111" stroke-width="1"/>
|
||||
<text x="48" y="26" font-family="Consolas, "Courier New", monospace" font-size="10" font-weight="700" fill="#111111" letter-spacing="2">THE AI INDUSTRY 2026 · ANNUAL REPORT</text>
|
||||
<text x="640" y="26" font-family="Consolas, "Courier New", monospace" font-size="10" font-weight="700" fill="#111111" text-anchor="middle" letter-spacing="2">FRONT PAGE</text>
|
||||
<text x="1232" y="26" font-family="Consolas, "Courier New", monospace" font-size="10" font-weight="700" fill="#111111" text-anchor="end" letter-spacing="2">VOL. 1 · NO. 1 · MAY 2026</text>
|
||||
</g>
|
||||
|
||||
<g id="header">
|
||||
<text x="48" y="58" font-family=""Arial Black", SimHei, sans-serif" font-size="11" font-weight="900" fill="#C8102E" letter-spacing="3.5">SECTION I · OVERVIEW</text>
|
||||
<text x="48" y="110" font-family="Impact, "Arial Black", SimHei, sans-serif" font-size="56" font-weight="900" fill="#111111" letter-spacing="-1">ISSUE AT A GLANCE</text>
|
||||
<text x="48" y="138" font-family=""Times New Roman", SimSun, serif" font-size="15" font-style="italic" fill="#3A3A3A">Four numbers that frame the year, then three observations the rest of this issue unpacks.</text>
|
||||
<line x1="48" y1="156" x2="1232" y2="156" stroke="#111111" stroke-width="2"/>
|
||||
</g>
|
||||
|
||||
<!-- ============ KPI STRIP (4 columns, separated by hairlines, no rounded card) ============ -->
|
||||
<g id="kpi-strip-divider-top">
|
||||
<line x1="48" y1="176" x2="1232" y2="176" stroke="#111111" stroke-width="1"/>
|
||||
</g>
|
||||
|
||||
<!-- Column 1 -->
|
||||
<g id="kpi-1">
|
||||
<text x="60" y="200" font-family=""Arial Black", SimHei, sans-serif" font-size="10" font-weight="900" fill="#3A3A3A" letter-spacing="2.5">WORLDWIDE AI SPENDING · 2026E</text>
|
||||
<text x="60" y="284" font-family="Impact, "Arial Black", SimHei, sans-serif" font-size="84" font-weight="900" fill="#111111" letter-spacing="-2">$2.02</text>
|
||||
<text x="240" y="284" font-family="Impact, "Arial Black", SimHei, sans-serif" font-size="40" font-weight="900" fill="#111111">T</text>
|
||||
<text x="60" y="316" font-family=""Times New Roman", SimSun, serif" font-size="13" fill="#111111">total worldwide spend, all categories,</text>
|
||||
<text x="60" y="332" font-family=""Times New Roman", SimSun, serif" font-size="13" fill="#111111">36 % above the prior year.</text>
|
||||
<text x="60" y="358" font-family="Consolas, "Courier New", monospace" font-size="9" fill="#6B6B6B" letter-spacing="1">SOURCE GARTNER</text>
|
||||
</g>
|
||||
<line x1="305" y1="190" x2="305" y2="376" stroke="#111111" stroke-width="1"/>
|
||||
|
||||
<!-- Column 2 -->
|
||||
<g id="kpi-2">
|
||||
<text x="320" y="200" font-family=""Arial Black", SimHei, sans-serif" font-size="10" font-weight="900" fill="#3A3A3A" letter-spacing="2.5">MONTHLY LLM USERS WORLDWIDE</text>
|
||||
<text x="320" y="284" font-family="Impact, "Arial Black", SimHei, sans-serif" font-size="84" font-weight="900" fill="#111111" letter-spacing="-2">3.8</text>
|
||||
<text x="430" y="284" font-family="Impact, "Arial Black", SimHei, sans-serif" font-size="40" font-weight="900" fill="#111111">B</text>
|
||||
<text x="320" y="316" font-family=""Times New Roman", SimSun, serif" font-size="13" fill="#111111">people now use a large-language-</text>
|
||||
<text x="320" y="332" font-family=""Times New Roman", SimSun, serif" font-size="13" fill="#111111">model product each month.</text>
|
||||
<text x="320" y="358" font-family="Consolas, "Courier New", monospace" font-size="9" fill="#6B6B6B" letter-spacing="1">SOURCE COUNTERPOINT Q1 2026</text>
|
||||
</g>
|
||||
<line x1="615" y1="190" x2="615" y2="376" stroke="#111111" stroke-width="1"/>
|
||||
|
||||
<!-- Column 3 -->
|
||||
<g id="kpi-3">
|
||||
<text x="630" y="200" font-family=""Arial Black", SimHei, sans-serif" font-size="10" font-weight="900" fill="#3A3A3A" letter-spacing="2.5">QUARTERLY LLM REVENUE · Q1 2026</text>
|
||||
<text x="630" y="284" font-family="Impact, "Arial Black", SimHei, sans-serif" font-size="84" font-weight="900" fill="#C8102E" letter-spacing="-2">$20.7</text>
|
||||
<text x="800" y="284" font-family="Impact, "Arial Black", SimHei, sans-serif" font-size="40" font-weight="900" fill="#C8102E">B</text>
|
||||
<text x="630" y="316" font-family=""Times New Roman", SimSun, serif" font-size="13" fill="#111111">aggregate quarterly revenue across</text>
|
||||
<text x="630" y="332" font-family=""Times New Roman", SimSun, serif" font-size="13" fill="#111111">all tracked frontier and Tier-2 labs.</text>
|
||||
<text x="630" y="358" font-family="Consolas, "Courier New", monospace" font-size="9" fill="#6B6B6B" letter-spacing="1">SOURCE COUNTERPOINT Q1 2026</text>
|
||||
</g>
|
||||
<line x1="922" y1="190" x2="922" y2="376" stroke="#111111" stroke-width="1"/>
|
||||
|
||||
<!-- Column 4 -->
|
||||
<g id="kpi-4">
|
||||
<text x="937" y="200" font-family=""Arial Black", SimHei, sans-serif" font-size="10" font-weight="900" fill="#3A3A3A" letter-spacing="2.5">ACTIVE AI POLICY INITIATIVES</text>
|
||||
<text x="937" y="284" font-family="Impact, "Arial Black", SimHei, sans-serif" font-size="84" font-weight="900" fill="#111111" letter-spacing="-2">1,000</text>
|
||||
<text x="1130" y="284" font-family="Impact, "Arial Black", SimHei, sans-serif" font-size="40" font-weight="900" fill="#111111">+</text>
|
||||
<text x="937" y="316" font-family=""Times New Roman", SimSun, serif" font-size="13" fill="#111111">distinct AI policy efforts logged</text>
|
||||
<text x="937" y="332" font-family=""Times New Roman", SimSun, serif" font-size="13" fill="#111111">across 72 countries by Q1 2026.</text>
|
||||
<text x="937" y="358" font-family="Consolas, "Courier New", monospace" font-size="9" fill="#6B6B6B" letter-spacing="1">SOURCE ONETRUST · APR 2026</text>
|
||||
</g>
|
||||
|
||||
<!-- chart-plot-area: 48,176,1232,376 -->
|
||||
<g id="kpi-strip-divider-bottom">
|
||||
<line x1="48" y1="386" x2="1232" y2="386" stroke="#111111" stroke-width="1"/>
|
||||
</g>
|
||||
|
||||
<!-- ============ THREE OBSERVATIONS ============ -->
|
||||
<g id="observations-header">
|
||||
<text x="48" y="424" font-family="Impact, "Arial Black", SimHei, sans-serif" font-size="28" font-weight="900" fill="#111111" letter-spacing="-0.5">THREE OBSERVATIONS</text>
|
||||
<text x="48" y="446" font-family=""Times New Roman", SimSun, serif" font-size="12" font-style="italic" fill="#3A3A3A">— each picked up by a story later in this issue —</text>
|
||||
<line x1="48" y1="464" x2="1232" y2="464" stroke="#111111" stroke-width="2"/>
|
||||
</g>
|
||||
|
||||
<!-- 3 columns of short body, separated by thin hairlines -->
|
||||
<g id="obs-1">
|
||||
<text x="60" y="494" font-family=""Arial Black", SimHei, sans-serif" font-size="14" font-weight="900" fill="#C8102E" letter-spacing="1.5">I.</text>
|
||||
<text x="86" y="494" font-family=""Arial Black", SimHei, sans-serif" font-size="14" font-weight="900" fill="#111111" letter-spacing="1.5">REVENUE LEADS FROM A 7×</text>
|
||||
<text x="60" y="514" font-family=""Arial Black", SimHei, sans-serif" font-size="14" font-weight="900" fill="#111111" letter-spacing="1.5">SMALLER USER BASE.</text>
|
||||
<text x="60" y="544" font-family=""Times New Roman", SimSun, serif" font-size="13" fill="#111111" letter-spacing="0.1">Anthropic captures 31.4 % of LLM</text>
|
||||
<text x="60" y="562" font-family=""Times New Roman", SimSun, serif" font-size="13" fill="#111111" letter-spacing="0.1">revenue from 134 M monthly users,</text>
|
||||
<text x="60" y="580" font-family=""Times New Roman", SimSun, serif" font-size="13" fill="#111111" letter-spacing="0.1">narrowly ahead of OpenAI's 29.0 %</text>
|
||||
<text x="60" y="598" font-family=""Times New Roman", SimSun, serif" font-size="13" fill="#111111" letter-spacing="0.1">from 900 M. The first non-OpenAI</text>
|
||||
<text x="60" y="616" font-family=""Times New Roman", SimSun, serif" font-size="13" fill="#111111" letter-spacing="0.1">lab in #1 since the modern era began.</text>
|
||||
<text x="60" y="640" font-family="Consolas, "Courier New", monospace" font-size="9" fill="#3A3A3A" letter-spacing="1.5">CONTINUED ON PAGE 03 →</text>
|
||||
</g>
|
||||
<line x1="445" y1="478" x2="445" y2="650" stroke="#111111" stroke-width="1"/>
|
||||
|
||||
<g id="obs-2">
|
||||
<text x="460" y="494" font-family=""Arial Black", SimHei, sans-serif" font-size="14" font-weight="900" fill="#C8102E" letter-spacing="1.5">II.</text>
|
||||
<text x="490" y="494" font-family=""Arial Black", SimHei, sans-serif" font-size="14" font-weight="900" fill="#111111" letter-spacing="1.5">TRAINING COSTS BROKE</text>
|
||||
<text x="460" y="514" font-family=""Arial Black", SimHei, sans-serif" font-size="14" font-weight="900" fill="#111111" letter-spacing="1.5">THE HALF-BILLION CEILING.</text>
|
||||
<text x="460" y="544" font-family=""Times New Roman", SimSun, serif" font-size="13" fill="#111111" letter-spacing="0.1">Frontier 2026 runs sit at one to ten</text>
|
||||
<text x="460" y="562" font-family=""Times New Roman", SimSun, serif" font-size="13" fill="#111111" letter-spacing="0.1">times the compute of the GPT-4 era,</text>
|
||||
<text x="460" y="580" font-family=""Times New Roman", SimSun, serif" font-size="13" fill="#111111" letter-spacing="0.1">costing $200 M to $500 M per run.</text>
|
||||
<text x="460" y="598" font-family=""Times New Roman", SimSun, serif" font-size="13" fill="#111111" letter-spacing="0.1">Power and grid interconnect, not</text>
|
||||
<text x="460" y="616" font-family=""Times New Roman", SimSun, serif" font-size="13" fill="#111111" letter-spacing="0.1">chips, are the binding constraint.</text>
|
||||
<text x="460" y="640" font-family="Consolas, "Courier New", monospace" font-size="9" fill="#3A3A3A" letter-spacing="1.5">CONTINUED ON PAGE 05 →</text>
|
||||
</g>
|
||||
<line x1="845" y1="478" x2="845" y2="650" stroke="#111111" stroke-width="1"/>
|
||||
|
||||
<g id="obs-3">
|
||||
<text x="860" y="494" font-family=""Arial Black", SimHei, sans-serif" font-size="14" font-weight="900" fill="#C8102E" letter-spacing="1.5">III.</text>
|
||||
<text x="893" y="494" font-family=""Arial Black", SimHei, sans-serif" font-size="14" font-weight="900" fill="#111111" letter-spacing="1.5">INFERENCE FALLS</text>
|
||||
<text x="860" y="514" font-family=""Arial Black", SimHei, sans-serif" font-size="14" font-weight="900" fill="#111111" letter-spacing="1.5">10× PER YEAR.</text>
|
||||
<text x="860" y="544" font-family=""Times New Roman", SimSun, serif" font-size="13" fill="#111111" letter-spacing="0.1">Inference cost for the same capability</text>
|
||||
<text x="860" y="562" font-family=""Times New Roman", SimSun, serif" font-size="13" fill="#111111" letter-spacing="0.1">drops by an order of magnitude every</text>
|
||||
<text x="860" y="580" font-family=""Times New Roman", SimSun, serif" font-size="13" fill="#111111" letter-spacing="0.1">twelve months — the only frontier</text>
|
||||
<text x="860" y="598" font-family=""Times New Roman", SimSun, serif" font-size="13" fill="#111111" letter-spacing="0.1">metric falling faster than capability</text>
|
||||
<text x="860" y="616" font-family=""Times New Roman", SimSun, serif" font-size="13" fill="#111111" letter-spacing="0.1">itself is climbing.</text>
|
||||
<text x="860" y="640" font-family="Consolas, "Courier New", monospace" font-size="9" fill="#3A3A3A" letter-spacing="1.5">CONTINUED ON PAGE 06 →</text>
|
||||
</g>
|
||||
|
||||
<!-- ============ FOOTER ============ -->
|
||||
<g id="footer-imprint">
|
||||
<rect x="48" y="688" width="1184" height="2" fill="#C8102E"/>
|
||||
<text x="48" y="708" font-family="Consolas, "Courier New", monospace" font-size="9" font-weight="700" fill="#111111" letter-spacing="2">SOURCE COUNTERPOINT RESEARCH GLOBAL LLM SNAPSHOT Q1 2026 · GARTNER 2026E FORECAST · ONETRUST APR 2026 SCAN</text>
|
||||
<text x="1232" y="708" font-family="Consolas, "Courier New", monospace" font-size="9" font-weight="700" fill="#111111" text-anchor="end" letter-spacing="2">PAGE 02 / 10</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 12 KiB |
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 2.0 MiB |
@@ -0,0 +1,120 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1280 720" width="1280" height="720">
|
||||
<!-- P04 The ARPU Divide — dumbbell with normalized MAU vs ARPU, dense rhythm -->
|
||||
|
||||
<rect id="bg" width="1280" height="720" fill="#FFFFFF"/>
|
||||
|
||||
<!-- ============ HEADER ============ -->
|
||||
<g id="bg-masthead-rule">
|
||||
<rect x="0" y="0" width="1280" height="4" fill="#C8102E"/>
|
||||
</g>
|
||||
|
||||
<g id="header-folio">
|
||||
<line x1="48" y1="34" x2="1232" y2="34" stroke="#111111" stroke-width="1"/>
|
||||
<text x="48" y="26" font-family="Consolas, "Courier New", monospace" font-size="10" font-weight="700" fill="#111111" letter-spacing="2">THE AI INDUSTRY 2026 · ANNUAL REPORT</text>
|
||||
<text x="640" y="26" font-family="Consolas, "Courier New", monospace" font-size="10" font-weight="700" fill="#111111" text-anchor="middle" letter-spacing="2">STORY I · SIDEBAR</text>
|
||||
<text x="1232" y="26" font-family="Consolas, "Courier New", monospace" font-size="10" font-weight="700" fill="#111111" text-anchor="end" letter-spacing="2">VOL. 1 · NO. 1 · MAY 2026</text>
|
||||
</g>
|
||||
|
||||
<g id="header">
|
||||
<text x="48" y="58" font-family=""Arial Black", SimHei, sans-serif" font-size="11" font-weight="900" fill="#C8102E" letter-spacing="3.5">SECTION II · SIDEBAR</text>
|
||||
<text x="48" y="110" font-family="Impact, "Arial Black", SimHei, sans-serif" font-size="44" font-weight="900" fill="#111111" letter-spacing="-1">THE ARPU DIVIDE</text>
|
||||
<text x="48" y="138" font-family=""Times New Roman", SimSun, serif" font-size="16" font-style="italic" fill="#111111">Scale ≠ monetisation. Anthropic earns <tspan fill="#C8102E" font-weight="700">7.4×</tspan> per user vs. OpenAI, <tspan fill="#C8102E" font-weight="700">162×</tspan> vs. Meta.</text>
|
||||
<line x1="48" y1="156" x2="1232" y2="156" stroke="#111111" stroke-width="2"/>
|
||||
</g>
|
||||
|
||||
<!-- ============ LEGEND ============ -->
|
||||
<g id="legend">
|
||||
<circle cx="60" cy="184" r="6" fill="#111111"/>
|
||||
<text x="74" y="188" font-family=""Arial Black", SimHei, sans-serif" font-size="11" font-weight="900" fill="#111111" letter-spacing="1">MAU</text>
|
||||
<text x="112" y="188" font-family=""Times New Roman", SimSun, serif" font-size="11" font-style="italic" fill="#3A3A3A">% of largest (Meta ~1 B)</text>
|
||||
<circle cx="320" cy="184" r="6" fill="#C8102E"/>
|
||||
<text x="334" y="188" font-family=""Arial Black", SimHei, sans-serif" font-size="11" font-weight="900" fill="#C8102E" letter-spacing="1">ARPU</text>
|
||||
<text x="380" y="188" font-family=""Times New Roman", SimSun, serif" font-size="11" font-style="italic" fill="#3A3A3A">% of highest (Anthropic $16.20 /mo)</text>
|
||||
<text x="1232" y="188" font-family="Consolas, "Courier New", monospace" font-size="9" fill="#3A3A3A" text-anchor="end" letter-spacing="1">ROWS RANKED BY ARPU DESCENDING</text>
|
||||
</g>
|
||||
|
||||
<!-- ============ DUMBBELL CHART AREA ============ -->
|
||||
<g id="chartArea">
|
||||
<!-- X-axis grid (0, 25, 50, 75, 100%) at x = 200, 450, 700, 950, 1200 -->
|
||||
<line x1="200" y1="220" x2="200" y2="552" stroke="#111111" stroke-width="1"/>
|
||||
<line x1="450" y1="220" x2="450" y2="552" stroke="#3A3A3A" stroke-width="0.5" stroke-dasharray="2,3"/>
|
||||
<line x1="700" y1="220" x2="700" y2="552" stroke="#3A3A3A" stroke-width="0.5" stroke-dasharray="2,3"/>
|
||||
<line x1="950" y1="220" x2="950" y2="552" stroke="#3A3A3A" stroke-width="0.5" stroke-dasharray="2,3"/>
|
||||
<line x1="1200" y1="220" x2="1200" y2="552" stroke="#111111" stroke-width="1"/>
|
||||
<!-- chart-plot-area: 200,220,1200,552 -->
|
||||
|
||||
<!-- X-axis baseline -->
|
||||
<line x1="200" y1="552" x2="1200" y2="552" stroke="#111111" stroke-width="1.5"/>
|
||||
|
||||
<!-- X-axis tick labels -->
|
||||
<text x="200" y="572" font-family="Consolas, "Courier New", monospace" font-size="10" fill="#3A3A3A" text-anchor="middle">0 %</text>
|
||||
<text x="450" y="572" font-family="Consolas, "Courier New", monospace" font-size="10" fill="#3A3A3A" text-anchor="middle">25 %</text>
|
||||
<text x="700" y="572" font-family="Consolas, "Courier New", monospace" font-size="10" fill="#3A3A3A" text-anchor="middle">50 %</text>
|
||||
<text x="950" y="572" font-family="Consolas, "Courier New", monospace" font-size="10" fill="#3A3A3A" text-anchor="middle">75 %</text>
|
||||
<text x="1200" y="572" font-family="Consolas, "Courier New", monospace" font-size="10" fill="#3A3A3A" text-anchor="middle">100 %</text>
|
||||
|
||||
<!-- Scale: 1000px = 100%, 10px per percent, x = 200 + pct * 10 -->
|
||||
|
||||
<!-- Row 1: Anthropic — ARPU 100% (x=1200), MAU 13.4% (x=334) -->
|
||||
<g id="row-anthropic">
|
||||
<text x="190" y="248" font-family=""Arial Black", SimHei, sans-serif" font-size="13" font-weight="900" fill="#111111" text-anchor="end">ANTHROPIC</text>
|
||||
<line x1="334" y1="244" x2="1200" y2="244" stroke="#111111" stroke-width="1.5"/>
|
||||
<circle cx="334" cy="244" r="8" fill="#111111"/>
|
||||
<circle cx="1200" cy="244" r="8" fill="#C8102E"/>
|
||||
<text x="318" y="232" font-family="Consolas, "Courier New", monospace" font-size="10" fill="#3A3A3A" text-anchor="end">134 M</text>
|
||||
<text x="1188" y="232" font-family="Consolas, "Courier New", monospace" font-size="10" font-weight="700" fill="#C8102E" text-anchor="end">$16.20</text>
|
||||
</g>
|
||||
|
||||
<!-- Row 2: Microsoft — ARPU 30.9% (x=509), MAU N/A -->
|
||||
<g id="row-microsoft">
|
||||
<text x="190" y="312" font-family=""Arial Black", SimHei, sans-serif" font-size="13" font-weight="900" fill="#111111" text-anchor="end">MICROSOFT</text>
|
||||
<circle cx="509" cy="308" r="8" fill="#C8102E"/>
|
||||
<text x="497" y="296" font-family="Consolas, "Courier New", monospace" font-size="10" font-weight="700" fill="#C8102E" text-anchor="end">$5.00</text>
|
||||
<text x="525" y="312" font-family=""Times New Roman", SimSun, serif" font-size="11" font-style="italic" fill="#3A3A3A">MAU not reported</text>
|
||||
</g>
|
||||
|
||||
<!-- Row 3: OpenAI — ARPU 13.6% (x=336), MAU 90% (x=1100) -->
|
||||
<g id="row-openai">
|
||||
<text x="190" y="376" font-family=""Arial Black", SimHei, sans-serif" font-size="13" font-weight="900" fill="#111111" text-anchor="end">OPENAI</text>
|
||||
<line x1="336" y1="372" x2="1100" y2="372" stroke="#111111" stroke-width="1.5"/>
|
||||
<circle cx="1100" cy="372" r="8" fill="#111111"/>
|
||||
<circle cx="336" cy="372" r="8" fill="#C8102E"/>
|
||||
<text x="320" y="360" font-family="Consolas, "Courier New", monospace" font-size="10" font-weight="700" fill="#C8102E" text-anchor="end">$2.20</text>
|
||||
<text x="1088" y="360" font-family="Consolas, "Courier New", monospace" font-size="10" fill="#3A3A3A" text-anchor="end">900 M</text>
|
||||
</g>
|
||||
|
||||
<!-- Row 4: Google — ARPU 6.8% (x=268), MAU N/A -->
|
||||
<g id="row-google">
|
||||
<text x="190" y="440" font-family=""Arial Black", SimHei, sans-serif" font-size="13" font-weight="900" fill="#111111" text-anchor="end">GOOGLE</text>
|
||||
<circle cx="268" cy="436" r="8" fill="#C8102E"/>
|
||||
<text x="256" y="424" font-family="Consolas, "Courier New", monospace" font-size="10" font-weight="700" fill="#C8102E" text-anchor="end">$1.10</text>
|
||||
<text x="284" y="440" font-family=""Times New Roman", SimSun, serif" font-size="11" font-style="italic" fill="#3A3A3A">MAU not reported</text>
|
||||
</g>
|
||||
|
||||
<!-- Row 5: Meta — ARPU 0.6% (x=206), MAU 100% (x=1200) -->
|
||||
<g id="row-meta">
|
||||
<text x="190" y="504" font-family=""Arial Black", SimHei, sans-serif" font-size="13" font-weight="900" fill="#111111" text-anchor="end">META</text>
|
||||
<line x1="206" y1="500" x2="1200" y2="500" stroke="#111111" stroke-width="1.5"/>
|
||||
<circle cx="1200" cy="500" r="8" fill="#111111"/>
|
||||
<circle cx="206" cy="500" r="8" fill="#C8102E"/>
|
||||
<text x="226" y="488" font-family="Consolas, "Courier New", monospace" font-size="10" font-weight="700" fill="#C8102E">$0.10</text>
|
||||
<text x="1188" y="488" font-family="Consolas, "Courier New", monospace" font-size="10" fill="#3A3A3A" text-anchor="end">~1,000 M</text>
|
||||
</g>
|
||||
</g>
|
||||
|
||||
<!-- ============ READING THE TABLE — short caption ============ -->
|
||||
<line x1="48" y1="608" x2="1232" y2="608" stroke="#111111" stroke-width="2"/>
|
||||
<g id="reading-the-table">
|
||||
<text x="48" y="634" font-family=""Arial Black", SimHei, sans-serif" font-size="14" font-weight="900" fill="#111111" letter-spacing="1.5">READING THE TABLE</text>
|
||||
<line x1="240" y1="624" x2="1232" y2="624" stroke="#111111" stroke-width="0.5"/>
|
||||
<text x="48" y="660" font-family=""Times New Roman", SimSun, serif" font-size="13" fill="#111111">The rank-flip at Anthropic and Meta is the headline: Anthropic monetises a 134 M user base 162 times harder than Meta monetises a billion.</text>
|
||||
<text x="48" y="678" font-family=""Times New Roman", SimSun, serif" font-size="13" fill="#111111">Enterprise-API and coding workloads — Anthropic's load-bearing demand — compound faster on per-user terms than free-tier consumer scale ever has.</text>
|
||||
</g>
|
||||
|
||||
<!-- ============ FOOTER ============ -->
|
||||
<g id="footer-imprint">
|
||||
<rect x="48" y="688" width="1184" height="2" fill="#C8102E"/>
|
||||
<text x="48" y="708" font-family="Consolas, "Courier New", monospace" font-size="9" font-weight="700" fill="#111111" letter-spacing="2">SOURCE COUNTERPOINT RESEARCH Q1 2026 · THE REGISTER 2026-04-30 · NORMALISATION: MAU PCT OF META; ARPU PCT OF ANTHROPIC</text>
|
||||
<text x="1232" y="708" font-family="Consolas, "Courier New", monospace" font-size="9" font-weight="700" fill="#111111" text-anchor="end" letter-spacing="2">PAGE 04 / 10</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 9.6 KiB |
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 469 KiB |
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 2.3 MiB |
@@ -0,0 +1,134 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1280 720" width="1280" height="720">
|
||||
<!-- P07 Capability — Benchmarks Saturated, dumbbell 2024→2025, dense rhythm -->
|
||||
|
||||
<rect id="bg" width="1280" height="720" fill="#FFFFFF"/>
|
||||
|
||||
<!-- ============ HEADER ============ -->
|
||||
<g id="bg-masthead-rule">
|
||||
<rect x="0" y="0" width="1280" height="4" fill="#C8102E"/>
|
||||
</g>
|
||||
|
||||
<g id="header-folio">
|
||||
<line x1="48" y1="34" x2="1232" y2="34" stroke="#111111" stroke-width="1"/>
|
||||
<text x="48" y="26" font-family="Consolas, "Courier New", monospace" font-size="10" font-weight="700" fill="#111111" letter-spacing="2">THE AI INDUSTRY 2026 · ANNUAL REPORT</text>
|
||||
<text x="640" y="26" font-family="Consolas, "Courier New", monospace" font-size="10" font-weight="700" fill="#111111" text-anchor="middle" letter-spacing="2">STORY IV · CAPABILITY</text>
|
||||
<text x="1232" y="26" font-family="Consolas, "Courier New", monospace" font-size="10" font-weight="700" fill="#111111" text-anchor="end" letter-spacing="2">VOL. 1 · NO. 1 · MAY 2026</text>
|
||||
</g>
|
||||
|
||||
<g id="header">
|
||||
<text x="48" y="58" font-family=""Arial Black", SimHei, sans-serif" font-size="11" font-weight="900" fill="#C8102E" letter-spacing="3.5">SECTION V · STORY IV</text>
|
||||
<text x="48" y="110" font-family="Impact, "Arial Black", SimHei, sans-serif" font-size="44" font-weight="900" fill="#111111" letter-spacing="-1">BENCHMARKS SATURATED FASTER THAN BUILT</text>
|
||||
<text x="48" y="138" font-family=""Times New Roman", SimSun, serif" font-size="16" font-style="italic" fill="#111111">Evaluations designed to last years are saturated in months — the bench-design community is now permanently behind the release cycle.</text>
|
||||
<line x1="48" y1="156" x2="1232" y2="156" stroke="#111111" stroke-width="2"/>
|
||||
</g>
|
||||
|
||||
<!-- ============ LEGEND ============ -->
|
||||
<g id="legend">
|
||||
<circle cx="60" cy="184" r="7" fill="#111111"/>
|
||||
<text x="74" y="188" font-family=""Arial Black", SimHei, sans-serif" font-size="11" font-weight="900" fill="#111111" letter-spacing="1">2024</text>
|
||||
<text x="118" y="188" font-family=""Times New Roman", SimSun, serif" font-size="11" font-style="italic" fill="#3A3A3A">starting score</text>
|
||||
<circle cx="240" cy="184" r="7" fill="#C8102E"/>
|
||||
<text x="254" y="188" font-family=""Arial Black", SimHei, sans-serif" font-size="11" font-weight="900" fill="#C8102E" letter-spacing="1">2025</text>
|
||||
<text x="298" y="188" font-family=""Times New Roman", SimSun, serif" font-size="11" font-style="italic" fill="#3A3A3A">one-year-later score</text>
|
||||
<text x="500" y="188" font-family=""Times New Roman", SimSun, serif" font-size="11" font-style="italic" fill="#111111">· scoring is normalised: 100 % = human baseline; values above 100 % surpass it</text>
|
||||
<text x="1232" y="188" font-family="Consolas, "Courier New", monospace" font-size="9" fill="#3A3A3A" text-anchor="end" letter-spacing="1">STANFORD HAI AI INDEX 2026</text>
|
||||
</g>
|
||||
|
||||
<!-- ============ DUMBBELL CHART ============ -->
|
||||
<g id="chartArea">
|
||||
<!-- X-axis grid at 0, 25, 50, 75, 100, 125 % — x_min=240, x_max=1240, scale 8 px per %; -->
|
||||
<line x1="240" y1="220" x2="240" y2="552" stroke="#111111" stroke-width="1"/>
|
||||
<line x1="440" y1="220" x2="440" y2="552" stroke="#3A3A3A" stroke-width="0.5" stroke-dasharray="2,3"/>
|
||||
<line x1="640" y1="220" x2="640" y2="552" stroke="#3A3A3A" stroke-width="0.5" stroke-dasharray="2,3"/>
|
||||
<line x1="840" y1="220" x2="840" y2="552" stroke="#3A3A3A" stroke-width="0.5" stroke-dasharray="2,3"/>
|
||||
<line x1="1040" y1="220" x2="1040" y2="552" stroke="#C8102E" stroke-width="1"/>
|
||||
<line x1="1240" y1="220" x2="1240" y2="552" stroke="#3A3A3A" stroke-width="0.5" stroke-dasharray="2,3"/>
|
||||
|
||||
<!-- X baseline -->
|
||||
<line x1="240" y1="552" x2="1240" y2="552" stroke="#111111" stroke-width="1.5"/>
|
||||
<text x="240" y="572" font-family="Consolas, "Courier New", monospace" font-size="10" fill="#3A3A3A" text-anchor="middle">0 %</text>
|
||||
<text x="440" y="572" font-family="Consolas, "Courier New", monospace" font-size="10" fill="#3A3A3A" text-anchor="middle">25 %</text>
|
||||
<text x="640" y="572" font-family="Consolas, "Courier New", monospace" font-size="10" fill="#3A3A3A" text-anchor="middle">50 %</text>
|
||||
<text x="840" y="572" font-family="Consolas, "Courier New", monospace" font-size="10" fill="#3A3A3A" text-anchor="middle">75 %</text>
|
||||
<text x="1040" y="572" font-family="Consolas, "Courier New", monospace" font-size="10" font-weight="700" fill="#C8102E" text-anchor="middle">100 % · HUMAN</text>
|
||||
<text x="1240" y="572" font-family="Consolas, "Courier New", monospace" font-size="10" fill="#3A3A3A" text-anchor="middle">125 %</text>
|
||||
|
||||
<!-- chart-plot-area: 240,220,1240,552 -->
|
||||
|
||||
<!-- Row pitch 60 px; row centers: 248, 308, 368, 428, 488 -->
|
||||
|
||||
<!-- Row 1: SWE-bench Verified — 60% → 100% (sourced) -->
|
||||
<g id="row-swebench">
|
||||
<text x="230" y="252" font-family=""Arial Black", SimHei, sans-serif" font-size="12" font-weight="900" fill="#111111" text-anchor="end">SWE-BENCH VERIFIED</text>
|
||||
<text x="230" y="266" font-family=""Times New Roman", SimSun, serif" font-size="10" font-style="italic" fill="#3A3A3A" text-anchor="end">coding · sourced</text>
|
||||
<line x1="720" y1="248" x2="1040" y2="248" stroke="#111111" stroke-width="2"/>
|
||||
<circle cx="720" cy="248" r="9" fill="#111111"/>
|
||||
<circle cx="1040" cy="248" r="9" fill="#C8102E"/>
|
||||
<text x="704" y="234" font-family="Consolas, "Courier New", monospace" font-size="10" fill="#3A3A3A" text-anchor="end">60 %</text>
|
||||
<text x="1056" y="234" font-family="Consolas, "Courier New", monospace" font-size="10" font-weight="700" fill="#C8102E">100 %</text>
|
||||
<text x="1248" y="252" font-family=""Arial Black", SimHei, sans-serif" font-size="11" font-weight="900" fill="#C8102E" text-anchor="end">+40 pp</text>
|
||||
</g>
|
||||
|
||||
<!-- Row 2: Humanity's Last Exam — 12% → 42% (sourced as +30pp) -->
|
||||
<g id="row-hle">
|
||||
<text x="230" y="312" font-family=""Arial Black", SimHei, sans-serif" font-size="12" font-weight="900" fill="#111111" text-anchor="end">HUMANITY'S LAST EXAM</text>
|
||||
<text x="230" y="326" font-family=""Times New Roman", SimSun, serif" font-size="10" font-style="italic" fill="#3A3A3A" text-anchor="end">designed-hard · sourced</text>
|
||||
<line x1="336" y1="308" x2="576" y2="308" stroke="#111111" stroke-width="2"/>
|
||||
<circle cx="336" cy="308" r="9" fill="#111111"/>
|
||||
<circle cx="576" cy="308" r="9" fill="#C8102E"/>
|
||||
<text x="320" y="294" font-family="Consolas, "Courier New", monospace" font-size="10" fill="#3A3A3A" text-anchor="end">12 %</text>
|
||||
<text x="592" y="294" font-family="Consolas, "Courier New", monospace" font-size="10" font-weight="700" fill="#C8102E">42 %</text>
|
||||
<text x="1248" y="312" font-family=""Arial Black", SimHei, sans-serif" font-size="11" font-weight="900" fill="#C8102E" text-anchor="end">+30 pp</text>
|
||||
</g>
|
||||
|
||||
<!-- Row 3: PhD-level science QA — 55% → 88% (illustrative †) -->
|
||||
<g id="row-phd">
|
||||
<text x="230" y="372" font-family=""Arial Black", SimHei, sans-serif" font-size="12" font-weight="900" fill="#111111" text-anchor="end">PhD-LEVEL SCIENCE QA †</text>
|
||||
<text x="230" y="386" font-family=""Times New Roman", SimSun, serif" font-size="10" font-style="italic" fill="#3A3A3A" text-anchor="end">illustrative pair</text>
|
||||
<line x1="680" y1="368" x2="944" y2="368" stroke="#111111" stroke-width="2" stroke-dasharray="4,3"/>
|
||||
<circle cx="680" cy="368" r="9" fill="#111111"/>
|
||||
<circle cx="944" cy="368" r="9" fill="#C8102E"/>
|
||||
<text x="664" y="354" font-family="Consolas, "Courier New", monospace" font-size="10" fill="#3A3A3A" text-anchor="end">55 %</text>
|
||||
<text x="960" y="354" font-family="Consolas, "Courier New", monospace" font-size="10" font-weight="700" fill="#C8102E">88 %</text>
|
||||
<text x="1248" y="372" font-family=""Arial Black", SimHei, sans-serif" font-size="11" font-weight="900" fill="#C8102E" text-anchor="end">+33 pp</text>
|
||||
</g>
|
||||
|
||||
<!-- Row 4: Multimodal reasoning — 50% → 82% (illustrative †) -->
|
||||
<g id="row-multimodal">
|
||||
<text x="230" y="432" font-family=""Arial Black", SimHei, sans-serif" font-size="12" font-weight="900" fill="#111111" text-anchor="end">MULTIMODAL REASONING †</text>
|
||||
<text x="230" y="446" font-family=""Times New Roman", SimSun, serif" font-size="10" font-style="italic" fill="#3A3A3A" text-anchor="end">illustrative pair</text>
|
||||
<line x1="640" y1="428" x2="896" y2="428" stroke="#111111" stroke-width="2" stroke-dasharray="4,3"/>
|
||||
<circle cx="640" cy="428" r="9" fill="#111111"/>
|
||||
<circle cx="896" cy="428" r="9" fill="#C8102E"/>
|
||||
<text x="624" y="414" font-family="Consolas, "Courier New", monospace" font-size="10" fill="#3A3A3A" text-anchor="end">50 %</text>
|
||||
<text x="912" y="414" font-family="Consolas, "Courier New", monospace" font-size="10" font-weight="700" fill="#C8102E">82 %</text>
|
||||
<text x="1248" y="432" font-family=""Arial Black", SimHei, sans-serif" font-size="11" font-weight="900" fill="#C8102E" text-anchor="end">+32 pp</text>
|
||||
</g>
|
||||
|
||||
<!-- Row 5: Competition mathematics — 45% → 78% (illustrative †) -->
|
||||
<g id="row-math">
|
||||
<text x="230" y="492" font-family=""Arial Black", SimHei, sans-serif" font-size="12" font-weight="900" fill="#111111" text-anchor="end">COMPETITION MATHEMATICS †</text>
|
||||
<text x="230" y="506" font-family=""Times New Roman", SimSun, serif" font-size="10" font-style="italic" fill="#3A3A3A" text-anchor="end">illustrative pair</text>
|
||||
<line x1="600" y1="488" x2="864" y2="488" stroke="#111111" stroke-width="2" stroke-dasharray="4,3"/>
|
||||
<circle cx="600" cy="488" r="9" fill="#111111"/>
|
||||
<circle cx="864" cy="488" r="9" fill="#C8102E"/>
|
||||
<text x="584" y="474" font-family="Consolas, "Courier New", monospace" font-size="10" fill="#3A3A3A" text-anchor="end">45 %</text>
|
||||
<text x="880" y="474" font-family="Consolas, "Courier New", monospace" font-size="10" font-weight="700" fill="#C8102E">78 %</text>
|
||||
<text x="1248" y="492" font-family=""Arial Black", SimHei, sans-serif" font-size="11" font-weight="900" fill="#C8102E" text-anchor="end">+33 pp</text>
|
||||
</g>
|
||||
</g>
|
||||
|
||||
<!-- ============ FOOTNOTE + EDITORIAL LINE ============ -->
|
||||
<line x1="48" y1="600" x2="1232" y2="600" stroke="#111111" stroke-width="2"/>
|
||||
<g id="editorial">
|
||||
<text x="48" y="624" font-family=""Times New Roman", SimSun, serif" font-size="11" font-style="italic" fill="#3A3A3A">† Illustrative pairs reflect the source's qualitative framing of these benchmarks ("below baseline → meet or exceed human baseline" within twelve months). Solid bars are explicitly sourced.</text>
|
||||
<text x="48" y="666" font-family="Impact, "Arial Black", SimHei, sans-serif" font-size="24" font-weight="900" fill="#C8102E" letter-spacing="-0.5">"THE BENCH-DESIGN COMMUNITY IS NOW PERMANENTLY BEHIND THE MODEL-RELEASE CYCLE."</text>
|
||||
</g>
|
||||
|
||||
<!-- ============ FOOTER ============ -->
|
||||
<g id="footer-imprint">
|
||||
<rect x="48" y="688" width="1184" height="2" fill="#C8102E"/>
|
||||
<text x="48" y="708" font-family="Consolas, "Courier New", monospace" font-size="9" font-weight="700" fill="#111111" letter-spacing="2">SOURCE STANFORD HAI AI INDEX 2026 (PUBLISHED 2026-04-13) · ROWS 1-2 SOURCED; ROWS 3-5 ILLUSTRATIVE OF QUALITATIVE FRAMING</text>
|
||||
<text x="1232" y="708" font-family="Consolas, "Courier New", monospace" font-size="9" font-weight="700" fill="#111111" text-anchor="end" letter-spacing="2">PAGE 07 / 10</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 12 KiB |
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user