2560190e06
- Change :hash to :cid throughout - Update cat asset: QmXrj6tSSn1vQXxxEY2Tyoudvt4CeeqR9gGQwSt7WFrhMZ - Update dog effect: QmT99H4MC5p18MGuxAeKGeXD71cGCzMNRxFfvt4FuCwpn6 - Update invert effect: QmPWaW5E5WFrmDjT6w8enqvtJhM8c5jvQu7XN1doHA3Z7J Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
150 lines
4.9 KiB
Markdown
150 lines
4.9 KiB
Markdown
# Art DAG Primitive Language
|
|
|
|
## Overview
|
|
|
|
Primitives enable declarative composition of audio-reactive video. The key insight: **data flows alongside media**.
|
|
|
|
```
|
|
audio → ANALYZE → data → BIND/MAP/COMPUTE → parameters → TRANSFORM → video
|
|
```
|
|
|
|
## Two Types of Flow
|
|
|
|
| Flow | Examples | Description |
|
|
|------|----------|-------------|
|
|
| **Media** | video, audio files | Actual content that gets transformed |
|
|
| **Data** | beat times, tempo, energy envelope | Analysis results that drive parameters |
|
|
|
|
## Primitives
|
|
|
|
### Source Primitives
|
|
|
|
| Primitive | Status | Description |
|
|
|-----------|--------|-------------|
|
|
| `SOURCE` | ✅ Implemented | Load single media file |
|
|
| `SOURCE_LIST` | ❌ Not implemented | Collect multiple inputs into a list |
|
|
| `PARAM` | ❌ Not implemented | Recipe parameter (data, not media) |
|
|
|
|
### Analysis Primitives
|
|
|
|
| Primitive | Status | Description |
|
|
|-----------|--------|-------------|
|
|
| `ANALYZE` | ❌ Not implemented | Extract features from media |
|
|
|
|
**ANALYZE features:**
|
|
|
|
| Feature | Output | Description |
|
|
|---------|--------|-------------|
|
|
| `beats` | `{beat_times: [], tempo: float}` | Beat positions |
|
|
| `downbeats` | `{downbeat_times: []}` | First beat of each bar |
|
|
| `tempo` | `{bpm: float, confidence: float}` | Tempo detection |
|
|
| `energy` | `{envelope: [{time, value}...]}` | Loudness over time |
|
|
| `spectrum` | `{bass: [], mid: [], high: []}` | Frequency bands over time |
|
|
| `onsets` | `{onset_times: []}` | Note/sound starts |
|
|
| `motion_tempo` | `{motion_bpm: float}` | Video motion periodicity |
|
|
|
|
### Data Processing Primitives
|
|
|
|
| Primitive | Status | Description |
|
|
|-----------|--------|-------------|
|
|
| `GROUP` | ❌ Not implemented | Chunk data (e.g., beats → measures) |
|
|
| `COMPUTE` | ❌ Not implemented | Arithmetic/expressions on data |
|
|
| `SELECT` | ❌ Not implemented | Conditional data selection |
|
|
| `BIND` | ❌ Not implemented | Map data ranges to parameter ranges |
|
|
|
|
**BIND example:**
|
|
```yaml
|
|
- source: energy.envelope # 0.0 → 1.0
|
|
target: saturation # mapped to 1.0 → 2.0
|
|
range: [1.0, 2.0]
|
|
attack_ms: 10 # response shaping
|
|
release_ms: 100
|
|
```
|
|
|
|
### Iteration Primitives
|
|
|
|
| Primitive | Status | Description |
|
|
|-----------|--------|-------------|
|
|
| `MAP` | ❌ Not implemented | Apply operation to each item in list |
|
|
| `RANDOM_SLICE` | ❌ Not implemented | Extract random segment from random pool item |
|
|
| `SEGMENT_AT` | ❌ Not implemented | Cut media at specified times |
|
|
|
|
**MAP operations:**
|
|
- `ANALYZE` - analyze each item
|
|
- `TRANSFORM` - apply effects to each item
|
|
- `COMPUTE` - calculate value for each item
|
|
- `RANDOM_SLICE` - extract random segment for each item
|
|
|
|
### Transform Primitives
|
|
|
|
| Primitive | Status | Description |
|
|
|-----------|--------|-------------|
|
|
| `SEGMENT` | ✅ Implemented | Extract time range |
|
|
| `RESIZE` | ✅ Implemented | Scale/crop/pad |
|
|
| `TRANSFORM` | ✅ Implemented | Static effects (color, blur, speed) |
|
|
| `TRANSFORM_DYNAMIC` | ❌ Not implemented | Time-varying effects from BIND |
|
|
|
|
### Compose Primitives
|
|
|
|
| Primitive | Status | Description |
|
|
|-----------|--------|-------------|
|
|
| `SEQUENCE` | ✅ Implemented | Concatenate in time |
|
|
| `LAYER` | ✅ Implemented | Stack spatially |
|
|
| `MUX` | ✅ Implemented | Combine video + audio |
|
|
| `BLEND` | ✅ Implemented | Blend two inputs |
|
|
|
|
## Patterns
|
|
|
|
### Pattern 1: Beat-Synced Cuts
|
|
|
|
```
|
|
music → ANALYZE(beats) → GROUP(4) → MAP(RANDOM_SLICE, videos) → SEQUENCE → MUX(music)
|
|
```
|
|
|
|
Audio drives cut timing, videos provide content.
|
|
|
|
### Pattern 2: Energy-Reactive Effects
|
|
|
|
```
|
|
music → ANALYZE(energy) → BIND(saturation, brightness) → TRANSFORM_DYNAMIC(video) → MUX
|
|
```
|
|
|
|
Audio amplitude drives visual intensity.
|
|
|
|
### Pattern 3: Tempo Matching
|
|
|
|
```
|
|
music → ANALYZE(tempo) ─┐
|
|
├→ COMPUTE(speed_factor) → TRANSFORM(speed) → SEQUENCE
|
|
videos → MAP(ANALYZE(motion_tempo)) ─┘
|
|
```
|
|
|
|
Video speed adjusted to match audio tempo.
|
|
|
|
### Pattern 4: Spectrum-Driven Layers
|
|
|
|
```
|
|
music → ANALYZE(spectrum) → BIND(bass→layer1_opacity, high→layer2_opacity)
|
|
↓
|
|
video1 ────────────────────→ LAYER ← video2
|
|
```
|
|
|
|
Different frequency bands control different visual layers.
|
|
|
|
## Design Principles
|
|
|
|
1. **Separation of concerns**: ANALYZE extracts data, BIND maps it, TRANSFORM applies it
|
|
2. **Composability**: Small primitives combine into complex behaviors
|
|
3. **Declarative**: Describe *what* you want, not *how* to compute it
|
|
4. **Reproducibility**: Seeds and deterministic operations ensure same inputs → same output
|
|
5. **Data as first-class**: Analysis results flow through the DAG like media
|
|
|
|
## Implementation Priority
|
|
|
|
1. `ANALYZE` (beats, energy, tempo) - foundation for audio-reactive
|
|
2. `BIND` - connects analysis to effects
|
|
3. `TRANSFORM_DYNAMIC` - applies time-varying effects
|
|
4. `MAP` - enables iteration over lists
|
|
5. `SOURCE_LIST` - multiple input handling
|
|
6. `GROUP`, `COMPUTE`, `SELECT` - data manipulation
|