recipes/PRIMITIVES.md

Art DAG Primitive Language

Overview

Primitives enable declarative composition of audio-reactive video. The key insight: data flows alongside media.

audio → ANALYZE → data → BIND/MAP/COMPUTE → parameters → TRANSFORM → video

Two Types of Flow

Flow	Examples	Description
Media	video, audio files	Actual content that gets transformed
Data	beat times, tempo, energy envelope	Analysis results that drive parameters

Primitives

Source Primitives

Primitive	Status	Description
SOURCE	✅ Implemented	Load single media file
SOURCE_LIST	❌ Not implemented	Collect multiple inputs into a list
PARAM	❌ Not implemented	Recipe parameter (data, not media)

Analysis Primitives

Primitive	Status	Description
ANALYZE	❌ Not implemented	Extract features from media

ANALYZE features:

Feature	Output	Description
beats	{beat_times: [], tempo: float}	Beat positions
downbeats	{downbeat_times: []}	First beat of each bar
tempo	{bpm: float, confidence: float}	Tempo detection
energy	{envelope: [{time, value}...]}	Loudness over time
spectrum	{bass: [], mid: [], high: []}	Frequency bands over time
onsets	{onset_times: []}	Note/sound starts
motion_tempo	{motion_bpm: float}	Video motion periodicity

Data Processing Primitives

Primitive	Status	Description
GROUP	❌ Not implemented	Chunk data (e.g., beats → measures)
COMPUTE	❌ Not implemented	Arithmetic/expressions on data
SELECT	❌ Not implemented	Conditional data selection
BIND	❌ Not implemented	Map data ranges to parameter ranges

BIND example:

- source: energy.envelope    # 0.0 → 1.0
  target: saturation         # mapped to 1.0 → 2.0
  range: [1.0, 2.0]
  attack_ms: 10              # response shaping
  release_ms: 100

Iteration Primitives

Primitive	Status	Description
MAP	❌ Not implemented	Apply operation to each item in list
RANDOM_SLICE	❌ Not implemented	Extract random segment from random pool item
SEGMENT_AT	❌ Not implemented	Cut media at specified times

MAP operations:

• ANALYZE - analyze each item
• TRANSFORM - apply effects to each item
• COMPUTE - calculate value for each item
• RANDOM_SLICE - extract random segment for each item

Transform Primitives

Primitive	Status	Description
SEGMENT	✅ Implemented	Extract time range
RESIZE	✅ Implemented	Scale/crop/pad
TRANSFORM	✅ Implemented	Static effects (color, blur, speed)
TRANSFORM_DYNAMIC	❌ Not implemented	Time-varying effects from BIND

Compose Primitives

Primitive	Status	Description
SEQUENCE	✅ Implemented	Concatenate in time
LAYER	✅ Implemented	Stack spatially
MUX	✅ Implemented	Combine video + audio
BLEND	✅ Implemented	Blend two inputs

Patterns

Pattern 1: Beat-Synced Cuts

music → ANALYZE(beats) → GROUP(4) → MAP(RANDOM_SLICE, videos) → SEQUENCE → MUX(music)

Audio drives cut timing, videos provide content.

Pattern 2: Energy-Reactive Effects

music → ANALYZE(energy) → BIND(saturation, brightness) → TRANSFORM_DYNAMIC(video) → MUX

Audio amplitude drives visual intensity.

Pattern 3: Tempo Matching

music → ANALYZE(tempo) ─┐
                        ├→ COMPUTE(speed_factor) → TRANSFORM(speed) → SEQUENCE
videos → MAP(ANALYZE(motion_tempo)) ─┘

Video speed adjusted to match audio tempo.

Pattern 4: Spectrum-Driven Layers

music → ANALYZE(spectrum) → BIND(bass→layer1_opacity, high→layer2_opacity)
                                    ↓
video1 ────────────────────→ LAYER ← video2

Different frequency bands control different visual layers.

Design Principles

1. Separation of concerns: ANALYZE extracts data, BIND maps it, TRANSFORM applies it
2. Composability: Small primitives combine into complex behaviors
3. Declarative: Describe what you want, not how to compute it
4. Reproducibility: Seeds and deterministic operations ensure same inputs → same output
5. Data as first-class: Analysis results flow through the DAG like media

Implementation Priority

1. ANALYZE (beats, energy, tempo) - foundation for audio-reactive
2. BIND - connects analysis to effects
3. TRANSFORM_DYNAMIC - applies time-varying effects
4. MAP - enables iteration over lists
5. SOURCE_LIST - multiple input handling
6. GROUP, COMPUTE, SELECT - data manipulation