Video Generation Grammar

1. First principle: generate editable shots, not vague videos

A useful generated asset must be designed for assembly.

GENERATED SHOT
= visual intent
+ stable subject/world specification
+ camera state
+ lighting state
+ temporal action
+ entry state
+ exit state
+ duration target
+ continuity keys
+ negative constraints
+ provenance
+ quality evidence

The generation system should operate at the shot level. A long sequence can then be constructed from verified shots, handles, transitions, and sound rather than relying on one uncontrolled generation.

2. Generation modes

TEXT_TO_VIDEO
IMAGE_TO_VIDEO
VIDEO_TO_VIDEO
KEYFRAME_TO_VIDEO
START_END_FRAME_INTERPOLATION
REFERENCE_GUIDED_VIDEO
CHARACTER_GUIDED_VIDEO
MOTION_GUIDED_VIDEO
CAMERA_GUIDED_VIDEO
DEPTH_GUIDED_VIDEO
POSE_GUIDED_VIDEO
MASKED_REGENERATION
OBJECT_INSERTION
OBJECT_REMOVAL
BACKGROUND_REPLACEMENT
OUTPAINTING
INPAINTING
STYLE_TRANSFORMATION
FRAME_INTERPOLATION
UPSCALE_AND_RESTORE
LIP_SYNC_OR_PERFORMANCE_TRANSFER
VIRTUAL_PRODUCTION_BACKGROUND
PROCEDURAL_3D_RENDER
COMPOSITED_HYBRID

Each mode has different truth, rights, continuity, and artifact risks. Store the mode with every output.

3. Prompt stack

Use structured fields rather than one paragraph.

PROMPT
= PURPOSE
+ SUBJECT_LOCK
+ ACTION
+ ENVIRONMENT
+ TIME_AND_WEATHER
+ CAMERA
+ LIGHTING
+ ART_DIRECTION
+ TEMPORAL_BEHAVIOR
+ ENTRY_STATE
+ EXIT_STATE
+ CONTINUITY_LOCKS
+ NEGATIVE_CONSTRAINTS
+ OUTPUT_CONSTRAINTS
+ PROVENANCE

Purpose

story beat
information owner
emotion
edit position
transition function
required evidence status

Subject lock

identity or entity ID
physical geometry
age range when relevant
wardrobe
hair/makeup
materials
props
left/right placement
expression baseline
rights and consent status

Action

verb
object
direction
path
speed
acceleration
start pose
end pose
interaction
cause
visible consequence

Avoid ambiguous action bundles. Prefer one primary action and at most one secondary action per short shot.

Environment

location geometry
scale
surfaces
objects
spatial layout
time period
weather
atmosphere
background activity
world rules

Camera

Reference the complete CameraVector from the camera grammar:

shot size
azimuth
elevation and height
lens/field of view
focus
depth of field
movement path and profile
stabilization
composition
frame rate and shutter character

Lighting

Reference the complete lighting state:

motivated sources
key direction and quality
ratio
color relationship
practical state
atmosphere
continuity locks

Art direction

medium
production design
color palette
texture
period
costume
surface wear
graphic density
realism/stylization
reference hierarchy

Temporal behavior

duration
phase 1 action
turn or event
phase 2 action
background motion
camera timing
focus timing
light timing
loop behavior

Negative constraints

identity drift
wardrobe drift
extra limbs/digits
object duplication
text mutation
logo mutation
geometry warping
camera teleportation
light-direction drift
flicker
boiling texture
background replacement
unmotivated zoom
unmotivated slow motion
unsafe content
unlicensed likeness or style imitation

4. Prompt object example

purpose:
  beat_id: beat_014
  function: REVEAL
  audience_change: "uncertain -> recognizes scale"
subject_lock:
  entity_id: machine_A
  geometry_ref: refs/machine_A_turntable
  material: brushed steel with black polymer panels
  position: center-left
  prohibited_changes: [panel_count, logo, wheel_count]
action:
  primary: conveyor starts moving
  direction: screen-left-to-right
  start_state: stationary
  end_state: stable operating speed
camera:
  shot_size: WIDE_FULL
  azimuth: THREE_QUARTER
  elevation: EYE_LEVEL
  lens_family: WIDE
  movement: SLOW_PUSH_IN
  duration_seconds: 6
lighting:
  motivation: overhead industrial fixtures plus doorway daylight
  key_direction: high camera-left
  contrast: MODERATE
  prohibited_changes: [time_of_day, key_direction]
entry_state: machine stationary; operator hand on control
exit_state: machine moving; operator steps back
negative_constraints: [flicker, duplicate parts, unreadable controls, camera jump]

5. Continuity bibles

Character bible

identity references
body proportions
face landmarks
skin/hair/eye state
wardrobe layers
accessories
voice
movement mannerisms
allowed emotional range
left/right distinctions
injury/dirt/wetness progression

World bible

map and geometry
object inventory
material palette
architecture
weather
time-of-day path
technology rules
signage and language
population/background rules

Prop bible

shape
scale
materials
markings
state changes
ownership
handedness
position continuity

Camera bible

sensor/format simulation
lens family
aspect and framing policy
movement language
stabilization language
depth-of-field policy
frame-rate/motion policy
forbidden camera behaviors

Lighting bible

source map
sun/time path
practical map
key direction per location
contrast range
color relationship
atmosphere range
transition rules

Performance bible

voice and cadence
gesture scale
posture
energy
emotional state per beat
gaze and eyeline
interaction rules

6. Shot-state continuity

Every generated shot must declare:

ENTRY = object positions + subject pose + camera state + light state + motion state
EXIT  = object positions + subject pose + camera state + light state + motion state

Adjacent shots are compatible when:

Exit(A) approximately satisfies Entry(B)
OR an explicit transition explains the discontinuity.

Compatibility dimensions:

identity
wardrobe
prop count/state
body pose
screen direction
gaze
camera side
location geometry
weather/time
light direction
motion vector
action phase

7. Coverage-generation grammar

For a beat, create a coverage matrix:

required story information
× shot function
× shot scale
× perspective
× entry/exit state
× format fitness

Example coverage set:

1. ESTABLISHING — space and subject relation
2. MASTER_ACTION — complete uninterrupted action
3. MEDIUM_PRIMARY — readable performance
4. CLOSE_EMPHASIS — key emotional/information turn
5. INSERT — mechanism or evidence
6. REACTION — consequence on observer
7. TRANSITION_OUT — usable exit movement or composition

Do not generate all combinations. Use constraints and utility scoring to find the smallest robust set.

8. Keyframe grammar

For complex action, define explicit temporal anchors:

K0 entry state
K1 anticipation
K2 action begins
K3 maximum change
K4 consequence
K5 exit state

For each anchor, store:

subject pose
object state
camera pose
focus owner
light state
background state

Generated motion is then validated between anchors.

9. Camera-angle generation matrix

For a subject/action, possible controlled variations include:

size: EWS | WS | MFS | MS | MCU | CU | ECU | INSERT
azimuth: FRONT | 3/4 | PROFILE | REAR_3/4 | REAR
vertical: TOP | HIGH | EYE | LOW | GROUND
relation: OBJECTIVE | POV | OTS | TWO_SHOT | REACTION | DETAIL
movement: STATIC | PUSH | PULL | TRACK | ARC | PAN | TILT | CRANE | HANDHELD
optics: ULTRA_WIDE | WIDE | NORMAL | SHORT_TELE | TELE | MACRO

Constraint examples:

IF identity consistency is weak
THEN limit simultaneous variation in angle, expression, wardrobe, and lighting.

IF an insert must prove a mechanism
THEN prioritize geometry and temporal action over cinematic depth of field.

IF the shot must bridge a cut
THEN match action phase and screen direction before aesthetic variation.

10. Generated reenactment grammar

Generated reenactments require:

clear editorial purpose
separation from documentary evidence
visible or contextual disclosure appropriate to risk
source basis
uncertainty label
no invented quotation presented as real
no false identity attribution
retained prompt/reference/version history

Possible labels:

ILLUSTRATION
SIMULATION
RECONSTRUCTION
GENERATED_REENACTMENT
SPECULATIVE_VISUALIZATION
DATA_DRIVEN_VISUALIZATION

11. Generation lifecycle

BRIEF
→ RIGHTS_AND_RISK_CHECK
→ CONTINUITY_BINDING
→ PROMPT_COMPILE
→ LOW_COST_PREVIEW
→ STRUCTURAL_QC
→ TARGETED_VARIATIONS
→ HUMAN_SELECTION
→ LOCAL_REPAIR
→ UPSCALE/FINISH
→ EDIT_INTEGRATION
→ FINAL_QA
→ PROVENANCE_RECORD

Generate low-resolution or short previews before expensive finishing where the tool permits it.

12. LAKA generation strategy

For a base prompt P0:

Baseline: seed/take variation within fixed prompt dimensions
Minor: vary one local dimension
Major: vary treatment while preserving beat and subject
Structural: vary coverage or temporal architecture
Paradigm: vary representational mode

Examples:

Minor: static medium → subtle push-in medium
Major: objective wide → subjective POV sequence
Structural: single shot → establish/insert/reaction coverage
Paradigm: literal reenactment → abstract data-space visualization

Variation guard:

Change no more dimensions than the selected LAKA level permits.
Lock all dimensions that protect identity, truth, continuity, rights, and brand.

13. Candidate scoring

GenerationScore
= 0.18 beat_fulfillment
+ 0.14 continuity
+ 0.12 subject_identity
+ 0.10 action_legibility
+ 0.09 camera_compliance
+ 0.09 lighting_compliance
+ 0.08 temporal_coherence
+ 0.07 editability
+ 0.05 technical_quality
+ 0.04 format_fitness
+ 0.04 novelty
- artifact_penalty
- rights_risk
- provenance_gap

Keep sub-scores and failure masks. A beautiful shot that fails continuity should not silently outrank a less spectacular usable shot.

14. Quality-control taxonomy

Identity and anatomy

face drift
body proportion drift
extra/missing anatomy
hand-object failure
lip/teeth deformation
gaze inconsistency
wardrobe mutation

Object and world

object duplication/disappearance
material mutation
scale drift
world geometry drift
text/logo mutation
background teleportation
reflection mismatch
shadow mismatch

Motion and time

motion discontinuity
speed discontinuity
physics violation
foot sliding
contact failure
camera jump
frame interpolation smear
flicker
texture boiling
loop seam

Cinematography

wrong shot size
wrong angle/lens behavior
unmotivated zoom
focus hunting
exposure drift
light-direction drift
color drift
rolling horizon

Editorial and ethical

beat not fulfilled
false documentary implication
unlicensed likeness
unsafe stereotype
unmarked speculation
provenance missing
quote or event invented

15. Repair grammar

REGENERATE_FULL_SHOT
REGENERATE_RANGE
MASK_AND_REPLACE
FREEZE_OR_HOLD
CUT_BEFORE_FAILURE
CUT_AFTER_FAILURE
COVER_WITH_INSERT
COVER_WITH_GRAPHIC
SPEED_ADJUST
STABILIZE
TRACK_AND_REPLACE
CROP_OR_REFRAME
USE_STILL_WITH_PARALLAX
ABSTRACT_THE_VISUAL
RECORD_PRACTICAL_PICKUP

Repair cost must be compared against fresh generation or practical capture.

16. Provenance record

asset_id
project_id
generation_mode
provider/tool/model/version when available
creation_timestamp
operator/agent
prompt object
negative constraints
seed or reproducibility data when available
input references and rights status
parent assets
edit history
review decisions
disclosure state
content-credential state
final usage locations

Never use provenance as a substitute for editorial disclosure when the visual could materially mislead.

17. Video-generation failure modes

LONG_PROMPT_WITHOUT_STRUCTURE
WHOLE_SCENE_GENERATION_WITHOUT_COVERAGE
NO_ENTRY_EXIT_STATE
IDENTITY_LOCK_TOO_WEAK
TOO_MANY_VARIABLES_CHANGED
MOTION_WITHOUT_CAUSAL_PHASES
BEAUTY_OVER_EDITABILITY
STYLE_OVER_TRUTH
UNMARKED_REENACTMENT
NO_PROMPT_OR_VERSION_HISTORY
REPAIR_COST_EXCEEDS_VALUE
PLATFORM_CROP_DESTROYS_INFORMATION

18. Acceptance test

Does the asset perform a named beat function?
Can it cut into and out of neighboring material?
Are entry and exit states explicit and compatible?
Are subject, world, camera, and lighting locks satisfied?
Does motion obey visible cause and contact?
Are text, logos, anatomy, hands, reflections, and shadows stable?
Is the representational status truthful and disclosed?
Are rights and provenance recorded?
Does the shot survive target aspect ratios and compression?