# 09 — Video Generation Grammar ## 1. First principle: generate editable shots, not vague videos A useful generated asset must be designed for assembly. ```text GENERATED SHOT = visual intent + stable subject/world specification + camera state + lighting state + temporal action + entry state + exit state + duration target + continuity keys + negative constraints + provenance + quality evidence ``` The generation system should operate at the shot level. A long sequence can then be constructed from verified shots, handles, transitions, and sound rather than relying on one uncontrolled generation. ## 2. Generation modes ```text TEXT_TO_VIDEO IMAGE_TO_VIDEO VIDEO_TO_VIDEO KEYFRAME_TO_VIDEO START_END_FRAME_INTERPOLATION REFERENCE_GUIDED_VIDEO CHARACTER_GUIDED_VIDEO MOTION_GUIDED_VIDEO CAMERA_GUIDED_VIDEO DEPTH_GUIDED_VIDEO POSE_GUIDED_VIDEO MASKED_REGENERATION OBJECT_INSERTION OBJECT_REMOVAL BACKGROUND_REPLACEMENT OUTPAINTING INPAINTING STYLE_TRANSFORMATION FRAME_INTERPOLATION UPSCALE_AND_RESTORE LIP_SYNC_OR_PERFORMANCE_TRANSFER VIRTUAL_PRODUCTION_BACKGROUND PROCEDURAL_3D_RENDER COMPOSITED_HYBRID ``` Each mode has different truth, rights, continuity, and artifact risks. Store the mode with every output. ## 3. Prompt stack Use structured fields rather than one paragraph. ```text PROMPT = PURPOSE + SUBJECT_LOCK + ACTION + ENVIRONMENT + TIME_AND_WEATHER + CAMERA + LIGHTING + ART_DIRECTION + TEMPORAL_BEHAVIOR + ENTRY_STATE + EXIT_STATE + CONTINUITY_LOCKS + NEGATIVE_CONSTRAINTS + OUTPUT_CONSTRAINTS + PROVENANCE ``` ### Purpose ```text story beat information owner emotion edit position transition function required evidence status ``` ### Subject lock ```text identity or entity ID physical geometry age range when relevant wardrobe hair/makeup materials props left/right placement expression baseline rights and consent status ``` ### Action ```text verb object direction path speed acceleration start pose end pose interaction cause visible consequence ``` Avoid ambiguous action bundles. Prefer one primary action and at most one secondary action per short shot. ### Environment ```text location geometry scale surfaces objects spatial layout time period weather atmosphere background activity world rules ``` ### Camera Reference the complete `CameraVector` from the camera grammar: ```text shot size azimuth elevation and height lens/field of view focus depth of field movement path and profile stabilization composition frame rate and shutter character ``` ### Lighting Reference the complete lighting state: ```text motivated sources key direction and quality ratio color relationship practical state atmosphere continuity locks ``` ### Art direction ```text medium production design color palette texture period costume surface wear graphic density realism/stylization reference hierarchy ``` ### Temporal behavior ```text duration phase 1 action turn or event phase 2 action background motion camera timing focus timing light timing loop behavior ``` ### Negative constraints ```text identity drift wardrobe drift extra limbs/digits object duplication text mutation logo mutation geometry warping camera teleportation light-direction drift flicker boiling texture background replacement unmotivated zoom unmotivated slow motion unsafe content unlicensed likeness or style imitation ``` ## 4. Prompt object example ```yaml purpose: beat_id: beat_014 function: REVEAL audience_change: "uncertain -> recognizes scale" subject_lock: entity_id: machine_A geometry_ref: refs/machine_A_turntable material: brushed steel with black polymer panels position: center-left prohibited_changes: [panel_count, logo, wheel_count] action: primary: conveyor starts moving direction: screen-left-to-right start_state: stationary end_state: stable operating speed camera: shot_size: WIDE_FULL azimuth: THREE_QUARTER elevation: EYE_LEVEL lens_family: WIDE movement: SLOW_PUSH_IN duration_seconds: 6 lighting: motivation: overhead industrial fixtures plus doorway daylight key_direction: high camera-left contrast: MODERATE prohibited_changes: [time_of_day, key_direction] entry_state: machine stationary; operator hand on control exit_state: machine moving; operator steps back negative_constraints: [flicker, duplicate parts, unreadable controls, camera jump] ``` ## 5. Continuity bibles ### Character bible ```text identity references body proportions face landmarks skin/hair/eye state wardrobe layers accessories voice movement mannerisms allowed emotional range left/right distinctions injury/dirt/wetness progression ``` ### World bible ```text map and geometry object inventory material palette architecture weather time-of-day path technology rules signage and language population/background rules ``` ### Prop bible ```text shape scale materials markings state changes ownership handedness position continuity ``` ### Camera bible ```text sensor/format simulation lens family aspect and framing policy movement language stabilization language depth-of-field policy frame-rate/motion policy forbidden camera behaviors ``` ### Lighting bible ```text source map sun/time path practical map key direction per location contrast range color relationship atmosphere range transition rules ``` ### Performance bible ```text voice and cadence gesture scale posture energy emotional state per beat gaze and eyeline interaction rules ``` ## 6. Shot-state continuity Every generated shot must declare: ```text ENTRY = object positions + subject pose + camera state + light state + motion state EXIT = object positions + subject pose + camera state + light state + motion state ``` Adjacent shots are compatible when: ```text Exit(A) approximately satisfies Entry(B) OR an explicit transition explains the discontinuity. ``` Compatibility dimensions: ```text identity wardrobe prop count/state body pose screen direction gaze camera side location geometry weather/time light direction motion vector action phase ``` ## 7. Coverage-generation grammar For a beat, create a coverage matrix: ```text required story information × shot function × shot scale × perspective × entry/exit state × format fitness ``` Example coverage set: ```text 1. ESTABLISHING — space and subject relation 2. MASTER_ACTION — complete uninterrupted action 3. MEDIUM_PRIMARY — readable performance 4. CLOSE_EMPHASIS — key emotional/information turn 5. INSERT — mechanism or evidence 6. REACTION — consequence on observer 7. TRANSITION_OUT — usable exit movement or composition ``` Do not generate all combinations. Use constraints and utility scoring to find the smallest robust set. ## 8. Keyframe grammar For complex action, define explicit temporal anchors: ```text K0 entry state K1 anticipation K2 action begins K3 maximum change K4 consequence K5 exit state ``` For each anchor, store: ```text subject pose object state camera pose focus owner light state background state ``` Generated motion is then validated between anchors. ## 9. Camera-angle generation matrix For a subject/action, possible controlled variations include: ```text size: EWS | WS | MFS | MS | MCU | CU | ECU | INSERT azimuth: FRONT | 3/4 | PROFILE | REAR_3/4 | REAR vertical: TOP | HIGH | EYE | LOW | GROUND relation: OBJECTIVE | POV | OTS | TWO_SHOT | REACTION | DETAIL movement: STATIC | PUSH | PULL | TRACK | ARC | PAN | TILT | CRANE | HANDHELD optics: ULTRA_WIDE | WIDE | NORMAL | SHORT_TELE | TELE | MACRO ``` Constraint examples: ```text IF identity consistency is weak THEN limit simultaneous variation in angle, expression, wardrobe, and lighting. IF an insert must prove a mechanism THEN prioritize geometry and temporal action over cinematic depth of field. IF the shot must bridge a cut THEN match action phase and screen direction before aesthetic variation. ``` ## 10. Generated reenactment grammar Generated reenactments require: ```text clear editorial purpose separation from documentary evidence visible or contextual disclosure appropriate to risk source basis uncertainty label no invented quotation presented as real no false identity attribution retained prompt/reference/version history ``` Possible labels: ```text ILLUSTRATION SIMULATION RECONSTRUCTION GENERATED_REENACTMENT SPECULATIVE_VISUALIZATION DATA_DRIVEN_VISUALIZATION ``` ## 11. Generation lifecycle ```text BRIEF → RIGHTS_AND_RISK_CHECK → CONTINUITY_BINDING → PROMPT_COMPILE → LOW_COST_PREVIEW → STRUCTURAL_QC → TARGETED_VARIATIONS → HUMAN_SELECTION → LOCAL_REPAIR → UPSCALE/FINISH → EDIT_INTEGRATION → FINAL_QA → PROVENANCE_RECORD ``` Generate low-resolution or short previews before expensive finishing where the tool permits it. ## 12. LAKA generation strategy For a base prompt `P0`: ```text Baseline: seed/take variation within fixed prompt dimensions Minor: vary one local dimension Major: vary treatment while preserving beat and subject Structural: vary coverage or temporal architecture Paradigm: vary representational mode ``` Examples: ```text Minor: static medium → subtle push-in medium Major: objective wide → subjective POV sequence Structural: single shot → establish/insert/reaction coverage Paradigm: literal reenactment → abstract data-space visualization ``` Variation guard: ```text Change no more dimensions than the selected LAKA level permits. Lock all dimensions that protect identity, truth, continuity, rights, and brand. ``` ## 13. Candidate scoring ```text GenerationScore = 0.18 beat_fulfillment + 0.14 continuity + 0.12 subject_identity + 0.10 action_legibility + 0.09 camera_compliance + 0.09 lighting_compliance + 0.08 temporal_coherence + 0.07 editability + 0.05 technical_quality + 0.04 format_fitness + 0.04 novelty - artifact_penalty - rights_risk - provenance_gap ``` Keep sub-scores and failure masks. A beautiful shot that fails continuity should not silently outrank a less spectacular usable shot. ## 14. Quality-control taxonomy ### Identity and anatomy ```text face drift body proportion drift extra/missing anatomy hand-object failure lip/teeth deformation gaze inconsistency wardrobe mutation ``` ### Object and world ```text object duplication/disappearance material mutation scale drift world geometry drift text/logo mutation background teleportation reflection mismatch shadow mismatch ``` ### Motion and time ```text motion discontinuity speed discontinuity physics violation foot sliding contact failure camera jump frame interpolation smear flicker texture boiling loop seam ``` ### Cinematography ```text wrong shot size wrong angle/lens behavior unmotivated zoom focus hunting exposure drift light-direction drift color drift rolling horizon ``` ### Editorial and ethical ```text beat not fulfilled false documentary implication unlicensed likeness unsafe stereotype unmarked speculation provenance missing quote or event invented ``` ## 15. Repair grammar ```text REGENERATE_FULL_SHOT REGENERATE_RANGE MASK_AND_REPLACE FREEZE_OR_HOLD CUT_BEFORE_FAILURE CUT_AFTER_FAILURE COVER_WITH_INSERT COVER_WITH_GRAPHIC SPEED_ADJUST STABILIZE TRACK_AND_REPLACE CROP_OR_REFRAME USE_STILL_WITH_PARALLAX ABSTRACT_THE_VISUAL RECORD_PRACTICAL_PICKUP ``` Repair cost must be compared against fresh generation or practical capture. ## 16. Provenance record ```text asset_id project_id generation_mode provider/tool/model/version when available creation_timestamp operator/agent prompt object negative constraints seed or reproducibility data when available input references and rights status parent assets edit history review decisions disclosure state content-credential state final usage locations ``` Never use provenance as a substitute for editorial disclosure when the visual could materially mislead. ## 17. Video-generation failure modes ```text LONG_PROMPT_WITHOUT_STRUCTURE WHOLE_SCENE_GENERATION_WITHOUT_COVERAGE NO_ENTRY_EXIT_STATE IDENTITY_LOCK_TOO_WEAK TOO_MANY_VARIABLES_CHANGED MOTION_WITHOUT_CAUSAL_PHASES BEAUTY_OVER_EDITABILITY STYLE_OVER_TRUTH UNMARKED_REENACTMENT NO_PROMPT_OR_VERSION_HISTORY REPAIR_COST_EXCEEDS_VALUE PLATFORM_CROP_DESTROYS_INFORMATION ``` ## 18. Acceptance test ```text Does the asset perform a named beat function? Can it cut into and out of neighboring material? Are entry and exit states explicit and compatible? Are subject, world, camera, and lighting locks satisfied? Does motion obey visible cause and contact? Are text, logos, anatomy, hands, reflections, and shadows stable? Is the representational status truthful and disclosed? Are rights and provenance recorded? Does the shot survive target aspect ratios and compression? ```