# 16 — Agent and Software Architecture ## 1. Architecture first principles The application is a semantic media operating system, not merely a nonlinear editor. ```text SOURCE MEDIA → immutable asset layer → time-coded semantic layer → story/decision graph → non-destructive timelines → render graph → QA/evidence layer ``` The system must keep the following separations: ```text source vs derived asset observation vs inference story fact vs narrative order content decision vs render decision machine proposal vs human approval creative preference vs hard constraint ``` ## 2. Core subsystems ```text INGEST_AND_PROVENANCE MEDIA_ANALYSIS TRANSCRIPTION_AND_DIARIZATION SEMANTIC_CONTEXT CLAIMS_AND_EVIDENCE CLIP_MINING STORY_PLANNING SHOT/COVERAGE_PLANNING GENERATION_ORCHESTRATION TIMELINE_EDITING GRAPHICS AUDIO COLOR/VFX VERSIONING_AND_DELIVERY QA_AND_REVIEW SEARCH_AND_RETRIEVAL ANALYTICS_AND_LEARNING ``` ## 3. Storage model ### Object storage ```text original media proxies waveforms thumbnails stills transcoded audio renders stems generated assets project archives ``` ### Relational database ```text projects assets people/entities transcript units clips beats sequences versions decisions rules reviews rights claims sources jobs ``` ### Search index ```text lexical transcript search metadata filters phonetic/fuzzy names OCR/document text when authorized ``` ### Vector index ```text semantic transcript retrieval clip similarity topic clustering visual similarity where supported ``` ### Graph store or graph tables ```text causal links question-answer links setup-payoff links claim-evidence links entity relations clip alternatives version dependencies continuity edges ``` Object storage holds bytes; databases hold decisions and relationships. ## 4. Immutable source principle ```text SOURCE ASSET = append-only identity + checksum + metadata DERIVED ASSET = parent reference + transformation + parameters + checksum ``` Never overwrite source media. Every render and repair declares its parentage. ## 5. Canonical time model Store time using rational values: ```text value / rate ``` Also maintain: ```text source timecode project timeline time sample position for audio word-level timestamps frame mapping ``` Avoid using floating-point seconds as the only source of truth. ## 6. Agent roster ```text INGEST_AGENT SYNC_AGENT TRANSCRIPT_AGENT DIARIZATION_AGENT CONTEXT_AGENT ENTITY_AGENT CLAIM_AGENT EVIDENCE_AGENT CLIP_AGENT STORY_AGENT HOST/INTERVIEW_AGENT COVERAGE_AGENT CAMERA_AGENT LIGHTING_AGENT GENERATION_AGENT EDIT_AGENT GRAPHICS_AGENT AUDIO_AGENT COLOR_VFX_AGENT ACCESSIBILITY_AGENT DELIVERY_AGENT QA_AGENT RIGHTS_RISK_AGENT VERSION_AGENT ANALYTICS_AGENT ``` Agents propose typed operations. They do not directly mutate approved timelines without permissions and audit records. ## 7. Agent contract Every agent call contains: ```text objective input object IDs allowed tools hard constraints soft preferences LAKA change level budget required output schema confidence requirement provenance requirement stop conditions ``` Every agent result contains: ```text proposal IDs operations scores assumptions uncertainties source references rule trace cost risk flags human-review requirements ``` ## 8. Orchestrator grammar ```text PLAN → DECOMPOSE → ASSIGN → EXECUTE → VALIDATE_SCHEMA → APPLY_RULES → SCORE → DIVERSIFY → HUMAN_GATE WHEN REQUIRED → COMMIT_VERSION → PROPAGATE ``` The orchestrator should prefer deterministic tools for measurable operations and generative agents for interpretation or candidate creation. ## 9. Project state machine ```text CREATED INGESTING ANALYZING TRANSCRIBED CONTEXTUALIZED CLIPPED STORY_PLANNED ROUGH_CUT REVIEW PICTURE_LOCK FINISHING DELIVERY_QA APPROVED PUBLISHED ARCHIVED ``` Substates may run in parallel, but approval gates are explicit. ## 10. Review state machine ```text PROPOSED AUTO_VALIDATED NEEDS_HUMAN CHANGES_REQUESTED APPROVED REJECTED SUPERSEDED ``` Approval is attached to a version/hash. A mutation invalidates affected downstream approvals. ## 11. Timeline architecture Use three linked timelines: ### Semantic timeline ```text beats, propositions, claims, speakers, story functions, dependencies ``` ### Editorial timeline ```text source ranges, order, tracks, transitions, markers, audio relationships ``` ### Render timeline ```text resizes, effects, color, mixes, captions, delivery-specific layouts ``` One semantic item may map to multiple editorial clips; one editorial timeline can produce multiple render timelines. ## 12. OTIO/interchange layer The interchange adapter maps: ```text SourceRange ↔ clip timing Track ↔ editorial track Transition ↔ supported transition Marker ↔ beat/claim/review metadata MediaReference ↔ external asset Metadata ↔ semantic IDs and provenance ``` Effects unsupported by the target application remain as metadata plus rendered references or application-specific sidecars. ## 13. UI workspace grammar ### Source workspace ```text media browser proxy status rights/provenance card sync groups technical metadata ``` ### Transcript workspace ```text speaker lanes word-level timecodes semantic units claims/entities context dependencies text-based rough editing ``` ### Clip workspace ```text clip cards story-role bins score facets boundary variants duplicate groups selection sets ``` ### Story workspace ```text beat board story graph open-loop panel claim/evidence panel emotional and tension curves version comparison ``` ### Shot workspace ```text coverage matrix camera vector editor lighting vector editor generation prompt compiler continuity bible ``` ### Timeline workspace ```text semantic/editorial/render overlays traditional tracks transcript link rule warnings version branches ``` ### Review workspace ```text frame/time-coded comments side-by-side versions rule trace approval gates QA evidence ``` ## 14. Transcript-to-timeline interaction Text edits compile into non-destructive timeline operations: ```text remove transcript range → lift/extract proposal move transcript unit → editorial move proposal select text → source range/clip mark beat → timeline marker and story node search concept → semantic retrieval ``` The UI must show when a text edit crosses source discontinuities, removes qualifiers, or requires visual coverage. ## 15. Search grammar Queries may combine: ```text exact words semantic meaning speaker topic/entity story role claim status emotion camera/visual quality duration source/date rights state approval state aspect fitness ``` Example: ```text speaker:guest AND role:REVEAL AND duration:8..25s AND context_complete>=0.85 AND rights:cleared NOT claim_status:contradicted ``` ## 16. Job and worker architecture Long operations become observable jobs: ```text queued running paused needs_input failed_retryable failed_terminal completed cancelled ``` Job record: ```text inputs parameters model/tool version progress events cost logs outputs checksums retries failure reason ``` No user-facing system should claim work is happening unless an actual job exists and reports state. ## 17. Human approval gates Mandatory gates: ```text narrative meaning sensitive factual claims participant dignity/likeness rights exceptions generated reenactment disclosure picture lock final publish ``` Human approval should be granular enough to avoid re-reviewing unaffected work. ## 18. Rule engine architecture Rules contain: ```text id domain priority hard/soft conditions actions rationale source/standard version tests ``` Execution: ```text normalize context → match conditions → resolve priority/conflicts → propose or enforce actions → record trace → allow authorized override with reason ``` ## 19. Candidate-generation architecture ```text BASELINE CANDIDATE → variation operators constrained by LAKA level → hard-constraint filtering → utility scoring → diversity selection → human comparison → feedback update ``` Avoid returning ten near-identical candidates. Diversity should span meaningful dimensions while preserving hard locks. ## 20. Security and trust boundaries ```text role-based access project isolation signed asset URLs secret management audit log model/tool allowlist malware scanning prompt-injection isolation PII controls retention/deletion policy export permissions ``` Transcript/media content is untrusted input. It may contain instructions visible or audible to models; analysis agents must treat those as content, not system commands. ## 21. Observability ```text trace_id project_id agent/tool/model rule matches latency cost error human override quality outcome ``` Dashboards: ```text pipeline state review backlog failed jobs fact-check backlog rights blockers cost by minute/version agent acceptance rate render health ``` ## 22. Learning loop Store feedback as: ```text proposal human choice rejection reason context outcome metrics LAKA vector ``` Learning may adjust soft weights and retrieval. It must not silently weaken hard rules for truth, consent, rights, or accessibility. ## 23. Minimum API surface ```text POST /projects POST /assets POST /jobs/transcribe POST /jobs/analyze POST /clips/mine POST /stories/generate POST /shots/plan POST /generations POST /timelines POST /versions POST /qa/run POST /reviews GET /search GET /projects/{id}/graph GET /versions/{id}/manifest ``` Use idempotency keys for writes and explicit version IDs for mutations. ## 24. Architecture failure modes ```text FILES_WITHOUT_SEMANTIC_IDS FLOAT_SECONDS_ONLY SOURCE_OVERWRITE ONE_TIMELINE_FOR_ALL_CONCERNS AGENT_MUTATES_WITHOUT_REVIEW NO_RULE_TRACE NO_VERSION_DEPENDENCY_GRAPH NO_PROVENANCE VECTOR_SEARCH_WITHOUT_FILTERS PLATFORM_RENDER LOGIC MIXED WITH STORY LOGIC UNOBSERVABLE_BACKGROUND_JOB PROMPT_INJECTION_FROM_MEDIA ```