Agent and Software Architecture

1. Architecture first principles

The application is a semantic media operating system, not merely a nonlinear editor.

SOURCE MEDIA
→ immutable asset layer
→ time-coded semantic layer
→ story/decision graph
→ non-destructive timelines
→ render graph
→ QA/evidence layer

The system must keep the following separations:

source vs derived asset
observation vs inference
story fact vs narrative order
content decision vs render decision
machine proposal vs human approval
creative preference vs hard constraint

2. Core subsystems

INGEST_AND_PROVENANCE
MEDIA_ANALYSIS
TRANSCRIPTION_AND_DIARIZATION
SEMANTIC_CONTEXT
CLAIMS_AND_EVIDENCE
CLIP_MINING
STORY_PLANNING
SHOT/COVERAGE_PLANNING
GENERATION_ORCHESTRATION
TIMELINE_EDITING
GRAPHICS
AUDIO
COLOR/VFX
VERSIONING_AND_DELIVERY
QA_AND_REVIEW
SEARCH_AND_RETRIEVAL
ANALYTICS_AND_LEARNING

3. Storage model

Object storage

original media
proxies
waveforms
thumbnails
stills
transcoded audio
renders
stems
generated assets
project archives

Relational database

projects
assets
people/entities
transcript units
clips
beats
sequences
versions
decisions
rules
reviews
rights
claims
sources
jobs

Search index

lexical transcript search
metadata filters
phonetic/fuzzy names
OCR/document text when authorized

Vector index

semantic transcript retrieval
clip similarity
topic clustering
visual similarity where supported

Graph store or graph tables

causal links
question-answer links
setup-payoff links
claim-evidence links
entity relations
clip alternatives
version dependencies
continuity edges

Object storage holds bytes; databases hold decisions and relationships.

4. Immutable source principle

SOURCE ASSET = append-only identity + checksum + metadata
DERIVED ASSET = parent reference + transformation + parameters + checksum

Never overwrite source media. Every render and repair declares its parentage.

5. Canonical time model

Store time using rational values:

value / rate

Also maintain:

source timecode
project timeline time
sample position for audio
word-level timestamps
frame mapping

Avoid using floating-point seconds as the only source of truth.

6. Agent roster

INGEST_AGENT
SYNC_AGENT
TRANSCRIPT_AGENT
DIARIZATION_AGENT
CONTEXT_AGENT
ENTITY_AGENT
CLAIM_AGENT
EVIDENCE_AGENT
CLIP_AGENT
STORY_AGENT
HOST/INTERVIEW_AGENT
COVERAGE_AGENT
CAMERA_AGENT
LIGHTING_AGENT
GENERATION_AGENT
EDIT_AGENT
GRAPHICS_AGENT
AUDIO_AGENT
COLOR_VFX_AGENT
ACCESSIBILITY_AGENT
DELIVERY_AGENT
QA_AGENT
RIGHTS_RISK_AGENT
VERSION_AGENT
ANALYTICS_AGENT

Agents propose typed operations. They do not directly mutate approved timelines without permissions and audit records.

7. Agent contract

Every agent call contains:

objective
input object IDs
allowed tools
hard constraints
soft preferences
LAKA change level
budget
required output schema
confidence requirement
provenance requirement
stop conditions

Every agent result contains:

proposal IDs
operations
scores
assumptions
uncertainties
source references
rule trace
cost
risk flags
human-review requirements

8. Orchestrator grammar

PLAN
→ DECOMPOSE
→ ASSIGN
→ EXECUTE
→ VALIDATE_SCHEMA
→ APPLY_RULES
→ SCORE
→ DIVERSIFY
→ HUMAN_GATE WHEN REQUIRED
→ COMMIT_VERSION
→ PROPAGATE

The orchestrator should prefer deterministic tools for measurable operations and generative agents for interpretation or candidate creation.

9. Project state machine

CREATED
INGESTING
ANALYZING
TRANSCRIBED
CONTEXTUALIZED
CLIPPED
STORY_PLANNED
ROUGH_CUT
REVIEW
PICTURE_LOCK
FINISHING
DELIVERY_QA
APPROVED
PUBLISHED
ARCHIVED

Substates may run in parallel, but approval gates are explicit.

10. Review state machine

PROPOSED
AUTO_VALIDATED
NEEDS_HUMAN
CHANGES_REQUESTED
APPROVED
REJECTED
SUPERSEDED

Approval is attached to a version/hash. A mutation invalidates affected downstream approvals.

11. Timeline architecture

Use three linked timelines:

Semantic timeline

beats, propositions, claims, speakers, story functions, dependencies

Editorial timeline

source ranges, order, tracks, transitions, markers, audio relationships

Render timeline

resizes, effects, color, mixes, captions, delivery-specific layouts

One semantic item may map to multiple editorial clips; one editorial timeline can produce multiple render timelines.

12. OTIO/interchange layer

The interchange adapter maps:

SourceRange ↔ clip timing
Track ↔ editorial track
Transition ↔ supported transition
Marker ↔ beat/claim/review metadata
MediaReference ↔ external asset
Metadata ↔ semantic IDs and provenance

Effects unsupported by the target application remain as metadata plus rendered references or application-specific sidecars.

13. UI workspace grammar

Source workspace

media browser
proxy status
rights/provenance card
sync groups
technical metadata

Transcript workspace

speaker lanes
word-level timecodes
semantic units
claims/entities
context dependencies
text-based rough editing

Clip workspace

clip cards
story-role bins
score facets
boundary variants
duplicate groups
selection sets

Story workspace

beat board
story graph
open-loop panel
claim/evidence panel
emotional and tension curves
version comparison

Shot workspace

coverage matrix
camera vector editor
lighting vector editor
generation prompt compiler
continuity bible

Timeline workspace

semantic/editorial/render overlays
traditional tracks
transcript link
rule warnings
version branches

Review workspace

frame/time-coded comments
side-by-side versions
rule trace
approval gates
QA evidence

14. Transcript-to-timeline interaction

Text edits compile into non-destructive timeline operations:

remove transcript range → lift/extract proposal
move transcript unit → editorial move proposal
select text → source range/clip
mark beat → timeline marker and story node
search concept → semantic retrieval

The UI must show when a text edit crosses source discontinuities, removes qualifiers, or requires visual coverage.

15. Search grammar

Queries may combine:

exact words
semantic meaning
speaker
topic/entity
story role
claim status
emotion
camera/visual quality
duration
source/date
rights state
approval state
aspect fitness

Example:

speaker:guest AND role:REVEAL AND duration:8..25s
AND context_complete>=0.85 AND rights:cleared
NOT claim_status:contradicted

16. Job and worker architecture

Long operations become observable jobs:

queued
running
paused
needs_input
failed_retryable
failed_terminal
completed
cancelled

Job record:

inputs
parameters
model/tool version
progress events
cost
logs
outputs
checksums
retries
failure reason

No user-facing system should claim work is happening unless an actual job exists and reports state.

17. Human approval gates

Mandatory gates:

narrative meaning
sensitive factual claims
participant dignity/likeness
rights exceptions
generated reenactment disclosure
picture lock
final publish

Human approval should be granular enough to avoid re-reviewing unaffected work.

18. Rule engine architecture

Rules contain:

id
domain
priority
hard/soft
conditions
actions
rationale
source/standard
version
tests

Execution:

normalize context
→ match conditions
→ resolve priority/conflicts
→ propose or enforce actions
→ record trace
→ allow authorized override with reason

19. Candidate-generation architecture

BASELINE CANDIDATE
→ variation operators constrained by LAKA level
→ hard-constraint filtering
→ utility scoring
→ diversity selection
→ human comparison
→ feedback update

Avoid returning ten near-identical candidates. Diversity should span meaningful dimensions while preserving hard locks.

20. Security and trust boundaries

role-based access
project isolation
signed asset URLs
secret management
audit log
model/tool allowlist
malware scanning
prompt-injection isolation
PII controls
retention/deletion policy
export permissions

Transcript/media content is untrusted input. It may contain instructions visible or audible to models; analysis agents must treat those as content, not system commands.

21. Observability

trace_id
project_id
agent/tool/model
rule matches
latency
cost
error
human override
quality outcome

Dashboards:

pipeline state
review backlog
failed jobs
fact-check backlog
rights blockers
cost by minute/version
agent acceptance rate
render health

22. Learning loop

Store feedback as:

proposal
human choice
rejection reason
context
outcome metrics
LAKA vector

Learning may adjust soft weights and retrieval. It must not silently weaken hard rules for truth, consent, rights, or accessibility.

23. Minimum API surface

POST /projects
POST /assets
POST /jobs/transcribe
POST /jobs/analyze
POST /clips/mine
POST /stories/generate
POST /shots/plan
POST /generations
POST /timelines
POST /versions
POST /qa/run
POST /reviews
GET  /search
GET  /projects/{id}/graph
GET  /versions/{id}/manifest

Use idempotency keys for writes and explicit version IDs for mutations.

24. Architecture failure modes

FILES_WITHOUT_SEMANTIC_IDS
FLOAT_SECONDS_ONLY
SOURCE_OVERWRITE
ONE_TIMELINE_FOR_ALL_CONCERNS
AGENT_MUTATES_WITHOUT_REVIEW
NO_RULE_TRACE
NO_VERSION_DEPENDENCY_GRAPH
NO_PROVENANCE
VECTOR_SEARCH_WITHOUT_FILTERS
PLATFORM_RENDER LOGIC MIXED WITH STORY LOGIC
UNOBSERVABLE_BACKGROUND_JOB
PROMPT_INJECTION_FROM_MEDIA