RookeryOS — Lighthouse Knowledge Layer

The state-driven knowledge system that powers agent intelligence

Overview

Lighthouse is RookeryOS's knowledge management layer — the source of truth for all information that agents need to do their work. It provides sub-100ms document lookups via Redis, semantic search via pgvector, and a full ingestion pipeline that processes documents from raw content to query-ready knowledge.

Every agent query starts with Lighthouse. Before making an LLM call, the Reasoner checks the knowledge index for existing answers. This reduces token usage by 90% and eliminates hallucination on known topics.

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Lighthouse Knowledge Layer                 │
│                                                              │
│  ┌─────────────┐    ┌──────────────┐    ┌────────────────┐  │
│  │  Ingestion   │    │  Storage     │    │  Query         │  │
│  │              │    │              │    │                │  │
│  │  Librarian   │───►│  PostgreSQL  │    │  Reasoner      │  │
│  │  Pipeline    │    │  (Source of  │◄──►│  (Index-first  │  │
│  │              │    │   Truth)     │    │   reasoning)   │  │
│  │  6 Steps:    │    │              │    │                │  │
│  │  PARSE       │    │  ┌────────┐  │    │  1. Path match │  │
│  │  VALIDATE    │    │  │pgvector│  │    │  2. Tag search │  │
│  │  FINGERPRINT │    │  │(embed) │  │    │  3. Vector     │  │
│  │  UPSERT      │    │  └────────┘  │    │  4. Temporal   │  │
│  │  INDEX       │    │              │    │     verify     │  │
│  │  EMBED       │    │  ┌────────┐  │    │                │  │
│  └─────────────┘    │  │ HTML   │  │    └───────┬────────┘  │
│                      │  │Renderer│  │            │           │
│  ┌─────────────┐    │  └────────┘  │            │           │
│  │  Redis Cache │    │              │            │           │
│  │  (Registry)  │◄───┤              │            │           │
│  │              │    │              │            │           │
│  │  Path→Meta   │    │              │            │           │
│  │  Tag→DocIDs  │    │              │            │           │
│  │  Fingerprint │    │              │            │           │
│  └─────────────┘    └──────────────┘            │           │
│                                                   │           │
│  ┌─────────────┐    ┌──────────────┐             │           │
│  │  API Routes  │    │  Specialized │             │           │
│  │              │    │  Services    │             │           │
│  │  /query      │◄───┤              │◄────────────┘           │
│  │  /ingest     │    │  Audit       │                          │
│  │  /documents  │    │  Processor   │                          │
│  │  /graph      │    │              │                          │
│  │  /health     │    │  Beacon      │                          │
│  │              │    │  Classifier  │                          │
│  │              │    │              │                          │
│  │              │    │  Verified    │                          │
│  │              │    │  Ledger      │                          │
│  │              │    │              │                          │
│  │              │    │  Briefing    │                          │
│  │              │    │  Generator   │                          │
│  │              │    │              │                          │
│  │              │    │  Pricing     │                          │
│  │              │    │  Engine      │                          │
│  └─────────────┘    └──────────────┘                          │
└─────────────────────────────────────────────────────────────┘

Core Components

1. Registry (`registry.ts`)

The Registry provides sub-100ms document lookups via Redis with PostgreSQL fallback.

Key Operations:

Operation	Redis Key	Purpose
`resolve(path)`	`lh:path:{tenant}:{path}`	Path → document metadata
`tagMembers(tag)`	`lh:tags:{tenant}:{tag}`	Tag → set of doc IDs
`register(doc)`	Both above	Index a document
`expire(path)`	Delete + TTL mark	Flag stale entry

Resolution Flow:

Query: "/knowledge/pricing/starter"
  → Redis GET lh:path:{tenant}:/knowledge/pricing/starter
    → Hit? Return metadata (<1ms)
    → Miss? PostgreSQL SELECT + cache result (~10ms)

SYSTEM_CORE Protection: Documents in the SYSTEM_CORE namespace (tenant_id = '00000000-...') are read-only for tenant pods. Agents can read platform knowledge but cannot modify it.

2. Librarian (`librarian.ts`)

The Librarian is the ingestion pipeline that transforms raw content into indexed knowledge.

Pipeline Steps:

Raw Content (Markdown + YAML frontmatter)
  │
  ├── 1. PARSE
  │     Extract YAML header → metadata
  │     Separate body → content
  │
  ├── 2. VALIDATE
  │     Mandatory fields: title, category, path
  │     Type checking and sanitization
  │
  ├── 3. FINGERPRINT
  │     SHA-256 hash of content
  │     Skip if hash matches existing (no-change detection)
  │
  ├── 4. UPSERT
  │     INSERT on new path, UPDATE on existing
  │     Set status, visibility, timestamps
  │
  ├── 5. INDEX
  │     Register path in Redis Registry
  │     Index tags in Redis Sets
  │
  └── 6. EMBED
        Generate vector embedding via pgvector
        Store alongside document for semantic search

Ingestion Sources:

Manual API calls (POST /lighthouse/ingest)
Agent outputs (ingestAgentOutput pipeline)
Document imports (YAML + Markdown files)
System core documents (platform knowledge)

3. Reasoner (`reasoner.ts`)

The Reasoner is index-first reasoning middleware that sits between agent queries and LLM calls.

Query Resolution Flow:

Agent Query: "What is the Starter plan price?"
  │
  ├── 1. Extract potential paths
  │     "/knowledge/pricing/starter", "/product/pricing/starter"
  │
  ├── 2. Registry lookup (Redis, <1ms)
  │     Try each extracted path
  │     SYSTEM_CORE fallback for platform docs
  │
  ├── 3. Hit? → Return document directly
  │     No LLM call needed. Token savings: ~90%
  │
  ├── 4. Miss? → Vector search (pgvector)
  │     Semantic similarity on embeddings
  │     Top-K results with relevance scores
  │
  ├── 5. Conflict Resolution
  │     Multiple active records for same path?
  │     Weighted scoring: recency × status × source
  │
  └── 6. Temporal Truth Verification
        Check document.status (active/deprecated/draft)
        Check last_verified against TTL
        Flag expired for re-verification

Key Principle: Every query checks Lighthouse first. LLM calls are the fallback, not the default.

4. HTML Renderer (`html-renderer.ts`)

Converts Markdown documents into self-contained HTML with the Lighthouse design system.

Features:

Category-based theming (Architecture=blue, Methodology=green, etc.)
Related document linking with sibling families
Navigation breadcrumbs
Print-optimized styles
Responsive layout with sidebar navigation
Syntax-highlighted code blocks

Usage: Documents rendered to HTML are stored in content_html column and served via the Lighthouse viewer in the Platform Admin console.

Database Schema

`lighthouse_documents` Table

Column	Type	Purpose
`tenant_id`	UUID	Owner (SYSTEM_CORE = platform docs)
`doc_id`	UUID	Unique identifier
`path`	TEXT	Canonical path (e.g., `/knowledge/pricing/starter`)
`title`	TEXT	Display title
`category`	TEXT	Classification (architecture, pricing, etc.)
`content`	TEXT	Markdown body
`content_html`	TEXT	Rendered HTML
`status`	TEXT	`active`, `deprecated`, `draft`, `pending_review`
`visibility`	TEXT	`public`, `internal`
`tags`	TEXT[]	PostgreSQL array for categorization
`content_hash`	TEXT	SHA-256 for change detection
`is_system_core`	BOOLEAN	Immutable anchor flag
`last_verified`	TIMESTAMPTZ	TTL expiration check
`embedding`	VECTOR	pgvector embedding for semantic search

Key Migrations

Migration	Purpose
`138_lighthouse_protocol.sql`	Core table structure
`144_lighthouse_visibility.sql`	Visibility controls
`152_lighthouse_bridge_tables.sql`	Bridge Protocol extensions
`157_lighthouse_pgvector.sql`	Vector search capabilities
`158_lighthouse_content_html.sql`	HTML artifact storage

API Endpoints

Method	Path	Purpose
`POST`	`/lighthouse/query`	Natural language query with optional path filter
`POST`	`/lighthouse/ingest`	Document ingestion via Librarian pipeline
`GET`	`/lighthouse/health`	System health check (Redis + PG connectivity)
`GET`	`/lighthouse/identity-health`	Agent identity health status
`GET`	`/lighthouse/documents`	Paginated document listing with filters
`GET`	`/lighthouse/documents/:docId`	Full document retrieval
`GET`	`/lighthouse/graph`	Force-directed graph visualization
`POST`	`/lighthouse/admin/regenerate-html`	Regenerate HTML artifacts

Specialized Services

Audit Processor (`audit-processor.ts`)

Processes VOSS audit transcripts using Vertex AI:

Transcribes conversation recordings
Extracts pain points and friction
Generates Kinetic Trees (operational process maps)
Classifies Beacons (high-impact opportunities)

Beacon Classifier (`beacon-classifier.ts`)

Classifies pain points from assessments:

Beacons: High-impact revenue opportunities
Stickies: Lower-priority friction points
Revenue impact scoring via Stripe integration

Verified Ledger (`verified-ledger.ts`)

Anti-hallucination boundary enforcement:

Tracks which facts have been verified
Flags unverifiable claims
Provides confidence scoring for agent outputs

Briefing Generator (`briefing-generator.ts`)

Executive briefing creation:

Aggregates data from multiple Lighthouse documents
Formats for email, Slack, and console display
Scheduled via Cron scheduler

Pricing Engine (`pricing-engine.ts`)

AI-driven pricing calculations:

Based on fiscal impact analysis
Stripe integration for validation
Tier comparison and recommendation generation

Sonar Pipeline (`sonar-pipeline.ts`)

Voice recording transcription integration:

Processes recordings from Sonar WebRTC
Extracts structured insights
Ingests into Lighthouse for agent access

Scheduled Jobs

Job	Schedule	Purpose
TTL Expiration	Every 6 hours	Flag documents past their TTL for re-verification
HTML Regeneration	On demand	Re-render HTML when templates change

Cron Configuration: apps/api/src/services/cron-scheduler.ts

Access Control

Visibility Levels

Level	Who Can See	Example
`public`	All tenants + unauthenticated	Pricing, features, API docs
`internal`	Same tenant only	Custom knowledge, agent outputs

Auto-Visibility by Path Prefix

Documents with these path prefixes are automatically set to public:

/knowledge/pricing
/knowledge/integrations
/knowledge/features
/knowledge/api
/knowledge/setup
/knowledge/onboarding
/knowledge/legal
/knowledge/security
/knowledge/sla
/product/connectors
/product/pricing

All other paths default to internal.

Integration Points

Agent Tools (Hudson Gateway)

Agents access Lighthouse through built-in tools:

Tool	Purpose
`read_from_lighthouse_memory`	Historical data lookup
`write_to_lighthouse_memory`	Knowledge persistence
`lighthouse_query`	Natural language query
`lighthouse_ingest`	Document creation

Console Integration

The Platform Admin console provides:

Document viewer with rendered HTML
Graph visualization of document relationships
Health monitoring for Registry and Database
Admin controls for regeneration and management

Bridge Protocol

Lighthouse powers the Bridge assessment pipeline:

Assessment results stored as Lighthouse documents
Beacons and Stickies classified and indexed
Value dossiers generated from knowledge layer
Client-facing HTML rendered from stored content

Performance Characteristics

Operation	Latency	Storage
Path lookup (Redis hit)	<1ms	~1KB per entry
Path lookup (Redis miss)	~10ms	PostgreSQL fallback
Semantic search (pgvector)	~50ms	Depends on corpus size
Full ingestion pipeline	~200ms	Document + embedding + cache
HTML rendering	~100ms	Self-contained HTML
TTL expiration scan	~5s	Full table scan (every 6h)

File Locations

apps/api/src/
├── routes/lighthouse.ts                    # API endpoints
├── services/lighthouse/
│   ├── registry.ts                         # Redis path registry
│   ├── librarian.ts                        # Ingestion pipeline
│   ├── reasoner.ts                         # Index-first reasoning
│   ├── html-renderer.ts                    # Markdown → HTML
│   ├── audit-processor.ts                  # VOSS audit processing
│   ├── verified-ledger.ts                  # Anti-hallucination
│   ├── beacon-classifier.ts                # Pain point classification
│   ├── transcript-ledger.ts                # Conversation capture
│   ├── briefing-generator.ts              # Executive briefings
│   ├── value-dossier-generator.ts         # Client value docs
│   ├── pricing-engine.ts                  # AI pricing
│   ├── sonar-pipeline.ts                  # Voice transcription
│   └── category-classifier.ts             # Auto-categorization
├── db/migrations/
│   ├── 138_lighthouse_protocol.sql
│   ├── 144_lighthouse_visibility.sql
│   ├── 152_lighthouse_bridge_tables.sql
│   ├── 157_lighthouse_pgvector.sql
│   └── 158_lighthouse_content_html.sql
└── services/cron-scheduler.ts             # TTL expiration job

apps/console/src/pages/
├── admin/workspace/Lighthouse.tsx          # Document viewer
├── admin/mission-control/Lighthouse.tsx    # Health dashboard
└── components/prism/LighthouseBlock.tsx    # Embeddable block

PreviousAgent Pod Lifecycle Next Sonar Voice Engine