RookeryOS — Agent Pod Lifecycle

From provisioning to teardown — the complete lifecycle of an agent pod

Lifecycle Overview

┌─────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐
│  CREATE  │───►│ PROVISION │───►│ BOOTSTRAP│───►│  RUNNING  │───►│ TEARDOWN │
│  Request │    │ K8s Pod   │    │ chelper  │    │  Active   │    │ Delete   │
└─────────┘    └──────────┘    └──────────┘    └──────────┘    └──────────┘
                                    │                                │
                                    ▼                                ▼
                               ┌──────────┐                    ┌──────────┐
                               │  FAILED   │                    │ CLEANUP  │
                               │  Retry    │                    │ PVC, NS  │
                               └──────────┘                    └──────────┘

Phase 1: Create Request

The lifecycle begins when a tenant or admin requests a new agent.

Request Sources

Source	Trigger
Agent Factory	User completes the agent creation wizard in Console
API	`POST /api/v1/agents` with agent configuration
Provisioner	Automatic re-provisioning after pod failure

Agent Configuration

interface AgentConfig {
  name: string;                    // Display name
  namespace: string;               // K8s namespace
  tier: 'starter' | 'pro' | 'business' | 'enterprise' | 'navigator';
  llm_model: string;              // e.g., 'zai/glm-4.5'
  soul: string;                   // SOUL.md content
  personality: PersonalityConfig;
  tools: ToolConfig[];
  connectors: ConnectorConfig[];
  enable_sidecar: boolean;        // Dual-container?
  enable_cookie_vault: boolean;
  enable_hitl: boolean;
  resources: ResourceConfig;
}

Tier Selection

Tier	Pod Type	Sidecar	Cookie Vault	PVC	Max Lifetime
Starter	Single	No	No	5Gi	Unlimited
Pro	Dual	Yes	Yes	10Gi	Unlimited
Business	Dual	Yes	Yes	20Gi	Unlimited
Enterprise	Dual	Yes	Yes	50Gi	Unlimited
Navigator	Single	No	No	None	24 hours

Phase 2: Provisioning

The Agent Provisioner creates the Kubernetes resources.

Provisioner Flow

API Request
  │
  ├── 1. Validate agent config
  │     Check required fields, tier limits, quotas
  │
  ├── 2. Generate namespace (production pods)
  │     Format: agent-blue-heron-{agent-id-prefix}
  │     Or use shared namespace (navigator)
  │
  ├── 3. Apply RBAC
  │     ServiceAccount, Role, RoleBinding
  │     Workload Identity binding (production only)
  │
  ├── 4. Apply NetworkPolicy
  │     Production: allow Platform API + internet
  │     Navigator: restricted egress, no cross-pod
  │
  ├── 5. Create PVC (production only)
  │     StorageClass: standard
  │     Size: based on tier
  │
  ├── 6. Generate pod manifest
  │     Template: agent-pod-v2.yaml (dual) or navigator-pod.yaml (single)
  │     Inject: env vars, secrets, resource limits
  │
  ├── 7. Create HTTPRoute
  │     Hostname: {agent-slug}.pods.agentworksstudio.com
  │     Backend: agent service
  │
  ├── 8. Create KEDA HTTPScaledObject
  │     Enable scale-to-zero
  │     Min: 0, Max: based on tier
  │
  └── 9. Submit to K8s API
        kubectl apply -f manifest.yaml
        Pod starts → Phase 3

Provisioner Files

apps/agent-provisioner/src/
├── provisioner.ts              # Production pod provisioning
└── navigator-provisioner.ts   # Navigator pod provisioning

Phase 3: Bootstrap (chelper)

Once the pod starts, chelper runs as a s6-overlay oneshot before Hudson Gateway starts.

Bootstrap Sequence

Pod Running
  │
  ├── tini (PID 1)
  │     └── s6-overlay
  │           │
  │           ├── init-config-gen (oneshot)
  │           │     1. Read env vars (AGENT_ID, TENANT_ID, etc.)
  │           │     2. Fetch agent config from Platform API
  │           │     3. Generate hudson.json config file
  │           │     4. Write to HUDSON_STATE_DIR
  │           │
  │           └── init-chelper (oneshot, depends on init-config-gen)
  │                 1. Wait for sidecar CDP (localhost:9222)
  │                 │    Timeout: 30 seconds
  │                 │    Retries: every 2 seconds
  │                 │
  │                 2. Validate PVC mount
  │                 │    Check HUDSON_STATE_DIR is writable
  │                 │    Create directory structure if fresh
  │                 │
  │                 3. Restore cookies from vault
  │                 │    IF ENABLE_COOKIE_VAULT=true:
  │                 │      Fetch encrypted cookies from API
  │                 │      Decrypt with DEK from Secret Manager
  │                 │      Inject into browser context
  │                 │
  │                 4. Exit 0 → Hudson can start
  │                    (or Exit 1 → pod fails, restart)

Bootstrap Failure Handling

Failure	Action
Sidecar CDP timeout	Pod restart (kubelet will retry)
PVC mount failure	Pod fails, manual intervention
Cookie vault error	Continue without cookies (non-fatal)
Config fetch failure	Pod restart with backoff

s6-overlay Service Graph

init-config-gen (oneshot)
  │
  └──► init-chelper (oneshot)
         │
         ├──► svc-nginx (longrun)
         │      nginx -g 'daemon off;'
         │
         └──► svc-hudson (longrun)
                node dist/entry.js gateway run

Phase 4: Running

Once bootstrapped, the pod serves agent requests.

Request Flow

Client (Console / API / Direct)
  │
  ├── HTTPS to {agent-slug}.pods.agentworksstudio.com
  │
  ├── GKE Gateway (34.102.157.95)
  │     ├── TLS termination
  │     └── HTTPRoute → pod IP
  │
  ├── Kernel :8080 (nginx)
  │     ├── /api/* → Hudson Gateway :3001
  │     ├── /socket.io/* → Hudson WebSocket
  │     └── /* → Static assets
  │
  └── Hudson Gateway
        ├── Parse request
        ├── Check Lighthouse knowledge
        ├── LLM call (if needed)
        ├── Tool execution
        │     ├── Browser actions → Sidecar :9222 (CDP)
        │     ├── Terminal commands → sandbox
        │     └── API calls → external services
        ├── Memory operations
        │     ├── Read SOUL.md
        │     └── Update memory files
        └── Return response

Browser Interaction Flow

Agent needs to interact with a website:
  │
  ├── Hudson → CDP → Sidecar Chromium
  │     Navigate to URL
  │     Page loads
  │
  ├── Login Detector checks for auth pages
  │     ├── No login → continue automation
  │     │     Browser-Use distills DOM → LLM tokens
  │     │     Agent processes content
  │     │
  │     └── Login detected → HITL Controller
  │           Pause agent execution
  │           WebSocket event → Console shows alert
  │           Human opens noVNC → authenticates
  │           Cookies captured → encrypted → stored
  │           Agent resumes with authenticated session
  │
  └── Agent completes task with browser data

Health Monitoring

Check	Interval	Action on Failure
Kernel health (`:8080/health`)	30s	Pod restart
Sidecar CDP (`:9222`)	60s	Sidecar restart (if possible)
Browser-Use (`:9223/health`)	60s	Service restart
PVC disk space	5m	Alert if >80%
Cookie freshness	On use	Re-auth if expired

Phase 5: Teardown

When an agent is deleted or its lifetime expires.

Teardown Triggers

Trigger	Action
Admin deletion	Via Console or API
Tenant deletion	Via Console
Navigator TTL	Auto-delete after 24 hours
Pod crash loop	Manual intervention
Billing failure	Grace period then delete

Teardown Flow

Delete Request
  │
  ├── 1. Stop accepting new requests
  │     KEDA HTTPScaledObject → scale to 0
  │
  ├── 2. Wait for in-flight requests
  │     Grace period: 30 seconds
  │
  ├── 3. Delete HTTPRoute
  │     Remove subdomain routing
  │
  ├── 4. Delete KEDA HTTPScaledObject
  │
  ├── 5. Delete pod
  │     kubectl delete pod -n {namespace}
  │
  ├── 6. Delete PVC
  │     kubectl delete pvc agent-workspace -n {namespace}
  │
  ├── 7. Delete namespace (production)
  │     kubectl delete namespace agent-blue-heron-{id}
  │     Removes all resources: RBAC, NetworkPolicy, etc.
  │
  ├── 8. Clean cookie vault
  │     DELETE FROM cookie_vault WHERE agent_id = $1
  │
  └── 9. Update agent status in database
        status: 'deleted'
        (Lighthouse documents preserved at tenant level)

Navigator Auto-Delete

Navigator pods have a TTL controller:

# Automatically deletes after 24 hours
metadata:
  annotations:
    navigator-ttl: "24h"

No PVC to clean up — all data is ephemeral.

Pod States

State	Description	User-Visible
`provisioning`	K8s resources being created	"Setting up your agent..."
`bootstrapping`	chelper running	"Initializing..."
`starting`	Hudson Gateway loading	"Almost ready..."
`healthy`	All checks passing	Green indicator
`degraded`	Sidecar or non-critical service down	Yellow indicator
`unhealthy`	Kernel or critical service down	Red indicator
`scaling_down`	KEDA scaling to zero	"Agent is sleeping"
`scaling_up`	KEDA scaling from zero	"Waking up agent..."
`terminating`	Graceful shutdown	"Agent shutting down"
`failed`	Bootstrap or runtime failure	"Agent encountered an error"

File Locations

apps/agent-provisioner/src/
├── provisioner.ts                    # Production provisioning logic
└── navigator-provisioner.ts         # Navigator provisioning logic

infrastructure/docker/rookery-kernel/
├── Dockerfile                        # Kernel image
└── rootfs/etc/s6-overlay/
    ├── s6-rc.d/
    │   ├── init-config-gen/          # Config generation service
    │   ├── init-chelper/             # Bootstrap validation service
    │   ├── svc-hudson/               # Hudson Gateway service
    │   └── svc-nginx/                # nginx service
    └── scripts/chelper               # Bootstrap executable

apps/hudson-gateway/src/bootstrap/
└── chelper.ts                        # TypeScript bootstrap logic

infrastructure/k8s/templates/
└── agent-pod-v2.yaml                 # Dual-container pod template

infrastructure/k8s/navigator/
└── navigator-pod.yaml                # Single-container template

PreviousDual-Container Architecture Next Lighthouse Knowledge Layer