diff --git a/plan.md b/plan.md index 7ca1706..d69d670 100644 --- a/plan.md +++ b/plan.md @@ -2,6 +2,7 @@ **Language:** Go **Inference:** llama.cpp (HTTP server), OpenAI-compatible API planned +**Storage:** SQLite (primary), PostgreSQL + pgvector (future) **Inspiration:** OpenHer, Agent Diva, nanobot --- @@ -9,7 +10,8 @@ ## Current Status (2026-06-10) **Builds:** yes -**Runs:** yes (requires llama.cpp on :8080) +**Runs:** yes (requires llama.cpp on :8080) + **Known issues:** - Prompt template is raw text — model dumps reasoning/self-correction meta-text instead of just the response @@ -32,6 +34,9 @@ User ┌─────┴──────────────────────────────┐ │ Core Agent Loop │ │ (internal/agent/) │ + │ - Message routing │ + │ - Tool invocation │ + │ - Context management │ └─────┬──────────────────────────────┘ │ ┌─────┴──────────────────────────────┐ @@ -47,7 +52,9 @@ User ┌─────┴──────────────────────────────┐ │ Memory Layer │ │ (internal/memory/) │ - │ ├─ RAG store (keywords → vectors) │ + │ ├─ Short-term (ephemeral context) │ + │ ├─ Long-term (SQLite → pgvector) │ + │ ├─ Episodic / Diary (sqlite) │ │ └─ Self-memory (promises,strats) │ └─────┬──────────────────────────────┘ │ @@ -61,6 +68,10 @@ Skills / Tools: ├─ web_fetch ├─ web_search (stub) └─ MCP protocol (future) + ├─ Obsidian vault + ├─ Gitea + ├─ Actual Budget + └─ Custom MCP tools ``` --- @@ -76,6 +87,8 @@ Skills / Tools: - [x] `internal/llama/client.go` — HTTP client for `POST /completion`, `POST /embedding`, `GET /health` - [x] `.env.example`, `Makefile`, `.gitignore`, `Dockerfile`, `docker-compose.yml` - [x] `AGENTS.md` with architecture, commands, conventions +- [x] `pkg/types/types.go` — Message, Role, Mood, ToolCall types +- [x] `pkg/personality/traits.go` — TraitValue, RelationshipMetrics, DiaryEntry, Observation types --- @@ -83,173 +96,313 @@ Skills / Tools: **Status: COMPLETE** (minor issues remain) -- [x] `internal/agent/agent.go` — message → memory search → prompt → llama → save → respond -- [x] `internal/agent/scheduler.go` — tick every 15m, check mood/diary/auto-message -- [x] `cmd/divad/main.go` — wires everything: config, DB, llama, TUI, scheduler, Matrix, API -- [x] Conversation persistence to SQLite (`conversations`, `messages` tables) -- [x] Bubble Tea TUI with viewport + input bar -- [ ] Prompt template is wrong — uses raw headers, no chat format. Model outputs meta-text +- [x] `internal/agent/agent.go` — message loop: receive → memory search → prompt → llama → save → respond +- [x] `internal/agent/scheduler.go` — background goroutine, configurable tick (default 15m), mood check, diary trigger, autonomous message queue +- [x] `cmd/divad/main.go` — wires everything: config, DB, llama health, TUI, scheduler, Matrix, API +- [x] Conversation persistence to SQLite (`conversations`, `messages` tables with indexes) +- [x] Bubble Tea TUI with viewport + input bar, Ctrl+C/Ctrl+Q quit +- [x] HTTP API (std lib `net/http`): `GET /health`, `POST /v1/chat`, `POST /v1/conversations` +- [ ] Prompt template is raw headers — model outputs self-correction/next-action meta-text instead of just the response - [ ] No token tracking — context window grows unbounded - [ ] TUI doesn't stream tokens (waits for full completion) +**Schema (`internal/storage/migrations.go`):** + +```sql +CREATE TABLE conversations ( + id TEXT PRIMARY KEY, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')) +); + +CREATE TABLE messages ( + id TEXT PRIMARY KEY, + conversation_id TEXT NOT NULL REFERENCES conversations(id), + role TEXT NOT NULL CHECK(role IN ('user','assistant','system','tool')), + content TEXT NOT NULL, + tool_call TEXT, + token_count INTEGER, + created_at TEXT NOT NULL DEFAULT (datetime('now')) +); + +CREATE INDEX IF NOT EXISTS idx_messages_conversation ON messages(conversation_id); + +CREATE TABLE context_state ( + conversation_id TEXT PRIMARY KEY REFERENCES conversations(id), + token_count INTEGER NOT NULL DEFAULT 0, + summary TEXT +); +``` + --- ## Phase 2 — Prompt Templating & Format Fix -**Goal:** Fix the model output. Swap raw prompt text for a configurable chat template so the model only returns the assistant response. +**Goal:** Fix model output. Swap raw prompt text for a configurable chat template so the model only returns the assistant response, not reasoning meta-text. - [ ] `internal/llama/template.go` — chat template formatter: - - ChatML (`<|im_start|>system...<|im_end|>`) - - Llama 3 instruct (`<|begin_of_text|><|start_header_id|>system<|end_header_id|>`) + - ChatML: `<|im_start|>system\n...<|im_end|>\n<|im_start|>user\n...<|im_end|>\n<|im_start|>assistant\n` + - Llama 3 instruct: `<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n...<|eot_id|>` - Configurable via env `CHAT_TEMPLATE=chatml` (default) -- [ ] Refactor `companion/promptbuilder.go` to output structured segments, then let template.go wrap them -- [ ] Strip system prompt of raw formatting — let the template handle structure -- [ ] Test: "hi" produces just "Hello!" not paragraphs of self-correction +- [ ] Refactor `companion/promptbuilder.go` to output structured segments (system context, personality, relationship, conversation history, user message) +- [ ] Let template.go wrap those segments in the chosen format +- [ ] Strip system prompt of raw formatting headers — let template handle structure +- [ ] Add `stop` tokens per template (`<|im_end|>`, `<|eot_id|>`) +- [ ] **Test:** "hi" produces just "Hello!" not paragraphs of self-correction --- ## Phase 3 — Long-Term Memory (RAG) -**Goal:** Embedding-based retrieval so the agent recalls across sessions. +**Goal:** Embedding-based retrieval so the agent recalls facts across sessions. -- [ ] `internal/memory/embed.go` — call llama.cpp `POST /embedding` to get vectors -- [ ] `internal/memory/store.go` — upgrade `Search()` from keyword `LIKE` to cosine similarity -- [ ] Store embeddings as BLOB in `memories` table -- [ ] Temporal decay — older memories scored lower -- [ ] Importance scoring — "remember this" / emotional content boosts score -- [ ] **Test:** Store info, restart, ask about stored info → recall +- [ ] `internal/memory/embed.go` — call llama.cpp `POST /embedding` to generate vectors +- [ ] SQLite vector storage: store embeddings as BLOB in `memories` table +- [ ] `internal/memory/store.go` — upgrade `Search()` from keyword `LIKE` to cosine similarity scan +- [ ] Temporal decay — older memories scored lower (weight = importance \* recency_factor) +- [ ] Importance scoring — user flags ("remember this"), emotional content, repeat mentions boost score + +```sql +CREATE TABLE memories ( + id TEXT PRIMARY KEY, + content TEXT NOT NULL, + embedding BLOB, -- float32 vector + source TEXT, -- 'conversation', 'reflection', 'user_input' + importance REAL DEFAULT 0.5, -- 0.0–1.0 + created_at TEXT NOT NULL DEFAULT (datetime('now')), + last_accessed TEXT +); + +CREATE INDEX IF NOT EXISTS idx_memories_importance ON memories(importance); +CREATE INDEX IF NOT EXISTS idx_memories_created ON memories(created_at); +``` + +- [ ] **Future:** Migrate to PostgreSQL + pgvector for production-scale vector search +- [ ] **Test:** Store info, restart agent, ask about stored info → verify recall --- ## Phase 4 — OpenAI-Compatible Adapter -**Goal:** Swap inference to any OpenAI-compatible API (OpenAI, vLLM, Ollama, etc.). +**Goal:** Swap inference to any OpenAI-compatible API (OpenAI, vLLM, Ollama, Groq, etc.). - [ ] `internal/llama/provider.go` — interface: - ```go - type Provider interface { - Complete(ctx, *Request) (*Response, error) - Embed(ctx, string) ([]float64, error) - Health() error - } - ``` + +```go +type Provider interface { + Complete(ctx context.Context, req *CompletionRequest) (*CompletionResponse, error) + Embed(ctx context.Context, content string) ([]float64, error) + Health() error +} +``` + - [ ] Rename `internal/llama/` → `internal/inference/` -- [ ] `llama.go` — implements Provider (existing code) -- [ ] `openai.go` — implements Provider via OpenAI-compatible REST API -- [ ] Config: `INFERENCE_PROVIDER=llama|openai`, `OPENAI_API_KEY`, `OPENAI_BASE_URL` -- [ ] **Test:** Both providers produce same interface, switchable via env +- [ ] `llama.go` — implements Provider (existing llama.cpp client wrapped) +- [ ] `openai.go` — implements Provider via OpenAI-compatible REST API (`/v1/chat/completions`, `/v1/embeddings`) +- [ ] Config: `INFERENCE_PROVIDER=llama|openai`, `OPENAI_API_KEY`, `OPENAI_BASE_URL`, `OPENAI_MODEL` +- [ ] Both providers produce same interface, switchable via env var +- [ ] **Test:** Same conversation, swap provider, verify behavior is consistent --- ## Phase 5 — Episodic Diary & Reflection -**Status: CODE DONE** — needs integration into the scheduled pipeline. +**Goal:** Daily diary entries + periodic self-reflection for narrative continuity. The diary is the emotional continuity layer. + +**Status: CODE DONE** — needs LLM integration for real summaries. - [x] `internal/companion/reflection.go` — `GenerateDiary`, `RecentObservations`, `AddObservation` -- [x] `diary_entries` table with date, summary, mood, topics, observations -- [x] `observations` table with confidence, category, applied flag -- [ ] Connect scheduler's diary trigger to actual LLM-generated summaries (currently writes stub text) +- [x] `diary_entries` table: + +```sql +CREATE TABLE diary_entries ( + id TEXT PRIMARY KEY, + date TEXT NOT NULL, -- YYYY-MM-DD + summary TEXT NOT NULL, + mood TEXT, + topics TEXT, -- JSON array + observations TEXT, -- JSON array + created_at TEXT NOT NULL DEFAULT (datetime('now')) +); +``` + +- [x] `observations` table: + +```sql +CREATE TABLE observations ( + id TEXT PRIMARY KEY, + content TEXT NOT NULL, + confidence REAL DEFAULT 0.5, + category TEXT, -- 'topic_interest', 'interaction_style', 'mood_pattern' + applied INTEGER DEFAULT 0, + created_at TEXT NOT NULL DEFAULT (datetime('now')) +); +``` + +- [ ] Connect scheduler's nightly diary trigger to LLM-generated summaries (currently writes stub text) - [ ] Reflection → Personality update pipeline: - - Observations accumulate → periodic analysis → bounded trait drift - - Max 0.5% change/day, max 5%/month + - Diary entries accumulate → periodic analysis → trait drift candidates + - Approval process: observations → candidates → bounded update + - Max 0.5% trait change/day, max 5%/month — slow evolution feels human, fast feels schizophrenic +- [ ] **Test:** Simulate conversations across multiple "days", verify diary entries, verify trait drift remains within bounds --- ## Phase 6 — MCP Tools & Integrations +**Goal:** Extensible tool system for web search, file access, external services. + **Status: PARTIAL** -- [x] `internal/tools/registry.go` — register/list/execute tools -- [x] `internal/tools/builtin.go` — `web_fetch`, `web_search` (stub) -- [ ] MCP protocol client (`internal/tools/mcp.go`) — stdio/TCP, list tools, execute, parse results -- [ ] Tool call loop in agent: LLM decides → agent executes → result appended → LLM finalizes -- [ ] Plugins: Obsidian vault, Gitea, Actual Budget +- [x] `internal/tools/registry.go` — register/list/execute tools with parameter definitions +- [x] `internal/tools/builtin.go` — `web_fetch` (working), `web_search` (stub) +- [ ] MCP protocol client (`internal/tools/mcp.go`): + - Connect to MCP servers via stdio or TCP + - List available tools with parameters + - Execute tool calls, parse results into context +- [ ] Tool call loop in agent: + - LLM decides to call tool → agent executes → result appended to context → LLM generates final response +- [ ] MCP-managed integrations: + - Obsidian vault (via `enquire-mcp` or custom) + - Gitea/Forgejo operations + - Actual Budget (spending tracking, categorization) +- [ ] **Test:** `web_fetch` tool, verify agent can search, summarize, cite sources --- ## Phase 7 — Matrix Integration +**Goal:** Agent lives as a Matrix bot, accessible via DM. Matrix feels like `@diva:homeserver.tld` instead of `Bot #4321`. + **Status: STUB** -- [x] `internal/channel/matrix.go` — config + start stub +- [x] `internal/channel/matrix.go` — config + start stub (disabled by default) - [ ] Full Matrix bot with `mautrix-go`: - Listen for invitations, join rooms - - Handle DMs with typing indicators + - Handle DMs with typing indicators, read receipts + - Message parsing & sending - Bot identity: `@diva:homeserver.tld` -- [ ] Sync TUI and Matrix state -- [ ] **Test:** Send DM, verify response, verify cross-session memory +- [ ] Config: homeserver URL, bot user token in env +- [ ] TUI and Matrix run concurrently — messages sync between both +- [ ] **Test:** Send DM to bot, verify response, verify cross-session memory --- ## Phase 8 — Self-Memory & Identity +**Goal:** Agent remembers its own state — promises, strategies, mistakes. "Bidirectional memory." + **Status: CODE DONE** -- [x] `internal/memory/self_memory.go` — promises, strategies -- [x] `agent_promises` table (id, promise, context, fulfilled, deadline) -- [x] `agent_strategies` table (id, situation, strategy, outcome) -- [ ] Wire into Prompt Composer: "You promised to X" context -- [ ] **Test:** "What did you promise me?" → recall +- [x] `internal/memory/self_memory.go`: + +```go +type Promise struct { + ID string + Promise string + Context string + Fulfilled bool + CreatedAt time.Time + Deadline *time.Time +} + +type Strategy struct { + ID string + Situation string + Strategy string + Outcome string + CreatedAt time.Time +} +``` + +- [x] `agent_promises` table (id, promise, context, fulfilled, created_at, deadline) +- [x] `agent_strategies` table (id, situation, strategy, outcome, created_at) +- [ ] Wire into Prompt Composer: recent promises, relevant strategies added to context +- [ ] **Test:** "What did you promise me yesterday?" → verify recall from self-memory --- ## Phase 9 — Safety Guardrails & Identity Stability -**Goal:** Prevent personality drift, maintain core safety. +**Goal:** Prevent personality drift, maintain core safety rules. -- [ ] `internal/companion/safety.go` — core identity (stable) vs state (dynamic) -- [ ] Personality safety: bounded traits, safety overrides personality, no tool access changes -- [ ] Prompt injection protection: user input wrapped in delimiters, strict template sections +- [ ] `internal/companion/safety.go`: + - Core Identity (never changes): values, communication style, boundaries, role definition + - State (dynamic): mood, interests, current projects, relationship opinions + - Separation: identity config (env/read-only) vs state (SQLite/writable) +- [ ] Personality safety rules: + - Traits bounded within configured min/max at all times + - Core safety rules override any personality state + - Permission policies cannot be modified by personality + - No tool access changes via personality evolution +- [ ] Prompt injection protection: + - User messages wrapped in separator tokens + - System prompt templates with strict, non-overridable sections + - Input sanitization at the agent boundary +- [ ] **Test:** Attempt prompt injection, verify guardrails hold --- ## Phase 10 — Polish: Streaming TUI, Token Tracking, Observability -**Goal:** Production-quality UX. +**Goal:** Production-quality UX and operations. -- [ ] Token streaming in TUI (char-by-char from llama.cpp SSE or chunked response) -- [ ] Context window manager — track tokens, sliding window when full, auto-summarize old messages +- [ ] Token streaming in TUI — char-by-char from llama.cpp SSE or chunked response +- [ ] Context window manager: + - Track total token count per conversation + - Sliding window when context exceeds limit + - Auto-summarize oldest messages to stay within context - [ ] `POST /v1/chat` streaming via SSE in API mode +- [ ] Structured JSON logging via `log/slog` (already done — custom handler with `[LEVEL] [TIME] [FILE:LINE]`) - [ ] Prometheus metrics (`GET /metrics`) -- [ ] Graceful shutdown — drain in-flight requests, save state +- [ ] Graceful shutdown — drain in-flight requests, save agent state +- [ ] **Test:** Full restart cycle, verify all state restored --- ## Phase 11 — Future Enhancements (Optional) -- [ ] Voice: whisper.cpp (STT) + piper/XTTS (TTS) -- [ ] VRM/Live2D avatar rendering -- [ ] Multi-user with per-user relationship tracking -- [ ] Multi-room Matrix -- [ ] Plugin system -- [ ] WebUI dashboard -- [ ] Home Assistant integration +- [ ] Voice interface: whisper.cpp (STT) + piper/XTTS (TTS) +- [ ] VRM/Live2D avatar rendering (like Open-LLM-VTuber, Soul of Waifu) +- [ ] Multi-user support with per-user relationship tracking +- [ ] Multi-room Matrix support +- [ ] Plugin system for community-contributed tools +- [ ] WebUI dashboard (like nanobot's WebUI) +- [ ] Mobile companion via Matrix (any Matrix client works) +- [ ] Home Assistant integration for physical automation +- [ ] PostgreSQL + pgvector migration for production-scale vector search --- -## Database Schema (15 tables across 2 DBs) +## Database Summary -`divacode.db`: +Two separate SQLite databases (clean separation of concerns): -- `conversations`, `messages`, `context_state`, `memories` +| Database | Purpose | Tables | +| -------------- | --------------- | ------------------------------------------------------- | +| `divacode.db` | Core agent data | `conversations`, `messages`, `context_state` | +| `divacode.db` | Memory | `memories` | +| `companion.db` | Personality | `personality_traits` | +| `companion.db` | Relationship | `relationship_metrics`, `shared_topics`, `inside_jokes` | +| `companion.db` | Reflection | `diary_entries`, `observations` | +| `companion.db` | Self-memory | `agent_promises`, `agent_strategies` | -`companion.db`: - -- `personality_traits`, `relationship_metrics`, `shared_topics`, `inside_jokes` -- `diary_entries`, `observations` -- `agent_promises`, `agent_strategies` +**Future:** `divacode.db` may be replaced by PostgreSQL + pgvector for production deployments with many users or large memory stores. --- ## Key Principles -1. **Smallest loop first** — get message → memory → llm → response working before adding complexity -2. **Personality is data, not prompt text** — traits in SQLite, composed dynamically -3. **Memory ≠ Personality** — factual memory separate from relationship/emotional state -4. **Slow evolution** — max 0.5% trait change/day, 5%/month -5. **Core Identity stable** — never allow personality to override safety rules -6. **Diary before reflection** — nightly diary is the emotional continuity layer -7. **Autonomous messaging** — scheduler makes it feel alive, not the LLM -8. **No config libs** — std lib `os.Getenv` only -9. **No HTTP routers** — std lib `net/http` only -10. **Error wrapping** — `fmt.Errorf("context: %w", err)`, never discard +1. **Build the smallest loop first** — get message → memory → LLM → response → diary → reflection working before adding complexity +2. **Personality is data, not prompt text** — traits stored in SQLite, composed dynamically by the prompt builder +3. **Memory ≠ Personality** — factual memory (Engram-style RAG) is separate from relationship/emotional state +4. **Slow evolution** — max 0.5% trait change/day, max 5%/month. Fast evolution feels schizophrenic, slow feels human +5. **Core Identity is stable** — identity.yaml (or env config) never changes much; state.json (SQLite) changes constantly +6. **Diary before reflection** — the nightly diary is the emotional continuity layer, more important than vector search +7. **Autonomous messaging** — the scheduler is what makes the character feel alive, not the LLM intelligence +8. **No secrets in code** — all config via env vars, `.env` in `.gitignore` +9. **No config libraries** — std lib `os.Getenv` only, never `caarlos0/env` or similar +10. **No HTTP routers** — std lib `net/http` only, no third-party routers (chi, gorilla, etc.) +11. **Error handling** — wrap errors with `fmt.Errorf("context: %w", err)`, never silent discards +12. **Test as you go** — `go test -race ./...` before every push +13. **Output readable code** — explain _why_, never _what_. No comments for self-documenting code