docs: restore detail in plan.md — schema SQL, table summary, principles, pgvector future, enriched phase descriptions

2026-06-10 01:08:08 +03:30
parent cd37d01cae
commit 9903c7b12b
1 changed files with 239 additions and 86 deletions
--- a/plan.md
+++ b/plan.md
@@ -2,6 +2,7 @@

 **Language:** Go  
 **Inference:** llama.cpp (HTTP server), OpenAI-compatible API planned  
+**Storage:** SQLite (primary), PostgreSQL + pgvector (future)  
 **Inspiration:** OpenHer, Agent Diva, nanobot

 ---
@@ -10,6 +11,7 @@

 **Builds:** yes  
 **Runs:** yes (requires llama.cpp on :8080)
+
 **Known issues:**

 - Prompt template is raw text — model dumps reasoning/self-correction meta-text instead of just the response
@@ -32,6 +34,9 @@ User
  ┌─────┴──────────────────────────────┐
  │        Core Agent Loop             │
  │  (internal/agent/)                 │
+  │  - Message routing                 │
+  │  - Tool invocation                 │
+  │  - Context management              │
  └─────┬──────────────────────────────┘
        │
  ┌─────┴──────────────────────────────┐
@@ -47,7 +52,9 @@ User
  ┌─────┴──────────────────────────────┐
  │         Memory Layer               │
  │  (internal/memory/)                │
-  │  ├─ RAG store (keywords → vectors) │
+  │  ├─ Short-term (ephemeral context) │
+  │  ├─ Long-term (SQLite → pgvector)  │
+  │  ├─ Episodic / Diary (sqlite)      │
  │  └─ Self-memory (promises,strats)  │
  └─────┬──────────────────────────────┘
        │
@@ -61,6 +68,10 @@ Skills / Tools:
  ├─ web_fetch
  ├─ web_search (stub)
  └─ MCP protocol (future)
+       ├─ Obsidian vault
+       ├─ Gitea
+       ├─ Actual Budget
+       └─ Custom MCP tools
 ```

 ---
@@ -76,6 +87,8 @@ Skills / Tools:
 - [x] `internal/llama/client.go` — HTTP client for `POST /completion`, `POST /embedding`, `GET /health`
 - [x] `.env.example`, `Makefile`, `.gitignore`, `Dockerfile`, `docker-compose.yml`
 - [x] `AGENTS.md` with architecture, commands, conventions
+- [x] `pkg/types/types.go` — Message, Role, Mood, ToolCall types
+- [x] `pkg/personality/traits.go` — TraitValue, RelationshipMetrics, DiaryEntry, Observation types

 ---

@@ -83,173 +96,313 @@ Skills / Tools:

 **Status: COMPLETE** (minor issues remain)

- [x] `internal/agent/agent.go` — message → memory search → prompt → llama → save → respond
- [x] `internal/agent/scheduler.go` — tick every 15m, check mood/diary/auto-message
- [x] `cmd/divad/main.go` — wires everything: config, DB, llama, TUI, scheduler, Matrix, API
- [x] Conversation persistence to SQLite (`conversations`, `messages` tables)
- [x] Bubble Tea TUI with viewport + input bar
- [ ] Prompt template is wrong — uses raw headers, no chat format. Model outputs meta-text
+- [x] `internal/agent/agent.go` — message loop: receive → memory search → prompt → llama → save → respond
+- [x] `internal/agent/scheduler.go` — background goroutine, configurable tick (default 15m), mood check, diary trigger, autonomous message queue
+- [x] `cmd/divad/main.go` — wires everything: config, DB, llama health, TUI, scheduler, Matrix, API
+- [x] Conversation persistence to SQLite (`conversations`, `messages` tables with indexes)
+- [x] Bubble Tea TUI with viewport + input bar, Ctrl+C/Ctrl+Q quit
+- [x] HTTP API (std lib `net/http`): `GET /health`, `POST /v1/chat`, `POST /v1/conversations`
+- [ ] Prompt template is raw headers — model outputs self-correction/next-action meta-text instead of just the response
 - [ ] No token tracking — context window grows unbounded
 - [ ] TUI doesn't stream tokens (waits for full completion)

+**Schema (`internal/storage/migrations.go`):**
+
+```sql
+CREATE TABLE conversations (
+    id          TEXT PRIMARY KEY,
+    created_at  TEXT NOT NULL DEFAULT (datetime('now')),
+    updated_at  TEXT NOT NULL DEFAULT (datetime('now'))
+);
+
+CREATE TABLE messages (
+    id              TEXT PRIMARY KEY,
+    conversation_id TEXT NOT NULL REFERENCES conversations(id),
+    role            TEXT NOT NULL CHECK(role IN ('user','assistant','system','tool')),
+    content         TEXT NOT NULL,
+    tool_call       TEXT,
+    token_count     INTEGER,
+    created_at      TEXT NOT NULL DEFAULT (datetime('now'))
+);
+
+CREATE INDEX IF NOT EXISTS idx_messages_conversation ON messages(conversation_id);
+
+CREATE TABLE context_state (
+    conversation_id TEXT PRIMARY KEY REFERENCES conversations(id),
+    token_count     INTEGER NOT NULL DEFAULT 0,
+    summary         TEXT
+);
+```
+
 ---

 ## Phase 2 — Prompt Templating & Format Fix

-**Goal:** Fix the model output. Swap raw prompt text for a configurable chat template so the model only returns the assistant response.
+**Goal:** Fix model output. Swap raw prompt text for a configurable chat template so the model only returns the assistant response, not reasoning meta-text.

 - [ ] `internal/llama/template.go` — chat template formatter:
-  - ChatML (`<|im_start|>system...<|im_end|>`)
-  - Llama 3 instruct (`<|begin_of_text|><|start_header_id|>system<|end_header_id|>`)
+  - ChatML: `<|im_start|>system\n...<|im_end|>\n<|im_start|>user\n...<|im_end|>\n<|im_start|>assistant\n`
+  - Llama 3 instruct: `<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n...<|eot_id|>`
  - Configurable via env `CHAT_TEMPLATE=chatml` (default)
- [ ] Refactor `companion/promptbuilder.go` to output structured segments, then let template.go wrap them
- [ ] Strip system prompt of raw formatting — let the template handle structure
- [ ] Test: "hi" produces just "Hello!" not paragraphs of self-correction
+- [ ] Refactor `companion/promptbuilder.go` to output structured segments (system context, personality, relationship, conversation history, user message)
+- [ ] Let template.go wrap those segments in the chosen format
+- [ ] Strip system prompt of raw formatting headers — let template handle structure
+- [ ] Add `stop` tokens per template (`<|im_end|>`, `<|eot_id|>`)
+- [ ] **Test:** "hi" produces just "Hello!" not paragraphs of self-correction

 ---

 ## Phase 3 — Long-Term Memory (RAG)

-**Goal:** Embedding-based retrieval so the agent recalls across sessions.
+**Goal:** Embedding-based retrieval so the agent recalls facts across sessions.

- [ ] `internal/memory/embed.go` — call llama.cpp `POST /embedding` to get vectors
- [ ] `internal/memory/store.go` — upgrade `Search()` from keyword `LIKE` to cosine similarity
- [ ] Store embeddings as BLOB in `memories` table
- [ ] Temporal decay — older memories scored lower
- [ ] Importance scoring — "remember this" / emotional content boosts score
- [ ] **Test:** Store info, restart, ask about stored info → recall
+- [ ] `internal/memory/embed.go` — call llama.cpp `POST /embedding` to generate vectors
+- [ ] SQLite vector storage: store embeddings as BLOB in `memories` table
+- [ ] `internal/memory/store.go` — upgrade `Search()` from keyword `LIKE` to cosine similarity scan
+- [ ] Temporal decay — older memories scored lower (weight = importance \* recency_factor)
+- [ ] Importance scoring — user flags ("remember this"), emotional content, repeat mentions boost score
+
+```sql
+CREATE TABLE memories (
+    id            TEXT PRIMARY KEY,
+    content       TEXT NOT NULL,
+    embedding     BLOB,              -- float32 vector
+    source        TEXT,              -- 'conversation', 'reflection', 'user_input'
+    importance    REAL DEFAULT 0.5,  -- 0.0–1.0
+    created_at    TEXT NOT NULL DEFAULT (datetime('now')),
+    last_accessed TEXT
+);
+
+CREATE INDEX IF NOT EXISTS idx_memories_importance ON memories(importance);
+CREATE INDEX IF NOT EXISTS idx_memories_created ON memories(created_at);
+```
+
+- [ ] **Future:** Migrate to PostgreSQL + pgvector for production-scale vector search
+- [ ] **Test:** Store info, restart agent, ask about stored info → verify recall

 ---

 ## Phase 4 — OpenAI-Compatible Adapter

-**Goal:** Swap inference to any OpenAI-compatible API (OpenAI, vLLM, Ollama, etc.).
+**Goal:** Swap inference to any OpenAI-compatible API (OpenAI, vLLM, Ollama, Groq, etc.).

 - [ ] `internal/llama/provider.go` — interface:
+
 ```go
 type Provider interface {
-      Complete(ctx, *Request) (*Response, error)
-      Embed(ctx, string) ([]float64, error)
+    Complete(ctx context.Context, req *CompletionRequest) (*CompletionResponse, error)
+    Embed(ctx context.Context, content string) ([]float64, error)
    Health() error
 }
 ```
+
 - [ ] Rename `internal/llama/` → `internal/inference/`
- [ ] `llama.go` — implements Provider (existing code)
- [ ] `openai.go` — implements Provider via OpenAI-compatible REST API
- [ ] Config: `INFERENCE_PROVIDER=llama|openai`, `OPENAI_API_KEY`, `OPENAI_BASE_URL`
- [ ] **Test:** Both providers produce same interface, switchable via env
+- [ ] `llama.go` — implements Provider (existing llama.cpp client wrapped)
+- [ ] `openai.go` — implements Provider via OpenAI-compatible REST API (`/v1/chat/completions`, `/v1/embeddings`)
+- [ ] Config: `INFERENCE_PROVIDER=llama|openai`, `OPENAI_API_KEY`, `OPENAI_BASE_URL`, `OPENAI_MODEL`
+- [ ] Both providers produce same interface, switchable via env var
+- [ ] **Test:** Same conversation, swap provider, verify behavior is consistent

 ---

 ## Phase 5 — Episodic Diary & Reflection

-**Status: CODE DONE** — needs integration into the scheduled pipeline.
+**Goal:** Daily diary entries + periodic self-reflection for narrative continuity. The diary is the emotional continuity layer.
+
+**Status: CODE DONE** — needs LLM integration for real summaries.

 - [x] `internal/companion/reflection.go` — `GenerateDiary`, `RecentObservations`, `AddObservation`
- [x] `diary_entries` table with date, summary, mood, topics, observations
- [x] `observations` table with confidence, category, applied flag
- [ ] Connect scheduler's diary trigger to actual LLM-generated summaries (currently writes stub text)
+- [x] `diary_entries` table:
+
+```sql
+CREATE TABLE diary_entries (
+    id            TEXT PRIMARY KEY,
+    date          TEXT NOT NULL,         -- YYYY-MM-DD
+    summary       TEXT NOT NULL,
+    mood          TEXT,
+    topics        TEXT,                  -- JSON array
+    observations  TEXT,                  -- JSON array
+    created_at    TEXT NOT NULL DEFAULT (datetime('now'))
+);
+```
+
+- [x] `observations` table:
+
+```sql
+CREATE TABLE observations (
+    id          TEXT PRIMARY KEY,
+    content     TEXT NOT NULL,
+    confidence  REAL DEFAULT 0.5,
+    category    TEXT,              -- 'topic_interest', 'interaction_style', 'mood_pattern'
+    applied     INTEGER DEFAULT 0,
+    created_at  TEXT NOT NULL DEFAULT (datetime('now'))
+);
+```
+
+- [ ] Connect scheduler's nightly diary trigger to LLM-generated summaries (currently writes stub text)
 - [ ] Reflection → Personality update pipeline:
-  - Observations accumulate → periodic analysis → bounded trait drift
-  - Max 0.5% change/day, max 5%/month
+  - Diary entries accumulate → periodic analysis → trait drift candidates
+  - Approval process: observations → candidates → bounded update
+  - Max 0.5% trait change/day, max 5%/month — slow evolution feels human, fast feels schizophrenic
+- [ ] **Test:** Simulate conversations across multiple "days", verify diary entries, verify trait drift remains within bounds

 ---

 ## Phase 6 — MCP Tools & Integrations

+**Goal:** Extensible tool system for web search, file access, external services.
+
 **Status: PARTIAL**

- [x] `internal/tools/registry.go` — register/list/execute tools
- [x] `internal/tools/builtin.go` — `web_fetch`, `web_search` (stub)
- [ ] MCP protocol client (`internal/tools/mcp.go`) — stdio/TCP, list tools, execute, parse results
- [ ] Tool call loop in agent: LLM decides → agent executes → result appended → LLM finalizes
- [ ] Plugins: Obsidian vault, Gitea, Actual Budget
+- [x] `internal/tools/registry.go` — register/list/execute tools with parameter definitions
+- [x] `internal/tools/builtin.go` — `web_fetch` (working), `web_search` (stub)
+- [ ] MCP protocol client (`internal/tools/mcp.go`):
+  - Connect to MCP servers via stdio or TCP
+  - List available tools with parameters
+  - Execute tool calls, parse results into context
+- [ ] Tool call loop in agent:
+  - LLM decides to call tool → agent executes → result appended to context → LLM generates final response
+- [ ] MCP-managed integrations:
+  - Obsidian vault (via `enquire-mcp` or custom)
+  - Gitea/Forgejo operations
+  - Actual Budget (spending tracking, categorization)
+- [ ] **Test:** `web_fetch` tool, verify agent can search, summarize, cite sources

 ---

 ## Phase 7 — Matrix Integration

+**Goal:** Agent lives as a Matrix bot, accessible via DM. Matrix feels like `@diva:homeserver.tld` instead of `Bot #4321`.
+
 **Status: STUB**

- [x] `internal/channel/matrix.go` — config + start stub
+- [x] `internal/channel/matrix.go` — config + start stub (disabled by default)
 - [ ] Full Matrix bot with `mautrix-go`:
  - Listen for invitations, join rooms
-  - Handle DMs with typing indicators
+  - Handle DMs with typing indicators, read receipts
+  - Message parsing & sending
  - Bot identity: `@diva:homeserver.tld`
- [ ] Sync TUI and Matrix state
- [ ] **Test:** Send DM, verify response, verify cross-session memory
+- [ ] Config: homeserver URL, bot user token in env
+- [ ] TUI and Matrix run concurrently — messages sync between both
+- [ ] **Test:** Send DM to bot, verify response, verify cross-session memory

 ---

 ## Phase 8 — Self-Memory & Identity

+**Goal:** Agent remembers its own state — promises, strategies, mistakes. "Bidirectional memory."
+
 **Status: CODE DONE**

- [x] `internal/memory/self_memory.go` — promises, strategies
- [x] `agent_promises` table (id, promise, context, fulfilled, deadline)
- [x] `agent_strategies` table (id, situation, strategy, outcome)
- [ ] Wire into Prompt Composer: "You promised to X" context
- [ ] **Test:** "What did you promise me?" → recall
+- [x] `internal/memory/self_memory.go`:
+
+```go
+type Promise struct {
+    ID        string
+    Promise   string
+    Context   string
+    Fulfilled bool
+    CreatedAt time.Time
+    Deadline  *time.Time
+}
+
+type Strategy struct {
+    ID        string
+    Situation string
+    Strategy  string
+    Outcome   string
+    CreatedAt time.Time
+}
+```
+
+- [x] `agent_promises` table (id, promise, context, fulfilled, created_at, deadline)
+- [x] `agent_strategies` table (id, situation, strategy, outcome, created_at)
+- [ ] Wire into Prompt Composer: recent promises, relevant strategies added to context
+- [ ] **Test:** "What did you promise me yesterday?" → verify recall from self-memory

 ---

 ## Phase 9 — Safety Guardrails & Identity Stability

-**Goal:** Prevent personality drift, maintain core safety.
+**Goal:** Prevent personality drift, maintain core safety rules.

- [ ] `internal/companion/safety.go` — core identity (stable) vs state (dynamic)
- [ ] Personality safety: bounded traits, safety overrides personality, no tool access changes
- [ ] Prompt injection protection: user input wrapped in delimiters, strict template sections
+- [ ] `internal/companion/safety.go`:
+  - Core Identity (never changes): values, communication style, boundaries, role definition
+  - State (dynamic): mood, interests, current projects, relationship opinions
+  - Separation: identity config (env/read-only) vs state (SQLite/writable)
+- [ ] Personality safety rules:
+  - Traits bounded within configured min/max at all times
+  - Core safety rules override any personality state
+  - Permission policies cannot be modified by personality
+  - No tool access changes via personality evolution
+- [ ] Prompt injection protection:
+  - User messages wrapped in separator tokens
+  - System prompt templates with strict, non-overridable sections
+  - Input sanitization at the agent boundary
+- [ ] **Test:** Attempt prompt injection, verify guardrails hold

 ---

 ## Phase 10 — Polish: Streaming TUI, Token Tracking, Observability

-**Goal:** Production-quality UX.
+**Goal:** Production-quality UX and operations.

- [ ] Token streaming in TUI (char-by-char from llama.cpp SSE or chunked response)
- [ ] Context window manager — track tokens, sliding window when full, auto-summarize old messages
+- [ ] Token streaming in TUI — char-by-char from llama.cpp SSE or chunked response
+- [ ] Context window manager:
+  - Track total token count per conversation
+  - Sliding window when context exceeds limit
+  - Auto-summarize oldest messages to stay within context
 - [ ] `POST /v1/chat` streaming via SSE in API mode
+- [ ] Structured JSON logging via `log/slog` (already done — custom handler with `[LEVEL] [TIME] [FILE:LINE]`)
 - [ ] Prometheus metrics (`GET /metrics`)
- [ ] Graceful shutdown — drain in-flight requests, save state
+- [ ] Graceful shutdown — drain in-flight requests, save agent state
+- [ ] **Test:** Full restart cycle, verify all state restored

 ---

 ## Phase 11 — Future Enhancements (Optional)

- [ ] Voice: whisper.cpp (STT) + piper/XTTS (TTS)
- [ ] VRM/Live2D avatar rendering
- [ ] Multi-user with per-user relationship tracking
- [ ] Multi-room Matrix
- [ ] Plugin system
- [ ] WebUI dashboard
- [ ] Home Assistant integration
+- [ ] Voice interface: whisper.cpp (STT) + piper/XTTS (TTS)
+- [ ] VRM/Live2D avatar rendering (like Open-LLM-VTuber, Soul of Waifu)
+- [ ] Multi-user support with per-user relationship tracking
+- [ ] Multi-room Matrix support
+- [ ] Plugin system for community-contributed tools
+- [ ] WebUI dashboard (like nanobot's WebUI)
+- [ ] Mobile companion via Matrix (any Matrix client works)
+- [ ] Home Assistant integration for physical automation
+- [ ] PostgreSQL + pgvector migration for production-scale vector search

 ---

-## Database Schema (15 tables across 2 DBs)
+## Database Summary

-`divacode.db`:
+Two separate SQLite databases (clean separation of concerns):

- `conversations`, `messages`, `context_state`, `memories`
+| Database       | Purpose         | Tables                                                  |
+| -------------- | --------------- | ------------------------------------------------------- |
+| `divacode.db`  | Core agent data | `conversations`, `messages`, `context_state`            |
+| `divacode.db`  | Memory          | `memories`                                              |
+| `companion.db` | Personality     | `personality_traits`                                    |
+| `companion.db` | Relationship    | `relationship_metrics`, `shared_topics`, `inside_jokes` |
+| `companion.db` | Reflection      | `diary_entries`, `observations`                         |
+| `companion.db` | Self-memory     | `agent_promises`, `agent_strategies`                    |

-`companion.db`:
-
- `personality_traits`, `relationship_metrics`, `shared_topics`, `inside_jokes`
- `diary_entries`, `observations`
- `agent_promises`, `agent_strategies`
+**Future:** `divacode.db` may be replaced by PostgreSQL + pgvector for production deployments with many users or large memory stores.

 ---

 ## Key Principles

-1. **Smallest loop first** — get message → memory → llm → response working before adding complexity
-2. **Personality is data, not prompt text** — traits in SQLite, composed dynamically
-3. **Memory ≠ Personality** — factual memory separate from relationship/emotional state
-4. **Slow evolution** — max 0.5% trait change/day, 5%/month
-5. **Core Identity stable** — never allow personality to override safety rules
-6. **Diary before reflection** — nightly diary is the emotional continuity layer
-7. **Autonomous messaging** — scheduler makes it feel alive, not the LLM
-8. **No config libs** — std lib `os.Getenv` only
-9. **No HTTP routers** — std lib `net/http` only
-10. **Error wrapping** — `fmt.Errorf("context: %w", err)`, never discard
+1. **Build the smallest loop first** — get message → memory → LLM → response → diary → reflection working before adding complexity
+2. **Personality is data, not prompt text** — traits stored in SQLite, composed dynamically by the prompt builder
+3. **Memory ≠ Personality** — factual memory (Engram-style RAG) is separate from relationship/emotional state
+4. **Slow evolution** — max 0.5% trait change/day, max 5%/month. Fast evolution feels schizophrenic, slow feels human
+5. **Core Identity is stable** — identity.yaml (or env config) never changes much; state.json (SQLite) changes constantly
+6. **Diary before reflection** — the nightly diary is the emotional continuity layer, more important than vector search
+7. **Autonomous messaging** — the scheduler is what makes the character feel alive, not the LLM intelligence
+8. **No secrets in code** — all config via env vars, `.env` in `.gitignore`
+9. **No config libraries** — std lib `os.Getenv` only, never `caarlos0/env` or similar
+10. **No HTTP routers** — std lib `net/http` only, no third-party routers (chi, gorilla, etc.)
+11. **Error handling** — wrap errors with `fmt.Errorf("context: %w", err)`, never silent discards
+12. **Test as you go** — `go test -race ./...` before every push
+13. **Output readable code** — explain _why_, never _what_. No comments for self-documenting code