n8r/nixos

Fork 0

Nate Anderson 02f7a3b960 Add memory and revamp skills plugin

2025-10-29 18:46:16 -06:00

23 KiB

Raw Blame History

LLMemory Architecture

System Overview

LLMemory is a three-layer system:

CLI Layer - User/agent interface (Commander.js)
Search Layer - Query processing and ranking (LIKE → FTS5 → Fuzzy)
Storage Layer - Persistent data (SQLite)

┌─────────────────────────────────────┐
│         CLI Layer                   │
│  (memory store/search/list/prune)   │
└──────────────┬──────────────────────┘
               │
┌──────────────▼──────────────────────┐
│         Search Layer                │
│  Phase 1: LIKE search               │
│  Phase 2: FTS5 + BM25 ranking       │
│  Phase 3: + Trigram fuzzy matching  │
└──────────────┬──────────────────────┘
               │
┌──────────────▼──────────────────────┐
│         Storage Layer               │
│  SQLite Database                    │
│  - memories (content, metadata)     │
│  - tags (normalized)                │
│  - memory_tags (many-to-many)       │
│  - memories_fts (FTS5 virtual)      │
│  - trigrams (fuzzy index)           │
└─────────────────────────────────────┘

Data Model

Phase 1 Schema (MVP)

┌─────────────────┐
│    memories     │
├─────────────────┤
│ id              │ PK
│ content         │ TEXT (max 10KB)
│ created_at      │ INTEGER (Unix timestamp)
│ entered_by      │ TEXT (agent name)
│ expires_at      │ INTEGER (nullable)
└─────────────────┘
         │
         │ 1:N
         ▼
┌─────────────────┐       ┌─────────────────┐
│  memory_tags    │ N:M   │      tags       │
├─────────────────┤       ├─────────────────┤
│ memory_id       │ FK ───│ id              │ PK
│ tag_id          │ FK    │ name            │ TEXT (unique, NOCASE)
└─────────────────┘       │ created_at      │ INTEGER
                          └─────────────────┘

Phase 2 Schema (+ FTS5)

Adds virtual table for full-text search:

┌─────────────────────┐
│   memories_fts      │ Virtual Table (FTS5)
├─────────────────────┤
│ rowid → memories.id │
│ content (indexed)   │
└─────────────────────┘
         │
         │ Synced via triggers
         ▼
┌─────────────────┐
│    memories     │
└─────────────────┘

Triggers:

memories_ai: INSERT into memories → INSERT into memories_fts
memories_au: UPDATE memories → UPDATE memories_fts
memories_ad: DELETE memories → DELETE from memories_fts

Phase 3 Schema (+ Trigrams)

Adds trigram index for fuzzy matching:

┌─────────────────┐
│    trigrams     │
├─────────────────┤
│ trigram         │ TEXT (3 chars)
│ memory_id       │ FK → memories.id
│ position        │ INTEGER (for proximity)
└─────────────────┘
         │
         │ Generated on insert/update
         ▼
┌─────────────────┐
│    memories     │
└─────────────────┘

Search Algorithm Evolution

Phase 1: LIKE Search

Algorithm:

function search_like(query, filters):
  # Case-insensitive wildcard matching
  sql = "SELECT * FROM memories WHERE LOWER(content) LIKE LOWER('%' || ? || '%')"
  
  # Apply filters
  if filters.tags:
    sql += " AND memory_id IN (SELECT memory_id FROM memory_tags WHERE tag_id IN (...))"
  
  if filters.after:
    sql += " AND created_at >= ?"
  
  # Exclude expired
  sql += " AND (expires_at IS NULL OR expires_at > now())"
  
  # Order by recency
  sql += " ORDER BY created_at DESC LIMIT ?"
  
  return execute(sql, params)

Strengths:

Simple, fast for small datasets
No dependencies
Predictable behavior

Weaknesses:

No relevance ranking
Slow for large datasets (full table scan)
No fuzzy matching
No phrase queries or boolean logic

Performance: O(n) where n = number of memories
Target: <50ms for <500 memories

Phase 2: FTS5 Search

Algorithm:

function search_fts5(query, filters):
  # Build FTS5 query
  fts_query = build_fts5_query(query)  # Handles AND/OR/NOT, quotes, prefixes
  
  # FTS5 MATCH with BM25 ranking
  sql = """
    SELECT m.*, mf.rank as relevance
    FROM memories_fts mf
    JOIN memories m ON m.id = mf.rowid
    WHERE memories_fts MATCH ?
      AND (m.expires_at IS NULL OR m.expires_at > now())
  """
  
  # Apply filters (same as Phase 1)
  # ...
  
  # Order by FTS5 rank (BM25 algorithm)
  sql += " ORDER BY mf.rank LIMIT ?"
  
  return execute(sql, params)

function build_fts5_query(query):
  # Transform grep-like to FTS5
  # "docker compose" → "docker AND compose"
  # "docker OR podman" → "docker OR podman" (unchanged)
  # '"exact phrase"' → '"exact phrase"' (unchanged)
  # "docker*" → "docker*" (unchanged)
  
  if has_operators(query):
    return query
  
  # Implicit AND
  terms = query.split()
  return " AND ".join(terms)

FTS5 Tokenization:

Tokenizer: porter unicode61 remove_diacritics 2
Porter: Stemming (running → run, databases → database)
unicode61: Unicode support
remove_diacritics: Normalize accented characters (café → cafe)

BM25 Ranking:

score = Σ(IDF(term) * (f(term) * (k1 + 1)) / (f(term) + k1 * (1 - b + b * |D| / avgdl)))

Where:
- IDF(term) = Inverse Document Frequency (rarer terms score higher)
- f(term) = Term frequency in document
- |D| = Document length
- avgdl = Average document length
- k1 = 1.2 (term frequency saturation)
- b = 0.75 (length normalization)

Strengths:

Fast search with inverted index
Relevance ranking (BM25)
Boolean operators, phrase queries, prefix matching
Scales to 100K+ documents

Weaknesses:

No fuzzy matching (typo tolerance)
FTS5 index overhead (~30% storage)
More complex setup (triggers needed)

Performance: O(log n) for index lookup
Target: <100ms for 10K memories

Phase 3: Fuzzy Search

Algorithm:

function search_fuzzy(query, filters):
  # Step 1: Try FTS5 exact match
  results = search_fts5(query, filters)
  
  # Step 2: If too few results, try fuzzy
  if len(results) < 5 and filters.fuzzy != false:
    fuzzy_results = search_trigram(query, filters)
    results = merge_dedup(results, fuzzy_results)
  
  # Step 3: Re-rank by combined score
  for result in results:
    result.score = calculate_combined_score(result, query)
  
  results.sort(by=lambda r: r.score, reverse=True)
  return results[:filters.limit]

function search_trigram(query, threshold=0.7, limit=10):
  # Extract query trigrams
  query_trigrams = extract_trigrams(query)  # ["doc", "ock", "cke", "ker"]
  
  # Find candidates by trigram overlap
  sql = """
    SELECT m.id, m.content, COUNT(DISTINCT tr.trigram) as matches
    FROM memories m
    JOIN trigrams tr ON tr.memory_id = m.id
    WHERE tr.trigram IN (?, ?, ?, ...)
      AND (m.expires_at IS NULL OR m.expires_at > now())
    GROUP BY m.id
    HAVING matches >= ?
    ORDER BY matches DESC
    LIMIT ?
  """
  
  min_matches = ceil(len(query_trigrams) * threshold)
  candidates = execute(sql, query_trigrams, min_matches, limit * 2)
  
  # Calculate edit distance and combined score
  scored = []
  for candidate in candidates:
    edit_dist = levenshtein(query, candidate.content[:len(query)*3])
    trigram_sim = candidate.matches / len(query_trigrams)
    normalized_edit = 1 - (edit_dist / max(len(query), len(candidate.content)))
    
    score = 0.6 * trigram_sim + 0.4 * normalized_edit
    
    if score >= threshold:
      scored.append((candidate, score))
  
  scored.sort(by=lambda x: x[1], reverse=True)
  return [c for c, s in scored[:limit]]

function extract_trigrams(text):
  # Normalize: lowercase, remove punctuation, collapse whitespace
  normalized = text.lower().replace(/[^\w\s]/g, ' ').replace(/\s+/g, ' ').trim()
  
  # Add padding for boundary matching
  padded = "  " + normalized + "  "
  
  # Sliding window of 3 characters
  trigrams = []
  for i in range(len(padded) - 2):
    trigram = padded[i:i+3]
    if trigram.strip().len() == 3:  # Skip whitespace-only
      trigrams.append(trigram)
  
  return unique(trigrams)

function levenshtein(a, b):
  # Wagner-Fischer algorithm with single-row optimization
  if len(a) == 0: return len(b)
  if len(b) == 0: return len(a)
  
  prev_row = [0..len(b)]
  
  for i in range(len(a)):
    cur_row = [i + 1]
    for j in range(len(b)):
      cost = 0 if a[i] == b[j] else 1
      cur_row.append(min(
        cur_row[j] + 1,        # deletion
        prev_row[j + 1] + 1,   # insertion
        prev_row[j] + cost     # substitution
      ))
    prev_row = cur_row
  
  return prev_row[len(b)]

function calculate_combined_score(result, query):
  # BM25 from FTS5 (if available)
  bm25_score = result.fts_rank if result.has_fts_rank else 0
  
  # Trigram similarity
  trigram_score = result.trigram_matches / len(extract_trigrams(query))
  
  # Edit distance (normalized)
  edit_dist = levenshtein(query, result.content[:len(query)*3])
  edit_score = 1 - (edit_dist / max(len(query), len(result.content)))
  
  # Recency boost (exponential decay over 90 days)
  days_ago = (now() - result.created_at) / 86400
  recency_score = max(0, 1 - (days_ago / 90))
  
  # Weighted combination
  score = (0.4 * bm25_score +
           0.3 * trigram_score +
           0.2 * edit_score +
           0.1 * recency_score)
  
  return score

Trigram Similarity (Jaccard Index):

similarity = |trigrams(query) ∩ trigrams(document)| / |trigrams(query)|

Example:
query = "docker"     → trigrams: ["doc", "ock", "cke", "ker"]
document = "dcoker"  → trigrams: ["dco", "cok", "oke", "ker"]

intersection = ["ker"] → count = 1
similarity = 1 / 4 = 0.25 (below threshold, but edit distance is 2)

Better approach: Edit distance normalized by length
edit_distance("docker", "dcoker") = 2
normalized = 1 - (2 / 6) = 0.67 (above threshold 0.6)

Strengths:

Handles typos (edit distance ≤2)
Partial matches ("docker" finds "dockerization")
Cascading strategy (fast exact, fallback to fuzzy)
Configurable threshold

Weaknesses:

Trigram table is large (~3x content size)
Slower than FTS5 alone
Tuning threshold requires experimentation

Performance: O(log n) + O(m) where m = trigram candidates
Target: <200ms for 10K memories with fuzzy

Memory Lifecycle

┌──────────┐
│  Store   │
└────┬─────┘
     │
     ▼
┌────────────────────┐
│  Validate:         │
│  - Length (<10KB)  │
│  - Tags (parse)    │
│  - Expiration      │
└────┬───────────────┘
     │
     ▼
┌────────────────────┐     ┌─────────────────┐
│  Insert:           │────▶│  Trigger:       │
│  - memories table  │     │  - Insert FTS5  │
│  - Link tags       │     │  - Gen trigrams │
└────┬───────────────┘     └─────────────────┘
     │
     ▼
┌────────────────────┐
│  Searchable        │
└────────────────────┘
     │
     │ (time passes)
     ▼
┌────────────────────┐
│  Expired?          │───No──▶ Continue
└────┬───────────────┘
     │ Yes
     ▼
┌────────────────────┐
│  Prune Command     │
│  (manual/auto)     │
└────┬───────────────┘
     │
     ▼
┌────────────────────┐     ┌─────────────────┐
│  Delete:           │────▶│  Trigger:       │
│  - memories table  │     │  - Delete FTS5  │
│  - CASCADE tags    │     │  - Delete tris  │
└────────────────────┘     └─────────────────┘

Query Processing Flow

Phase 1 (LIKE)

User Query: "docker networking"
     │
     ▼
Parse Query: Extract terms, filters
     │
     ▼
Build SQL: LIKE '%docker%' AND LIKE '%networking%'
     │
     ▼
Apply Filters: Tags, dates, agent
     │
     ▼
Execute: Sequential scan through memories
     │
     ▼
Order: By created_at DESC
     │
     ▼
Limit: Take top N results
     │
     ▼
Format: Plain text / JSON / Markdown

Phase 2 (FTS5)

User Query: "docker AND networking"
     │
     ▼
Parse Query: Identify operators, quotes, prefixes
     │
     ▼
Build FTS5 Query: "docker AND networking" (already valid)
     │
     ▼
FTS5 MATCH: Inverted index lookup
     │
     ▼
BM25 Ranking: Calculate relevance scores
     │
     ▼
Apply Filters: Tags, dates, agent (on results)
     │
     ▼
Order: By rank (BM25 score)
     │
     ▼
Limit: Take top N results
     │
     ▼
Format: With relevance scores

Phase 3 (Fuzzy)

User Query: "dokcer networking"
     │
     ▼
Try FTS5: "dokcer AND networking"
     │
     ▼
Results: 0 (no exact match)
     │
     ▼
Trigger Fuzzy: Extract trigrams
     │
     ├─▶ "dokcer" → ["dok", "okc", "kce", "cer"]
     └─▶ "networking" → ["net", "etw", "two", ...]
     │
     ▼
Find Candidates: Query trigrams table
     │
     ▼
Calculate Similarity: Trigram overlap + edit distance
     │
     ├─▶ "docker" → similarity = 0.85 (good match)
     └─▶ "networking" → similarity = 1.0 (exact)
     │
     ▼
Filter: Threshold ≥ 0.7
     │
     ▼
Re-rank: Combined score (trigram + edit + recency)
     │
     ▼
Merge: With FTS5 results (dedup by ID)
     │
     ▼
Limit: Take top N results
     │
     ▼
Format: With relevance scores

Indexing Strategy

Phase 1 Indexes

-- Recency queries (ORDER BY created_at DESC)
CREATE INDEX idx_memories_created ON memories(created_at DESC);

-- Expiration filtering (WHERE expires_at > now())
CREATE INDEX idx_memories_expires ON memories(expires_at) 
  WHERE expires_at IS NOT NULL;

-- Tag lookups (JOIN on tag_id)
CREATE INDEX idx_tags_name ON tags(name);

-- Tag filtering (JOIN memory_tags on memory_id)
CREATE INDEX idx_memory_tags_tag ON memory_tags(tag_id);

Query plans:

-- Search query uses indexes:
EXPLAIN QUERY PLAN
SELECT * FROM memories WHERE created_at > ? ORDER BY created_at DESC;
-- Result: SEARCH memories USING INDEX idx_memories_created

EXPLAIN QUERY PLAN
SELECT * FROM memories WHERE expires_at > strftime('%s', 'now');
-- Result: SEARCH memories USING INDEX idx_memories_expires

Phase 2 Indexes (+ FTS5)

-- FTS5 creates inverted index automatically
CREATE VIRTUAL TABLE memories_fts USING fts5(content, ...);
-- Generates internal tables: memories_fts_data, memories_fts_idx, memories_fts_config

FTS5 Index Structure:

Term → Document Postings List

"docker" → [1, 5, 12, 34, 56, ...]
"compose" → [5, 12, 89, ...]
"networking" → [5, 34, 67, ...]

Query "docker AND compose" → intersection([1,5,12,34,56], [5,12,89]) = [5, 12]

Phase 3 Indexes (+ Trigrams)

-- Trigram lookups (WHERE trigram IN (...))
CREATE INDEX idx_trigrams_trigram ON trigrams(trigram);

-- Cleanup on memory deletion (CASCADE via memory_id)
CREATE INDEX idx_trigrams_memory ON trigrams(memory_id);

Trigram Index Structure:

Trigram → Memory IDs

"doc" → [1, 5, 12, 34, ...]  (all memories with "doc")
"ock" → [1, 5, 12, 34, ...]  (all memories with "ock")
"cke" → [1, 5, 12, ...]      (all memories with "cke")

Query "docker" trigrams ["doc", "ock", "cke", "ker"]
→ Find intersection: memories with all 4 trigrams (or ≥ threshold)

Performance Optimization

Database Configuration

-- WAL mode for better concurrency
PRAGMA journal_mode = WAL;

-- Memory-mapped I/O for faster reads
PRAGMA mmap_size = 268435456;  -- 256MB

-- Larger cache for better performance
PRAGMA cache_size = -64000;  -- 64MB (negative = KB)

-- Synchronous writes (balance between speed and durability)
PRAGMA synchronous = NORMAL;  -- Not FULL (too slow), not OFF (unsafe)

-- Auto-vacuum to prevent bloat
PRAGMA auto_vacuum = INCREMENTAL;

Query Optimization

// Use prepared statements (compiled once, executed many times)
const searchStmt = db.prepare(`
  SELECT * FROM memories 
  WHERE LOWER(content) LIKE LOWER(?) 
  ORDER BY created_at DESC 
  LIMIT ?
`);

// Transaction for bulk inserts
const insertMany = db.transaction((memories) => {
  for (const memory of memories) {
    insertStmt.run(memory);
  }
});

Trigram Pruning

// Prune common trigrams (low information value)
// E.g., "the", "and", "ing" appear in most memories
const pruneCommonTrigrams = db.prepare(`
  DELETE FROM trigrams
  WHERE trigram IN (
    SELECT trigram FROM trigrams
    GROUP BY trigram
    HAVING COUNT(*) > (SELECT COUNT(*) * 0.5 FROM memories)
  )
`);

// Run after bulk imports
pruneCommonTrigrams.run();

Result Caching

// LRU cache for frequent queries
const LRU = require('lru-cache');
const queryCache = new LRU({
  max: 100,           // Cache 100 queries
  ttl: 1000 * 60 * 5  // 5 minute TTL
});

function search(query, filters) {
  const cacheKey = JSON.stringify({ query, filters });
  
  if (queryCache.has(cacheKey)) {
    return queryCache.get(cacheKey);
  }
  
  const results = executeSearch(query, filters);
  queryCache.set(cacheKey, results);
  return results;
}

Error Handling

Database Errors

try {
  db.prepare(sql).run(params);
} catch (error) {
  if (error.code === 'SQLITE_BUSY') {
    // Retry after backoff
    await sleep(100);
    return retry(operation, maxRetries - 1);
  }
  
  if (error.code === 'SQLITE_CONSTRAINT') {
    // Validation error (content too long, duplicate tag, etc.)
    throw new ValidationError(error.message);
  }
  
  if (error.code === 'SQLITE_CORRUPT') {
    // Database corruption - suggest recovery
    throw new DatabaseCorruptError('Database corrupted, run: memory recover');
  }
  
  // Unknown error
  throw error;
}

Migration Errors

async function migrate(targetVersion) {
  const currentVersion = getCurrentSchemaVersion();
  
  // Backup before migration
  await backupDatabase(`backup-v${currentVersion}.db`);
  
  try {
    db.exec('BEGIN TRANSACTION');
    
    // Run migrations
    for (let v = currentVersion + 1; v <= targetVersion; v++) {
      await runMigration(v);
    }
    
    db.exec('COMMIT');
    console.log(`Migrated to version ${targetVersion}`);
  } catch (error) {
    db.exec('ROLLBACK');
    console.error('Migration failed, rolling back');
    
    // Restore backup
    await restoreDatabase(`backup-v${currentVersion}.db`);
    throw error;
  }
}

Security Considerations

Input Validation

// Prevent SQL injection (prepared statements)
const stmt = db.prepare('SELECT * FROM memories WHERE content LIKE ?');
stmt.all(`%${userInput}%`);  // Safe: userInput is parameterized

// Validate content length
if (content.length > 10000) {
  throw new ValidationError('Content exceeds 10KB limit');
}

// Sanitize tags (only alphanumeric, hyphens, underscores)
const sanitizeTag = (tag) => tag.replace(/[^a-z0-9\-_]/gi, '');

Sensitive Data Protection

// Warn if sensitive patterns detected
const sensitivePatterns = [
  /password\s*[:=]\s*\S+/i,
  /api[_-]?key\s*[:=]\s*\S+/i,
  /token\s*[:=]\s*\S+/i,
  /secret\s*[:=]\s*\S+/i
];

function checkSensitiveData(content) {
  for (const pattern of sensitivePatterns) {
    if (pattern.test(content)) {
      console.warn('⚠️  Warning: Potential sensitive data detected');
      console.warn('Consider storing credentials in a secure vault instead');
      return true;
    }
  }
  return false;
}

File Permissions

# Database file should be user-readable only
chmod 600 ~/.config/opencode/memories.db

# Backup files should have same permissions
chmod 600 ~/.config/opencode/memories-backup-*.db

Scalability Limits

Phase 1 (LIKE)

Max memories: ~500 (performance degrades beyond)
Query latency: O(n) - linear scan
Storage: ~250KB for 500 memories

Phase 2 (FTS5)

Max memories: ~50K (comfortable), 100K+ (possible)
Query latency: O(log n) - index lookup
Storage: +30% for FTS5 index (~325KB for 500 memories)

Phase 3 (Fuzzy)

Max memories: 100K+ (with trigram pruning)
Query latency: O(log n) + O(m) where m = fuzzy candidates
Storage: +200% for trigrams (~750KB for 500 memories)
- Mitigated by pruning common trigrams

Migration Triggers

Phase 1 → Phase 2:

Dataset > 500 memories
Query latency > 500ms
Manual user request

Phase 2 → Phase 3:

User reports needing fuzzy search
High typo rates in queries
Manual user request

Future Enhancements

Vector Embeddings (Phase 4?)

Semantic search ("docker" → "containerization")
Requires embedding model (~100MB)
SQLite-VSS extension
Hybrid: BM25 (lexical) + Cosine similarity (semantic)

Automatic Summarization

LLM-generated summaries for long memories
Reduces token usage in search results
Trade-off: API dependency

Memory Versioning

Track edits to memories
Show history
Revert to previous version

Conflict Detection

Identify contradictory memories
Suggest consolidation
Flag for review

Collaborative Features

Share memories between agents
Team-wide memory pool
Privacy controls

Document Version: 1.0
Last Updated: 2025-10-29
Status: Planning Complete, Implementation Pending

23 KiB Raw Blame History

LLMemory Architecture

System Overview

Data Model

Phase 1 Schema (MVP)

Phase 2 Schema (+ FTS5)

Phase 3 Schema (+ Trigrams)

Search Algorithm Evolution

Phase 1: LIKE Search

Phase 2: FTS5 Search

Phase 3: Fuzzy Search

Memory Lifecycle

Query Processing Flow

Phase 1 (LIKE)

Phase 2 (FTS5)

Phase 3 (Fuzzy)

Indexing Strategy

Phase 1 Indexes

Phase 2 Indexes (+ FTS5)

Phase 3 Indexes (+ Trigrams)

Performance Optimization

Database Configuration

Query Optimization

Trigram Pruning

Result Caching

Error Handling

Database Errors

Migration Errors

Security Considerations

Input Validation

Sensitive Data Protection

File Permissions

Scalability Limits

Phase 1 (LIKE)

Phase 2 (FTS5)

Phase 3 (Fuzzy)

Migration Triggers

Future Enhancements

Vector Embeddings (Phase 4?)

Automatic Summarization

Memory Versioning

Conflict Detection

Collaborative Features

23 KiB

Raw Blame History