KB Synchronization Architecture¶

This document describes the synchronization mechanism for knowledge base operations when multiple users work with the same KB.

Problem Statement¶

When multiple users work with the same knowledge base (shared GitHub repository):

Concurrent modifications can lead to merge conflicts
Race conditions may cause lost updates
Git conflicts are difficult to resolve automatically
Data consistency must be maintained

Solution: Operation Serialization + Git Pull¶

Core Components¶

1. KBSyncManager¶

Location: src/knowledge_base/sync_manager.py

Purpose: Serialize KB operations to prevent conflicts

Features: - File-based locks (cross-process synchronization) - Async locks (in-process synchronization) - Per-KB locking (different KBs can be accessed in parallel) - Singleton pattern for global coordination

Usage:

from src.knowledge_base.sync_manager import get_sync_manager

sync_manager = get_sync_manager()

async with sync_manager.with_kb_lock(kb_path, "operation_name"):
    # Your KB operations here (create, update, delete)
    # Only one user can execute this block at a time for this KB
    pass

2. GitOperations.pull()¶

Location: src/knowledge_base/git_ops.py

Purpose: Pull latest changes before KB operations

Features: - Checks for uncommitted changes - Validates remote configuration - Detects merge conflicts - Returns success status and message

Usage:

git_ops = GitOperations(kb_path, enabled=True)
success, message = git_ops.pull(remote="origin", branch="main")

if not success:
    # Handle error (conflicts, network issues, etc.)
    print(f"Pull failed: {message}")

3. BaseKBService (Common Functionality)¶

Location: src/services/base_kb_service.py

Purpose: Base class that consolidates common functionality for KB-related services

Responsibilities: - Git operations setup and management - Agent working directory configuration - Rate limiting enforcement - Auto-commit and push operations - GitHub URL generation for file links - File change formatting for user notifications - Links/relations filtering and display - Safe message editing with timeout handling

Key Methods: - _setup_git_ops(): Initialize Git operations with user credentials - _get_agent_working_dir(): Determine agent working directory based on KB_TOPICS_ONLY setting - _configure_agent_working_dir(): Set agent working directory - _check_rate_limit(): Enforce rate limits before agent API calls - _auto_commit_and_push(): Commit and push changes to remote - _get_github_base_url(): Generate GitHub URLs for file links - _format_file_changes(): Format file changes for notifications - _filter_and_format_links(): Filter and format link relations - _safe_edit_message(): Safely edit messages with timeout handling

4. NoteCreationService Integration¶

Location: src/services/note_creation_service.py

Inherits from: BaseKBService, INoteCreationService

Flow (Note mode - /note):

1. User sends message
2. Acquire KB lock (wait if another user is working)
3. Pull latest changes from GitHub (using base class Git operations)
4. Process message with agent (with rate limiting from base class)
5. Save note to KB
6. Commit and push changes (using base class auto-commit)
7. Release KB lock

5. AgentTaskService Integration¶

Location: src/services/agent_task_service.py

Inherits from: BaseKBService, IAgentTaskService

Flow (Agent mode - /agent):

1. User sends task request
2. Acquire KB lock (wait if another user is working)
3. Parse task and prepare context
4. Execute task with agent (may read/write KB files, with rate limiting)
5. Auto-commit and push changes (using base class auto-commit)
6. Return results to user
7. Release KB lock

Note: Both services inherit from BaseKBService to minimize code duplication and share common functionality while maintaining their specific responsibilities. This follows the DRY (Don't Repeat Yourself) principle and Single Responsibility Principle.

Synchronization Flow¶

Single User Scenario¶

sequenceDiagram
    participant U as User
    participant S as NoteService
    participant L as KBSyncManager
    participant G as GitOps
    participant GH as GitHub

    U->>S: Create note
    S->>L: Acquire lock
    L-->>S: Lock acquired
    S->>G: Pull latest
    G->>GH: git pull
    GH-->>G: Up to date
    S->>S: Process & save
    S->>G: Commit & push
    G->>GH: git push
    S->>L: Release lock
    L-->>S: Lock released
    S-->>U: Success

Multi-User Scenario¶

sequenceDiagram
    participant U1 as User 1
    participant U2 as User 2
    participant S as NoteService
    participant L as KBSyncManager
    participant G as GitOps
    participant GH as GitHub

    U1->>S: Create note
    S->>L: Acquire lock (User 1)
    L-->>S: Lock acquired

    U2->>S: Create note
    S->>L: Acquire lock (User 2)
    Note over L: User 2 waits...

    S->>G: Pull latest (User 1)
    G->>GH: git pull
    S->>S: Process & save (User 1)
    S->>G: Commit & push (User 1)
    G->>GH: git push
    S->>L: Release lock (User 1)

    L-->>S: Lock acquired (User 2)
    S->>G: Pull latest (User 2)
    G->>GH: git pull
    Note over G,GH: Gets User 1's changes
    S->>S: Process & save (User 2)
    S->>G: Commit & push (User 2)
    G->>GH: git push
    S->>L: Release lock (User 2)

Lock Types¶

File Lock (FileLock)¶

Purpose: Cross-process synchronization

Location: .kb_operations.lock in KB directory

Characteristics: - Works across different Python processes - Works across different machines (if KB is on shared filesystem) - Timeout: 300 seconds (5 minutes) - Automatically released on exception

Async Lock (asyncio.Lock)¶

Purpose: In-process synchronization

Characteristics: - Works within same Python process - Prevents race conditions in async code - No timeout (relies on file lock timeout) - Acquired before file lock

Lock Hierarchy¶

Acquire Async Lock
    ↓
Acquire File Lock
    ↓
Perform KB Operations
    ↓
Release File Lock
    ↓
Release Async Lock

Why both? - Async lock: Fast, prevents issues in same process - File lock: Slower, but works across processes/machines

Error Handling¶

Git Pull Failures¶

Uncommitted changes:

Error: "Repository has uncommitted changes"
Action: User must commit or stash manually

Merge conflicts:

Error: "Merge conflict during pull"
Action: User must resolve conflicts manually

No remote configured:

Error: "Remote 'origin' not found"
Action: Configure remote or work locally

Remote branch doesn't exist:

Warning: "Remote branch 'branch_name' doesn't exist on origin"
Action: Automatically creates branch locally and pushes to remote

Authentication errors:

Error: "Failed to push (authentication error): could not read Username"
Action: Configure git credentials or switch to SSH
Automatic solution: Set GITHUB_USERNAME and GITHUB_TOKEN in .env file
Manual solutions:
  1. Use SSH instead of HTTPS: git remote set-url origin git@github.com:user/repo.git
  2. Configure git credential helper: git config credential.helper store
  3. Use a personal access token: https://github.com/settings/tokens

Automatic HTTPS Authentication:

The system automatically configures HTTPS authentication if credentials are provided in environment variables:

# .env file
GITHUB_USERNAME=your_github_username
GITHUB_TOKEN=ghp_your_personal_access_token

When GitOperations is initialized with these credentials: - It automatically updates HTTPS remote URLs to include authentication - Only GitHub HTTPS remotes without existing credentials are modified - SSH remotes remain unchanged - This enables push operations without manual credential configuration - Configuration is idempotent (safe to call multiple times) - If configuration fails for one remote, other remotes are still processed

Security safeguards: - Files outside the repository cannot be added to git (prevents path traversal) - Branch switching requires successful stashing of uncommitted changes (prevents data loss) - Detached HEAD state is detected and prevents dangerous operations - All file paths are validated and resolved before git operations

Lock Timeout¶

If a user's operation takes > 5 minutes: - Lock automatically released - Warning logged - Next user can proceed

Exception Handling¶

Locks are always released, even on exception:

async with sync_manager.with_kb_lock(kb_path):
    # If exception occurs here
    raise SomeError()
    # Locks are still released properly

Configuration¶

Settings¶

config.yaml:

# Enable Git operations
KB_GIT_ENABLED: true

# Auto-push after commit (recommended for multi-user)
KB_GIT_AUTO_PUSH: true

# Remote and branch
KB_GIT_REMOTE: origin
KB_GIT_BRANCH: main

.env file (for HTTPS authentication):

# GitHub credentials for automatic HTTPS authentication
GITHUB_USERNAME=your_github_username
GITHUB_TOKEN=ghp_your_personal_access_token

How to get GitHub token: 1. Go to https://github.com/settings/tokens 2. Click "Generate new token" → "Generate new token (classic)" 3. Set expiration and select scopes: repo (full control of private repositories) 4. Copy token and add to .env file

Lock Timeout¶

Modify in sync_manager.py:

FileLock(str(lock_file), timeout=300)  # 5 minutes

Best Practices¶

For Users¶

Enable auto-push: Ensures changes are immediately available to others
Keep notes focused: Shorter operations = less waiting
Monitor for errors: Check for pull/push failures

For Developers¶

Always use sync manager: Never modify KB without lock
Pull before operations: Ensures latest version
Handle pull errors: Don't proceed if pull fails
Keep operations atomic: Complete entire operation within lock
Log lock events: Track who's waiting and why

For System Administrators¶

Monitor lock files: Check for stale locks
Set reasonable timeouts: Balance between patience and responsiveness
Use separate KBs: For independent teams/projects
Regular backups: Even with synchronization, backups are essential

Performance Considerations¶

Lock Contention¶

Low contention (few users): Negligible impact
High contention (many users): Users may wait
Solution: Use separate KBs per team/project

Network Latency¶

Pull/push operations: Depend on network speed
Lock waiting: Not affected by network
Solution: Use faster network or local-first workflow

Lock File I/O¶

File lock overhead: ~10-50ms
Async lock overhead: ~1ms
Total overhead: < 100ms per operation

Testing¶

Unit Tests¶

tests/test_kb_sync_manager.py: Sync manager tests
tests/test_agent_task_service_kb_lock.py: Agent mode KB locking tests
tests/test_git_ops.py: Git operations tests

Integration Tests¶

Simulate multi-user scenarios:

async def test_concurrent_users():
    # Create two users
    # Both try to create notes simultaneously
    # Verify operations are serialized
    # Verify both notes are saved

Agent Mode Locking Tests:

async def test_agent_tasks_serialized_for_same_kb():
    # Create two users with agent mode
    # Both try to execute tasks simultaneously on same KB
    # Verify operations are serialized
    # Verify both tasks complete successfully

Future Improvements¶

Optimistic locking: Allow concurrent reads
Lock queue visibility: Show users their position in queue
Automatic conflict resolution: Smart merge for simple conflicts
Lock analytics: Track wait times and contention
Distributed locks: Support for multi-server deployments

KB Synchronization Architecture¶

Problem Statement¶

Solution: Operation Serialization + Git Pull¶

Core Components¶

1. KBSyncManager¶

2. GitOperations.pull()¶

3. BaseKBService (Common Functionality)¶

4. NoteCreationService Integration¶

5. AgentTaskService Integration¶

Synchronization Flow¶

Single User Scenario¶

Multi-User Scenario¶

Lock Types¶

File Lock (FileLock)¶

Async Lock (asyncio.Lock)¶

Lock Hierarchy¶

Error Handling¶

Git Pull Failures¶

Lock Timeout¶

Exception Handling¶

Configuration¶

Settings¶

Lock Timeout¶

Best Practices¶

For Users¶

For Developers¶

For System Administrators¶

Performance Considerations¶

Lock Contention¶

Network Latency¶

Lock File I/O¶

Testing¶

Unit Tests¶

Integration Tests¶

Future Improvements¶

Related Documentation¶