WAL & Crash Recovery

PocketDB uses a Write-Ahead Log (WAL) to ensure durability and enable automatic crash recovery.

Installation

Add PocketDB to your Cargo.toml:

[dependencies]
pocketdb = "0.1"

Or via the command line:

cargo add pocketdb

How WAL Works

The WAL records all write operations before they're applied to the main database file:

1. A write operation is first appended to the WAL file inside .pocketdb/
2. The operation is then applied to the main .pdb file
3. Once applied, the WAL record can be cleaned up

This ensures that if the process crashes mid-operation, the WAL contains enough information to either complete or roll back the operation.

File Structure

The .pdb file is the single source of truth. The .pocketdb/ directory is auto-created alongside it for WAL crash recovery:

your_project/
├── mydata.pdb              # Your database (single file, all data)
└── .pocketdb/              # Auto-created, optional (WAL for crash recovery)
    └── mydata.pdb-wal      # WAL file

The .pocketdb/ directory is optional — if deleted, the database still works and the WAL recreates on next commit.

Automatic Recovery

When you open a database, WAL recovery is performed automatically:

use pocketdb::Database;

// Recovery happens automatically on open
let db = Database::open("mydata.pdb")?;
// Any incomplete transactions in the WAL are rolled back
// Committed transactions that weren't applied are replayed

Recovery Process

1. Open the WAL file — read any existing WAL records from .pocketdb/
2. Replay committed transactions — apply any operations that were committed but not yet written to the main file
3. Roll back incomplete transactions — discard any operations from transactions that weren't committed
4. Clean up the WAL — truncate the WAL file after successful recovery

Durability Guarantees

With WAL enabled, PocketDB provides:

• Crash Safety: Committed data is never lost
• Atomicity: Each transaction is either fully applied or not at all
• Durability: Once commit() returns, data is persisted

Sync on Close

When the Database is dropped, it automatically syncs to disk:

{
    let db = Database::open("mydata.pdb")?;
    let users = db.collection("users")?;
    users.insert(doc! { "name" => "Alice" })?;
    // db is dropped here, data is synced to disk
}

Validating Database Integrity

Check if the database has any corrupted pages:

use pocketdb::Database;

let db = Database::open("mydata.pdb")?;
if db.validate()? {
    println!("All pages valid");
} else {
    eprintln!("Corruption detected — run Database::repair()");
}

Repairing a Corrupted Database

If validation fails, use the static repair method to attempt recovery:

use pocketdb::Database;

let corrupted = Database::repair("mydata.pdb")?;
println!("Found and handled {} corrupted pages", corrupted);

The repair method:

1. Validates the file header
2. Reads every page and checks checksums
3. Identifies corrupted pages
4. Cleans up stale WAL files

Cleaning Up Orphaned WAL Files

If you delete a .pdb file but the .pocketdb/ directory still contains WAL files, clean them up:

use pocketdb::Database;

// Returns true if cleanup was performed
let cleaned = Database::cleanup_orphaned("mydata.pdb")?;
if cleaned {
    println!("Orphaned WAL files removed");
}

This is safe to call even if the database file exists — it only removes WAL files when the .pdb file is missing.

Compacting

Flush all data and clean up the WAL:

use pocketdb::Database;

let db = Database::open("mydata.pdb")?;
db.compact()?;

Best Practices

1. Let the WAL manage itself — don't manually delete WAL files
2. Use transactions for multi-document atomicity
3. Handle errors — check return values from insert, update, delete
4. Validate after crashes — run db.validate() to check integrity
5. Backup regularly — see Backup & Export

File Format

PocketDB uses a custom binary file format:

• Magic bytes: PKDB (0x504B4442)
• Versioned format for forward compatibility
• CRC32 checksums on pages for integrity verification
• WAL records are also checksummed

// If a checksum mismatch is detected, PocketError::PageChecksumMismatch is returned
// This indicates file corruption — use Database::repair() or restore from backup