AccelerateSearch

A self-hosted, production-grade search engine written in Rust.

AccelerateSearch combines the developer experience of Meilisearch with the analytical power of Elasticsearch, all in a single binary that runs on Linux, macOS, and Windows.

Warning

The project is in active development. APIs, on-disk formats, configuration keys, and CLI flags may change between releases. Pin a specific commit or release tag for stability.

Features

• Blazing-fast full-text search with BM25 ranking
• Lock-free concurrent reads with DashMap and parking_lot
• FST-backed term dictionaries for O(log n) prefix lookups and autocomplete
• Vector and hybrid search with scalar / product / binary quantization
• Fuzzy matching with bounded Damerau-Levenshtein typo tolerance
• Complex filter expressions (field = "value" AND rating > 4 OR location GEO_BBOX …)
• Facet distributions and stats for every field
• Per-collection settings: ranking rules, synonyms, stop words, typo tolerance, embedders, distinct field, …
• Webhooks that fire on document and index events
• Tenant tokens for short-lived, scoped access from the browser
• API keys with expiry, scopes, and per-collection ACLs
• Prometheus metrics at /metrics, structured logs via tracing
• Snapshots (tar + zstd) for backup and restore
• Single binary with no external services required

Where to go next

• Architecture overview
• REST API reference
• Configuration reference
• Deployment guide
• Development guide
• Rust API docs (cargo doc)

Project layout

crates/
  api/          REST handlers, DTOs, OpenAPI schema
  auth/         master key, API keys, tenant tokens
  cache/        LRU + TTL cache
  collections/  collection metadata service
  config/       TOML config & validation
  documents/    document service
  filters/      filter expression parser & evaluator
  hybrid/       RRF, score normalization
  highlighting/ <em> highlighting
  indexing/     tokenization + inverted index + FST
  metrics/      Prometheus exporter
  models/       shared data types
  search/       BM25, ranking, query parser
  security/     rate limit, CORS, audit logger
  server/       HTTP lifecycle, banner
  storage/      StorageBackend trait + redb
  synonyms/     synonym map storage and lookup
  tasks/        async task queue
  telemetry/    tracing-subscriber setup
  typo/         Damerau-Levenshtein
  utils/        hash, random, time helpers
  validation/   input validation & sanitization
  vector/       embedding types + quantization
config/         default.toml
docs/           this mdbook source
.github/        CI + release + docs workflows

Author

Muhammad Fiaz — contact@muhammadfiaz.com

License

Apache-2.0

AccelerateSearch Architecture

AccelerateSearch is a self-hosted, production-grade search engine written in Rust. This document describes the high-level architecture, the crate dependency graph, and the data flow during a search and an indexing request.

Goals

• Single binary that serves the full REST API on Linux, macOS, and Windows.
• Pluggable storage backend (default: embedded redb).
• Fast keyword + vector search with a 10 ms target p99 for sub-million-document collections on commodity hardware.
• Meilisearch-style developer experience: tasks, settings, scopes, webhooks, tenant tokens.
• Elasticsearch-style power: complex filter expressions, facet distributions, multi-index search, ranking rules.
• OpenSearch-level observability: Prometheus metrics, structured logging via tracing.

Crate Dependency Graph

The workspace contains 30 library crates and one binary. The binary (accelerate) is a thin shell that wires them together; the server lifecycle (crates/server) owns the actix-web setup, banner, and graceful-shutdown logic. Everything else is layered on top of the api crate, which holds the HTTP handlers.

                      ┌─────────────────────┐
                      │      accelerate     │  (root binary)
                      └──────────┬──────────┘
                                 │
                      ┌──────────▼──────────┐
                      │       server        │  (HTTP lifecycle, banner)
                      └──────────┬──────────┘
                                 │
            ┌────────────────────┼────────────────────┐
            │                    │                    │
     ┌──────▼──────┐     ┌───────▼───────┐    ┌───────▼───────┐
     │     api     │     │  scheduler    │    │  telemetry    │
     └──────┬──────┘     └───────────────┘    └───────────────┘
            │
   ┌────────┼────────┬───────────┬────────────┬──────────────┐
   │        │        │           │            │              │
┌──▼──┐  ┌──▼──┐  ┌───▼───┐  ┌────▼────┐  ┌────▼────┐  ┌─────▼─────┐
│auth │  │search│  │indexing│  │documents│  │filters │  │ collections│
└──┬──┘  └──┬──┘  └───┬───┘  └────┬────┘  └────┬────┘  └─────┬─────┘
   │        │         │           │            │             │
   │    ┌───▼───┐     │           │            │             │
   │    │ cache │     │           │            │             │
   │    └───────┘     │           │            │             │
   │                 │           │            │             │
┌──▼──────┐     ┌────▼────┐  ┌────▼────┐  ┌────▼────┐  ┌────▼────┐
│security │     │ storage │  │ facets  │  │  typo   │  │  hybrid │
└─────────┘     └────┬────┘  └─────────┘  └─────────┘  └─────────┘
                     │
              ┌──────▼──────┐
              │    redb     │  (embedded key-value store)
              └─────────────┘

Cross-cutting helpers that all crates can depend on:

Data Flow: Search Request

1. actix-web receives the HTTP request at /api/v1/collections/{uid}/search.
2. The middleware stack runs in order: tracing → rate limit → auth.
3. The search handler validates the request and looks up the collection in the in-memory CollectionStore.
4. SearchEngine::search_with_rules consults the result cache. On hit, the cached response is returned immediately.
5. On miss, the engine:
   • Loads the collection’s InvertedIndex from the IndexStore (cached in a DashMap keyed by CollectionId).
   • Resolves synonym expansion for the query terms.
   • Applies typo tolerance (bounded Damerau-Levenshtein expansion).
   • Scores candidates with BM25 (crates/search::bm25).
   • Applies the filter (recursive-descent parser → evaluator) on hydrated documents.
   • Applies user-requested sorting and the ruleset (pinned, hidden, sort/filter overrides).
   • Computes facet distributions.
6. The response is JSON-serialised with a processingTimeMs field and returned.
7. Successful responses are stored in the result cache (TTL + LRU).

Data Flow: Indexing Request

1. POST /api/v1/collections/{uid}/documents is received.
2. The documents handler validates every document and calls DocumentService::add_or_replace.
3. The service runs the IndexingPipeline which:
   • Tokenises each searchable field with the Analyzer (Unicode NFC, lowercase, stop-word removal, optional stemming).
   • Updates the in-memory InvertedIndex (per-field term frequencies, per-document field lengths).
   • Recomputes BM25 collection statistics.
   • Rebuilds the FST-backed term dictionary for O(log n) prefix lookups (used by autocomplete).
   • Persists the documents to storage::TABLE_DOCUMENTS and the postings / terms / field-lengths / stats to the matching TABLE_* tables.
4. The result cache is invalidated for the collection.

Concurrency Model

• The server runs actix-web with one Tokio worker per CPU core.
• Shared in-memory state lives in DashMap instances (per-collection indexes, hooks, rulesets, key cache).
• Long-running mutators (e.g. RwLock over an index) use parking_lot for lower contention than std::sync.
• Background jobs (scheduler) run on a Notify-gated Tokio task that can be cancelled on shutdown.
• Result caching uses an LRU + TTL TtlCache (parking_lot::Mutex<LruCache<K, Entry<V>>>).
• Hot-reloadable config is wrapped in arc-swap so readers never block writers.

Storage

The default StorageBackend is an embedded redb key-value store. The schema is table-based, with the following tables defined in crates/storage:

A different backend (RocksDB, Sled, …) can be plugged in by implementing the StorageBackend trait and swapping the wiring in crates/server::run.

Configuration

crates/config parses config/default.toml (or the path supplied via --config), then layers CLI overrides on top, then environment variables, then the built-in defaults. Validation is performed with the validator crate. See docs/configuration.md for the full key reference.

Security

• Master key (SHA-256 hashed) gates all /api/v1/* routes except a whitelist (/health, /version, /metrics, /swagger-ui/*).
• API keys are scoped by Permission and an optional collection list.
• Tenant tokens are HS256 JWTs with a short (≤ 1 h) lifetime.
• Rate limiting uses governor keyed by client IP.
• Security headers (X-Content-Type-Options, CSP, HSTS, …) are added on every response.
• All user-supplied strings are sanitised (control characters stripped, whitespace runs collapsed) before storage or query parsing.

AccelerateSearch REST API

All routes are versioned under /api/v1/. The OpenAPI specification is served at /api-docs/openapi.json and the Swagger UI at /swagger-ui/.

The response format for errors is:

{
  "error": "code_snake_case",
  "message": "Human-readable description.",
  "code": 404
}

System (no auth required)

All other endpoints require the master key or a scoped API key in the Authorization: Bearer <key> header.

Collections

Per-setting endpoints (leaf GET / PUT / DELETE)

The following “leaf” settings each have their own GET/PUT/DELETE trio so clients can manage individual settings without round-tripping the full CollectionSettings blob:

The embedders leaf uses PATCH (not PUT) because the embedder configuration is a JSON object that is deep-merged, not replaced.

Documents

Search

Search request body

{
  "q": "rust search",
  "offset": 0,
  "limit": 20,
  "filter": "rating >= 4 AND status = \"active\"",
  "facets": ["category", "brand"],
  "attributes_to_retrieve": ["id", "title", "price"],
  "attributes_to_highlight": ["title", "body"],
  "sort": ["price:asc"],
  "show_ranking_score": true,
  "hybrid": { "semantic_ratio": 0.5, "embedder": "default" },
  "vector": [0.1, 0.2, 0.3],
  "distinct": "sku"
}

Search response

{
  "query": "rust search",
  "hits": [
    {
      "document": { "id": "1", "title": "Rust in Action", "price": 39.95 },
      "formatted": { "title": "<em>Rust</em> in Action" },
      "ranking_score": 0.86
    }
  ],
  "offset": 0,
  "limit": 20,
  "estimatedTotalHits": 142,
  "processingTimeMs": 4,
  "facetDistribution": { "category": { "counts": { "book": 87, "video": 55 } } }
}

Autocomplete response

{
  "query": "rus",
  "suggestions": [
    { "term": "rust",     "total_term_freq": 1234 },
    { "term": "rusty",    "total_term_freq":  17  },
    { "term": "russian",  "total_term_freq":   9  }
  ],
  "processingTimeMs": 1
}

Filter expression grammar

expr        := or
or          := and ( "OR" and )*
and         := not ( "AND" not )*
not         := "NOT" not | atom
atom        := "(" expr ")" | comparison
comparison  := field op value
op          := "=" | "!=" | ">" | ">=" | "<" | "<=" | "TO"
            | "IN" | "NOT" "IN" | "EXISTS" | "IS" "NULL" | "IS" "NOT" "NULL"
            | "CONTAINS" | "STARTS_WITH" | "ENDS_WITH" | "LIKE"
            | "GEO_BBOX" lat lng lat lng
            | "GEO_RADIUS" lat lng meters
value       := number | string | bool | null | array

LIKE patterns use % for “any” and _ for “single character”.

Tasks

API Keys

Tenant Tokens

Snapshots

Indexes (alias for collections)

The /api/v1/indexes/* routes are aliases for /api/v1/collections/* and accept the same payloads. They exist for Meilisearch compatibility on the search-rules endpoint (which lives at /api/v1/indexes/{uid}/settings/rules).

Search Rules (curated queries)

Hooks (webhooks)

Network and experimental features

OpenAPI and Swagger

Both are gated by the [api_docs] TOML section. Disabling either key returns a 404 for that path.

Configuration Reference

Configuration is parsed at startup from (in order of precedence):

1. CLI flags (e.g. --host 0.0.0.0 --port 7700)
2. Environment variables (e.g. ACCELERATE_HOST, ACCELERATE_PORT)
3. config/default.toml (the file shipped with the binary)
4. Built-in defaults

The path to the TOML file can be overridden with --config <file> or ACCELERATE_CONFIG=/path/to/file.

Warning

The project is in active development. Configuration keys may be renamed, removed, or have their defaults changed between releases. Always read the config/default.toml of the release you are running for the authoritative reference.

[server]

[server.tls]

[api_docs]

[data]

[auth]

[search]

[indexing]

[vector]

[logging]

[metrics]

[snapshots]

[updates]

[rate_limit]

[telemetry]

[cache]

[cors]

Environment variables

Every CLI flag is also exposed as an ACCELERATE_* environment variable. For example, --host 0.0.0.0 is equivalent to ACCELERATE_HOST=0.0.0.0. Sensitive values that are commonly set via the environment in production:

Precedence example

# config/default.toml contains:  host = "localhost", port = 7700
# env vars:                       ACCELERATE_PORT=8080
# CLI:                            --host 0.0.0.0

# Effective:
#   host = 0.0.0.0   (CLI > env > TOML > default)
#   port = 8080      (env > TOML > default)

Deployment

Single-node (default)

# Build
cargo build --release

# Run
./target/release/accelerate

A data/ directory is created on first run and holds the embedded redb store, snapshots, daily logs, and the API key store.

TLS

Generate a self-signed cert for local development:

mkcert -install
mkcert accelerate.local

Set in config/default.toml:

[server]
host = "0.0.0.0"
port = 7700

[server.tls]
enabled = true
cert_path = "./accelerate.local.pem"
key_path = "./accelerate.local-key.pem"

For production, use certs from Let’s Encrypt or your corporate CA.

systemd unit

# /etc/systemd/system/accelerate.service
[Unit]
Description=AccelerateSearch
After=network.target

[Service]
Type=simple
User=accelerate
WorkingDirectory=/var/lib/accelerate
Environment=ACCELERATE_CONFIG=/etc/accelerate/config.toml
ExecStart=/usr/local/bin/accelerate
Restart=on-failure
RestartSec=5
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

sudo useradd -r -s /usr/sbin/nologin accelerate
sudo install -m755 target/release/accelerate /usr/local/bin/
sudo install -d -o accelerate -g accelerate /var/lib/accelerate
sudo install -d -m755 /etc/accelerate
sudo cp config/default.toml /etc/accelerate/config.toml
sudo systemctl daemon-reload
sudo systemctl enable --now accelerate

Docker

docker build -t accelerate:local .
docker run --rm -p 7700:7700 -v "$PWD/data:/data" accelerate:local

docker-compose.yml is provided in the repository root for a single-command bring-up that exposes port 7700.

Reverse proxy

nginx (with TLS termination)

server {
    listen 443 ssl http2;
    server_name accelerate.example.com;

    ssl_certificate     /etc/letsencrypt/live/accelerate.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/accelerate.example.com/privkey.pem;

    client_max_body_size 10m;

    location / {
        proxy_pass http://127.0.0.1:7700;
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Capacity planning

A single node comfortably handles:

• 5 M documents × 1 KB each
• 30 K QPS search (cache-warm, single collection)
• 5 K QPS index (small batch updates)

For higher throughput, scale horizontally behind a load balancer. The embedded redb backend is single-node only; use the network cluster skeleton in crates/cluster to coordinate a fleet.

Documentation site (GitHub Pages)

The user guide is an mdbook site that is published to GitHub Pages via the .github/workflows/docs.yml workflow.

One-time setup

1. Create the GitHub repository AccelerateSearch (the URL slug must match the directory name for the project page to live at muhammad-fiaz.github.io/AccelerateSearch/).
2. In Settings → Pages, set Source to GitHub Actions.
3. (Optional) Configure a custom domain in Settings → Pages → Custom domain and add a CNAME file in site/ from the workflow.
4. Grant the workflow the pages: write and id-token: write permissions (already declared in the workflow file).

Build locally

# One-off: install the mdbook binary
cargo install mdbook --locked --version 0.4.43

# Build the user guide into ./docs/book/
(cd docs && mdbook build)

# Build cargo doc into ./target/doc/
cargo doc --no-deps --workspace --target-dir target

# Combine both into a single ./site/ directory ready for Pages
mkdir -p site
cp -R docs/book/. site/
mkdir -p site/rust-api
cp -R target/doc/. site/rust-api/
touch site/.nojekyll

Deploy

Every push to main rebuilds and deploys the site. To force a rebuild without a code change, go to the Actions tab, select Docs, and click Run workflow.

The site is served at https://muhammad-fiaz.github.io/AccelerateSearch/.

Backup and restore

# Snapshot
curl -X POST -H "Authorization: Bearer $MASTER" \
     http://localhost:7700/api/v1/snapshots \
     -d '{"name":"nightly-2026-06-03"}'

# Download the underlying files (tar+zstd snapshot)
curl -O -H "Authorization: Bearer $MASTER" \
     http://localhost:7700/api/v1/snapshots/nightly-2026-06-03

# Restore
curl -X POST -H "Authorization: Bearer $MASTER" \
     http://localhost:7700/api/v1/snapshots/nightly-2026-06-03/restore

Always stop the server (or pause writes via --read-only) before restoring, to avoid in-flight write conflicts.

Health checks

curl -fsS http://localhost:7700/health
# {"status":"available"}

Configure your orchestrator to restart the container on a non-200 response.

Upgrades

1. Drain writes (optional): set [search] readonly = true.
2. Stop the old binary.
3. Install the new binary.
4. Start the new binary; the embedded store is upgraded in place.
5. Re-enable writes.

Schema migrations live in crates/storage and run automatically on startup. They are additive and idempotent; a rollback to a previous version is always supported.

Development

Prerequisites

• Rust 1.85+ (edition 2024; 1.88+ recommended for the latest dependencies)
• A C toolchain (gcc, clang, or MSVC)
• On Linux: pkg-config and mimalloc’s usual build deps
• git, curl, and cargo (the toolchain)

Build

git clone https://github.com/muhammad-fiaz/AccelerateSearch
cd AccelerateSearch
cargo build

The binary lands at target/debug/accelerate.

Run

cargo run -- --config config/default.toml

Logs are emitted to stdout and logs/accelerate-YYYY-MM-DD.log.

Test

cargo test --workspace --no-fail-fast

• Unit tests live next to the code they cover (#[cfg(test)] mod tests).
• Property-based tests use proptest for the filter parser (filters::evaluator) and the tokenizer (indexing::analyzer).
• Every crate that exposes a public type is documented with cargo doc --no-deps --workspace; the CI fails on any rustdoc warning.

Lint and format

cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings

The CI workflow runs both on every push; failing either blocks the build.

Benchmarks

benchmark/ is a standalone project that spins up a 1M-document collection and measures indexing and search throughput.

cd benchmark
cargo run --release

For micro-benchmarks, use cargo bench in the crate of interest (currently only filters).

Project layout

crates/         # library crates
  api/          # HTTP handlers, DTOs, OpenAPI
  auth/         # master key, API keys, tenant tokens
  cache/        # LRU + TTL cache
  cluster/      # cluster skeleton (TODO)
  collections/  # collection metadata service
  config/       # TOML config, validation
  documents/    # document service (add, update, delete, get, list)
  errors/       # AppError and From impls
  facets/       # facet distribution
  filters/      # filter parser & evaluator
  hybrid/       # hybrid query fusion (RRF)
  highlighting/ # <em> highlight
  indexing/     # tokenization + inverted index + FST term dict
  metrics/      # Prometheus exporter
  models/       # shared data types
  replication/  # replication skeleton (TODO)
  scheduler/    # cron + interval jobs
  search/       # BM25, ranking, query parser
  security/     # rate limit, CORS, audit
  server/       # HTTP lifecycle, banner
  sharding/     # sharding skeleton (TODO)
  snapshots/    # tar+zstd snapshots
  storage/      # StorageBackend trait + redb
  synonyms/     # synonym map storage and lookup
  tasks/        # async task queue
  telemetry/    # tracing-subscriber setup
  typo/         # Damerau-Levenshtein
  utils/        # helpers (hash, random, time)
  validation/   # input validation + sanitization
  vector/       # embedding types + quantization
config/         # default.toml
docs/           # mdbook user guide (this site)
benchmark/      # standalone benchmark project
.github/        # CI + release + docs workflows

Adding a new feature

1. Decide the layer. Filters belong in filters/, ranking tweaks in search/, persistence in storage/, etc. The crate boundaries exist to keep compile times low; honour them.
2. Define the data type in models. Public types live there so they can be shared across crates without circular deps.
3. Write a failing unit test first. Tests are colocated with code.
4. Implement the feature. No unwrap outside tests; no unsafe.
5. Add an integration test if the feature is exposed via HTTP.
6. Update docs/api.md if a new route was added, and docs/configuration.md if a new config key was added.
7. Run cargo fmt, cargo clippy, and cargo test before opening a PR.

Adding a new crate

1. mkdir crates/<name> && cd crates/<name>
2. Copy a minimal Cargo.toml from a sibling crate; pin the workspace deps and inherit the lints table.
3. Add it to [workspace.dependencies] in the workspace Cargo.toml if other crates need to depend on it, otherwise just to the [workspace] members list (default).
4. Update docs/architecture.md to show the new crate in the diagram.

Releasing

1. Bump versions: cargo set-version --workspace 0.X.0 (manual edit acceptable).
2. Update CHANGELOG.md (chronological, newest first).
3. Tag the commit: git tag v0.X.0.
4. Push the tag; .github/workflows/release.yml cross-compiles for six targets and publishes a draft release.

Common pitfalls

• unwrap in non-test code will fail clippy. Use ?, .expect("…") with a justification, or convert the error via From into AppError.
• Forgetting to invalidate the search cache after a document write. See crates/api/src/v1/documents.rs for the pattern.
• Using println! for logging. Use tracing::{info, warn, error} so the structured-logging pipeline picks it up.
• Adding a new env var that doesn’t go through crates/config. All configuration must be TOML-driven; env vars are an escape hatch only.

Rust API Reference

The full Rust API documentation is generated with cargo doc and lives in the target/doc/ directory of the workspace. It is regenerated on every push to main by the GitHub Actions docs workflow and published at:

https://muhammad-fiaz.github.io/AccelerateSearch/rust-api/

Build locally

cargo doc --no-deps --workspace --target-dir target
# Optional: open in your default browser
cargo doc --no-deps --workspace --target-dir target --open

RUSTDOCFLAGS="-D warnings" is set in the CI, so any doc warning fails the build.

Crate index