LYNX logoLYNXSearch Engine

What ships next

Roadmap

LYNX — An independent, privacy-focused web search engine — no query logs, explainable ranking, and crawler ethics as a structural constraint.

Phase 0 — Architecture

active
  • Accept ADRs 0001–0021
  • Give every open question an owner and a first experiment
  • A reviewer outside the core team can follow the design without asking what anything means

Phase 1 — Search prototype

planned
  • Query processor: lexer, parser, AST, operators, normalisation
  • SearchIndexReader trait behind an in-memory fake and a real Tantivy reader
  • BM25F ranker with field weights, phrase queries, freshness and quality
  • End-to-end /api/v1/search with the full middleware stack
  • Evaluation harness: judged query set, NDCG@10, Recall@50, MRR@10, golden rankings

Phase 2 — Crawler

planned
  • Frontier with per-host budgets and politeness tokens
  • robots.txt as a policy input, never a trust input
  • SSRF safety gate, streaming downloader caps, parser bounds
  • Extracted-text-only indexing with canonicalisation and tracking-param stripping

Phase 3 — Real search engine

planned
  • Scale the crawl with better frontier prioritisation, sitemaps and feeds
  • Link graph and domain-capped PageRank authority
  • Near-duplicate clusters via SimHash and MinHash
  • Full operator set including filetype:, intitle:, inurl: and before:/after:
  • Suggestions from corpus statistics, never from query logs
  • Multi-language detection, stopwords, stemmers and index fields

Phase 4 — Privacy hardening

planned
  • Privacy invariants enforced as tests, not as review comments
  • Ship lynx-verify-privacy as a release artefact so anyone can check the claims
  • Zero-downtime index generations with 24h rollback

Phase 5 — Astra (AI answers)

planned
  • Optional grounded answers with deterministic citation validation
  • Prompt-injection defences as an architectural property, not a prompt
  • Self-hosted Llama-3-8B path via vLLM or Ollama

Phase 6 — Scale & distribution

planned
  • Multi-shard index with documented capacity targets
  • Horizontal crawl worker scaling with a measured claim-latency trigger
  • Federated and third-party embedding path