Hybrid search, reranking, vector stores & permissions

The online path: embed query → retrieve candidates → rerank → assemble context → generate → cite. Two upgrades separate a demo from production: hybrid search and a reranker.

Hybrid search + rerank

• Dense (vector) search captures semantics; sparse (BM25/keyword) captures exact terms — product codes, names, acronyms. Combine them (hybrid) and fuse scores (e.g. Reciprocal Rank Fusion).
• Retrieve a wide candidate set (top-50/100), then apply a cross-encoder reranker to reorder and keep the top-5. Rerankers read the query+chunk together, so they're far more precise than the first-stage vector score.
• This retrieve-wide-then-rerank-narrow funnel is the single most impactful quality upgrade in RAG.

Vector stores

Options: pgvector (reuse Postgres, great to start), Pinecone/Weaviate/Qdrant/Milvus (purpose-built, scale + filtering), or Elastic/OpenSearch (hybrid built in). Under the hood most use ANN indexes (HNSW) — approximate nearest neighbor, trading a little recall for big speed. Pick based on scale, existing infra, and metadata-filtering needs — don't fixate on the brand. And say whether the corpus needs a vector index at all: for navigable, structured corpora (code, filesystems), an agent driving keyword/grep search as a tool is often competitive or better — vectors earn their keep on large, fresh, permissioned, unstructured corpora.

Permission-filtered retrieval

In any real enterprise system, retrieval must respect access control. The reflex answer — index everything, filter later — leaks data. Correct pattern:

1. Store ACLs / tenant IDs / security labels as metadata on every chunk at ingestion.
2. Filter at query time by the caller's identity — ideally as a pre-filter inside the vector search, so forbidden docs never enter the candidate set.
3. Re-check permissions on the final chunks before they hit the prompt (documents get re-shared/revoked).