Vector Databases for RAG:
How They Work, When You Need One & How to Choose

REFERENCE GUIDE

What is a vector database for RAG?

A vector database for RAG stores embeddings and their document identifiers, supports similarity retrieval and often stores the metadata required for policy filters. In retrieval-augmented generation, a user query is converted into a compatible embedding, relevant allowed chunks are retrieved and the language model uses this evidence when producing an answer. The database does not itself verify whether the generated answer is factual.

Think of an engineering manual: a question like “Why are requests slowing after we hit our quota?” might retrieve a chunk titled “API rate-limit saturation.” Semantic vectors help when words differ. A separate lexical signal can be much stronger for exact terms such as ERR_API_QUOTA_17. This is why real systems should benchmark exact, lexical, semantic and hybrid retrieval rather than assuming vectors solve every problem.

REFERENCE GUIDE

Vector database vs SQL database: when is PostgreSQL enough?

The phrase vector database vs traditional database is misleading if it suggests that SQL cannot search vectors. The open-source pgvector extension adds vector types and exact or approximate nearest-neighbor searching inside PostgreSQL. PostgreSQL also provides full-text search, joins, transactions and row-level security. Test what the existing stack can do before adding a new service.

REFERENCE GUIDE

Vector store vs vector database: definitions are not guarantees

Vector store is a broad term for a persistence and retrieval component. Vector database generally indicates searchable vectors with database-like operations. Vendor naming is not a specification: ask for persistence guarantees, index consistency, deletion semantics, tenancy, backup/restore, migration, metrics and service limits.

A hosted “vector store” can be operationally more complete than a small self-managed “vector database.” Conversely, an attractive vector API is not a guarantee that your compliance, ACL and disaster recovery constraints are satisfied. Require evidence for the deployed version and plan, not a generic feature checklist.

REFERENCE GUIDE

An end-to-end RAG reference architecture: from document to verified answer

1. Ingest: extract documents, capture source revision, owners, retention policy and stable identifiers.
2. Chunk: preserve headings, citations, adjacent context and canonical chunk IDs; tune chunking on real queries.
3. Embed: pin the embedding model, dimensions, normalization and distance metric. Store a model version with each chunk.
4. Index: insert vectors with text, tenant/ACL metadata, timestamps and deletion information.
5. Authorize: derive the allowed document scope from trusted application identity before executing retrieval.
6. Retrieve: run dense and, where useful, lexical queries over allowed material. Fuse and optionally rerank a deep enough candidate set.
7. Generate and cite: supply only authorized evidence; require abstention when relevant support is unavailable.
8. Evaluate: measure retrieval recall, relevance, citation correctness, update visibility and answer faithfulness separately.

Choosing RAG itself versus fine-tuning or long context is covered in RAG vs Fine-Tuning vs Long Context. Choosing the embedding model is covered in Best Embedding Models for RAG. This guide addresses the missing retrieval infrastructure layer.

REFERENCE GUIDE

How vector search works: cosine distance, HNSW and IVFFlat

Vectors are compared using a metric such as cosine distance, Euclidean distance or inner product. Those metrics are not interchangeable without considering vector normalization and the training objective. Exact search evaluates the requested eligible candidates and provides an algorithmic reference. Approximate nearest-neighbor (ANN) methods accelerate search by changing the work performed, possibly sacrificing some recall.

The foundational HNSW research is Malkov and Yashunin. pgvector documents exact defaults and separately configurable HNSW/IVFFlat indexes. Selective predicates can influence filtered ANN results: benchmark within the authorized candidate set instead of treating global top-k as guaranteed filtered top-k.

CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE rag_chunks (
  chunk_id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  tenant_id TEXT NOT NULL,
  source_id TEXT NOT NULL,
  content TEXT NOT NULL,
  embedding vector(3) NOT NULL
);
CREATE INDEX rag_chunks_tenant ON rag_chunks(tenant_id);
CREATE INDEX rag_chunks_ann ON rag_chunks
  USING hnsw (embedding vector_cosine_ops);

-- :tenant_id must come from authenticated application context.
SELECT chunk_id, source_id
FROM rag_chunks
WHERE tenant_id = :tenant_id
ORDER BY embedding <=> '[0.6,0.2,0.7]'
LIMIT 5;

REFERENCE GUIDE

Hybrid search for RAG: combine semantic retrieval, BM25 and RRF

Hybrid search combines vector-based semantic candidates with lexical or sparse matches. Error codes, part numbers and version identifiers often benefit from keyword search, whereas paraphrased questions benefit from a strong embedding model. Qdrant, Weaviate, Pinecone and Milvus publish different hybrid-search interfaces; those differences must be evaluated with the same labeled queries.

Reciprocal Rank Fusion (RRF) merges rankings by summing rank-based contributions; one common equation is RRF(d) = Σ 1/(k + rank(d)). The constant, candidate depth and product defaults vary. Never directly add raw BM25 scores to cosine similarity scores without normalization: their scales are different. Both retrieval branches must enforce the same authorization scope.

def rrf(rankings, constant=60):
    scores = {}
    for ranking in rankings:
        for rank, document_id in enumerate(ranking, 1):
            scores[document_id] = (
                scores.get(document_id, 0) + 1/(constant + rank)
            )
    return sorted(scores, key=lambda d: (-scores[d], d))

# Both input rankings must already be ACL scoped.

REFERENCE GUIDE

Filtering, tenancy and access control: why query order matters

One misleading retrieval design is global vector top-k followed by tenant post-filtering. The global top-k can contain only other-tenant candidates; dropping them afterwards may yield zero results even when several relevant authorized records exist. An appropriately scoped search, or a database's documented filtered ANN strategy, is needed. Vendor filtering implementations differ, so verify correctness and access controls empirically.

Tenant metadata supplied by the language model is not an authority signal. Your application must bind identities, filter candidates before exposing text, and test deletions and changed permissions. PostgreSQL row-level security and vector-engine server-side filters each have distinct security semantics; neither removes the need for threat modeling.

ORIGINAL SXF RESEARCH

SXF original experiment: 240 synthetic vectors and 40 filtered queries

We constructed 240 two-dimensional synthetic vectors (120 Red, 120 Blue) and evaluated 40 generated query vectors at k=5 (20 per tenant). Red points are deliberately closer to the query angles; the experiment isolates whether filter placement loses eligible neighbors. Two strategies were compared against a tenant-scoped exact top-five ground truth.

We executed this exact algorithm locally and verified these outcomes. The Blue query still has 120 eligible documents, but global top-five selects Red neighbors. Post-filtering then discards all five. This demonstrates candidate starvation, not a general property of every real vector engine or an estimate of the probability of failure.

VENDOR-NEUTRAL SELECTION

pgvector vs Qdrant vs Weaviate vs Pinecone vs Milvus

This comparison summarizes documented features, not a ranking of measured speed. Pricing, availability, service configurations, index defaults and cloud regions vary. Always verify a vendor's current documentation and your plan before procurement.

“Best” depends on your workload. Database selection cannot be established from a static list of vector counts or a single vendor-provided benchmark. Consider the number of filtered eligible candidates, query distribution, write/delete rate, concurrent traffic, operator experience, residency and true cost per accepted answer.

INTERACTIVE BUYER TOOL

Interactive SXF Vector Database Decision Matrix

Select the constraints of your retrieval workload. This tool builds an evaluation plan without pretending to measure vendor speed, assigning sponsored scores or choosing a universal winner.

REFERENCE GUIDE

How to benchmark vector databases fairly

A credible vector database evaluation requires the same embedding model, corpus, chunking, query labels, index objective and authorization scope for every engine. Different hardware/service tiers or cold/warm cache profiles are not directly comparable without explicit disclosure.

1. Define representative tasks and relevance judgments before selecting a vendor.
2. Freeze document IDs, embeddings, dimension, metric, indexing defaults and data revisions.
3. Establish tenant-scoped exact ground truth and separately annotate human-relevant documents.
4. Run lexical, semantic, selective-filter, multilingual, no-answer and exact identifier tests.
5. Measure warm/cold cache, concurrent reads, writes, deletes, reindexing and recovery.
6. Record configurations, failures, timeouts, distributions and costs instead of one headline score.
7. Repeat the benchmark when changing embedding models or data distributions.

REFERENCE GUIDE

Vector database pricing, sizing and total cost

Raw vector payload is only the start of the capacity model. For one million vectors with 1,536 float32 dimensions, raw payload is 6.144 GB (decimal) before index structures, text, metadata, copies, replicas, cache, backup and memory allocator overhead. Compression can reduce payload while changing accuracy and index needs. Vendor pricing must be checked per service tier and usage shape.

For model-level inference spend, use the SXF AI Model Cost Calculator. The important unit is a verified, authorized answer—not cost per vector alone.

REFERENCE GUIDE

Production operations: versioning, indexing, deletion, backups and observability

Maintain an explicit chain from source document → chunk ID → embedding version → vector ID. Without it, permission changes and embedding migrations are hard to reconcile. Treat deletions and ACL updates as first-class correctness events, not delayed maintenance.

For the wider deployment envelope—including queues, state and release management—see How to Deploy AI Agents in Production. For when agent memory should be retained, reviewed and forgotten, see AI Agent Memory.

REFERENCE GUIDE

Four practical vector RAG architecture patterns

These are patterns, not a ranking. The right answer can also be no vector database when deterministic SQL or keyword retrieval solves user needs more reliably and cheaply.

REFERENCE GUIDE

Migrating vector databases without breaking retrieval

1. Export document IDs, source versions, embeddings, metadata and ACL model—not vectors alone.
2. Pin the existing embedding model, dimension, similarity function and chunking contract.
3. Build the target index and replay in-flight updates and deletes.
4. Shadow the same authorized queries through both systems and compare exact recall and human relevance.
5. Validate tail latency, search cost, deletion and tenant isolation at expected load.
6. Canary by cohort and keep a rollback route for outstanding index updates.
7. Delete old replicas only after access and retention requirements are verified.

Exportability and the cost of leaving a managed system matter in long-lived enterprise deployments. Document the exit plan before signing a commercial agreement.

REFERENCE GUIDE

Ten common mistakes when choosing a vector database for RAG

1. Buying a specialized database before measuring a PostgreSQL/exact baseline.
2. Assuming semantic search outperforms literal retrieval for every query.
3. Comparing providers with different embeddings, hardware or source documents.
4. Applying tenant filtering after an unrestricted global top-k search.
5. Trusting a model to determine its own document permissions.
6. Changing embedding versions without rebuilding and validating the index.
7. Confusing geometric ANN recall with user answer correctness.
8. Ignoring update/deletion lag, stale chunks and source provenance.
9. Reporting only p50 latencies from toy data with a warm cache.
10. Calling vector storage price the entire retrieval system cost.

REFERENCE GUIDE

Vector database for RAG: frequently asked questions

REFERENCE GUIDE

Primary sources, experiment disclosure and editorial methodology

Capabilities in this guide are grounded in official documentation, Postgres and pgvector primary sources and published algorithmic research. We have not conducted comparative hosted vendor benchmarks; no latency, recall superiority, uptime or actual provider pricing is implied by the table. The original SXF demonstration tests only filtered retrieval order in synthetic exact dot-product search. Our evaluation worksheet is a template, not a dataset of measured provider outcomes.

SXF INTERNAL LINKS

Continue through the SXF retrieval and engineering library