We Deleted Pinecone From Production. Latency Went Down.

Day 9: You Probably Don't Need a Vector Database

Vector databases got pitched as a mandatory line item the moment your product had 'AI' in the name — right up there with 'you need an LLM' and 'you need an orchestration framework.' Here's the part nobody puts in the stack diagram: most RAG and agent features never come close to the scale or query complexity that would justify one. Picking your retrieval infra before you've actually looked at your query pattern is the single most common unforced error I watch teams make in week one of a retrieval project.

The deletion

A few months into running retrieval for an internal agent product, we ripped Pinecone out of production. Not a gradual migration — we deleted the index, cut the API key, and pulled the client out of the codebase in a single PR. Nothing broke. p99 latency on the retrieval call dropped by about a third, simply because we'd removed a network hop to a third-party service. The on-call pages for Pinecone rate limits and timeout spikes just stopped coming in. That's when it became obvious: we'd been paying rent on infrastructure we never needed.

Rewind: why we picked it in the first place

When we built the first version, the decision wasn't driven by a requirement — it was driven by a checklist. Every RAG blog post and every 'AI-native stack' diagram drew the same four boxes: LLM API, embedding model, vector database, orchestration framework. We copied the shape of the stack without asking whether our data or our query pattern actually needed that shape. Pinecone was the 'AI-native' choice, so it landed in the architecture doc before anyone had measured corpus size, recall requirements, or write throughput.

The tell we ignored for too long

Our corpus was about 40,000 chunks — small enough to sit comfortably in memory on a single Postgres instance. And every query we actually ran filtered hard on metadata first: tenant_id, doc_type, date range. The embedding similarity search only ever ran on top of that already-filtered subset, never across the whole corpus. Put plainly: we weren't doing vector search. We were doing search — filtered, structured, relational search — with cosine similarity bolted on as one extra scoring dimension. A dedicated ANN (approximate nearest neighbor) engine solves a different problem: ranking similarity across tens of millions of unfiltered vectors at low latency. That was never our problem.

The 10-minute query pattern audit

Before you choose retrieval infra, sit down with whoever owns the feature and run through five questions. Ten minutes now saves a migration later.

• ▹Recall target — do you need 99%+ recall (medical, legal, compliance retrieval), or is 'good enough top-k' fine because the agent re-ranks or reads multiple chunks anyway?
• ▹Corpus size — under a few million vectors, a single-node index is fast; past that, you start trading memory and index-build time for query speed.
• ▹Filter complexity — is similarity actually your primary axis, or is it a scoring dimension bolted onto tenant/date/type filters that already do most of the narrowing?
• ▹Write frequency — is this corpus append-mostly (docs, tickets), or does it churn (session memory, live agent state) in a way that stresses index rebuild/update costs?
• ▹Who else queries this data — does the same table get joined, filtered, or reported on by non-AI parts of the product? If so, a separate vector store means a second source of truth to keep in sync.

What pgvector actually cost us to adopt

The migration itself was small: add the pgvector extension, add an embedding column to the existing chunks table, backfill embeddings, add one index. The real decision was HNSW vs IVFFlat. IVFFlat is cheaper to build and works fine if your data distribution is stable — but recall degrades if you don't retrain the list count as the corpus grows. HNSW costs more memory and a longer build time, but gives better recall/latency tradeoffs without retraining. At our size — 40k rows, growing slowly — it barely mattered which one we picked. The bigger win was everything that disappeared: no separate service to monitor, no second auth/network boundary for security review, no vendor rate limits, and backups plus point-in-time recovery came free because the embeddings now lived in the same database as everything else.

-- pgvector: same table as your relational data, one extra column
ALTER TABLE chunks ADD COLUMN embedding vector(1536);

CREATE INDEX chunks_embedding_hnsw
  ON chunks USING hnsw (embedding vector_cosine_ops);

-- metadata filter runs first, similarity ranks what's left
SELECT id, content
FROM chunks
WHERE tenant_id = $1
  AND doc_type = 'policy'
  AND created_at > now() - interval '90 days'
ORDER BY embedding <=> $2
LIMIT 10;

Where a dedicated vector DB earns its keep

This isn't 'never use one.' There are real cases where a dedicated engine is the right call, and being honest about them matters just as much as the deletion story:

• ▹Billion-scale corpora, where index build/update time on a general-purpose database becomes the actual bottleneck, not just an inconvenience.
• ▹Multi-region, multi-tenant read replication at a scale your general-purpose DB's replication story was never built for.
• ▹Workloads that are genuinely pure similarity search at high QPS with almost no structured pre-filtering — the metadata-filter tell from earlier simply doesn't apply here.
• ▹Teams that need dedicated ANN tuning knobs — quantization, hybrid sparse+dense search, custom distance metrics — as a core product differentiator, not a nice-to-have.

The checklist for your next 'we need a vector database' meeting

• ▹What's our actual corpus size and growth rate — not what we expect in three years, what it is today?
• ▹What fraction of our queries are similarity-first versus metadata-filtered-first?
• ▹What does adding a second data store actually cost us — in sync overhead, security review, on-call surface — and is that cost justified by a performance need we've measured, or by a stack diagram we copied?

Where this fits in the 30-day arc

Tomorrow we build the retrieval layer on top of whatever storage you land on today — chunking strategy, embedding choice, and the actual retrieve-then-rank pipeline your agent calls. That's where this decision compounds: the storage choice you make today decides which queries are cheap tomorrow and which are expensive, and retrofitting a storage swap after the retrieval layer is already built is exactly the migration this lesson is meant to help you skip.

Extend your knowledge

• ▹Read the pgvector README on GitHub (pgvector/pgvector) for the current index types and distance operators before your next migration.
• ▹On SQLite? Look at sqlite-vec — the equivalent 'don't add a service' option for smaller or embedded deployments.
• ▹Run the 10-minute query pattern audit from this lesson against your current or planned RAG feature before your next infra meeting.
• ▹Tomorrow: building the retrieval layer — chunking, embedding choice, retrieve-then-rank — on top of the storage decision you make today.