How to Add Semantic Search to an Existing Product
Last updated October 2026
The short answer: leave your keyword search in place and run vector retrieval next to it. Embed your records with a hosted model, keep the vectors in a database you already operate, apply the same permission filters to both paths, combine the two result lists with Reciprocal Rank Fusion, and release the whole thing behind a feature flag.
Why do semantic search retrofits go wrong?
Most tutorials picture a brand-new app with a single table, no tenants and no search box that customers already depend on. Your situation is different. You have keyword search that works, records that change by the minute, and access rules that decide who can see what. Calling the embedding API is the simple bit. The real work is keeping the vector index in step with your source of truth, guaranteeing that no query ever surfaces a record the user isn’t allowed to see, and proving the new results beat the old ones before you switch the old ones off.
The second classic mistake is ripping out keyword search altogether. Semantic retrieval understands meaning well but handles exact strings poorly. In its explainer on hybrid search, Redis says semantic search “can struggle with precise identifiers like product codes, API names, and technical terminology that need exact matching.” When people look things up by SKU, invoice number, error code or customer name, a vector-only search makes the product feel worse from the first day. Hybrid search solves that, and for a retrofit it should be the starting design.
What do you need to add semantic search to a product you already run?
There are five pieces. In most products only one of them counts as new infrastructure, and sometimes none of them do.
| Component | Practical default | Concrete value to plan around |
|---|---|---|
| Embedding model | Hosted API model | OpenAI text-embedding-3-small: $0.02 per 1M tokens, 1,536 dimensions by default, 8,192 max input tokens |
| Vector storage | pgvector if you already run Postgres | HNSW and IVFFlat indexes support the vector type up to 2,000 dimensions |
| Keyword retrieval | Your existing full-text search | Unchanged; it becomes one of two ranked lists |
| Fusion | Reciprocal Rank Fusion | Score = 1 / (k + rank) per list, k = 60 by convention |
| Reranker (optional) | Cross-encoder or hosted rerank API | Rerank the top 150 fused results, return the top 20 |
Those embedding figures come from OpenAI’s embeddings guide, which also puts text-embedding-3-small at roughly 62,500 pages per dollar, assuming about 800 tokens per page. The bigger cost questions sit at query time and whenever you re-embed, and we walk through those in LLM cost control patterns.
If Postgres is already in your stack, try pgvector before you shop for a dedicated vector database. The pgvector README spells out the index tradeoff: HNSW gives a better balance of speed and recall than IVFFlat, at the price of slower builds and more memory, while IVFFlat builds faster and uses less memory but queries more slowly. Keeping vectors beside the rows they describe gives you one backup, one permission model and no sync job running between two databases.
How do you add semantic search to an existing product, step by step?
Here is the order we follow when we retrofit search into a product that’s already live. Every step can ship on its own, and customers see no change until step 7.
- Gather the queries that fail today. Pull 60 to 90 days of search logs and mark zero-result queries, queries the user immediately rewrote, and queries that got no clicks. That list becomes both your test set and your business case.
- Define what a “document” is. It could be a support article, a product, a ticket, or one paragraph inside any of those. Long records need chunking, while short ones usually don’t. Record which fields get embedded (title, body, tags) and which remain keyword-only (IDs, codes, names).
- Add an embedding column and a backfill job. Save the model name and version alongside every vector. Switching models later means re-embedding everything, and you’ll need to know which rows are out of date.
- Keep embeddings current on every write. Connect record creation and updates to a queue that re-embeds the rows that changed. Embed the text rather than the row ID, and skip the API call when none of the embedded fields changed.
- Put permissions inside the query. Tenant ID, workspace and role filters have to run on the vector path exactly the way they run on the keyword path. Don’t filter permissions in application code after retrieval.
- Merge the two lists with RRF. Run keyword and vector retrieval in parallel, take the top 50 to 100 results from each, and combine them by rank.
- Ship it behind a flag and compare. Send a slice of traffic to the hybrid results and track click-through, zero-result rate and reformulation rate against the control group.
Step 5 needs extra care. The pgvector README cautions that “with approximate indexes, filtering is applied after the index is scanned,” which means a heavily filtered query can come back with fewer results than you requested. Version 0.8.0 introduced iterative index scans that automatically read further into the index when they need to. In a multi-tenant product where any one tenant is a small slice of the table, switch those on or partition by tenant. Otherwise your best customers will quietly get half-empty search results.
How does Reciprocal Rank Fusion combine keyword and vector results?
BM25 scores and vector distances sit on different scales, so adding them together makes no sense. RRF sets the raw scores aside and works from each result’s position. According to the ParadeDB RRF reference, every branch contributes its weight divided by (k + rank), those contributions are summed, and 60 is the usual value for k. ParadeDB recommends a lower k, 20 to 40, when one branch is much stronger than the other, and 80 to 100 when the two are about even.
Take an example. A help article comes 3rd in vector search and 1st in keyword search, so its fused score is 1/63 + 1/61, roughly 0.0323. Another article ranks 1st in vector search but never shows up in the keyword list, so it scores 1/61, roughly 0.0164. When both methods agree, that result wins, which is exactly how a product search box should behave. RRF needs no training data and no score normalization, which makes it the right choice for version one.
How much better does hybrid search actually perform?
How much you gain depends on your corpus, so test against your own failed queries. The most helpful public figures come from Anthropic’s contextual retrieval write-up, which measured top-20 retrieval failure rates over several knowledge bases:
| Retrieval setup | Top-20 failure rate | Reduction vs. baseline |
|---|---|---|
| Baseline embeddings | 5.7% | None |
| Contextual embeddings | 3.7% | 35% |
| Contextual embeddings plus contextual BM25 | 2.9% | 49% |
| Plus reranking | 1.9% | 67% |
Two takeaways carry straight over to a retrofit. Adding BM25 on top of embeddings was among the biggest single gains, which backs the case for keeping keyword search instead of replacing it. And putting a reranker over hybrid retrieval reduced failures further still. In that setup, the first retrieval step pulled 150 chunks for reranking and handed the top 20 onward. The write-up also reports that generating the contextualized chunks was a one-time cost of $1.02 per million document tokens.
How do you know semantic search is ready to ship?
Don’t judge relevance by typing in five searches whose answers you already know. Build the evaluation set before you build the feature, as we explain in how to write evals for an AI feature. This is the minimum we expect before widening a rollout:
- A labeled set of at least a few hundred real queries, covering the failed ones from step 1 plus exact-match queries such as SKUs and IDs.
- Zero regressions on exact-match queries. When a customer searches for an invoice number, that invoice is the first result every single time.
- A lower zero-result rate and higher click-through in the flagged group than in the control.
- A permission test suite that runs each query as users from different tenants and roles, and fails if any record leaks.
- p95 latency that stays within the budget your search box already has. Embedding the query adds a network round trip, so cache embeddings for common queries.
A lot of the problems that appear later, like stale embeddings, chunk boundaries that cut the answer in two, and recall that silently drops after a model change, are the same ones we cover in RAG failure modes in production. Semantic search is the retrieval half of RAG, so the same monitoring applies.
How long does it take to add semantic search to a product?
Say your product runs on Postgres, already has a search box, and holds a corpus in the low millions of records. Plan for four phases: roughly a week for query log analysis and the evaluation set, one to two weeks for the embedding column, backfill and sync pipeline, a week for hybrid fusion and permission tests, and one to two weeks of flagged rollout. That fits within the 30-day plan in our pillar on shipping your first production AI feature in 30 days.
Schedules slip when the sync pipeline gets treated as an afterthought, or when permissions are tacked on after a demo. If you’d rather have a fixed price than an estimate, our $24,500 AI Integration Sprint is designed for exactly this sort of scoped feature, and you don’t pay the final milestone if delivery misses day 30. The engineers who run it have 10+ years of experience on average.
Frequently asked questions
Should semantic search replace keyword search?
No. Keep keyword search and run vector search next to it. Keyword search covers exact identifiers such as SKUs, codes and names, and vector search covers meaning and synonyms. Combine the two ranked lists with Reciprocal Rank Fusion.
Do I need a separate vector database?
Not at the start, usually. If you already run Postgres, pgvector keeps embeddings beside your existing rows and supports HNSW indexes. A dedicated vector database starts to pay off at very large scale or when you have specialized filtering needs.
How much does it cost to embed an existing corpus?
For most products, hardly anything. OpenAI prices text-embedding-3-small at $0.02 per million tokens and estimates about 62,500 pages per dollar. Re-embedding after model changes, plus query-time costs, matter more than the initial backfill.
What is the biggest risk when adding semantic search?
Records leaking across permission boundaries. Put tenant and role filters inside the vector query, test them with users from different tenants, and watch for approximate indexes that return too few results when filters are heavy.
Book a Code Review
If semantic search is on your roadmap, the riskiest code isn’t the model. It’s everything around it: sync jobs, permission filters and fallbacks. Having a senior engineer review your data model and search path before you build shows you where those risks are. Book a Code Review.