Small Embedding Models: Running Meaning-Based Search on a Server Without a GPU

When an AI assistant looks for the part of your documents that answers a question, it does not read every document. It compares the meaning of the question with the meaning of each stored passage. That comparison depends on embeddings: lists of numbers, called vectors, that represent the meaning of a piece of text so that passages with similar meaning end up close together. Producing embeddings is done by a model, and the model you choose affects accuracy, languages, speed, cost and privacy. Many people assume this needs an expensive GPU or a paid API. It does not. Small, open embedding models run comfortably on an ordinary server CPU, and for most small business knowledge bases they perform well. This guide explains what embeddings do, why small models are a sensible default, what you give up compared with larger ones, what quantisation means, how language coverage differs, how to choose and how to test a change safely. It also describes Crescelo's approach: a small English model that runs locally by default, with an opt-in multilingual model for workspaces that serve customers in other languages.

What an embedding model does

Given a piece of text, the model outputs a fixed-length vector, for example 384 numbers. Texts with similar meaning have vectors that point in similar directions, so the system can rank passages by similarity to the question. This is why a visitor can ask 'how much does it set me back?' and find your pricing passage, even though the words differ.

The same model must be used to embed the documents and the questions, which is why changing models means re-embedding stored content.

Why a CPU is enough

• Embedding is much lighter than generating text. It is a single pass through a small network, not a long token-by-token generation.
• Small models have tens of millions of parameters, not billions.
• A question is short, so embedding it takes a fraction of a second.
• Indexing documents happens once, in the background, and can be batched.
• Small models can be compressed (quantised) with little loss of accuracy, making them faster and lighter.

The heavy computation in a chat is the language model that writes the reply, and that runs at the AI provider. The embedding step can run on your own server without special hardware.

What quantisation means

Model weights are stored as numbers. Full precision uses 32-bit floating-point numbers. Quantisation stores them with fewer bits, such as 8-bit integers, which reduces memory and speeds up computation on ordinary CPUs at a small cost in accuracy. For embedding models the loss is usually tiny, which is why quantised versions are common for CPU deployment. ONNX is a common portable format for running such models efficiently without a heavy machine learning framework.

Small versus large models

For a service business with a few dozen pages of clear content, the difference in retrieval quality between a small and a large model is usually modest, especially when hybrid search and reranking are used.

Language coverage matters more than size

The biggest practical limit of many small models is language. An English-only model represents Spanish or Arabic text poorly, so customers writing in those languages may not find the right passages. Multilingual small models exist and handle many languages at a slightly lower peak English accuracy. Crescelo's default is a small English model. Workspaces serving multilingual customers can switch to a small multilingual model by an opt-in setting, after which stored content is re-embedded. In our own English test set, the default found the right passage among its top five results 95.3% of the time, and the multilingual option 90.7%; in a small cross-language probe the multilingual model handled Spanish and French questions against English passages better. These are results on our own test set, not universal figures, so test with your content.

How to choose

1. List the languages your customers really use.
2. Prepare twenty to fifty real questions with the passages that should answer them.
3. Measure how often the right passage appears in the top few results for each candidate model.
4. Compare speed and memory on your server.
5. Choose the smallest model that meets your accuracy needs, and record the results.

See benchmarking retrieval with simple metrics for how to run this test.

Switching models safely

• Changing the embedding model requires re-embedding all stored passages, because vectors from different models cannot be compared.
• Test on a small assistant first.
• Keep the old model available until the migration is complete, so search keeps working; Crescelo tags each stored chunk with the model that produced it and re-embeds in the background.
• Avoid a half-migrated state where some passages use the old model and some the new; mixed states give poor results.
• Re-run your test questions afterwards.

Privacy and cost benefits

Running embeddings locally means your documents and visitors' questions are not sent to an external embedding service, which simplifies your data protection story and removes a per-request cost. The language model that writes replies is a separate matter and depends on the provider you use; see bring your own AI key explained.

Common mistakes

• Assuming you need a GPU.
• Using an English-only model for multilingual customers.
• Switching models without re-embedding.
• Choosing by benchmark rankings instead of testing your own questions.
• Ignoring memory limits on small servers.

Frequently asked questions

Try retrieval on your own documents in the playground, see the knowledge base feature and pricing.

Further reading

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al.): The original research paper behind retrieval-augmented generation.
• Google Search Central: creating helpful, people-first content: What Google looks for in useful pages.