Inference on Qdrant - Vector Search Engine

Inference API

info@qdrant.tech (Andrey Vasnetsov) — Mon, 01 Jan 0001 00:00:00 +0000

Inference API

In Qdrant’s API, wherever you can use regular vectors, you can also use Inference Objects. An Inference Object is an object that contains the necessary information to generate a vector embedding from an input, such as text or an image, using a specified embedding model. The API supports three types of Inference Objects:

Document object, used for text inference

// Document
{
 // Text input
 text: "Your text",
 // Name of the model, to do inference with
 model: "<the-model-to-use>",
 // Extra parameters for the model, Optional
 options: {}
}

Image object, used for image inference

BM25

info@qdrant.tech (Andrey Vasnetsov) — Mon, 01 Jan 0001 00:00:00 +0000

Server-side Inference: BM25

BM25 (Best Matching 25) is a ranking function for text search. BM25 uses sparse vectors that represent documents, where each dimension corresponds to a word. Qdrant can generate these sparse embeddings from input text directly on the server.

While upserting points, provide the text and the qdrant/bm25 embedding model:

PUT /collections/{collection_name}/points
{
 "points": [
 {
 "id": 1,
 "vector": {
 "my-bm25-vector": {
 "text": "Recipe for baking chocolate chip cookies",
 "model": "qdrant/bm25"
 }
 }
 }
 ]
}

from qdrant_client import QdrantClient, models

client.upsert(
 collection_name="{collection_name}",
 points=[
 models.PointStruct(
 id=1,
 vector={
 "my-bm25-vector": models.Document(
 text="Recipe for baking chocolate chip cookies",
 model="Qdrant/bm25",
 )
 },
 )
 ],
)

import { QdrantClient } from "@qdrant/js-client-rest";

client.upsert("{collection_name}", {
 points: [
 {
 id: 1,
 vector: {
 'my-bm25-vector': {
 text: 'Recipe for baking chocolate chip cookies',
 model: 'Qdrant/bm25',
 },
 },
 },
 ],
});

use qdrant_client::{
 Payload, Qdrant,
 qdrant::{DocumentBuilder, PointStruct, UpsertPointsBuilder},
};
use std::collections::HashMap;

client
 .upsert_points(UpsertPointsBuilder::new(
 "{collection_name}",
 vec![PointStruct::new(
 1,
 HashMap::from([(
 "my-bm25-vector".to_string(),
 DocumentBuilder::new("Recipe for baking chocolate chip cookies", "qdrant/bm25")
 .build(),
 )]),
 Payload::default(),
 )],
 ))
 .await?;

import static io.qdrant.client.PointIdFactory.id;
import static io.qdrant.client.ValueFactory.value;
import static io.qdrant.client.VectorFactory.vector;
import static io.qdrant.client.VectorsFactory.namedVectors;

import io.qdrant.client.QdrantClient;
import io.qdrant.client.QdrantGrpcClient;
import io.qdrant.client.grpc.Points.Document;
import io.qdrant.client.grpc.Points.Image;
import io.qdrant.client.grpc.Points.PointStruct;
import java.util.List;
import java.util.Map;

 client
 .upsertAsync(
 "{collection_name}",
 List.of(
 PointStruct.newBuilder()
 .setId(id(1))
 .setVectors(
 namedVectors(
 Map.of(
 "my-bm25-vector",
 vector(
 Document.newBuilder()
 .setModel("qdrant/bm25")
 .setText("Recipe for baking chocolate chip cookies")
 .build()))))
 .build()))
 .get();

using Qdrant.Client;
using Qdrant.Client.Grpc;

await client.UpsertAsync(
 collectionName: "{collection_name}",
 points: new List<PointStruct>
 {
 new()
 {
 Id = 1,
 Vectors = new Dictionary<string, Vector>
 {
 ["my-bm25-vector"] = new Document()
 {
 Model = "qdrant/bm25",
 Text = "Recipe for baking chocolate chip cookies",
 },
 },
 },
 }
);

import (
	"context"

	"github.com/qdrant/go-client/qdrant"
)

client.Upsert(context.Background(), &qdrant.UpsertPoints{
	CollectionName: "{collection_name}",
	Points: []*qdrant.PointStruct{
		{
			Id: qdrant.NewIDNum(uint64(1)),
			Vectors: qdrant.NewVectorsMap(map[string]*qdrant.Vector{
				"my-bm25-vector": qdrant.NewVectorDocument(&qdrant.Document{
					Model: "qdrant/bm25",
					Text: "Recipe for baking chocolate chip cookies",
				}),
			}),
		},
	},
})

Qdrant uses the model to generate the embeddings and stores the point with the resulting vector. Retrieving the point shows the embeddings that were generated:

Cloud Inference

info@qdrant.tech (Andrey Vasnetsov) — Mon, 01 Jan 0001 00:00:00 +0000

Qdrant Cloud Inference

Clusters on Qdrant Managed Cloud can access embedding models that are hosted on Qdrant Cloud. For a list of available models, visit the Inference tab of the Cluster Detail page in the Qdrant Cloud Console. Here, you can also enable Cloud Inference for a cluster if it’s not already enabled.

Several embedding models can be used for free with Qdrant Cloud Inference, also in combination with clusters on the Qdrant Cloud free tier. Free models are identified by the “Cost: Free” label in the Inference tab of the Cluster Detail page.

External Providers

info@qdrant.tech (Andrey Vasnetsov) — Mon, 01 Jan 0001 00:00:00 +0000

External Embedding Model Providers

Qdrant Cloud can act as a proxy for the APIs of external embedding model providers:

OpenAI
Cohere
Jina AI
OpenRouter

This enables you to access any of the embedding models provided by these providers through the Qdrant API.

To use an external provider’s embedding model, you need an API key from that provider. For example, to access OpenAI models, you need an OpenAI API key. Qdrant does not store or cache your API keys; they must be provided with each inference request.

Matryoshka Models

info@qdrant.tech (Andrey Vasnetsov) — Mon, 01 Jan 0001 00:00:00 +0000

Reduce Vector Dimensionality with Matryoshka Models

Matryoshka Representation Learning (MRL) is a technique used to train embedding models to produce vectors that can be reduced in size with minimal loss of information. On Qdrant Cloud, for supported models, you can specify the mrl parameter in the options object to reduce the vector size to the desired dimension.

MRL on Qdrant Cloud helps minimize costs and latency when you need multiple sizes of the same vector. Instead of making several inference requests for each vector size, the inference service only generates embeddings for the full-sized vector and then reduces the vector to each requested smaller size.