Milvus#

pip install "needtoknow[milvus]"

The backend runs Milvus’s vector search with the permission filter applied inside the search, so the k nearest chunks the subject may read come back however few of the collection’s chunks that is. It needs pymilvus 2.6 or newer and a Milvus that supports filter templating. CI runs needtoknow’s conformance kit against Milvus 3.0.2 standalone. Zilliz Cloud serves the same API through the same client, but CI does not test it.

from pymilvus import MilvusClient

from needtoknow.backends.milvus import MilvusBackend, create_collection

client = MilvusClient(uri="http://localhost:19530")  # or Zilliz Cloud: uri=..., token=...
create_collection(client, "docs", dim=768)
backend = MilvusBackend(client, "docs", embed=embed)  # embed: str -> list[float]
backend.add(documents)

create_collection makes a collection with a string primary key (id), the vector (embedding, HNSW, cosine), the text (content), the metadata (metadata, JSON) and the permission terms (needtoknow_terms, with an inverted index). Field names, the metric (COSINE or IP; scores are Milvus’s similarity, higher is closer) and the HNSW parameters can be changed; pass the same field names to MilvusBackend.

Tuning#

ef (default 100) is HNSW’s search breadth: higher finds more of the true nearest chunks and takes longer. It is raised to k when k is larger, as Milvus requires.
search_params replaces the search parameters entirely, for a collection with another index type, for example {"params": {"nprobe": 16}} for IVF.
consistency_level (default "Strong") makes a chunk searchable as soon as add returns. "Bounded" is faster when a short delay after writes is acceptable; it never affects which chunks a subject may read, only how soon new ones appear.
For a native aretrieve, pass async_client=AsyncMilvusClient(...); otherwise async calls run in a thread.

How permissions are indexed#

The backend uses the same design as Elasticsearch and OpenSearch: one array field, needtoknow_terms, holds a type-tagged term for every string and integer in the chunk’s metadata, so 1 and "1" stay apart and booleans, floats, nested lists and objects never match. A filter becomes ARRAY_CONTAINS_ANY conditions on that field, joined with AND and OR, and an In with no values matches nothing without a call to Milvus.

Values never enter the expression. Milvus filters are text expressions. needtoknow writes only placeholders into the expression and passes the values separately, as Milvus’s filter templating (filter_params) allows:

from needtoknow import In
from needtoknow.backends.milvus import filter_to_expression

filter_to_expression(In("allowed_groups", ["hr", "eng"]))
# ('ARRAY_CONTAINS_ANY(needtoknow_terms, {p0})', {'p0': ['allowed_groups\x1fs:eng', 'allowed_groups\x1fs:hr']})

A group name or user id containing quotes, brackets or operators is therefore only ever a value, and cannot change what the filter selects. CI checks this against a real Milvus.

The metadata itself is stored in the JSON field and returned as written, so the retriever’s re-check sees the true permissions. The terms field is not consulted by the re-check: if the two ever disagree, a chunk can be hidden from a pre-filtered search, never leaked.

Limits#

max_terms (default 4,096) caps the tagged terms per chunk, one per string or integer in its metadata; a chunk shared with more groups than that needs a larger value, within what your Milvus allows for an array field. Inserting a chunk with more fails, so nothing is stored with partial permissions.
max_term_length (default 1,024) caps one term: the field name, two characters and the value.
max_content_length (default 65,535, Milvus’s largest VARCHAR) caps the chunk text.
max_id_length (default 512) caps document ids.

Collections written by other tools#

A row whose needtoknow_terms is empty is invisible to a pre-filtered search, though post-filtering still finds it and checks its metadata. Fill the field in your ingestion pipeline with permission_terms:

from needtoknow.backends.milvus import permission_terms

row = {"id": doc_id, "embedding": vector, "content": text, "metadata": metadata}
row["needtoknow_terms"] = permission_terms(metadata)
client.upsert(collection_name="docs", data=[row])

Keyword search and hybrid search#

mode="keyword" searches the chunks’ text with BM25, under the same permission filter and its parameters. Milvus computes BM25 from a sparse vector it fills from the text itself, which needs a function in the schema when the collection is created: create_collection(..., text_search=True) adds it, with Milvus’s standard analyzer, which keeps every word. One collection then serves both halves of hybrid search:

from needtoknow import RankFusion

create_collection(client, "docs", dim=768, text_search=True)
vector = MilvusBackend(client, "docs", embed=embed)
keyword = MilvusBackend(client, "docs", mode="keyword")
retriever = PermissionedRetriever(RankFusion(vector, keyword), authorizer)

The query’s text is sent as the search data, separate from the filter expression, so it cannot change what the filter selects. Scores are Milvus’s BM25, higher for better matches. Milvus needs every chunk’s vector, so write chunks through a backend with embed. A collection created without text_search cannot gain it later; create a new one and load it again. CI runs the conformance kit against Milvus 3.0.2 in keyword mode and fused with vector search.