Redis vector search

pip install "needtoknow[redis]"

The backend runs a KNN query on the Redis Query Engine with the permission filter as part of the same query, so the k nearest chunks the subject may read come back however few of the index’s chunks that is. It needs Redis 8 or later, where the Query Engine is built in, or Redis Stack. CI runs needtoknow’s conformance kit against Redis 8.10.2. The same needtoknow[redis] extra serves the shared permission cache.

import redis

from needtoknow.backends.redis import RedisBackend, create_index

client = redis.Redis.from_url(REDIS_URL)
create_index(client, "chunks", dim=768)  # hashes under "needtoknow:", HNSW, cosine
backend = RedisBackend(client, "chunks", embed=embed)  # embed: str -> list[float]
backend.add(documents)

Each chunk is a hash under the prefix (needtoknow: plus the document id) with the fields needtoknow_id, content, metadata (JSON), needtoknow_terms and embedding (float32). A write is searchable as soon as it returns; replacing a chunk replaces the whole hash. ef_runtime sets HNSW’s search breadth. Scores are cosine similarity, 1 minus Redis’s cosine distance. For a native aretrieve, pass async_client=redis.asyncio.Redis(...); otherwise async calls run in a thread.

HNSW is approximate. Redis indexes a hash again whenever any of its fields changes, so set_metadata removes and re-adds the chunk’s vector in the graph, and a small graph that changes often can miss a chunk; it never returns one the filter excludes. create_index(..., algorithm="FLAT") compares the query with every vector instead: exact, and fast enough up to some tens of thousands of chunks. CI checks both, exactly on FLAT.

How permissions are indexed

The backend uses the same type-tagged terms as Elasticsearch, OpenSearch and Milvus, so 1 and "1" stay apart and booleans, floats, nested lists and objects never match. Redis TAG queries have their own syntax (braces, |, @, *), and TAG matching ignores case by default, so each term is stored hex-encoded:

from needtoknow import In
from needtoknow.backends.redis import filter_to_query

filter_to_query(In("allowed_groups", ["hr"]))
# '@needtoknow_terms:{616c6c6f7765645f67726f7570731f733a6872}'

A hex string holds only 0-9 and a-f, so no group name or user id can add query syntax, and "HR" and "hr" stay different values. CI checks both on a real server. And becomes an intersection and Or a union of TAG conditions, and an In with no values matches nothing without a call to Redis. The metadata is stored as JSON and returned exactly as written, so the retriever’s re-check sees the true permissions.

Hashes written by other tools

A hash without needtoknow_terms is invisible to a pre-filtered search, though post-filtering still finds it and checks its metadata. Fill the field in your ingestion pipeline:

from needtoknow.backends.redis import permission_terms

client.hset(key, mapping={"needtoknow_terms": permission_terms(metadata)})

Keyword search and hybrid search

mode="keyword" searches the chunks’ text with BM25 under the same permission filter. The index must hold the text: create it with content_field=, which keeps every word (the index has no stop words). One index then serves both halves of hybrid search:

from needtoknow import RankFusion

create_index(client, "chunks", dim=768, content_field="content")
vector = RedisBackend(client, "chunks", embed=embed)
keyword = RedisBackend(client, "chunks", mode="keyword")  # needs no embed
retriever = PermissionedRetriever(RankFusion(vector, keyword), authorizer)

The query is split into words (letters, digits and underscores), and each word is sent as a query parameter rather than written into the query, so no query text can add syntax: the permission condition always applies. A query keeps its first 64 distinct words, and any of them may match. Scores are Redis’s BM25 (BM25STD), higher for better matches. A keyword backend without embed writes chunks with no vector, which keyword search finds and vector search does not.

An existing index can gain the text field with FT.ALTER chunks SCHEMA ADD content TEXT, and Redis indexes the stored hashes in the background. An index created by an earlier needtoknow keeps Redis’s default stop words, so common English words in a query match nothing there. CI runs the conformance kit against Redis 8.10.2 in keyword mode and fused with vector search.