Redis vector search#
pip install "wardowl[redis]"
The backend runs a KNN query on the Redis Query Engine with the permission filter as part of the
same query, so the k nearest chunks the subject may read come back however few of the index’s
chunks that is. It needs Redis 8 or later, where the Query Engine is built in, or Redis Stack. CI
runs wardowl’s conformance kit against
Redis 8.10.2. The same wardowl[redis] extra serves the shared permission cache.
import redis
from wardowl.backends.redis import RedisBackend, create_index
client = redis.Redis.from_url(REDIS_URL, socket_connect_timeout=2, socket_timeout=5)
create_index(client, "chunks", dim=768) # hashes under "wardowl:", HNSW, cosine
backend = RedisBackend(client, "chunks", embed=embed) # embed: str -> list[float]
backend.add(documents)
Each chunk is a hash under the prefix (wardowl: plus the document id) with the fields
wardowl_id, content, metadata (JSON), wardowl_terms and embedding (float32). A write
is searchable as soon as it returns; replacing a chunk replaces the whole hash. ef_runtime sets
HNSW’s search breadth. Scores are cosine similarity, 1 minus Redis’s cosine distance. For a native
aretrieve, pass async_client=redis.asyncio.Redis(...); otherwise async calls run in a thread.
An index covers every hash whose key starts with its prefix. Two indexes on the default prefix therefore index each other’s chunks, and two backends that write the same chunk id overwrite one hash. Give each index its own prefix, and pass the same prefix to its backends:
create_index(client, "acme_chunks", dim=768, prefix="acme:")
acme = RedisBackend(client, "acme_chunks", embed=embed, prefix="acme:")
Prefixes are compared as plain strings, so end each with a separator: t1: and t10: are apart,
but every key under t10 also starts with t1.
HNSW is approximate. Redis indexes a hash again whenever any of its fields changes, so set_metadata
removes and re-adds the chunk’s vector in the graph, and a small graph that changes often can miss
a chunk; it never returns one the filter excludes. create_index(..., algorithm="FLAT") compares
the query with every vector instead: exact, and fast enough up to some tens of thousands of chunks.
CI checks both, exactly on FLAT.
Timeouts. By default the Query Engine answers a query that runs out of time with the results
it found so far, which look complete. In production, make it fail instead: set
search-on-timeout fail in the Redis 8 configuration (FT.CONFIG SET ON_TIMEOUT FAIL on Redis
Stack), and a timed-out search raises. A client using RESP3 (protocol=3) also receives a
warning with such partial results, and the backend raises RuntimeError when it sees one; RESP2
replies carry no warning, so with RESP2 the server setting is the only protection. On the client’s
side, redis-py waits for an answer without limit by default: set socket_connect_timeout and
socket_timeout, as in the example above, so that a Redis that hangs raises instead of holding up
retrievals.
How permissions are indexed#
The backend uses the same type-tagged terms as Elasticsearch, OpenSearch
and Milvus, so 1 and "1" stay apart and booleans,
floats, nested lists and objects never match. Redis TAG queries have their own syntax (braces,
|, @, *), and TAG matching ignores case by default, so each term is stored hex-encoded:
from wardowl import In
from wardowl.backends.redis import filter_to_query
filter_to_query(In("allowed_groups", ["hr"]))
# '@wardowl_terms:{616c6c6f7765645f67726f7570731f733a6872}'
A hex string holds only 0-9 and a-f, so no group name or user id can add query syntax, and
"HR" and "hr" stay different values. CI checks both on a real server. And becomes an
intersection and Or a union of TAG conditions, and an In with no values matches nothing without
a call to Redis; filter_to_query raises NothingMatches for a filter that can match nothing,
since a query without the expression matches every hash. The metadata is stored as JSON and
returned exactly as written, so the retriever’s re-check sees the true permissions.
Hashes written by other tools#
A hash without wardowl_terms is invisible to a pre-filtered search, though post-filtering
still finds it and checks its metadata. Fill the field in your ingestion pipeline:
from wardowl.backends.redis import permission_terms
client.hset(key, mapping={"wardowl_terms": permission_terms(metadata)})
Keyword search and hybrid search#
mode="keyword" searches the chunks’ text with BM25 under the same permission filter. The index
must hold the text: create it with content_field=, which keeps every word (the index has no
stop words). One index then serves both halves of hybrid search:
from wardowl import RankFusion
create_index(client, "chunks", dim=768, content_field="content")
vector = RedisBackend(client, "chunks", embed=embed)
keyword = RedisBackend(client, "chunks", mode="keyword") # needs no embed
retriever = PermissionedRetriever(RankFusion(vector, keyword), authorizer)
The query is split into words (letters, digits and underscores), and each word is sent as a query
parameter rather than written into the query, so no query text can add syntax: the permission
condition always applies. A query keeps its first 64 distinct words, and any of them may match.
Scores are Redis’s BM25 (BM25STD), higher for better matches. A keyword backend without embed
writes chunks with no vector, which keyword search finds and vector search does not.
An existing index can gain the text field with FT.ALTER chunks SCHEMA ADD content TEXT, and Redis
indexes the stored hashes in the background. An index created by an earlier wardowl keeps Redis’s
default stop words, so common English words in a query match nothing there. CI runs the conformance
kit against Redis 8.10.2 in keyword mode and fused with vector search.