Elasticsearch and OpenSearch#

pip install "wardowl[elasticsearch]"   # or "wardowl[opensearch]"

Both backends run an approximate kNN search with the permission filter applied during the search, so the k nearest chunks the subject may read come back however few of the index’s documents that is. Elasticsearch 8.12 or newer and OpenSearch 2.9 or newer are supported; CI runs wardowl’s conformance kit against Elasticsearch 9.5.5 and OpenSearch 3.9.0.

Elasticsearch#

from elasticsearch import Elasticsearch

from wardowl.backends.elasticsearch import ElasticsearchBackend, create_index

client = Elasticsearch("http://localhost:9200")
create_index(client, "docs", dim=768)
backend = ElasticsearchBackend(client, "docs", embed=embed)  # embed: str -> list[float]

The filter goes in the knn clause’s filter. Elasticsearch documents that this filter “is applied during approximate kNN search to ensure that k matching documents are returned”; a filter placed beside the knn clause would apply afterwards and could return fewer, so wardowl never does that. num_candidates= (default 100, at least k, at most 10,000) is how many neighbours each shard considers: higher finds more of the true nearest chunks and takes longer.

Elasticsearch quantizes dense vectors by default (int8_hnsw below 384 dimensions, bbq_hnsw from 384 in 9.1 and later). That saves memory and costs some recall; pass create_index(..., index_options={"type": "hnsw"}) to keep full-precision vectors. On a million 384-dimensional chunks, the default with 100 candidates found 0.62 of the true top 10 for a user who may read 1% of them; int8_hnsw with num_candidates=500 found 0.98 at a similar median latency (11 ms against 13), in the same 2.2 GiB, and full-precision hnsw found 1.00 (Benchmarks). For a native aretrieve, pass async_client=AsyncElasticsearch(...) (install elasticsearch[async]).

OpenSearch#

from opensearchpy import OpenSearch

from wardowl.backends.opensearch import OpenSearchBackend, create_index

client = OpenSearch(hosts=["http://localhost:9200"])
create_index(client, "docs", dim=768)  # engine="lucene" (default) or "faiss"
backend = OpenSearchBackend(client, "docs", embed=embed)

The filter goes in the knn query’s filter, which the Lucene and Faiss engines apply during the search (efficient k-NN filtering). create_index makes a knn_vector field with cosine similarity and HNSW; pass parameters={"m": 16, "ef_construction": 128} to tune the graph. For a native aretrieve, pass async_client=AsyncOpenSearch(...) (install opensearch-py[async]).

Approximate search and recall#

Both backends always run an approximate kNN search (HNSW, or a quantized form of it), and neither has a switch for exact search. The engines may search exactly on their own when a filter admits few documents, but the answer does not say which ran. A result can therefore miss some of the nearest chunks the user may read and still say complete=True (Concepts). To find more of them:

Elasticsearch: raise num_candidates (100 by default), the neighbours each shard considers; the backend sends at least the k the retriever asks for. Keep full-precision vectors with create_index(..., index_options={"type": "hnsw"}), or, on Elasticsearch 8.13 and later, use {"type": "flat"}, which compares the query with every vector the filter admits: exact, and slower as the index grows.
OpenSearch: the query takes no candidate count. With the Lucene engine (the default), HNSW examines as many neighbours as the query’s k, which is the number of chunks the retriever asks for: k, more when post-filtering, or a reranker’s depth. With the Faiss engine, the index setting index.knn.algo_param.ef_search (100 by default) sets it, as in create_index(..., engine="faiss", settings={"index": {"knn.algo_param.ef_search": 256}}); it can also be changed on a live index.

The benchmarks measure what these settings cost and buy on a million chunks.

How permissions are indexed#

Keyword fields in both engines hold strings, so a terms query for the integer 1 also matches the string "1", and a field’s type is fixed by the first document that used it. The reference semantics keep 1 and "1" apart and never match booleans or floats. So both backends index one keyword field, wardowl_terms, with a type-tagged term for every string and integer in the chunk’s metadata, scalars and list elements alike:

Metadata	`wardowl_terms`
`{"allowed_groups": ["hr", "eng"]}`	`allowed_groups␟s:eng`, `allowed_groups␟s:hr`
`{"level": 1}`	`level␟i:1`
`{"level": "1"}`	`level␟s:1`
`{"public": true, "score": 0.9}`	none: no filter ever matches booleans or floats

(␟ is the ASCII unit separator, \x1f; field names may not contain it.) A filter becomes a terms query on these tags, And a bool filter, Or a bool should, and an In with no values matches nothing without a call to the engine. filter_to_query raises NothingMatches for a filter that can match nothing, since a search without a filter matches everything. The metadata itself is stored with enabled: false: kept in _source and returned exactly as written, so the retriever’s re-check sees the true permissions, but never indexed, so arbitrary metadata cannot add mappings to the index. The index uses dynamic: false for the same reason.

Indexes written by other tools#

A document without wardowl_terms is invisible to a pre-filtered search: its metadata may say it is readable, but no tag matches. Post-filtering still finds it and checks its metadata. To make such an index fully searchable, add the field to each document in your ingestion pipeline:

from wardowl.backends.elasticsearch import TERMS_FIELD, permission_terms

document[TERMS_FIELD] = permission_terms(document["metadata"])

and map wardowl_terms as keyword. A document without a metadata object carries no permissions, so the re-check denies it.

Limits#

Both engines return at most 10,000 hits per search, so a search with k above 10,000 raises ValueError rather than return 10,000 hits as if no more matched. A terms query holds at most index.max_terms_count values (65,536 by default), which bounds how many groups a single subject’s filter can name; see Operations.

Partial results#

Every search sends allow_partial_search_results=false, so a shard that fails or times out fails the search instead of leaving its hits out. An answer that still reports a failed shard (_shards.failed above 0) or timed_out: true raises RuntimeError naming the engine, rather than return its hits as complete.