Glossary
Retrieval and RAG terms
A glossary of the retrieval and RAG engineering terms used in the Greenskin Labs engineering notes. Each entry is a short, standalone definition of how the term is used in the Rules Oracle pipeline.
Approximate Nearest Neighbor search
A vector search method that trades exact nearest-neighbor guarantees for speed by indexing embeddings in structures like HNSW or IVFFlat. Good enough for retrieval when you need millisecond queries over millions of chunks, not mathematically perfect ranking.
Byte-Pair Encoding
A subword tokenization algorithm that merges frequent character pairs into tokens. BPE vocabularies handle rare words and proper names more predictably than word-level splitting, which matters when chunk boundaries depend on exact token counts.
chunk
A retrieval-sized slice of source text: one rule section, table, ability cell, or similar unit, usually with metadata (heading path, page number) attached. Chunks are what get embedded, searched, and passed into the answer model — not whole PDFs.
concept aliases
Alternate terms mapped to a canonical concept at query time (e.g. warlord → hero). Alias expansion changes what gets embedded for dense retrieval without rewriting the user's question in the prompt — surfacing chunks written in canonical vocabulary.
context window
The rule text actually passed into the answer model for one query — here, the top-ranked chunks assembled after retrieval. Everything in the window is visible to generation; everything left out might as well not exist for that answer.
corpus
The complete set of source documents ingested, chunked, and indexed for retrieval — here, the PDF rulebooks for a game system. The corpus is swappable; the pipeline architecture is not.
cosine similarity
A score for how aligned two embedding vectors are, typically between 0 and 1 for similar text. Higher means the chunks occupy similar directions in vector space. In the FAQ supersession work, a replacement pair topped out around 0.59 — too low to treat vector similarity as the primary match signal.
embedding
A numeric vector that represents text meaning in a high-dimensional space. Similar passages land near each other; unlike wording can still cluster if the concepts overlap. Dense retrieval compares a question embedding against chunk embeddings to find semantic neighbors.
FAQ supersession
When an errata or FAQ chunk replaces an older rulebook chunk as the authoritative source. Superseded parent chunks are flagged at ingest time so query-time retrieval skips them and answers cite the current rule, not the retired one.
k-Nearest Neighbors
A retrieval method that ranks candidates by distance (or similarity) in embedding space and returns the top k matches. In RAG, k-NN on chunk embeddings is the dense retrieval lane — "find the chunks whose vectors are closest to the question vector."
OCR
Optical Character Recognition: software that turns pixels in a scan or photo into machine-readable text. Layout-aware OCR also tries to preserve structure (headings, columns, tables). On hostile PDFs, OCR quality varies page to page and is often why naive text extraction fails.
pgvector
A Postgres extension that stores and queries high-dimensional vectors using approximate nearest-neighbor (ANN) indexes. Used here to run semantic similarity search directly in the application database without a separate vector store.
Recall@5
Retrieval eval metric: the fraction of questions where at least one correct chunk appears in the top five retrieved results. A Recall@5 of 0.74 means 74% of golden questions had the right chunk in the top five.
Reciprocal Rank Fusion
Combines ranked result lists from multiple retrieval lanes by inverse rank position. The k parameter (default 60) controls score decay from rank 1 downward, making the merge less sensitive to large score gaps at the top.
Retrieval-Augmented Generation
A pattern where a language model answers using retrieved document chunks instead of parametric memory alone. Retrieval grounds the answer in your corpus; generation turns those chunks into natural-language responses (ideally with citations).
vector similarity
A score for how close two embedding vectors are in meaning-space. Higher similarity means the passages are semantically nearer; lower means farther apart. Cosine similarity is one common measure. Used here as a fallback signal when section names and text search cannot link an FAQ to its parent rule.