0

Glossary

Engineering-focused definitions with analogies, use cases, and interview tips. 69 terms covering AI Engineering, GenAI, RAG, and Agentic AI.

ABCDEFGHIKLMOPQRSTV

A

Agent

Simple Definition

An AI system that can perceive, reason, and act autonomously to achieve goals.

Technical Definition

A software entity powered by an LLM that operates in a loop: observe environment → reason → select tools → execute actions → update state, until the task is complete.

Analogy

Like a personal assistant who reads your email, checks your calendar, books meetings, and reports back — without you micromanaging each step.

Where It's Used

Customer support bots, coding assistants, research agents, autonomous workflows.

Related Concepts

Agent LoopTool CallingReActMemory

Interview Tip

Emphasize the loop (perceive → plan → act → observe) and how you handle failures and hallucinations in agent systems.

Agent Loop

Simple Definition

The repeating cycle an AI agent follows: think, act, observe results, and repeat until done.

Technical Definition

An iterative control flow where the LLM receives state (user input, tool outputs, memory), produces a decision (text or tool call), the runtime executes actions, appends observations to context, and continues until a termination condition is met.

Analogy

Like a detective who gathers clues, forms a theory, tests it, and repeats until the case is solved.

Where It's Used

Autonomous agents, coding assistants, workflow automation, multi-step research tools.

Related Concepts

AgentReActTool CallingReflection

Interview Tip

Draw the loop diagram and explain termination conditions, max iterations, and error recovery strategies.

Attention

Simple Definition

A mechanism that lets models focus on the most relevant parts of input when making predictions.

Technical Definition

A weighted sum over value vectors, where weights are computed from query-key dot products, allowing dynamic focus on different input positions.

Analogy

Like highlighting the important sentences in a long document before answering a question about it.

Where It's Used

Transformers, translation, summarization, all modern LLMs.

Related Concepts

TransformerKV CacheRoPEContext Window

Interview Tip

Know the Q, K, V intuition and why attention solves the long-range dependency problem in RNNs.

AutoGen

Simple Definition

A Microsoft framework for building multi-agent conversations where AI agents collaborate on tasks.

Technical Definition

An open-source framework providing agent abstractions, conversation patterns (group chat, sequential), and tool integration for orchestrating multiple LLM-powered agents with human-in-the-loop support.

Analogy

Like a conference room where several specialists discuss a problem until they reach a solution.

Where It's Used

Multi-agent research, code generation pipelines, collaborative problem-solving workflows.

Related Concepts

AgentCrewAILangGraphTool Calling

Interview Tip

Compare AutoGen's conversational multi-agent model with LangGraph's graph-based orchestration.

B

BM25

Simple Definition

A classic keyword-based ranking algorithm used to find the most relevant documents for a search query.

Technical Definition

A probabilistic retrieval function based on term frequency, inverse document frequency, and document length normalization (Okapi BM25), scoring documents by lexical overlap with the query.

Analogy

Like a librarian who finds books by matching exact words and phrases, not just meaning.

Where It's Used

Hybrid search, RAG retrieval baselines, Elasticsearch, traditional search engines.

Related Concepts

Hybrid SearchRetrieverRe-rankingSemantic Search

Interview Tip

Explain when BM25 beats embeddings (exact matches, rare terms) and why hybrid search combines both.

C

Caching

Simple Definition

Storing previously computed results so they can be reused instead of recalculated.

Technical Definition

In LLM systems, caching spans prompt caching (reusing prefix KV states), embedding cache (storing vector lookups), semantic cache (returning prior answers for similar queries), and response cache at the API layer.

Analogy

Like keeping answers to frequently asked questions on a sticky note instead of looking them up every time.

Where It's Used

Production LLM APIs, RAG pipelines, cost optimization, latency reduction.

Related Concepts

KV CacheLatencyInferenceRate Limiting

Interview Tip

Distinguish KV cache (inference), semantic cache (application), and prompt caching (provider feature) — each solves different problems.

Chain-of-Thought

Simple Definition

Prompting a model to show its step-by-step reasoning before giving a final answer.

Technical Definition

A prompting technique where the model generates intermediate reasoning steps (explicit or via 'let's think step by step') before producing a conclusion, improving performance on complex reasoning tasks.

Analogy

Like showing your work on a math test — the steps help you and the grader catch mistakes.

Where It's Used

Math problems, logical reasoning, complex Q&A, agent planning.

Related Concepts

Prompt EngineeringFew-shotReActReflection

Interview Tip

Mention CoT prompting vs fine-tuned reasoning models (o1-style) and when explicit reasoning helps vs adds latency.

Chunking

Simple Definition

Breaking large documents into smaller pieces for processing and retrieval.

Technical Definition

The process of splitting documents into segments (fixed-size, semantic, or recursive) with optional overlap, balancing retrieval granularity against context window limits and embedding quality.

Analogy

Like cutting a textbook into index cards — each card covers one topic so you can find the right page quickly.

Where It's Used

RAG ingestion pipelines, long document Q&A, knowledge base indexing.

Related Concepts

RAGEmbeddingContext WindowDocument Loader

Interview Tip

Discuss chunk size tradeoffs: too small loses context, too large dilutes relevance. Mention overlap and semantic chunking.

Context Window

Simple Definition

The maximum amount of text (in tokens) a model can process in a single request.

Technical Definition

The fixed-size input buffer (measured in tokens) that the model's attention mechanism can attend to during a single forward pass.

Analogy

Like working memory — you can only hold so much information in your head at once before older details fade.

Where It's Used

Long document Q&A, chat history management, RAG chunk selection.

Related Concepts

TokenKV CacheRAGChunking

Interview Tip

Discuss strategies for handling documents larger than the context window: chunking, summarization, RAG.

Cosine Similarity

Simple Definition

A measure of how similar two vectors are based on the angle between them.

Technical Definition

The cosine of the angle between two vectors: cos(θ) = (A·B) / (||A|| × ||B||), ranging from -1 to 1, commonly used for comparing embedding vectors regardless of magnitude.

Analogy

Like comparing the direction two arrows point — similar meanings point the same way, even if one arrow is longer.

Where It's Used

Vector search, RAG retrieval, recommendation systems, clustering.

Related Concepts

EmbeddingVector SearchSemantic SearchHNSW

Interview Tip

Explain why cosine similarity is preferred over Euclidean distance for normalized embeddings.

CrewAI

Simple Definition

A framework for orchestrating teams of AI agents with defined roles working together on tasks.

Technical Definition

A Python framework modeling agent crews with role-based agents, task delegation, sequential/hierarchical processes, and tool integration for collaborative multi-agent workflows.

Analogy

Like assembling a project team — a researcher, writer, and editor each doing their part toward one deliverable.

Where It's Used

Content pipelines, research automation, multi-role business workflows.

Related Concepts

AgentAutoGenLangGraphTool Calling

Interview Tip

Describe role-based agent design: how you define goals, backstories, and delegation between crew members.

Cross Encoder

Simple Definition

A model that scores how relevant a query-document pair is by processing them together.

Technical Definition

A transformer model that jointly encodes query and document in a single forward pass, producing a relevance score via cross-attention — more accurate than bi-encoder similarity but slower.

Analogy

Like reading a question and answer side-by-side to judge fit, instead of comparing two separate summaries.

Where It's Used

Re-ranking in RAG, search result refinement, relevance scoring.

Related Concepts

Re-rankingRetrieverEmbeddingHybrid Search

Interview Tip

Contrast bi-encoder (fast retrieval) vs cross-encoder (accurate re-ranking) — use both in a two-stage pipeline.

D

Distillation

Simple Definition

Training a smaller model to mimic the behavior of a larger, more capable model.

Technical Definition

Knowledge distillation transfers knowledge from a teacher model to a student model using soft labels (probability distributions) rather than hard labels, compressing capability into smaller architectures.

Analogy

Like a master chef teaching an apprentice — the apprentice learns techniques without years of independent experimentation.

Where It's Used

Model compression, edge deployment, cost reduction, faster inference.

Related Concepts

QuantizationInferenceFine TuningLatency

Interview Tip

Explain teacher-student training and when distillation preserves vs loses capability.

Document Loader

Simple Definition

A component that reads and parses documents from various sources into processable text.

Technical Definition

An ingestion utility that extracts text and metadata from files (PDF, DOCX, HTML, etc.), APIs, or databases, handling encoding, layout parsing, and format-specific extraction.

Analogy

Like a scanner that digitizes paper documents so a computer can read them.

Where It's Used

RAG ingestion pipelines, knowledge base setup, data preprocessing.

Related Concepts

ChunkingRAGEmbeddingSemantic Search

Interview Tip

Mention challenges with PDFs (tables, images), and tools like Unstructured, LlamaParse, or custom parsers.

E

Embedding

Simple Definition

A numerical vector representation of text, images, or other data that captures semantic meaning.

Technical Definition

A dense, fixed-dimensional vector in ℝⁿ produced by an encoder model, where semantically similar inputs map to nearby points in vector space.

Analogy

Like GPS coordinates for meaning — words with similar meanings end up close together on the map.

Where It's Used

Semantic search, RAG, recommendation systems, clustering.

Related Concepts

Embeddings ModelVector DBCosine SimilarityRAG

Interview Tip

Explain cosine similarity and why embedding model choice dramatically affects RAG quality.

Embeddings Model

Simple Definition

A specialized model trained to convert text into meaningful vector representations.

Technical Definition

An encoder model (e.g., text-embedding-3, BGE, E5) trained with contrastive or dual-encoder objectives to produce dense vectors optimized for semantic similarity and retrieval tasks.

Analogy

Like a translator that converts sentences into a universal 'meaning language' computers can compare.

Where It's Used

RAG indexing, semantic search, clustering, deduplication.

Related Concepts

EmbeddingVector DBRetrieverCosine Similarity

Interview Tip

Know how to evaluate embedding models: MTEB benchmarks, domain fit, dimension size vs quality tradeoffs.

Evaluation

Simple Definition

Systematically measuring how well an AI system performs on defined criteria.

Technical Definition

The process of assessing LLM outputs using automated metrics (BLEU, ROUGE, faithfulness), LLM-as-judge, human evaluation, and domain-specific benchmarks across accuracy, relevance, safety, and latency.

Analogy

Like grading an exam with a rubric — you need clear criteria to know if the student actually learned.

Where It's Used

RAG pipelines, agent systems, fine-tuning validation, production monitoring.

Related Concepts

RAGASTracingHallucinationObservability

Interview Tip

Describe an eval pipeline: golden datasets, automated metrics, human review, and continuous monitoring in production.

F

Few-shot

Simple Definition

Providing a few examples in the prompt to teach the model the desired output format or behavior.

Technical Definition

An in-context learning technique where 1-10 input-output examples are prepended to the prompt, enabling task adaptation without weight updates by leveraging the model's pattern-matching capabilities.

Analogy

Like showing someone two solved examples before asking them to solve a similar problem.

Where It's Used

Classification, formatting, style matching, quick prototyping without fine-tuning.

Related Concepts

Prompt EngineeringChain-of-ThoughtSystem PromptFine Tuning

Interview Tip

Contrast few-shot prompting vs fine-tuning: few-shot is fast but limited by context window and example quality.

Fine Tuning

Simple Definition

Adapting a pre-trained model to perform better on a specific task or domain.

Technical Definition

Continued training of a base model on task-specific data, adjusting weights (full fine-tune) or adapter layers (LoRA/QLoRA) to specialize behavior.

Analogy

A general doctor specializing in cardiology — same medical foundation, but deep expertise in one area.

Where It's Used

Domain-specific chatbots, style transfer, classification, custom assistants.

Related Concepts

LoRAQLoRAPEFTSFT

Interview Tip

Know when to fine-tune vs RAG vs prompt engineering — fine-tuning changes behavior, RAG adds knowledge.

Function Calling

Simple Definition

An API feature where models output structured requests to invoke predefined functions.

Technical Definition

A provider-specific implementation of tool calling where the model receives function schemas in the API request and returns structured JSON specifying which function to call with what arguments.

Analogy

Like a remote control with labeled buttons — the model presses the right button instead of doing the work itself.

Where It's Used

OpenAI/Anthropic APIs, agent frameworks, API integrations.

Related Concepts

Tool CallingStructured OutputsMCPAgent

Interview Tip

Function calling and tool calling are often used interchangeably — know your provider's specific API format.

G

Grounding

Simple Definition

Connecting AI responses to verified external sources or facts to improve accuracy.

Technical Definition

The practice of anchoring model outputs to retrieved documents, knowledge bases, or tool results, with citations and faithfulness checks to reduce hallucination and enable verification.

Analogy

Like a journalist citing sources — claims are only as trustworthy as the evidence behind them.

Where It's Used

RAG systems, enterprise chatbots, legal/medical AI, search-augmented generation.

Related Concepts

RAGHallucinationEvaluationRAGAS

Interview Tip

Distinguish grounding (data source) from guardrails (behavior constraints) — both reduce risk but address different failure modes.

Guardrails

Simple Definition

Safety controls that prevent AI systems from producing harmful, off-topic, or non-compliant outputs.

Technical Definition

Input/output validation layers using classifiers, rule engines, or LLM judges to filter toxic content, PII, off-topic responses, and policy violations before or after generation.

Analogy

Like guardrails on a highway — they keep the car on the road without dictating the destination.

Where It's Used

Production chatbots, enterprise AI, regulated industries, customer-facing apps.

Related Concepts

Prompt InjectionHallucinationEvaluationObservability

Interview Tip

Layer defenses: input validation, system prompt constraints, output filtering, and human review for high-stakes decisions.

H

Hallucination

Simple Definition

When an AI model generates confident but factually incorrect information.

Technical Definition

A generation artifact where the model produces plausible-sounding but ungrounded outputs, often due to training data gaps, overconfidence in next-token prediction, or lack of retrieval grounding.

Analogy

Like a student who didn't study but writes a very convincing essay with made-up citations.

Where It's Used

Any LLM application — especially critical in medical, legal, and financial domains.

Related Concepts

RAGGroundingGuardrailsEvaluation

Interview Tip

List mitigation strategies: RAG, grounding, confidence scoring, human-in-the-loop, evaluation pipelines.

HNSW

Simple Definition

A fast algorithm for finding similar vectors in large databases.

Technical Definition

Hierarchical Navigable Small World graphs — a multi-layer graph index for approximate nearest neighbor search with O(log n) query complexity, balancing recall and speed for high-dimensional vectors.

Analogy

Like a highway system with express lanes — you skip most roads and jump to the right neighborhood quickly.

Where It's Used

Vector databases (Pinecone, Qdrant, Weaviate), large-scale semantic search.

Related Concepts

Vector DBVector SearchEmbeddingCosine Similarity

Interview Tip

Know HNSW parameters: ef_construction, M, ef_search — they trade build time, memory, and recall.

I

Inference

Simple Definition

Running a trained AI model to generate predictions or responses.

Technical Definition

The forward pass of a neural network at serving time: tokenization → model computation (attention, FFN) → sampling → detokenization, optimized via batching, KV caching, and quantization.

Analogy

Like a chef cooking from a recipe — the recipe (training) is done, now they're making the actual dish.

Where It's Used

All LLM deployments, API serving, edge devices, batch processing.

Related Concepts

LatencyThroughputKV CacheQuantization

Interview Tip

Distinguish training vs inference costs and optimizations — inference is where production economics matter.

K

KV Cache

Simple Definition

A performance optimization that stores previous attention computations during text generation.

Technical Definition

Cached key and value tensors from prior tokens' attention computations, avoiding recomputation during autoregressive decoding and reducing latency from O(n²) to O(n) per step.

Analogy

Keeping your notes from earlier chapters instead of re-reading the entire book for each new sentence.

Where It's Used

LLM inference, streaming, production deployments.

Related Concepts

InferenceTransformerRoPECaching

Interview Tip

Explain memory tradeoff: KV cache speeds up generation but increases GPU memory usage linearly with sequence length.

L

LangGraph

Simple Definition

A framework for building stateful, multi-step AI workflows as directed graphs.

Technical Definition

A LangChain library modeling agent workflows as graphs with nodes (functions/LLM calls), edges (conditional routing), and shared state, supporting cycles, persistence, and human-in-the-loop checkpoints.

Analogy

Like a flowchart for your AI app — each box is a step, arrows show what happens next, and you can loop back.

Where It's Used

Complex agent workflows, multi-step pipelines, production agent orchestration.

Related Concepts

Agent LoopReActCrewAITracing

Interview Tip

Contrast LangGraph's explicit state machine model with simpler chain-based approaches for complex agents.

Latency

Simple Definition

The time it takes for an AI system to respond after receiving a request.

Technical Definition

End-to-end response time measured as time-to-first-token (TTFT) and time-to-last-token (TTLT), affected by model size, batching, network, retrieval steps, and hardware.

Analogy

Like how long you wait on hold before someone picks up the phone.

Where It's Used

Production SLAs, user experience optimization, real-time applications.

Related Concepts

ThroughputInferenceStreamingCaching

Interview Tip

Break down latency budget: retrieval + LLM TTFT + generation + tool calls. Know what to optimize first.

LoRA

Simple Definition

A lightweight fine-tuning method that trains small adapter matrices instead of the full model.

Technical Definition

Low-Rank Adaptation injects trainable rank-decomposition matrices (A×B) into attention layers, freezing base weights and training only ~0.1-1% of parameters for efficient task specialization.

Analogy

Like adding a small plugin to an app instead of rewriting the entire codebase.

Where It's Used

Domain fine-tuning, style adaptation, multi-adapter serving, resource-constrained training.

Related Concepts

QLoRAPEFTFine TuningQuantization

Interview Tip

Explain rank selection tradeoff: higher rank = more capacity but more memory. LoRA is the default PEFT method.

M

MCP

Simple Definition

Model Context Protocol — a standard for connecting AI models to external tools and data sources.

Technical Definition

An open protocol defining how LLM clients discover and invoke tools, access resources, and use prompts from MCP servers via standardized JSON-RPC transport.

Analogy

USB-C for AI — one standard port to connect any tool or data source to any AI model.

Where It's Used

IDE integrations, agent tool ecosystems, enterprise AI platforms.

Related Concepts

Tool CallingAgentFunction CallingObservability

Interview Tip

Explain the client-server architecture and how MCP differs from ad-hoc function calling.

Memory

Simple Definition

How AI agents store and recall information across conversations or tasks.

Technical Definition

Persistent state mechanisms including short-term (conversation buffer/summary), long-term (vector store of past interactions), and entity memory (structured facts about users), enabling context beyond the context window.

Analogy

Like a colleague who takes notes during meetings and remembers your preferences from last month.

Where It's Used

Chatbots, personal assistants, multi-session agents, customer support.

Related Concepts

AgentContext WindowEmbeddingRAG

Interview Tip

Describe memory types and tradeoffs: buffer vs summary vs vector retrieval. Mention when to use external DB vs in-context.

MoE

Simple Definition

Mixture of Experts — a model architecture where only some neural network parts activate per input.

Technical Definition

A sparse architecture with multiple expert FFN sub-networks and a gating/router network that selects top-k experts per token, enabling larger total parameters with lower active compute per forward pass.

Analogy

Like a hospital where each patient sees only the relevant specialists, not every doctor in the building.

Where It's Used

Mixtral, GPT-4 (rumored), large-scale efficient models.

Related Concepts

TransformerInferenceThroughputDistillation

Interview Tip

Explain the efficiency tradeoff: more total params but only fraction active — great for scaling with controlled inference cost.

Multimodal

Simple Definition

AI models that can process and generate multiple types of data — text, images, audio, video.

Technical Definition

Models with unified encoders or cross-modal attention processing heterogeneous inputs (vision + language), enabling tasks like image captioning, visual Q&A, and document understanding with charts.

Analogy

Like a person who can read, look at pictures, and listen to speech — not just one sense.

Where It's Used

GPT-4V, Gemini, Claude vision, document AI, content moderation.

Related Concepts

EmbeddingTransformerRAGInference

Interview Tip

Know multimodal RAG: how to embed and retrieve images, tables, and text in unified pipelines.

O

Observability

Simple Definition

The ability to understand what's happening inside a running AI system through logs, metrics, and traces.

Technical Definition

Comprehensive monitoring of LLM applications including request tracing, token/cost tracking, latency histograms, retrieval quality, error rates, and drift detection via platforms like LangSmith, Arize, or Datadog.

Analogy

Like a car dashboard showing speed, fuel, and engine warnings — you need visibility to drive safely.

Where It's Used

Production AI deployments, SRE for ML, cost management, debugging.

Related Concepts

TracingEvaluationLatencyRate Limiting

Interview Tip

Go beyond logging — explain the three pillars: metrics, logs, traces, and what to alert on in LLM apps.

Ollama

Simple Definition

A tool for running open-source LLMs locally on your machine.

Technical Definition

A local inference runtime that packages and serves quantized open-weight models (Llama, Mistral, etc.) with a simple CLI and REST API, handling model downloads, GPU/CPU routing, and context management.

Analogy

Like having a private AI assistant on your laptop instead of calling a cloud service.

Where It's Used

Local development, privacy-sensitive apps, offline inference, prototyping.

Related Concepts

InferenceQuantizationvLLMLatency

Interview Tip

Know when local inference makes sense: privacy, cost at scale, latency — vs cloud for latest models and scale.

P

PEFT

Simple Definition

Parameter-Efficient Fine-Tuning — methods to adapt large models by training only a small subset of parameters.

Technical Definition

A family of techniques (LoRA, QLoRA, prefix tuning, adapters) that freeze most base model weights and train lightweight modules, reducing memory, storage, and training time.

Analogy

Like customizing a car with aftermarket parts instead of rebuilding the engine.

Where It's Used

Fine-tuning on limited hardware, multi-task serving, rapid experimentation.

Related Concepts

LoRAQLoRAFine TuningQuantization

Interview Tip

PEFT is the umbrella term — name specific methods (LoRA most common) and when each applies.

Plan Execute

Simple Definition

An agent pattern that first creates a full plan, then executes each step sequentially.

Technical Definition

A two-phase agent architecture: a planner LLM decomposes the task into steps, then an executor agent carries out each step (with optional re-planning on failure), separating strategic planning from tactical execution.

Analogy

Like writing a recipe before cooking — you think through all steps first, then follow them one by one.

Where It's Used

Complex multi-step tasks, research agents, workflow automation.

Related Concepts

ReActAgent LoopLangGraphChain-of-Thought

Interview Tip

Compare Plan-Execute vs ReAct: planning upfront vs interleaved reasoning-action. Plan-Execute is better for structured tasks.

Prompt Engineering

Simple Definition

The practice of crafting effective inputs to get better outputs from AI models.

Technical Definition

The systematic design of prompts including system instructions, few-shot examples, output format constraints, chain-of-thought triggers, and iterative refinement to maximize task performance without model changes.

Analogy

Like learning to ask good questions — the same expert gives better answers to well-framed questions.

Where It's Used

All LLM applications, prototyping, production prompt templates.

Related Concepts

System PromptFew-shotChain-of-ThoughtTemperature

Interview Tip

Show a before/after prompt improvement. Mention versioning, A/B testing, and prompt templates in production.

Prompt Injection

Simple Definition

An attack where malicious input manipulates an AI system to ignore its instructions.

Technical Definition

A security vulnerability where adversarial text in user input or retrieved documents overrides system prompts, causing unintended model behavior or data exfiltration.

Analogy

Someone slipping a note into your assistant's inbox saying 'ignore all previous instructions.'

Where It's Used

Any LLM application accepting user input — especially RAG and agent systems.

Related Concepts

GuardrailsHallucinationRAGEvaluation

Interview Tip

Discuss defense layers: input sanitization, output filtering, privilege separation, prompt isolation.

PydanticAI

Simple Definition

A Python framework for building type-safe AI agents with structured outputs.

Technical Definition

A framework by the Pydantic team combining LLM calls with Pydantic model validation, dependency injection, tool registration, and structured output parsing for reliable agent development.

Analogy

Like building with LEGO blocks that only fit together the right way — type safety catches mistakes early.

Where It's Used

Production Python agents, APIs requiring validated outputs, type-safe tool calling.

Related Concepts

Structured OutputsAgentFunction CallingTool Calling

Interview Tip

Highlight PydanticAI's strength: validated structured outputs and Python-native developer experience.

Q

QLoRA

Simple Definition

Fine-tuning large models on a single GPU by combining quantization with LoRA adapters.

Technical Definition

Quantized Low-Rank Adaptation: base model weights stored in 4-bit NF4 quantization, with LoRA adapters trained in higher precision, enabling fine-tuning 65B+ models on consumer GPUs.

Analogy

Like compressing a textbook to fit in your bag, then adding sticky notes with your own annotations.

Where It's Used

Fine-tuning on limited hardware, open-source model customization, research.

Related Concepts

LoRAPEFTQuantizationFine Tuning

Interview Tip

QLoRA made fine-tuning accessible — explain 4-bit NF4 + double quantization + LoRA stack.

Quantization

Simple Definition

Reducing model precision to make models smaller and faster with minimal quality loss.

Technical Definition

Converting model weights from FP32/FP16 to lower bit-width representations (INT8, INT4, NF4), reducing memory footprint and increasing inference throughput via optimized kernels.

Analogy

Like converting a high-res photo to a smaller file — slightly less detail, but much faster to share.

Where It's Used

Edge deployment, cost reduction, local inference (Ollama), production serving.

Related Concepts

InferenceQLoRADistillationvLLM

Interview Tip

Know quantization formats: GPTQ, AWQ, GGUF. Explain accuracy vs speed tradeoff at different bit levels.

R

RAG

Simple Definition

Retrieval-Augmented Generation — enhancing LLM responses with relevant external knowledge.

Technical Definition

A pipeline that retrieves relevant documents from a knowledge base using embedding similarity, injects them into the prompt context, and generates grounded responses.

Analogy

An open-book exam — the model can look up answers in reference materials before responding.

Where It's Used

Enterprise chatbots, document Q&A, knowledge bases, customer support.

Related Concepts

EmbeddingVector DBChunkingRe-ranking

Interview Tip

Walk through the full pipeline: ingest → chunk → embed → store → retrieve → rerank → generate → evaluate.

RAGAS

Simple Definition

A framework for automatically evaluating the quality of RAG systems.

Technical Definition

Retrieval-Augmented Generation Assessment: an open-source library computing metrics like faithfulness, answer relevancy, context precision, and context recall using LLM-based evaluators.

Analogy

Like a quality inspector checking if a student's open-book answers actually used the textbook correctly.

Where It's Used

RAG pipeline evaluation, CI/CD for AI, benchmarking retrieval strategies.

Related Concepts

EvaluationRAGGroundingHallucination

Interview Tip

Name key RAGAS metrics: faithfulness (grounded in context?), answer relevancy, context precision/recall.

Rate Limiting

Simple Definition

Controlling how many requests a user or system can make in a given time period.

Technical Definition

Throttling mechanisms (token bucket, sliding window) applied at API gateway, provider, or application level to manage cost, prevent abuse, and ensure fair resource allocation across tenants.

Analogy

Like a bouncer at a club letting in 100 people per hour — keeps things manageable.

Where It's Used

Production API serving, multi-tenant SaaS, cost control, DDoS prevention.

Related Concepts

LatencyThroughputCachingObservability

Interview Tip

Discuss rate limiting at multiple levels: provider quotas, app-level per-user limits, and queue-based backpressure.

Re-ranking

Simple Definition

Re-scoring retrieved documents with a more accurate model to improve result ordering.

Technical Definition

A second-stage retrieval step applying a cross-encoder or LLM-based scorer to the top-k candidates from initial retrieval, reordering by relevance before context injection.

Analogy

Like a hiring manager reviewing the top 10 resumes more carefully after HR does an initial screen.

Where It's Used

RAG pipelines, search engines, recommendation systems.

Related Concepts

Cross EncoderRetrieverHybrid SearchRAG

Interview Tip

Two-stage retrieval is standard: fast bi-encoder for recall, cross-encoder for precision. Know the latency cost.

ReAct

Simple Definition

An agent pattern where the model alternates between reasoning and taking actions.

Technical Definition

Reasoning + Acting: the LLM generates interleaved Thought → Action → Observation steps, using natural language reasoning to decide which tools to invoke and updating plans based on results.

Analogy

Like a mechanic who thinks aloud ('the engine sounds rough'), tries a fix, checks the result, and adjusts.

Where It's Used

Tool-using agents, question answering with search, multi-step problem solving.

Related Concepts

Agent LoopTool CallingChain-of-ThoughtPlan Execute

Interview Tip

ReAct is the foundational agent pattern — explain the Thought/Action/Observation trace format.

Reflection

Simple Definition

An agent technique where the AI critiques and improves its own outputs before finalizing.

Technical Definition

A self-correction loop where the model evaluates its draft output against criteria, identifies errors or gaps, and regenerates improved responses — often implemented as a critic-reflector agent pair.

Analogy

Like proofreading your email before sending — you catch mistakes you'd miss if you hit send immediately.

Where It's Used

Code generation, writing assistants, complex reasoning agents, quality improvement.

Related Concepts

Agent LoopChain-of-ThoughtEvaluationReAct

Interview Tip

Reflection adds latency but improves quality — know when the tradeoff is worth it (high-stakes outputs).

Retriever

Simple Definition

The component in a RAG system that finds relevant documents for a given query.

Technical Definition

A retrieval module executing similarity search (dense, sparse, or hybrid) over an indexed corpus, returning top-k document chunks with scores for downstream re-ranking and generation.

Analogy

Like a research librarian who quickly finds the most relevant books for your question.

Where It's Used

RAG pipelines, search systems, knowledge management.

Related Concepts

EmbeddingBM25Hybrid SearchRe-ranking

Interview Tip

Discuss retriever design choices: embedding model, top-k size, metadata filters, and hybrid strategies.

RLHF

Simple Definition

Training AI models using human feedback to align outputs with human preferences.

Technical Definition

Reinforcement Learning from Human Feedback: collect human preference rankings, train a reward model, then optimize the LLM policy via PPO or similar RL to maximize reward while staying close to the base model.

Analogy

Like training a dog with treats — good behavior gets rewarded until it becomes habit.

Where It's Used

ChatGPT alignment, instruction-following models, safety tuning.

Related Concepts

SFTFine TuningEvaluationGuardrails

Interview Tip

Know the RLHF pipeline: SFT → reward model → RL optimization. Mention alternatives like DPO and constitutional AI.

RoPE

Simple Definition

A technique that helps transformers understand the position of words in a sequence.

Technical Definition

Rotary Position Embedding applies rotation matrices to query and key vectors based on token position, encoding relative positional information in attention without explicit positional embeddings.

Analogy

Like numbered seats in a theater — everyone knows their position relative to others.

Where It's Used

Llama, Mistral, GPT-NeoX, most modern LLMs.

Related Concepts

AttentionTransformerKV CacheContext Window

Interview Tip

RoPE enables better length extrapolation than absolute positional embeddings — know why it matters for long context.

S

Semantic Kernel

Simple Definition

Microsoft's SDK for integrating AI services into applications with plugins and planners.

Technical Definition

An open-source orchestration SDK providing abstractions for prompts, plugins (tools), memory, and planners that decompose goals into steps across .NET, Python, and Java applications.

Analogy

Like a Swiss Army knife for AI — standardized pieces you snap together for any application.

Where It's Used

Enterprise .NET/Python apps, Microsoft ecosystem integrations, plugin-based agents.

Related Concepts

AgentTool CallingMemoryPlan Execute

Interview Tip

Semantic Kernel's plugin model is its core abstraction — compare with LangChain's tool ecosystem.

SFT

Simple Definition

Supervised Fine-Tuning — training a model on labeled input-output examples.

Technical Definition

The first stage of alignment: fine-tuning a base model on curated instruction-response pairs using standard supervised learning (cross-entropy loss) to teach instruction-following behavior.

Analogy

Like teaching with a textbook of worked examples — the student learns the expected format and style.

Where It's Used

Instruction tuning, chat model training, domain adaptation, RLHF pipeline first stage.

Related Concepts

Fine TuningRLHFFew-shotPEFT

Interview Tip

SFT is step 1 before RLHF. Know data quality requirements: diverse, high-quality instruction-response pairs.

Streaming

Simple Definition

Sending AI responses token-by-token as they're generated instead of waiting for the complete answer.

Technical Definition

Server-sent events or WebSocket delivery of autoregressively generated tokens, reducing perceived latency (TTFT) and enabling real-time UI updates during long generations.

Analogy

Like watching a live sports broadcast instead of waiting for the full replay.

Where It's Used

Chat UIs, voice assistants, real-time applications, long-form generation.

Related Concepts

InferenceLatencyKV CacheObservability

Interview Tip

Streaming improves UX but complicates error handling and token counting — mention both sides.

Structured Outputs

Simple Definition

Forcing AI models to respond in a specific, machine-readable format like JSON.

Technical Definition

Constraining model generation to valid JSON/XML/schema via constrained decoding, grammar-based sampling, or post-processing validation, ensuring parseable outputs for downstream systems.

Analogy

Like filling out a form with specific fields instead of writing a free-form essay.

Where It's Used

API integrations, data extraction, agent tool calls, form filling.

Related Concepts

Function CallingTool CallingPydanticAIPrompt Engineering

Interview Tip

Know approaches: JSON mode, function calling schemas, constrained decoding, and Pydantic validation as fallback.

System Prompt

Simple Definition

Instructions that define the AI's role, behavior, and constraints for a conversation.

Technical Definition

A persistent message (system role) prepended to the conversation that sets persona, capabilities, output format, safety guidelines, and tool usage rules, processed with higher priority than user messages.

Analogy

Like an employee handbook — it defines how the assistant should behave in every interaction.

Where It's Used

All production chatbots, agents, API integrations.

Related Concepts

Prompt EngineeringGuardrailsFew-shotPrompt Injection

Interview Tip

System prompts are your first line of defense — discuss layering: system prompt + guardrails + output validation.

T

Temperature

Simple Definition

A setting that controls how random or deterministic an AI model's responses are.

Technical Definition

A scaling factor applied to logits before softmax: T→0 makes output deterministic (argmax), T→1 uses raw distribution, T>1 increases entropy and randomness.

Analogy

A creativity dial — low temperature gives predictable answers, high temperature gives creative surprises.

Where It's Used

All LLM API calls — tune per use case (0 for code, 0.7 for creative writing).

Related Concepts

Top PInferencePrompt Engineering

Interview Tip

Know practical defaults: 0 for extraction/classification, 0.3-0.7 for chat, 0.8+ for creative tasks.

Throughput

Simple Definition

How many requests or tokens an AI system can process per unit of time.

Technical Definition

Serving capacity measured in requests/sec or tokens/sec, optimized via batching, continuous batching (vLLM), model parallelism, and hardware scaling.

Analogy

Like how many customers a restaurant can serve per hour — not just how fast one meal is cooked.

Where It's Used

Production capacity planning, cost modeling, SLA design.

Related Concepts

LatencyInferencevLLMQuantization

Interview Tip

Throughput vs latency tradeoff: batching increases throughput but may increase individual request latency.

Token

Simple Definition

The basic unit of text that AI models process — roughly a word or part of a word.

Technical Definition

A subword unit from a tokenizer vocabulary (BPE, SentencePiece) that models consume as input/output, with typical English text averaging ~0.75 words per token.

Analogy

Like syllables or word fragments — 'unbelievable' might be 3 tokens: 'un', 'believ', 'able'.

Where It's Used

All LLM APIs, billing, context window limits, chunking strategies.

Related Concepts

TokenizationContext WindowInferenceLatency

Interview Tip

Tokens drive cost and context limits — know how to estimate token counts and why they matter for pricing.

Tokenization

Simple Definition

The process of breaking text into tokens that a model can understand.

Technical Definition

Converting raw text to integer token IDs using subword algorithms (BPE, WordPiece, SentencePiece), handling whitespace, special tokens, and vocabulary mapping for model input.

Analogy

Like cutting a sentence into puzzle pieces that the model's dictionary recognizes.

Where It's Used

All LLM preprocessing, embedding pipelines, context window management.

Related Concepts

TokenContext WindowEmbeddingInference

Interview Tip

Different models use different tokenizers — same text can have different token counts across models.

Tool Calling

Simple Definition

When an LLM decides to invoke external functions or APIs to complete a task.

Technical Definition

A structured output pattern where the model generates a JSON schema specifying a function name and arguments, which the runtime executes and returns results to the model.

Analogy

A manager who doesn't do everything themselves — they delegate tasks to specialists and use the results.

Where It's Used

Agents, API integrations, database queries, code execution.

Related Concepts

AgentFunction CallingMCPStructured Outputs

Interview Tip

Describe the loop: model requests tool → runtime executes → result fed back → model continues.

Top P

Simple Definition

A sampling setting that limits choices to the most likely tokens whose combined probability reaches P.

Technical Definition

Nucleus sampling: at each step, select the smallest set of tokens whose cumulative probability ≥ p (e.g., 0.9), then sample from that set, dynamically adjusting candidate pool size.

Analogy

Like only considering the top candidates who together have 90% of the votes, ignoring long-shot options.

Where It's Used

Text generation, creative writing, chat applications alongside temperature.

Related Concepts

TemperatureInferencePrompt Engineering

Interview Tip

Top-p and temperature are often used together. Top-p prevents sampling from the long tail of unlikely tokens.

Tracing

Simple Definition

Recording the full execution path of an AI application for debugging and monitoring.

Technical Definition

Distributed tracing of LLM calls, retrievals, tool invocations, and latencies using tools like LangSmith, Phoenix, or OpenTelemetry spans.

Analogy

A flight recorder for your AI app — see exactly what happened when something went wrong.

Where It's Used

Production AI systems, agent debugging, cost monitoring.

Related Concepts

ObservabilityEvaluationLatencyAgent Loop

Interview Tip

Mention key metrics to trace: token usage, latency per step, retrieval relevance scores, error rates.

Transformer

Simple Definition

The neural network architecture that powers virtually all modern LLMs.

Technical Definition

An encoder-decoder (or decoder-only) architecture using multi-head self-attention and feed-forward layers, with positional encoding, enabling parallel processing of sequences.

Analogy

A team of experts who each read the entire document simultaneously and share notes about what's important.

Where It's Used

GPT, BERT, Claude, Gemini, and all frontier models.

Related Concepts

AttentionRoPEMoEKV Cache

Interview Tip

Draw the encoder-decoder diagram and explain why transformers replaced RNNs for NLP.

V

Vector DB

Simple Definition

A database optimized for storing and searching high-dimensional vector embeddings.

Technical Definition

A specialized datastore using approximate nearest neighbor (ANN) algorithms (HNSW, IVF) for sub-linear similarity search over dense vector collections with metadata filtering.

Analogy

A library organized by meaning instead of alphabetical order — you find books on similar topics instantly.

Where It's Used

RAG pipelines, semantic search, recommendation engines.

Related Concepts

EmbeddingHNSWVector SearchRAG

Interview Tip

Compare Pinecone, Weaviate, Qdrant, Chroma — know tradeoffs between managed vs self-hosted.

vLLM

Simple Definition

A high-performance library for serving large language models in production.

Technical Definition

An inference engine featuring PagedAttention for efficient KV cache memory management, continuous batching, tensor parallelism, and OpenAI-compatible API serving for high-throughput LLM deployment.

Analogy

Like an express highway system for AI — moves many requests efficiently instead of one at a time.

Where It's Used

Production LLM serving, self-hosted deployments, high-throughput APIs.

Related Concepts

InferenceThroughputKV CacheQuantization

Interview Tip

vLLM's PagedAttention is the key innovation — explain how it reduces KV cache memory fragmentation.