← Glossary

LoRA (Low-Rank Adaptation)

Technique

Definition

A parameter-efficient fine-tuning technique that adds small trainable rank-decomposition matrices to frozen model layers. Achieves near full fine-tuning performance while training less than 1% of parameters. Industry standard for adapting LLMs.

Related Terms

QLoRA (Quantized LoRA)

Combines 4-bit quantization with LoRA fine-tuning, enabling fine-tuning of 65B+ parameter models on a single consumer GPU. Published by Tim Dettmers et al. (2023). Made democratized fine-tuning of large models practical.

Fine-tuning

Continuing to train a pre-trained model on a domain-specific or task-specific dataset to specialize its behavior. Ranges from full fine-tuning (updating all weights) to parameter-efficient methods like LoRA and QLoRA.

Quantization

Reducing the numerical precision of model weights (e.g., from 32-bit float to 4-bit integer) to shrink model size and speed up inference with minimal accuracy loss. Enables running large models on consumer hardware. Key for local deployments.

All Terms

AgentAttention MechanismChain-of-ThoughtContext WindowDiffusion ModelDistillationDPO (Direct Preference Optimization)EmbeddingFew-shot PromptingFine-tuningFunction CallingGAN (Generative Adversarial Network)GRPO (Group Relative Policy Optimization)HallucinationInferenceLLM (Large Language Model)LoRA (Low-Rank Adaptation)MCP (Model Context Protocol)MultimodalPrompt EngineeringQLoRA (Quantized LoRA)QuantizationRAG (Retrieval-Augmented Generation)RLHF (Reinforcement Learning from Human Feedback)TemperatureTokenizerTool UseTop-p (Nucleus Sampling)TransformerVector DatabaseZero-shot Prompting