NEWS & INSIGHTS

Understanding LoRA: adapting an AI model with small weight updates

How LoRA freezes a pretrained model and learns small update matrices, what the original GPT-2 experiment shows, and which training, deployment and patent-practice distinctions matter.

LoRA concept diagram: the same input follows the frozen W₀ path and the trainable A-to-B path scaled by α/r, and the outputs are added.
An explanatory diagram created for the source article distinguishes the frozen original computation from the trainable update path. It is neither a figure from the paper nor a screenshot of a product.

LoRA adapts a pretrained AI model by keeping its existing weights fixed and learning an update represented by two small matrices. The aim is to reduce the work involved in training and storing a separate adaptation for each business task. A hypothetical online-store example and the paper’s experiments help distinguish what becomes smaller from what the system still needs.

This article examines LoRA: Low-Rank Adaptation of Large Language Models by Edward J. Hu, Yelong Shen and their co-authors, using arXiv version 2. The paper was first submitted on 17 June 2021 and revised on 16 October 2021.

1. Why adapting a model for each business creates extra work

Customer support and product descriptions call for different outputs, even when they use the same language model. A support response may need to explain a procedure calmly and in order, while a product description should present features in a consistent format. Training on examples of these tasks is called fine-tuning. The question is whether every weight in the model must be updated each time.

Weights are numerical values inside a model that influence how information passes to the next calculation. Full fine-tuning treats the existing weights throughout the model as trainable. For a large model, this means managing not only the values being updated but also gradients and optimizer states. Keeping a complete model for each task adds storage requirements.

LoRA asks whether the changes needed for a task can be represented with fewer trainable values while retaining the pretrained model. It does not remove the original model or replace it with a smaller one. Instead, it adds a small trainable update path alongside the computation the model already performs.

2. Keeping the original calculation and learning only an update

Consider one calculation in the model as a matrix, a rectangular array of numbers. If the original weight matrix is W₀, LoRA represents the change to be added as the product of two smaller matrices, A and B. The same input passes through W₀ and through A followed by B; their outputs are added.

W₀ stays fixed, while A and B are trained. The paper writes the forward calculation as W₀x + BAx and also scales the update by α/r. Here x is the input and r is the width of the intermediate connection between the two small matrices. Freezing the original weights does not stop their computation: they remain in use but are not updated by training.

Low rank limits the number of independent directions available to express the update. Rather than train every entry of a large matrix independently, two matrices construct the change through a narrow intermediate dimension. A smaller r reduces the trainable values but also restricts the changes that can be represented. It does not mean discarding supposedly unimportant words or documents.

For a weight matrix with d rows and k columns, the original matrix has d×k entries. A has shape r×k and B has shape d×r, giving r×k + d×r trainable entries. Applying LoRA to more layers or more kinds of weights increases the total. The rank alone therefore does not determine training cost for the whole model.

3. An online-store example and the size of two matrices

Imagine an online store collecting examples written by its support staff. It pairs a question such as “How do I request an exchange?” with a response explaining the required steps. The base-model weights remain fixed, while the small LoRA matrices are adjusted to reduce the difference between the training examples and the model’s output.

A separate set of product-description examples can train another update for the same base model. At use time, the system selects the adapter for the relevant task. An adapter here means the learned update parameters. Training two adapters does not automatically combine their abilities, nor does it make either adapter compatible with a different base model.

For an illustrative calculation, take one weight matrix with 4,096 rows and 4,096 columns. Updating every entry gives 16,777,216 trainable values. With LoRA at r=8, A has 8×4,096 entries and B has 4,096×8, for a total of 65,536.

Illustrative LoRA parameter count for a 4,096×4,096 matrix: 16,777,216 full-matrix entries versus 65,536 entries in A and B at r=8; W₀ remains necessary.
An illustrative calculation created for the source article compares the trainable entries in two small matrices for one weight matrix. It does not measure total model size, memory usage or processing speed. The original W₀ is still required.

This explains why the task-specific update can be stored compactly. Real support responses, however, also need current exchange policies and order information, not just the desired writing style. Incorrect training examples affect learning. Adapting the model and supplying accurate, current business information must be designed together.

4. Reading the results of the original experiment

Table 3 of version 2 compares GPT-2 Medium on the E2E NLG Challenge. This is a data-to-text task: restaurant attributes and values are provided, and the model generates a description. The table below reproduces only two rows from that comparison.

GPT-2 Medium on the E2E text-generation task
MethodTrainable parametersBLEU
Full fine-tuning354.92M68.2
LoRA0.35M70.4 ± 0.1

M means million, and the parameter figures count trainable parameters rather than the entire model. BLEU measures overlap between generated text and reference wording. It is neither the probability that a customer-support answer is correct nor a percentage measure of factual accuracy. In this experiment, LoRA substantially reduced trainable parameters while reporting a higher BLEU score than the selected baseline.

LoRA used r=4 for both the query and value projection weights in attention. Appendix D.3 and Table 11 specify five passes through the training data, or five epochs, and a beam size of 10 for generation. The authors averaged results over three random seeds, selecting the best epoch from each run. The ±0.1 is the confidence-interval notation reported in the table; the full fine-tuning score is a baseline taken from earlier work.

These results do not promise better support answers for every business. Restaurant descriptions and online-store support have different inputs and objectives. Evaluation data specific to the intended task should examine missing information, factual errors and output format together.

5. Training cost and inference cost are different questions

LoRA reduces the variables being trained, but it does not remove the resources needed to load the base model and process inputs. Training memory also depends on model size, input length, batch size and numerical precision. A small adapter file does not, by itself, establish that training will fit on a GPU of a particular capacity.

After training, the product of the two matrices, including its scaling, can be added to the original weights. Inference with these merged weights need not execute a separate update path. Merging is a deployment choice; keeping adapters separate and switching between them is another. It would be inaccurate to claim that every unmerged deployment has no additional latency.

The official Hugging Face PEFT documentation explains LoRA configuration and merging. Using merge_and_unload produces a conventional model that no longer retains PEFT functions for disabling or switching the adapter. An operating design must balance the convenience of managing small task-specific files against that of running a merged model.

A larger r does not always improve results. Unsuitable training data or placement of the update matrices may increase cost without producing the desired quality. Comparing settings against the same task-specific evaluation data makes it easier to explain which change caused an improvement.

6. Documenting technical differences for development and patent discussions

Development records should identify the base model and version, the training-data design, the layers or weights adapted, the rank r and whether deployment merges the weights. Linking each adapter to the base model used to train it helps prevent incompatible combinations. Quality comparisons should also record evaluation data and generation settings.

The following is a separate patent-practice perspective for preparing a technical consultation, not a finding of the paper’s experiments. Rather than describe an in-house development only as “reducing costs with LoRA,” explain the processing steps: what inputs determine the update selected, how task-specific training data is organized, and how adapters are matched to base models.

If a development prevents the wrong adapter from being connected during a task switch, the relevant explanation concerns the identifying information, the timing of selection and application, and the error-prevention flow. This does not assert the existence of a newly granted patent or guarantee patentability. Separating the known LoRA architecture from the specific additional processing makes the technical difference easier to discuss.

For related reading, PagedAttention addresses KV-cache storage during inference, while FlashAttention-2 addresses the distribution of attention-computation work. LoRA focuses on which parameters are trained when adapting a model. Distinguishing a training bottleneck from an inference bottleneck clarifies why each technique is being considered.

The original figure labels and forms are preserved; the text explains the source material.

Related service: AI patents

Discuss the specific technical differences in your AI development with IPLEX. Explore our AI projects, publications and consultation information.

Explore IPLEX’s AI expertise

Original Tistory article

LoRA paper: Low-Rank Adaptation of Large Language Models

Hugging Face PEFT documentation

LoRA paper PDF: section 4.1, Table 3, Appendix D.3 and Table 11

Microsoft LoRA implementation and experiments

Hugging Face PEFT: saving merged models and capability differences

PagedAttention: how vLLM manages KV cache and GPU memory

FlashAttention-2: making long-context attention run efficiently on GPUs

Read the Korean source

This article reflects the information available when it was published. Contact us to discuss your circumstances.
Discuss this topic ↗All articles

Put your IP strategy into practice.