Pretraining, fine-tuning, and in-context learning
Overview
Pretraining, fine-tuning, and in-context learning (ICL) refer to distinct stages and mechanisms through which a language model acquires or applies task behavior. The primary distinction is whether a training procedure updates the model parameters via loss gradients, or whether the parameters remain frozen () while the model conditions on demonstrations supplied in the current input prompt.
This article focuses on autoregressive language models that predict the next token given preceding context tokens. While pretraining and fine-tuning objectives can also include masked token prediction or preference optimization depending on the architecture, the integer token sequences described in Tokens and tokenizers form the inputs and targets here.

In the diagram above, pretraining and fine-tuning pass through loss computation and an optimizer to modify checkpoint weights, whereas in-context learning processes the current prompt sequence through frozen weights to produce an output.
Updating weights by predicting the next token
In autoregressive (causal) pretraining, the model is trained to maximize the conditional probability of the actual next token given the preceding tokens by minimizing negative log-likelihood1. At each step, gradients are computed and applied by an optimizer with learning rate across a large training corpus.
Training token sequence
-> Compute next-token probabilities with current weights
-> Compare against ground-truth next token to compute loss
-> Backpropagate gradients and run optimizer step
-> Proceed to next training step with updated weightsThe training objective that increases the log-probability of ground-truth tokens must be distinguished from the decoding rule used at generation time (such as greedy decoding, beam search, or temperature and top- sampling)2. Regardless of the decoding rule, a high token probability assigned by the model does not by itself guarantee factual truth.
Adapting a pretrained model to a target task
Fine-tuning initializes from pretrained checkpoint weights and performs additional gradient updates using data and a loss function tailored to a target task. This may update all parameters (full fine-tuning) or freeze the base weight matrices while updating only a small set of trainable parameters such as low-rank adapter matrices in parameter-efficient fine-tuning (PEFT, e.g., LoRA)3. In either case, the defining boundary lies between generating outputs from an existing checkpoint and running backpropagation to update trainable parameters.
Radford et al., Improving Language Understanding by Generative Pre-Training (2018), pretrain a Transformer language model on a large unlabeled text corpus and then fine-tune it on labeled downstream tasks (classification, entailment, similarity, and multiple-choice question answering) by formatting each task’s inputs into a structured token sequence.

The figure above is excerpted from Radford et al. (2018), Figure 1 (PDF page 4; source page). The task-specific input transformations on the right are paired with supervised gradient updates to the Transformer and linear output heads—distinct from merely placing a few examples inside an inference prompt.
Supplying demonstrations inside the current context
In-context learning supplies task instructions or input-output demonstration pairs followed by a query directly inside the input sequence, prompting the model to generate a response without updating model weights (). Brown et al., Language Models are Few-Shot Learners (NeurIPS 2020), evaluated this capability across zero-shot, one-shot, and few-shot settings without task-specific gradient updates4.
Translate the following words from Korean to English.
고양이 -> cat
강아지 -> dog
토끼 ->The demonstrations in this prompt specify both the translation task and the arrow output format. Even when a model completes the pattern accurately, reading the prompt during inference does not permanently modify the checkpoint weights . While the ability to follow in-context patterns emerges from prior pretraining and fine-tuning, the forward-pass computation during inference must be distinguished from parameter-updating training runs.

The diagram above is excerpted from Brown et al. (2020), Figure 1.1 (PDF page 3; source page). The horizontal purple arrow represents pretraining that updates weights across many training sequences, whereas the vertical blue arrows represent in-context learning that conditions on earlier demonstrations within a single sequence.
The Python and C++20 examples below contrast fine-tuning (which updates parameter by a gradient step) against in-context learning (where during inference).
w_pretrained = 1.20
lr = 0.10
grad_loss = -0.80 # Loss gradient computed on labeled target examples
# 1. Fine-tuning: updates parameters via gradient descent.
w_fine_tuned = w_pretrained - lr * grad_loss
# 2. In-context learning: keeps parameters frozen during inference.
w_icl = w_pretrained
print(f"fine_tuned_w={w_fine_tuned:.3f} delta_w={w_fine_tuned - w_pretrained:.3f}")
print(f"icl_frozen_w={w_icl:.3f} delta_w={w_icl - w_pretrained:.3f}")Output:
fine_tuned_w=1.280 delta_w=0.080
icl_frozen_w=1.200 delta_w=0.000The same comparison in C++20 is shown below:
#include <iomanip>
#include <iostream>
int main() {
const double wPretrained = 1.20;
const double lr = 0.10;
const double gradLoss = -0.80;
// 1. Fine-tuning: parameters are updated by gradient steps.
const double wFineTuned = wPretrained - lr * gradLoss;
// 2. In-context learning: parameters remain frozen at inference (delta_w = 0).
const double wIcl = wPretrained;
std::cout << std::fixed << std::setprecision(3);
std::cout << "fine_tuned_w=" << wFineTuned
<< " delta_w=" << (wFineTuned - wPretrained) << '\n';
std::cout << "icl_frozen_w=" << wIcl
<< " delta_w=" << (wIcl - wPretrained) << '\n';
return 0;
}Output:
fine_tuned_w=1.280 delta_w=0.080
icl_frozen_w=1.200 delta_w=0.000What persists to subsequent calls
Process | What changes during the step | How to reuse the result in future calls |
|---|---|---|
Pretraining | Trainable model parameters | Load the resulting pretrained checkpoint |
Fine-tuning | All or a subset of trainable parameters (e.g., LoRA adapters) | Load the fine-tuned checkpoint or adapter weights |
In-context learning | Current input token sequence and forward-pass activations (such as KV cache) | Provide the required instructions and demonstrations again in the next prompt |
- Process
Pretraining
- What changes during the step
Trainable model parameters
- How to reuse the result in future calls
Load the resulting pretrained checkpoint
- Process
Fine-tuning
- What changes during the step
All or a subset of trainable parameters (e.g., LoRA adapters)
- How to reuse the result in future calls
Load the fine-tuned checkpoint or adapter weights
- Process
In-context learning
- What changes during the step
Current input token sequence and forward-pass activations (such as KV cache)
- How to reuse the result in future calls
Provide the required instructions and demonstrations again in the next prompt
Applications that persist chat histories in a database or retrieve external documents (RAG) to prepend to future prompts rely on separate application-level storage and retrieval pipelines (see State). Observing that a service remembers an earlier turn does not imply that the model weights were fine-tuned; one must inspect whether checkpoint parameters changed or whether stored context was re-injected into the prompt.
See also
Footnotes
- Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever, "Improving Language Understanding by Generative Pre-Training," OpenAI Technical Report, 2018, §3.1-3.2. https://cdn.openai.com/research-covers/language-unsupervised/languageunderstandingpaper.pdf ↩
- Hugging Face Transformers Documentation, "Causal language modeling." https://huggingface.co/docs/transformers/tasks/language_modeling ↩
- Edward J. Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models," International Conference on Learning Representations (ICLR), 2022. https://arxiv.org/abs/2106.09685 ↩
- Tom B. Brown et al., "Language Models are Few-Shot Learners," Advances in Neural Information Processing Systems (NeurIPS) 33, 2020, §1-2, Figure 1.1. https://arxiv.org/abs/2005.14165 ↩