From Scale to Capability Scaling Laws and Emergence

1. Scaling Laws

Large language models are usually pretrained with Next Token Prediction. Given the preceding tokens, the model predicts a probability distribution over the next token.

If the model assigns probability (p) to the correct token, the corresponding loss can be written as:

$$L=-\log p$$

The higher the probability assigned to the correct token, the lower the loss. Training continuously adjusts model parameters so that the correct token receives higher probability.

This gives us a relatively unified metric for training quality: Cross Entropy Loss.

Once loss is available as a common metric, model scale no longer has to be compared only by running a collection of downstream tasks. We can directly study how loss changes as parameter count, training data, and compute increase.

In 2020, OpenAI’s paper Scaling Laws for Neural Language Models studied this problem systematically. It focused on three main quantities: model parameters (N), training data (D), and training compute (C). Here, (D) is usually measured by the number of training tokens.

One of the most important empirical findings was that loss often follows an approximate Power Law with respect to these scaling variables.

Power Laws

When only the amount of training data increases, loss can be approximated by:

$$L(D)\approx\left(\frac{D_c}{D}\right)^{\alpha_D}$$

In Kaplan et al.’s experiments:

$$\alpha_D\approx0.095$$

More data lowers the loss, but the exponent is small. Increasing the dataset by 10× does not reduce the loss by 10×.

Parameter count follows a similar pattern:

$$L(N)\approx\left(\frac{N_c}{N}\right)^{\alpha_N}$$

with:

$$\alpha_N\approx0.076$$

Training compute also exhibits an approximate power-law relationship:

$$L(C_{\min})\approx\left(\frac{C_c}{C_{\min}}\right)^{\alpha_C}$$

with:

$$\alpha_C\approx0.050$$

All three curves show the same general behavior: scaling usually keeps helping, but with diminishing returns. Larger models, more data, and more compute can still reduce loss, but achieving the same additional improvement requires increasingly more resources.

The value of Scaling Laws is not that they reveal a single “best model size.” Their real value is that they turn the relationship between model quality and resource investment into an empirical pattern that can be fitted and extrapolated.

---

2.Training Compute

In discussions of Scaling Laws, Compute appears everywhere, but compute is not simply the number of GPUs being used.

For a classic Dense Transformer, training compute can be roughly approximated as:

$$C\approx6NBS$$

where (B) is the Batch Size and (S) is the number of training steps. The number of training tokens (D) is approximately:

$$D\approx BS$$

which gives the commonly used approximation:

$$C\approx6ND$$

Training cost therefore depends on both parameter count and the number of processed tokens. Doubling model size increases compute if the amount of training data stays fixed. Keeping the model unchanged while training on more tokens also requires additional compute.

Under a fixed compute budget, model size and training data therefore compete for the same resources.

Batch Size and Training Steps

Batch Size is the amount of data used to compute the gradient for one parameter update.

If the batch is too small, gradient estimates contain more noise and parameter updates fluctuate more. Increasing the batch size makes gradients more stable and can improve parallel efficiency, but the benefit does not continue indefinitely.

Once the batch size exceeds the Critical Batch Size, further increases bring much smaller optimization gains even though computational cost continues to rise.

Training for more steps has a similar diminishing-return effect. Longer training generally continues to reduce loss:

$$S\uparrow\Rightarrow L\downarrow$$

but the improvement usually slows later in training. Simply extending the same training run indefinitely is not an efficient use of compute.

Compute itself is only a budget. Training efficiency depends on how that budget is allocated.

---

3. Models and Data

If the dataset remains fixed while model size keeps increasing, performance usually improves quickly at first. Eventually, however, the gains begin to shrink.

The reason is straightforward. Model capacity has increased, but the amount of information provided by the training data has not. This creates a Data Bottleneck: the model has many parameters, but not enough data to train that capacity effectively.

The reverse situation also exists. If the model is too small, giving it far more data may not help much because the model lacks enough capacity to absorb the information. This is a Model Bottleneck.

Model size and data size therefore cannot be considered independently. Larger models need more training data, and the harder question is how quickly the two should grow relative to each other.

---

4. Kaplan’s Allocation

Kaplan’s work did more than describe Scaling Laws. It also studied Compute-Optimal training: given a fixed training compute budget, how should parameters and data be chosen to minimize loss?

The paper estimated:

$$N_{\mathrm{opt}}\propto C^{0.74}$$ $$D_{\mathrm{opt}}\propto C^{0.27}$$

The exponents are very different. As compute increases, the compute-optimal model size grows much faster than the amount of training data.

This naturally pushes training strategy toward larger models. With more compute available, a greater share of the additional budget is spent on increasing parameter count.

Many large language models later scaled rapidly in this direction. Two years later, however, Chinchilla introduced an important correction.

---

5. Chinchilla’s Correction

In 2022, DeepMind’s Training Compute-Optimal Large Language Models revisited how model size and data should be balanced under a fixed compute budget.

Its main argument was that many large models at the time were actually undertrained: their parameter counts were already very large, but they had not seen enough training tokens to make full use of that capacity.

Chinchilla found relationships closer to:

$$N_{\mathrm{opt}}\propto C^{0.5}$$ $$D_{\mathrm{opt}}\propto C^{0.5}$$

Model size and training data should grow at roughly similar rates.

If compute increases by 100×, a rough interpretation is:

$$N\times10,\qquad D\times10$$

rather than spending most of the additional compute on parameters alone.

I initially tended to think of Chinchilla as a rejection of Kaplan. The more useful distinction is that the two do not disagree on whether scaling works. Both accept that increasing scale can continue to reduce loss. The disagreement is about how a fixed compute budget should be divided between parameters and data.

Comparison	Kaplan	Chinchilla
Year	2020	2022
Main focus	How loss scales as models grow	How to balance parameters and data under fixed compute
Model size	Grows roughly with compute to the power of 0.74	Grows roughly with compute to the power of 0.5
Data size	Grows roughly with compute to the power of 0.27	Grows roughly with compute to the power of 0.5
Training tendency	Spend more new compute on model size	Scale model size and data more evenly

This correction shifted large-model training away from simply chasing parameter count and toward Compute-Optimal Training.

How Chinchilla Verified It

Chinchilla did not train a single model and fit one curve. It repeatedly searched for the best model-data combination under fixed compute from several different angles.

Training Curve Envelope trains many models of different sizes and records their loss against compute. At budgets such as (10^{18}), (10^{19}), and (10^{20}) FLOPs, the model with the lowest loss is selected.

Connecting these optimal points forms an Efficient Frontier / Envelope, which can then be used to study how the optimal model size and optimal data size change with compute.

IsoFLOP takes the opposite view and fixes compute directly.

Because:

$$C\approx6ND$$

a fixed compute budget forces model size and training tokens to trade off against each other. A larger model can process fewer tokens; a smaller model can be trained on more data.

The resulting curve is typically U-shaped. On one side, the model is too small and capacity becomes the bottleneck. On the other, the model is too large but sees too little data. Somewhere in the middle lies a lower-loss combination.

A third approach models loss directly as a function of model size and data:

$$L(N,D)=E+\frac{A}{N^\alpha}+\frac{B}{D^\beta}$$

This expression separates loss into three parts. (E) represents irreducible error. The second term decreases as model size (N) grows and captures the penalty from insufficient model capacity. The third decreases as data size (D) grows and captures the penalty from insufficient training data.

With the compute constraint (C\approx6ND), the question becomes: under a fixed compute budget, what values of (N) and (D) minimize loss?

The exact exponents differ slightly across the three methods, but they support the same broad conclusion: model size and data should scale more evenly than Kaplan’s earlier estimate suggested.

---

6. Limits of Scaling

Scaling Laws are often summarized in terms of parameters, data, and compute, but real training systems are more complicated than three variables.

Learning Rate

Learning Rate controls the size of each parameter update. If it is too large, training can become unstable. If it is too small, convergence becomes unnecessarily slow.

Larger batch sizes usually produce more stable gradient estimates, which often allows a larger learning rate. Batch Size and Learning Rate are therefore not completely independent hyperparameters.

Architecture

Kaplan’s classic experiments were mainly based on Dense Transformers. With total parameter count held fixed, changes in depth, width, and the number of Attention Heads had relatively limited influence on the overall scaling trend.

That result should not be applied blindly to all modern architectures. MoE (Mixture of Experts), GQA (Grouped Query Attention), and MQA (Multi-Query Attention) change computational efficiency and parameter usage, which also changes what “the same parameter count” or “the same compute” actually means.

Context

When predicting a token, having more preceding context usually makes the prediction easier. In the classic experiments, longer context generally reduced loss.

Tokens near the beginning of a sequence have very little context, while later tokens can use much more information from previous positions. Larger models also tend to make better use of that context.

---

7. Distribution Shift

Training data and test data do not always come from the same distribution. A model may be trained on WebText2, for example, and evaluated on Books, Wikipedia, or Common Crawl. This is an Out-of-Distribution (OOD) setting.

Loss usually increases on OOD data because the test distribution has changed. Even so, the broad scaling trend often remains: larger models still tend to achieve lower loss.

This suggests that Scaling Laws have some degree of stability across distributions, but it should not be interpreted as “larger models automatically solve distribution shift.” If the training data lacks a certain kind of knowledge, or the test distribution changes substantially, increasing parameter count alone does not guarantee that the problem disappears.

---

8. Emergent Abilities

Scaling Laws usually describe Cross Entropy Loss, which tends to change smoothly as models grow.

Later work shifted attention from average loss to specific tasks and found a different pattern. On some tasks, smaller models perform close to random guessing, while performance rises sharply once model scale enters a certain range.

This behavior is commonly described as Emergent Abilities.

Tasks that have shown this kind of pattern include:

• Modular Arithmetic
• IPA Transliteration
• Word Unscramble
• Persian QA

Here, “emergence” describes the shape of the task-performance curve. It does not by itself prove that some internal capability suddenly appeared inside the model at a precise scale.

Few-shot and ICL

Emergence research often involves Few-shot Prompting, because some capabilities appear without any additional parameter training. The model is simply given a few examples in context.

For example:

Review: This movie sucks.
Sentiment: negative.

Review: I love this movie.
Sentiment:

If the model outputs positive, it has inferred the task pattern from the examples already present in the prompt.

This is usually called Few-shot Prompting. More broadly, when a model uses examples, instructions, or patterns in the current context to perform a task, the behavior can be understood as In-context Learning (ICL). Few-shot Prompting is one common form of ICL. The most direct difference from Fine-tuning is that the model’s parameters are not updated during inference.

Reasoning and Instructions

Several methods for improving reasoning or instruction-following have also shown scale-dependent behavior.

Chain-of-Thought (CoT) uses intermediate reasoning steps to help the model solve more complex problems. Smaller models may gain little from CoT, while larger models can benefit much more, which is why CoT has sometimes been described as showing emergent behavior.

Instruction Tuning trains models on natural-language instructions such as summarization, translation, question answering, reasoning, and code generation. In some experiments, the benefit is limited for smaller models and becomes much larger as model scale increases.

Scratchpad follows a similar idea: the model keeps intermediate computation steps before producing the final answer. On complex arithmetic and multi-step reasoning tasks, larger models are often better able to use those intermediate steps.

These observations lead to a deeper question: if underlying capabilities improve continuously, why can task accuracy appear to jump suddenly?

---

9. Why Emergence Can Look Sudden

One common explanation comes from multi-step tasks.

Suppose a task requires (L) consecutive steps, and each step is completed correctly with probability (p). If every step must be correct, the overall success probability is approximately:

$$p^L$$

For a task with 10 steps:

$$0.8^{10}\approx10.7\%$$ $$0.9^{10}\approx34.9\%$$ $$0.97^{10}\approx73.7\%$$

The per-step accuracy rises continuously from 80% to 97%, but the final task success rate jumps from about 10.7% to 73.7%. There is no sharp breakpoint in the underlying capability, yet the task-level metric becomes very steep.

This does not explain every reported case of emergence, but it shows one plausible mechanism: continuously improving low-level capabilities can produce a sharply nonlinear task curve once they are composed across multiple steps. No sudden internal “phase transition” is required.

---

10. The Evaluation Debate

Emergent Abilities later became controversial, and one important reason is the Evaluation Metric.

Exact Match is a simple example. A completely correct answer receives 1; anything else receives 0.

Suppose model quality is actually improving continuously:

20 → 40 → 60 → 80 → 100

Exact Match might still record:

0 → 0 → 0 → 0 → 1

The resulting curve makes the model look as if it suddenly “acquired” the capability at the final scale point. Part of the apparent discontinuity, however, may come from the discrete evaluation metric rather than a genuinely discontinuous change inside the model.

A safer interpretation of Emergence is therefore that some task metrics do show very steep improvements with scale, but whether this represents a truly discontinuous capability transition depends on the task and the way it is measured.

---

11. Scaling and Emergence

Scaling Laws and Emergence describe different levels of behavior. The former mainly studies average Cross Entropy Loss; the latter asks whether a specific task can be completed.

As model scale increases, loss may decrease smoothly:

3.0 → 2.9 → 2.8 → 2.7 → 2.6

A complex task, however, may depend on language understanding, factual knowledge, context processing, logical reasoning, and mathematical ability at the same time. If even one component remains too weak, the final task can still fail.

As these underlying capabilities improve, several of them may reach a usable level within the same range of model scales. Average loss can therefore keep following a smooth Scaling Law while task accuracy rises sharply.

The two curves are not contradictory. They simply describe different aspects of model behavior.

---

12. Key Takeaways

• Scaling Laws describe empirical relationships between model scale and loss. As parameter count, training data, and compute increase, loss usually decreases according to an approximate power law, with diminishing returns.
• Training compute can be roughly approximated as C ≈ 6ND, where (N) is model size and (D) is the number of training tokens. Parameters and data therefore consume the same compute budget.
• Compute-Optimal Training is not about making the largest possible model. It asks how to divide a fixed compute budget between model size and training data to minimize loss.
• Kaplan allocated more of additional compute toward model parameters, while Chinchilla argued that many early large models were undertrained and that model size and training tokens should grow more evenly.
• Scaling behavior depends on more than (N), (D), and (C). Batch Size, Learning Rate, architecture, context length, and data distribution all affect real training.
• Emergent Abilities describe sharp improvements on some specific tasks as model scale increases. They do not by themselves prove that a model undergoes a sudden internal qualitative change at a particular scale.
• Smoothly improving low-level capabilities can become sharply nonlinear after multi-step composition, and discrete metrics such as Exact Match can make the transition look even more abrupt. This is why Scaling Laws and Emergence are compatible: the former mainly describes average loss, while the latter describes task-level performance.

---

13. References

• Jared Kaplan et al., 2020. Scaling Laws for Neural Language Models
• Jordan Hoffmann et al., 2022. Training Compute-Optimal Large Language Models
• Jason Wei et al., 2022. Emergent Abilities of Large Language Models
• Rylan Schaeffer, Brando Miranda, Sanmi Koyejo, 2023. Are Emergent Abilities of Large Language Models a Mirage?
• ZOIMI https://github.com/Infrasys-AI/AIInfra/blob/main/00Summary/02StandardScaling.md