Before Learning AI Infra, I Went Back to Review C++

Recently, as I started seriously filling in the low-level foundations for AI Infra, I went back and reviewed C++ once again.

This note is not intended to cover all of C++. It only organizes a set of fundamentals that repeatedly show up later when reading Runtime, Tensor, CUDA, and operator code. I am still learning as well, and I will continue updating this article as I learn more. If anything here is inaccurate, corrections and discussion are welcome.

As we move further into AI Infra, we will quickly encounter code like this:

// T is a type parameter. It can be replaced with concrete types such as float or int during instantiation.
template <typename T>
class Tensor {
public:
    T* data();              // Return the address of the beginning of the data
    const T* data() const;  // Return the address of the beginning of the data, but do not allow modification through it

private:
    std::unique_ptr<T[]> storage_;  // Exclusively owns an array of T
};

Low-level function interfaces are also often written like this:

// input / weight are inputs, and output stores the result
void launch_rmsnorm(
    const half* input,
    const half* weight,
    half* output,
    int hidden_size
);

1. Objects and Lifetime

int x = 10;

The most direct description of this line is: “an integer variable x is defined with the value 10.” That is correct, but when we look further down the stack, there is more to care about: the type of x, how much space it occupies, its address, its lifetime, and whether it can be accessed through its address. Going even lower, we also need to consider whether compiler optimizations may change the form in which it actually exists.

Object

Variables are already familiar in C. In C++, we also frequently use a broader concept: Object. Variables such as x and y can both be viewed as objects.

An Object is an entity that occupies storage and has a type and a lifetime while the program is running.

For example:

int x = 10;
float y = 3.14f;

x and y are objects of different types.

A type does more than determine “what something is called.” It also affects how much storage it needs, how its binary representation should be interpreted, what operations are allowed, and how the object is created and destroyed.

sizeof

Objects occupy storage space, and sizeof is used to obtain the number of bytes occupied by the representation of a type or object.

#include <iostream>

int main() {
    std::cout << sizeof(int) << '\n';
    std::cout << sizeof(float) << '\n';
    std::cout << sizeof(double) << '\n';
}

For example, on a typical desktop platform, you may see sizeof(int) == 4.

But that does not mean C++ requires int to always be 4 bytes.

The C++ language does not require int to have a fixed size of 4 bytes on every implementation. The concrete size of fundamental types is implementation-dependent. Therefore, when this note discusses low-level data widths, it will try to use explicit descriptions such as FP32 / INT8 / 32-bit, rather than treating sizeof(int) on one machine as a universal rule.

In AI Infra, explicit data formats are even more common: FP32 is 32 bits, FP16 and BF16 are 16 bits, and INT8 is 8 bits.

Address

Besides size, an object also needs an address to determine where it is stored.

#include <iostream>

int main() {
    int x = 10;

    std::cout << &x << '\n';
}

&x obtains the address of x. The output may look like:

0x7ffd12345678

A simplified memory diagram is:

Memory Address

0x1000    [  x = 10  ]
0x1004    [  ...     ]
0x1008    [  ...     ]

An address determines the specific storage location associated with a read or write operation.

Scope

An address describes where an object is located; Scope describes where a name is visible in the source code.

int main() {
    int a = 1;

    {
        int b = 2;

        // Both a and b can be used here
    }

    // a can still be used here
    // but the name b is no longer in scope
}

Scope mainly describes where a name can be used in source code.

Lifetime

Scope and Lifetime are easy to confuse. A name becoming invisible does not necessarily mean the object no longer exists, and the reverse is also true. Lifetime describes whether an object still exists at runtime.

The following example shows a problem that can occur when Scope and Lifetime are separated:

// p does not point to any variable yet
int* p = nullptr;

{
    int x = 10;
    p = &x;
}

// x's lifetime has already ended
// p still stores the old address

At this point, the pointer variable p itself still exists, but the x it originally pointed to no longer exists.

Dereferencing p afterward:

std::cout << *p;

is undefined behavior.

Scope vs Lifetime

• Scope: whether a name can be written and used at a given point in the code;
• Lifetime: whether the object still exists at runtime.

Stack and Heap

Once lifetime is understood, the stack and heap should no longer be treated as the only way to decide whether an object exists. They describe common forms of storage organization, while lifetime describes when an object’s existence begins and ends.

A common simplified process address-space model looks like this:

Process Address Space
├── Code
├── Global / Static
├── Heap
└── Stack

This diagram is useful, but it is only a simplified model.

For example:

int main() {
    int x = 10;
}

At the introductory level, this is usually described as “the local variable x is stored on the stack.”

After real compiler optimization, however, the compiler may:

• keep it only in a register;
• eliminate it completely;
• avoid allocating a traditional stack slot for it.

Dangling Pointer

The situation where “the pointer still exists, but the object is already gone” is dangerous. It has a common name: Dangling Pointer.

// p does not point to any variable yet
int* p = nullptr;

void f() {
    int x = 42;
    p = &x;
}

int main() {
    f();
    std::cout << *p << '\n';
}

The problem in this code is lifetime. p is a global variable, so it exists for a long time.

The key issue is the x it points to.

x is a local automatic object inside f(), and its lifetime ends when execution leaves the corresponding scope.

After f() returns, the lifetime of the local variable x ends.

So although p still stores the original address, that address no longer refers to a live x. At this point, p becomes a Dangling Pointer, and dereferencing it is undefined behavior.

The following program can print both the size and address of variables:

#include <iostream>

int main() {
    int x = 10;
    double y = 3.14;

    std::cout << "sizeof(x) = " << sizeof(x) << '\n';
    std::cout << "address x = " << &x << '\n';
    std::cout << "address y = " << &y << '\n';
}

There is no need to analyze the exact difference between the two addresses here. The important point is to distinguish a variable’s value, type, address, and lifetime.

2. Pointers and References

Objects have addresses. Pointers store addresses and use those addresses to access the original objects.

Pointer

int x = 10;
int* p = &x;  // p stores the address of x

The relationship between the pointer and the object can be drawn as:

p

┌────────────┐
│ 0x1000     │
└─────┬──────┘
      │
      ▼

0x1000
┌────────────┐
│ x = 10     │
└────────────┘

This relationship can be summarized as follows: x is an int; &x is the address of x; p stores that address; and *p accesses the original int object through that address.

Dereference

After storing an address, we still need a way to access the original object through that address. C++ uses *p for this operation, which is called Dereference.

int x = 10;
int* p = &x;  // p stores the address of x

*p = 99;  // Find x through p and change x to 99

std::cout << x << '\n';

The output is 99.

The reason is that *p does not create a new variable. It accesses the original object through the address stored in p.

nullptr (Null Pointer)

A pointer does not always point to a valid object. When we need to explicitly represent “there is currently no target object,” we can use nullptr.

// p does not point to any variable yet
int* p = nullptr;

This means p currently does not point to an int object that can be safely dereferenced.

A null pointer must not be directly dereferenced:

*p = 42;

This is undefined behavior.

Arrays and Pointers

One of the most common use cases for pointers is arrays. Since an array is already a sequence of elements stored contiguously, the address of where the array begins is very important.

// Here, the array name a can be understood as the position of the first element
int a[4] = {10, 20, 30, 40};

In many expressions, an array is converted to a pointer to its first element.

int* p = a;

So we can access elements using indexing:

p[0] == 10
p[1] == 20

or use equivalent pointer arithmetic:

*(p + 2) == a[2]

Pointer Arithmetic

Pointer arithmetic is used to move from the current element position to another element position in an array.

For an int* p, the expression

p + 1

means moving to the position of the next int element.

If sizeof(int) == 4 on the current platform, then at the low-level address representation, p + 1 will usually move forward by 4 bytes.

This is the foundation of linear indexing for Tensors later:

// Row row and column col correspond to this position in the one-dimensional array
data[row * cols + col]

Pointer + Length

Knowing only where an array begins is not enough. A function also needs to know where the array ends. That is why low-level interfaces often pass both an “address” and a “length.”

// data points to the beginning of the array, and n is the number of elements
void scale(float* data, int n, float alpha) {
    for (int i = 0; i < n; ++i) {
        data[i] *= alpha;
    }
}

The two parameters describe the memory location and the number of elements: data points to the start of the array, and n represents the number of elements.

In engineering code, std::size_t is often used for lengths. This example temporarily uses int to demonstrate the “address + length” interface pattern. The later vector and MiniTensor examples will use std::size_t.

CUDA Kernels also use this design extensively:

kernel(const float* x, float* y, int n);

Reference

Pointers require explicit handling of nullptr and dereferencing. If an interface semantically requires an object to exist and wants to directly operate on that original object, a Reference can be used.

// r is a reference to x, that is, an alias
int x = 10;  // Create an int variable
int& r = x;  // r is a reference (alias) to x

r = 20;  // This actually modifies x

std::cout << x << '\n';

The simplest intuition for a Reference is: give an already existing object another name.

Pointer vs Reference

Pointers and references both allow a function to access an original object, but they express different semantics.

Question	T*	T&
Can represent “no object”	Yes, with nullptr	Usually not used to represent a null value
Can be redirected elsewhere	Yes	No rebinding after binding
Access	*p	Used directly like the object
Common use	Arrays, low-level memory, optional objects	Function parameters that must refer to an existing object

Interface design can distinguish between them based on semantics:

If a parameter must correspond to an existing object, a reference T& is often more direct. If “no object” is allowed, a pointer T* can express that using nullptr. A raw pointer itself does not say who is responsible for releasing the resource; that is an ownership issue covered in the next section.

const

Pointers and references determine how the original object is accessed. const further restricts access permissions by expressing “read-only, not writable” in the type system.

Pointer to Const

float x = 1.0f;
const float* p = &x;

Here, the pointed-to float cannot be modified through p.

Const Pointer

float x = 1.0f;
float* const p = &x;

p itself cannot be redirected, but *p can still be modified.

Double const

float x = 1.0f;
const float* const p = &x;

const in Kernel Interfaces

In Kernel interfaces, const can directly distinguish read-only inputs from writable outputs.

// a and b are read-only, while c is used to write the result
void vector_add(
    const float* a,
    const float* b,
    float* c,
    std::size_t n
);

So just by looking at the function declaration, we know that a and b are read-only inputs, while c is used to write output.

In this sense, the function signature itself serves as part of the interface documentation.

Parameter Passing

Combining values, pointers, references, and const gives several common parameter-passing styles in C++.

Pass by Value

void f(int x) {
    x = 100;
}

int a = 10;
f(a);

a does not change because of x = 100 inside the function.

The function receives its own parameter value.

Pass by Reference

void f(int& x) {
    x = 100;
}

int a = 10;
f(a);

After the call, the original a becomes 100, because the parameter x is a reference to a, meaning it is an alias.

Pass by Pointer

void f(int* x) {
    if (x != nullptr) {
        *x = 100;
    }
}

int a = 10;
f(&a);

const T&

For small values such as int, making a copy is usually inexpensive. But when objects become larger, copying may be costly. That is why const T& is extremely common: it avoids copying while preventing the function from modifying the object.

void print_tensor(const Tensor& t);

If we instead write:

void print_tensor(Tensor t);

the object may be copied.

Using:

const Tensor& t

allows:

• no copy;
• read-only access.

Move Semantics further affect the cost model of value passing. For now, it is enough to know that copying large objects may have a noticeable cost.

3. Memory Ownership

Pointers solve the problem of how to access memory, but low-level code must also define who is responsible for releasing a resource. This is the problem solved by Ownership.

The most direct approach is dynamic allocation:

int* p = new int(42);

After using it:

delete p;

These few lines look simple enough.

In real projects, however, there are early returns, exceptions, object copies, multiple objects storing the same address, thread concurrency, and many other situations. Manually remembering “a delete is needed here” quickly becomes unreliable.

new / delete

Single objects and arrays use different deallocation forms:

int* p = new int(42);
delete p;

int* a = new int[100];
delete[] a;

The basic pairing rule is: new pairs with delete, and new[] pairs with delete[].

Memory Leak

The biggest problem with manual new / delete is not the syntax. It is how easy it is to forget to release memory. The most typical consequence is a Memory Leak.

// If the function returns early, the delete[] below will never execute
void f(bool early_return) {
    int* p = new int[1000];

    if (early_return) {
        return;
    }

    delete[] p;
}

When early_return == true, the function returns before delete[] p has a chance to execute, so the dynamically allocated memory is never released. If a program repeatedly allocates memory this way without releasing it, unused memory accumulates over time. That is a memory leak.

Double Free

Forgetting to release a resource is a problem, but releasing the same resource twice is also a problem. This is called Double Free.

// The same memory must not be released twice
int* p = new int(42);

delete p;
delete p;  // Error

Releasing the same resource twice can cause serious errors.

Ownership

Memory Leak and Double Free point to the same underlying issue: responsibility for releasing a resource must be clear. This is what Ownership is about.

The core idea of Ownership is: whoever owns a resource is responsible for releasing it at the appropriate time.

For example:

float* p;

float* p only stores an address. It does not tell us whether p owns the memory, nor who will eventually release it. Large systems therefore need clear ownership conventions.

Construction, Destruction, Copying, and Moving

When a resource is placed inside a class, the lifetime of the object directly affects the lifetime of the resource. Several special member functions are especially important in C++: a Constructor initializes an object when it is created; a Destructor performs cleanup when the object’s lifetime ends; copy construction and copy assignment define what it means to copy an object; move construction and move assignment define whether a resource can be transferred from one object to another.

Later, resource-managing classes will contain declarations like:

CudaBuffer(const CudaBuffer&) = delete;
CudaBuffer& operator=(const CudaBuffer&) = delete;

Here, = delete explicitly tells the compiler that the type cannot be copied. For types that directly manage files, locks, CPU / GPU memory, or other resources, these operations must be designed together with Ownership.

RAII

Having clear release responsibility is still not enough. Resource release should preferably be tied to object lifetime instead of relying on manually remembering every delete. Modern C++ mainly uses RAII for this.

RAII stands for Resource Acquisition Is Initialization. Its core idea is that resources are automatically released along with the objects that manage them.

The basic RAII pattern is: acquire the resource when the object is created, manage the resource while the object is alive, and automatically release the resource when the object is destroyed.

Rule of Zero

The Rule of Zero is a further engineering application of RAII: if standard library types can already manage the resource, let those types handle the resource lifetime instead of repeatedly writing custom resource-management code.

For example, when implementing a floating-point Buffer, we use class formally for the first time. Here, think of a class as a type that organizes a group of data together with functions that operate on that data. Details such as public and private access control are not expanded here.

A direct implementation would manually allocate:

new float[n]

A more suitable approach here is to let a standard container manage the resource:

class FloatBuffer {
public:
    explicit FloatBuffer(std::size_t n)
        : data_(n) {}

    float* data() {
        return data_.data();
    }

    const float* data() const {
        return data_.data();
    }

    std::size_t size() const {
        return data_.size();
    }

private:
    std::vector<float> data_;
};

Here, std::vector is the type actually responsible for managing the memory. FloatBuffer does not need to write manual release logic. When the vector is destroyed, it automatically cleans up the memory it manages.

Therefore, FloatBuffer does not need to implement its own destructor, copy constructor, move constructor, copy assignment operator, or move assignment operator just for resource management.

That is the Rule of Zero: whenever standard library resource-management types can do the job, avoid manually implementing resource lifetime logic.

Risks of Raw Pointers

The following class shows why ownership must be explicit: a raw pointer stores only an address and does not automatically tell the compiler who is responsible for releasing the resource.

// This design is risky: after copying the object, two objects may point to the same memory
class BadBuffer {
public:
    explicit BadBuffer(std::size_t n)
        : data_(new float[n]) {}

    ~BadBuffer() {
        delete[] data_;
    }

private:
    float* data_;
};

The problem appears when the object is copied:

BadBuffer a(100);
BadBuffer b = a;

With the default copy behavior, the most dangerous part is that the address stored in data_ is simply copied as well.

It may become:

a.data_ ─┐
         ├──→ same array
b.data_ ─┘

The result is that when a is destroyed, the memory is released once, and when b is destroyed, the same memory is released again, causing a Double Free.

Therefore, if a type directly owns a resource, its copy and move semantics must be designed together.

This is the reason behind the Rule of Five.

unique_ptr (Exclusive Smart Pointer)

If a resource should truly have only one owner, C++ can express that relationship directly in the type using unique_ptr.

auto lets the compiler infer the variable type from the initialization expression. It will be discussed further in the templates section.

#include <memory>

auto p = std::make_unique<int>(42);

unique_ptr expresses unique ownership, so it cannot be copied directly:

auto q = p;  // Copying is not allowed

This prevents a second owner of the same resource from being created by copying.

std::move

unique_ptr cannot be copied, but its ownership can be transferred from one object to another through a move.

auto p = std::make_unique<int>(42);
auto q = std::move(p);  // Transfer ownership of the memory to q

After the move, ownership belongs to q, and p no longer owns the original object.

A Common Misunderstanding About std::move

The name std::move can be misleading: the function itself does not directly move a large block of data.

From a usage perspective, std::move(x) indicates that subsequent operations may try to use move semantics.

More precisely, std::move(x) itself only converts x into a form that can trigger move semantics. How the resource is actually transferred is determined by the type’s move constructor and move assignment operator.

Moving a vector

Suppose we first create a very large vector:

std::vector<float> a(10'000'000);

A direct copy:

std::vector<float> b = a;

may require copying a large number of elements. If we instead move it:

std::vector<float> b = std::move(a);

for a standard std::vector, move construction can usually take over the dynamic storage managed by the original container without performing a deep element-by-element copy. Special allocator-related conditions can affect some move-assignment cases.

After the move, a is still a valid object, but we should not assume it still contains the same elements it had before the move. The standard commonly describes this as valid but unspecified: the object is still valid, but its exact remaining state is not guaranteed.

shared_ptr (Shared Smart Pointer)

Not every resource can have exactly one owner forever. If multiple objects genuinely need to jointly determine how long a resource remains alive, shared_ptr can be used.

// p1 and p2 point to the same object together
auto p1 = std::make_shared<int>(42);
auto p2 = p1;  // p2 and p1 point to the same object

shared_ptr expresses shared ownership. The managed object is destroyed only after the last Owner disappears.

But it is not a universal solution.

Costs and risks include:

• reference counting, which tracks how many shared_ptrs jointly own the object;
• more complicated ownership relationships;
• cyclic references, where two objects keep each other alive through shared_ptr;
• safe reference-count updates do not imply that the pointed-to object can be freely modified by multiple threads at the same time.

weak_ptr (Weak Smart Pointer)

Shared ownership can introduce new problems such as cyclic references. weak_ptr can observe an object managed by shared_ptr without participating in the count that determines whether the object should remain alive.

One of its most common uses is breaking cyclic references.

Ownership Decisions

Prefer storing the object directly as a value member
├─ Possible → use a direct value object
└─ Not possible
   ↓
Check whether a dynamic lifetime is needed
├─ No → do not allocate dynamically
└─ Yes
   ↓
Check whether there is only one Owner
├─ Yes → unique_ptr
└─ No
   ↓
Confirm whether shared ownership is truly necessary
├─ Yes → shared_ptr
└─ No → redesign the Ownership model

Types such as T*, T&, std::span, and std::string_view are usually more suitable for expressing non-owning access: they temporarily use a resource but are not responsible for releasing it.

GPU Buffer

A GPU Buffer often has an interface structure like this:

// Focus only on the structure here: copying is forbidden, but the internal resource can be transferred to another object
class CudaBuffer {
public:
    explicit CudaBuffer(std::size_t bytes);
    ~CudaBuffer();

    CudaBuffer(const CudaBuffer&) = delete;
    CudaBuffer& operator=(const CudaBuffer&) = delete;

    CudaBuffer(CudaBuffer&& other) noexcept;
    CudaBuffer& operator=(CudaBuffer&& other) noexcept;

private:
    void* ptr_ = nullptr;
    std::size_t bytes_ = 0;
};

A common design for resource objects such as GPU Buffers is: copying is forbidden, but moving is allowed. The reason is straightforward—GPU memory resources should usually have one clearly defined Owner responsible for releasing them.

This is where C++ ownership concepts connect directly to CUDA.

4. Contiguous Memory and Data Layout

Once ownership is clear, performance concerns begin to focus heavily on data layout. Contiguous memory is especially important for arrays, Tensors, and GPU Kernels.

Problems with Raw Arrays

A raw array can still be allocated like this:

float* p = new float[n];

But raw arrays push responsibilities such as deallocation, length tracking, copying, capacity growth, and exception safety back onto the caller. The standard library already provides more mature resource-management tools.

std::vector

What a vector Really Is

std::vector is a dynamically sized array that automatically manages its memory. Its most important connection to a C array is that its elements are still stored contiguously in memory.

std:: indicates that a name comes from the C++ standard library. For example, std::vector is the standard library’s vector. Although using can be used to omit some prefixes, this article keeps std:: to make standard library types easier to distinguish from project-defined types.

std::vector<float> v = {1.0f, 2.0f, 3.0f};

Its memory relationship can be simplified as:

v.data()
   │
   ▼
[1.0][2.0][3.0]

vector is also useful for comparing pass-by-value, pass-by-reference, and pass-by-const-reference.

Passing vector as a Parameter

Three common forms are:

void f1(std::vector<int> v);
void f2(std::vector<int>& v);
void f3(const std::vector<int>& v);

f1:

void f1(std::vector<int> v);

The parameter is a new vector object.

If an existing vector is passed in, this form will usually copy its elements.

f2:

void f2(std::vector<int>& v);

No copy is made here, and the caller’s vector can be modified.

f3:

void f3(const std::vector<int>& v);

No copy is made here, and the object cannot be modified through v.

Therefore, when a large vector only needs to be read, it is very common to write:

const std::vector<int>&

Contiguous Memory and Performance

The most direct manifestation of vector’s contiguity is that we can obtain the address of its first element:

float* p = v.data();

This allows the data to be passed to low-level C/C++ APIs.

In CPU code, the starting address returned by vector.data() can be passed to low-level numerical functions. In CUDA, a similar idea is used with a device pointer to pass the location of data to a GPU Kernel.

More importantly, contiguous data is generally easier to use efficiently with Cache and SIMD.

size vs capacity

Automatic growth of a vector relies on capacity management. Before understanding reallocation, we need to distinguish size from capacity.

std::vector<int> v;

std::cout << v.size();
std::cout << v.capacity();

size is the number of elements that currently exist. capacity is the maximum number of elements the current storage can hold without reallocating memory.

reserve

reserve adjusts available capacity but does not create that number of elements.

std::vector<int> v;
v.reserve(1000);  // Reserve capacity without adding elements

reserve(1000) means increasing the capacity so that it can hold at least 1000 elements, but it does not create those elements. Since the vector above was initially empty, size() is still 0 afterward.

resize

resize actually changes the current number of elements.

v.resize(1000);  // Adjust the number of elements to 1000

So reserve and resize are completely different: the former mainly changes capacity, while the latter changes the actual element count.

Growth and Invalidation

When growth triggers reallocation, a vector may move to a new memory location. Previously saved element addresses, pointers, or iterators may then become invalid.

For example, first save the address of an element:

std::vector<int> v;
v.push_back(1);

int* p = &v[0];

Then continue calling push_back:

v.push_back(...);

When capacity is insufficient, the vector may reallocate:

allocate larger memory
↓
move elements
↓
release old memory

After reallocation, p still stores the old memory address, so it may become a dangling pointer.

Iterator

Besides indexing and pointers, the C++ standard library heavily uses Iterators to represent positions inside containers. An iterator is a more general abstraction of “current position” than a raw pointer.

for (auto it = v.begin(); it != v.end(); ++it) {
    std::cout << *it << '\n';
}

One of the most common traversal forms is:

for (int x : v) {
    std::cout << x << '\n';
}

<algorithm>

With iterators, algorithms such as sorting no longer need to be tied to a specific container type. The standard library’s <algorithm> makes extensive use of this design.

#include <algorithm>

std::sort(v.begin(), v.end());

A major design pattern in the C++ standard library is:

Container
   ↓
Iterator
   ↓
Algorithm

rather than implementing the same algorithm API separately for every container.

Complexity and Hardware

Common operation complexities can be remembered roughly like this:

Operation	Complexity intuition
vector[i]	O(1)
push_back	amortized O(1)
insertion in the middle of a vector	O(n)
sort	O(n log n)
average lookup in unordered_map	average O(1)
lookup in map	O(log n)

Performance analysis must distinguish algorithmic complexity from hardware behavior: the same asymptotic complexity can still lead to very different real hardware performance.

For example, even though both are O(n), scanning a contiguous vector is often much friendlier to the CPU Cache than randomly jumping among many scattered pointers.

The Effect of reserve

Compare these two implementations:

std::vector<int> v;

for (int i = 0; i < 1'000'000; ++i) {
    v.push_back(i);
}

and:

std::vector<int> v;
v.reserve(1'000'000);

for (int i = 0; i < 1'000'000; ++i) {
    v.push_back(i);
}

The speed difference between them may not be dramatic on every machine.

The main purpose of reserve is to reduce repeated reallocation while the container grows.

Alignment and Padding

At a low level, knowing only “how many elements there are” is not enough. We also need to know exactly how data is laid out in memory. This introduces Alignment and Padding.

Consider the following structure:

struct A {
    char c;
    int x;
};

Intuitively, it may look like 1 + 4 = 5 bytes.

If we print:

std::cout << sizeof(A);

on a common platform, sizeof(A) will likely be 8.

This is because int usually needs to satisfy a particular alignment requirement.

The compiler may therefore lay it out like this:

offset 0      char c
offset 1~3    padding
offset 4~7    int x

Alignment and GPU

Later, during performance optimization, concepts such as SIMD-aligned loads, vectorized reads and writes, float4, Tensor alignment, and GPU memory access will continue to appear. All of them are related to how data is laid out in memory.

So low-level performance analysis requires not only knowing the number of elements, but also the exact memory layout.

TensorView

Suppose we have:

std::vector<float> storage(1024);

If we only need to represent 200 elements starting from storage[100], there is no need to copy the underlying storage.

We can define a non-owning View:

// TensorView only remembers the data address and length; it is not responsible for deleting the data
struct TensorView {
    float* data = nullptr;
    std::size_t size = 0;
};

Buffer / vector owns the data, while TensorView only stores information such as the address and length. It is a non-owning View that is not responsible for releasing the data.

Modern Tensor frameworks rely heavily on similar ideas.

std::span (C++20)

// span only represents "where a block of data is and how long it is"; it is not responsible for deleting it
#include <span>

void scale(std::span<float> x, float alpha) {
    for (float& v : x) {
        v *= alpha;
    }
}

std::span represents a non-owning view over a contiguous sequence of T.

std::span entered the standard library in C++20. If a project uses C++17, understanding the basic “pointer + size” pattern is enough for now.

5. Templates and Compile Time

The same logic in AI Infra often needs to support many data types such as FP32, FP16, BF16, and INT8. That means the “type” itself also needs to become a parameter.

Why Templates Are Needed

Suppose we already have:

float add(float a, float b);

Later, we may also want to support double, int, half, bfloat16, and other types.

Writing a separate copy of the same logic for every type creates a large amount of duplication. Templates solve exactly this problem: “same logic, different types.”

Function Template

A function template parameterizes the type itself. We first write one generic implementation, and then during instantiation the compiler determines which concrete type, such as int or float, corresponds to T.

template <typename T>
T add(T a, T b) {
    return a + b;
}

During instantiation, we can simply write:

int a = add(1, 2);
float b = add(1.5f, 2.5f);

What Does a Template Actually Do?

The core idea of templates is to parameterize types and let the compiler generate the corresponding implementation for concrete types. Numerical libraries often need the same logic to support FP32, FP16, BF16, INT8, and other data types, so templates are extremely common.

Class Template

The same idea can also be applied to classes. Then a Tensor class does not need separate copies of its implementation for float, int, and other types.

template <typename T>
class Tensor {
public:
    T* data() {
        return data_.data();
    }

private:
    std::vector<T> data_;
};

This gives us, for example:

Tensor<float> a;
Tensor<int> b;

auto

auto x = 10;      // int
auto y = 3.14f;   // float

auto is especially useful when:

• the type is already obvious;
• types such as iterators would be long to write explicitly;
• the concrete type in template code depends on context.

But do not make interfaces harder to read just to save a few characters.

constexpr

constexpr int kBlockSize = 256;

This means the value can participate in compile-time constant evaluation when the relevant language rules are satisfied.

In CUDA template code, you will often see:

template <int BLOCK_SIZE>

which moves certain parameters to compile time.

6. Multithreading and Concurrency

Multithreading allows the CPU to process more work at the same time, but the central challenge of concurrency is how multiple threads safely access shared state.

std::thread

#include <iostream>
#include <thread>

void worker(int id) {
    std::cout << "worker " << id << '\n';
}

int main() {
    std::thread t1(worker, 1);  // Start thread 1
    std::thread t2(worker, 2);  // Start thread 2

    t1.join();  // Wait for thread 1 to finish
    t2.join();  // Wait for thread 2 to finish
}

join() waits for the target thread to finish. There is no guarantee which of the two threads prints first; execution order is determined by scheduling.

Lambda

Threads, condition variables, and Benchmarks later in the article use Lambdas multiple times. For now, a Lambda can be understood as “a small temporary function written directly where it is needed”:

auto f = [] {
    std::cout << "hello\n";
};

f();

The square brackets [] are called the Capture List. They determine whether the Lambda can use variables from the surrounding scope. Three common forms are:

• []: capture no external variables;
• [=]: capture used external values by value by default;
• [&]: capture external objects by reference by default.

When reference capture is used in concurrent code, object lifetime and thread synchronization must also be considered. It is not enough for the syntax to compile.

Shared State

Creating threads itself is not very complicated. The real design problem is synchronization when multiple threads share and modify the same state.

When Thread A and Thread B access shared mutable data at the same time, we must consider whether their accesses conflict.

Data Race

If multiple threads access the same variable at the same time, at least one thread modifies it, and the program does not provide proper synchronization, a Data Race may occur.

// If multiple threads execute ++counter at the same time, the result may be wrong
int counter = 0;

void work() {
    for (int i = 0; i < 100000; ++i) {
        ++counter;
    }
}

When multiple threads run this code at the same time, conflicting accesses to counter are not synchronized, which creates a Data Race. In C++, a Data Race causes undefined behavior, so this is not merely an ordinary logical error where “a few increments may occasionally be lost.”

The Problem with ++counter

Conceptually, it may involve:

load counter
↓
+1
↓
store counter

With two threads:

A load 10
B load 10
A store 11
B store 11

The value should have increased by 2, but it ends up as only 11.

The sequence above is only one possible interleaving. At the language level, the real problem is that unsynchronized conflicting accesses constitute a Data Race.

Mutex

A Mutex protects shared data from concurrent modification by limiting how many threads can enter a Critical Section at the same time.

// mutex acts like a lock: only one thread can modify counter at a time
#include <mutex>

std::mutex m;
int counter = 0;

void work() {
    for (int i = 0; i < 100000; ++i) {
        std::lock_guard<std::mutex> lock(m);  // Lock on entry and automatically unlock on scope exit
        ++counter;
    }
}

lock_guard locks the mutex when it is created and automatically unlocks it when leaving the current scope. In essence, this is RAII applied to lock lifetime management.

Critical Section

The small piece of key code protected by a lock is called a Critical Section.

{
    std::lock_guard<std::mutex> lock(m);  // Lock on entry and automatically unlock on scope exit

    ++counter;
}

The longer the Critical Section, the longer the lock is held and the longer other threads may need to wait. So correctness comes first, and only then should lock contention be reduced.

std::atomic

If the problem is only a simple counter, sometimes we do not need to lock an entire code block. We can use std::atomic. It can make specified operations atomic, so other threads do not observe the operation in a partially completed intermediate state.

// Incrementing counter by 1 is atomic
#include <atomic>

std::atomic<int> counter{0};

void work() {
    for (int i = 0; i < 100000; ++i) {
        counter.fetch_add(1);  // Safely add 1
    }
}

atomic guarantees atomicity for the specified operations on that variable.

But atomic operations only guarantee the atomicity of the relevant operations. They do not make the entire program automatically thread-safe. Relationships among multiple shared variables still require additional synchronization design.

Condition Variable

A Mutex solves mutual exclusion, while a Condition Variable solves the problem of waiting for a condition to change. When the condition is not satisfied, a thread can block; when the condition changes, it can be awakened.

Suppose a consumer thread currently has no task to process.

An inefficient busy-waiting implementation would be:

// This loop keeps running and continuously occupies the CPU
while (queue.empty()) {
}

This loop continuously consumes CPU time. A better approach is to block and wait:

no task
→ sleep and wait

task arrives
→ producer notifies

Producer-Consumer

The classic use case for a condition variable is producer-consumer: one thread pushes tasks into a queue, while another waits to take tasks from it.

The waiting condition below is: either there is a task in the queue, or the program is preparing to finish.

// The consumer waits when there is no task; after the producer adds a task, it wakes the consumer
#include <condition_variable>
#include <mutex>
#include <queue>

std::mutex m;
std::condition_variable cv;
std::queue<int> tasks;
bool finished = false;

void producer() {
    {
        std::lock_guard<std::mutex> lock(m);  // Lock on entry and automatically unlock on scope exit
        tasks.push(42);  // Add one task
    }

    cv.notify_one();  // Wake one waiting consumer
}

void consumer() {
    std::unique_lock<std::mutex> lock(m);  // wait releases the lock first and reacquires it after waking

    cv.wait(lock, [] {
        return !tasks.empty() || finished;
    });

    if (!tasks.empty()) {
        int task = tasks.front();
        tasks.pop();  // Remove the task from the queue

        // Use task
        (void)task;
    }
}

The relationship can be simplified as:

shared condition
→ protected by mutex

condition not satisfied
→ wait

state changes
→ notify

wait and Predicate

A thread being awakened does not mean the Predicate is necessarily true, so the condition still needs to be checked again after waking. Common situations include:

• multiple threads being awakened;
• the condition already being changed by another thread;
• spurious wakeups, where a thread may wake even though no real task has arrived.

Therefore, waiting logic should be based on the Predicate rather than treating a notify as proof that the condition is already satisfied.

Deadlock

If multiple locks form a cycle of waiting dependencies, a Deadlock occurs: threads wait for one another to release resources, and none can continue.

The most typical case is two threads acquiring locks in opposite orders:

Thread A:
lock M1
wait M2

Thread B:
lock M2
wait M1

forming:

A waits for B
↑           ↓
└───────────┘

Common engineering approaches include:

• use a consistent lock acquisition order;
• avoid holding multiple locks at the same time when possible;
• shorten Critical Sections;
• use tools such as std::scoped_lock.

More Threads Do Not Always Mean More Speed

If a single task takes only 1 ms, repeatedly creating and destroying a new thread for every task may cost more than the actual computation.

Other possible costs include:

• context switch: switching the CPU between different threads has a cost;
• Cache Interference: multiple threads may compete for Cache capacity;
• lock contention: threads may all be waiting for the same lock;
• Memory Bandwidth bottleneck: data movement cannot keep up with computation.

A Runtime usually reuses Worker Threads instead of creating a new thread for every task. This is the basic idea behind a Thread Pool.

Concurrency in a Runtime

An inference engine may contain a flow like:

Request Queue
     ↓
Scheduler Thread
     ↓
Worker
     ↓
GPU Launch

So the mutexes, condition variables, queues, and thread lifetime concepts learned here will all appear again later in an inference Runtime.

7. Compilation and Build Systems

Correct code alone does not mean a project can run. Real C++ projects are usually split across many .h and .cpp files, and they still need to go through compilation and linking before becoming an executable program or library.

Build Process

Suppose a project currently contains only main.cpp, tensor.cpp, and tensor.h.

At a high level, the process looks like this:

Source
  ↓
Preprocess
  ↓
Compile
  ↓
Object File
  ↓
Link
  ↓
Executable

Header

When code is split across multiple files, another .cpp file first needs to know that “a function with this interface exists.” Header files are commonly used for such declarations.

tensor.h:

#pragma once

float add(float a, float b);

Its purpose is to declare this function interface to other Translation Units, where a Translation Unit can be understood as one independently compiled unit of source code.

Definition

A declaration only tells the compiler “what this function looks like.” The code that actually performs the work still needs a Definition.

tensor.cpp:

float add(float a, float b) {
    return a + b;
}

The actual implementation is defined here.

undefined reference

For example, main.cpp may already know the declaration of a function through a header file, but if the linker cannot find the actual implementation, compilation may succeed while linking fails. This is when the common undefined reference error may appear.

Static and Dynamic Libraries

Common libraries can be divided into two categories.

Static Library

On Linux, static libraries commonly use the .a suffix.

During linking, the required object code is integrated into the final result.

Dynamic Library

On Linux, dynamic libraries commonly use the .so suffix.

They are loaded and resolved through dynamic linking mechanisms at runtime.

When working with CUDA / PyTorch later, libraries such as libcudart.so, libcublas.so, and libtorch.so will appear frequently.

ABI (Application Binary Interface)

At the library level, another compatibility issue exists below the source-code level: how compiled binaries actually call and cooperate with one another. This is the ABI, or Application Binary Interface.

It concerns how compiled binaries work together, including:

• calling convention: how function parameters and return values are passed;
• data layout: how data is arranged at the binary level;
• symbol naming: what function and variable names become after compilation;
• binary interface: how separately compiled results interact with one another.

Therefore, even if source code looks identical, incompatible compilers, standard libraries, or binary interfaces can still cause linking or runtime problems.

CMake

As the number of project files grows, handwritten compilation commands quickly become difficult to maintain. CMake is not a compiler. It describes source files, build targets, and dependencies in a project.

A minimal CMakeLists.txt can look like:

cmake_minimum_required(VERSION 3.20)

project(MiniTensor LANGUAGES CXX)

add_executable(app
    src/main.cpp
)

target_compile_features(app
    PRIVATE
        cxx_std_17
)

Then configure and build the project from the command line:

cmake -S . -B build
cmake --build build

Target

In a large CMake project, treating everything as a collection of global flags quickly becomes confusing. A better perspective is to view every library and executable as a Target.

For example:

# First build a library, then build demo, and finally let demo use the library
add_library(minitensor
    src/vector_add.cpp
)

target_include_directories(minitensor
    PUBLIC
        ${CMAKE_CURRENT_SOURCE_DIR}/include
)

target_compile_features(minitensor
    PUBLIC
        cxx_std_17
)

add_executable(demo
    src/main.cpp
)

target_link_libraries(demo
    PRIVATE
        minitensor
)

The dependency relationship can be understood as:

minitensor
     ↑
     │ link
     │
    demo

PRIVATE / PUBLIC / INTERFACE

The three visibility levels can be summarized as follows: PRIVATE applies only to the current Target; PUBLIC applies both to the current Target and to Targets that use it; INTERFACE applies only to other Targets that use the current Target.

Later, this can be understood in more depth together with:

• include dirs;
• compile features;
• link libraries.

References

The terminology and engineering principles in this note primarily refer to the following first-party or official materials. One important clarification: the C++ Core Guidelines are modern C++ engineering guidelines, not the ISO C++ standard text itself. For definitive language specifications, the corresponding ISO C++ standard should be used. This note mainly uses the Guidelines to cross-check principles related to resource management, interface design, and concurrent programming.

C++ Core Guidelines

https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines

The sections especially worth reading include Resource Management (R), Concurrency and Parallelism (CP), Functions (F), and Classes (C).

When specific problems arise later, it is especially useful to revisit:

• RAII;
• Ownership;
• unique_ptr;
• lock RAII;
• avoiding returning references or pointers to local objects.

Standard C++ Foundation

https://isocpp.org/

This is an entry point for information about the C++ standards ecosystem, the committee, and official community resources.

Official CMake Tutorial

https://cmake.org/cmake/help/latest/guide/tutorial/index.html

It is worth actually going through the official tutorial sections on Executables, Libraries, Usage Requirements, Testing, Installing, and Dependencies.

CMake Buildsystem Manual

https://cmake.org/cmake/help/latest/manual/cmake-buildsystem.7.html

This manual is more suitable for further understanding Targets, Usage Requirements, and Transitive Properties.