GPU architecture

GPU execution model, memory hierarchy, occupancy

Lars Pastewka

Learning objectives

After this lecture you can

  • sketch the topology of a GPU node and rank its interconnect bandwidths
  • explain the SIMT thread hierarchy (grid, block, warp) and how it maps to SMs
  • navigate the GPU memory hierarchy and quote its key latencies
  • explain how GPUs hide memory latency and compute occupancy limits

A node with accelerators

PCIe is slower than you think – sometimes slower than the network.

CPU vs. GPU: design philosophy

  • CPU: few complex cores, large automatic caches, latency-optimized
  • GPU: thousands of simple cores, small caches, throughput-optimized

GPU code must expose tens of thousands of independent threads.

SIMT execution model

  • GPU threads execute the same function (kernel) on different data
  • Threads are grouped into thread blocks (or workgroups)
  • Within a block, threads execute in warps (NVIDIA, 32 threads) or wavefronts (AMD, 64 threads)
  • All threads in a warp execute the same instruction at the same time

Thread hierarchy

  • Grid = all blocks of one kernel launch (independent, any order)
  • Block = threads that cooperate via shared memory + barriers; scheduled onto one SM
  • Warp = smallest scheduling unit (lockstep execution)

Streaming multiprocessors / compute units

  • SM (NVIDIA) / CU (AMD): the physical execution unit
  • Contains: arithmetic units, register file, shared memory, warp schedulers
  • A GPU has 50–150 SMs/CUs
  • Hardware distributes blocks across SMs; once assigned, a block runs to completion

Warp divergence

  • if/else inside a warp: both branches execute, inactive lanes are masked
  • Worst case: throughput drops to the fraction of active lanes
  • Mitigation: branch on block index, not thread index – entire warps go one way

GPU memory hierarchy

Three numbers worth remembering:

  • registers ~1 cycle, shared/L1 ~30 cycles, HBM ~400 cycles

Registers and shared memory

Registers – per-thread, fastest, finite budget shared across all active threads

  • More registers per thread → fewer active threads → less latency hiding
  • Overflow: register spilling to slow global memory

Shared memory – per-block, programmer-managed (48–164 KB per SM)

  • Enables tiling and inter-thread communication; bank conflicts serialize access

Latency hiding: the core trick

  • A warp stalls on a memory access → the scheduler instantly switches to another warp
  • Enough resident warps → the memory latency disappears behind useful work

Occupancy

\[\text{Occupancy} = \frac{\text{active warps per SM}}{\text{max warps per SM}}\]

Limited by: registers/thread, shared memory/block, block size.

Aim for ≥50% – but 50% with good data reuse can beat 100% that hammers global memory.

Profile with ncu (NVIDIA) or rocprof (AMD).

Worked example: register-limited kernel

Kernel uses 48 registers/thread; SM has 65,536 registers, max 2,048 threads (64 warps).

  • Register file fits \(\lfloor 65{,}536 / 48 \rfloor = 1{,}365\) threads
  • Warps of 32: \(\lfloor 1{,}365 / 32 \rfloor = 42\) active warps = 1,344 threads
  • Occupancy: \(42/64 \approx 66\%\) – register-limited, regardless of block size

Reducing to 32 registers/thread allows all 2,048 threads (100%).

Vendor nomenclature

Concept CUDA (NVIDIA) ROCm/HIP (AMD) SYCL (Intel)
Scheduling unit Warp (32) Wavefront (64) Sub-group
Thread group Thread block Workgroup Work-group
Processing unit SM CU EU
On-chip memory Shared memory LDS SLM
Compiler nvcc hipcc icpx

What this means for your LBM code

  • One thread per lattice site: a \(1000^2\) grid = \(10^6\) threads – exactly what the GPU wants
  • Kokkos parallel_for + RangePolicy → kernel launch; the runtime picks block sizes
  • Collision is uniform work → no warp divergence; boundaries need care
  • Performance hinges on how warps access memory – coalescing depends on data layout → next lecture

Summary

  • A node couples CPU and GPUs via PCIe; GPU–GPU (NVLink) and even the network can be faster
  • SIMT hierarchy: grid → block → warp; warps execute in lockstep, divergence costs throughput
  • Memory hierarchy: registers ~1 cycle, shared/L1 ~30, HBM ~400
  • GPUs hide latency by warp switching, not caching – occupancy measures how much hiding is possible
  • Registers and shared memory per SM limit occupancy

Reading