Parallel hardware architectures
Flynn’s taxonomy, levels of parallelism, and the programming models that match them
Learning objectives
- Classify parallel computers with Flynn’s taxonomy
- Distinguish shared, distributed, and hybrid memory architectures
- Name the three levels of parallelism in a modern node and the programming model for each
- Read the bandwidth hierarchy of a GPU cluster: NVLink > PCIe > InfiniBand
- Explain why this zoo of models motivates a portability layer like Kokkos
Flynn’s taxonomy
How many instruction streams, how many data streams?
| Single instruction |
SISD – classic serial CPU |
SIMD – vector units, GPUs |
| Multiple instruction |
MISD – rare (fault-tolerant pipelines) |
MIMD – multicore, clusters |
- Modern HPC machines are MIMD of SIMD: many independent cores, each operating on vectors
- GPUs push the SIMD idea to the extreme
Shared, distributed, hybrid memory
![]()
Hierarchical parallelism: SIMD lanes inside cores, cores sharing memory within a node, nodes connected by a network, GPUs attached to each node.
- Shared memory: all threads see one address space (within a node)
- Distributed memory: each process owns its memory; data moves by explicit messages (between nodes)
- Hybrid: every real cluster is both – plus accelerators
Level 1: Vectorization (SIMD)
One instruction, many data elements.
- CPU vector units: SSE, AVX – e.g.
ADDPS adds 4 floats in one instruction
- GPUs are SIMD taken to the extreme (SIMT): an NVIDIA H200 has 16,896 CUDA cores; threads execute in warps of 32
- Programming models: CUDA, HIP, SYCL, OpenACC, OpenMP (target), OpenCL
Level 2: Thread-level parallelism
Multiple threads inside one process.
- Threads share the memory of their process – no explicit data movement
- Beware: shared access requires synchronization (races!)
- In HPC, the dominant model is OpenMP
Level 3: Process-level parallelism
Multiple processes, possibly on different machines.
- Each process has its own memory – no process can see another’s data
- Data is moved by explicit copies: messages over the network
- The HPC standard: MPI (Message Passing Interface)
Mapping the levels onto a cluster
![]()
A compute node: CPU with DRAM, four GPUs with HBM, NVLink between GPUs, PCIe to the CPU, NIC/InfiniBand to other nodes.
- A cluster = many such nodes on an InfiniBand fabric
- Every link in this picture has a different bandwidth…
The bandwidth hierarchy
- NVLink (GPU ↔︎ GPU, same node): ~900 GB/s
- PCIe 5 (CPU ↔︎ GPU): ~128–256 GB/s
- InfiniBand NDR (node ↔︎ node): ~50 GB/s
The further data travels, the slower the link.
Keep data on the GPU; communicate as locally – and as rarely – as possible.
CPU vs GPU: two design philosophies
![]()
CPU: few fat, latency-optimized cores with large caches and control logic. GPU: many lean, throughput-optimized cores.
- CPU: minimize the time to finish one task (latency)
- GPU: maximize the number of tasks finished per second (throughput)
One programming model per hardware level
| Distributed memory (node ↔︎ node) |
MPI |
| Shared memory (cores in a node) |
OpenMP, pthreads |
| SIMD (vector lanes in a core) |
intrinsics, auto-vectorization |
| Accelerator (GPU) |
CUDA, HIP, SYCL |
That is four models for one machine. Kokkos spans all on-node levels from a single C++ source – which is why this course is built around it.
Summary
![]()
One C++ source, many backends, many kinds of hardware.
- Flynn: modern machines are MIMD of SIMD
- Three levels of parallelism: SIMD lanes → threads (shared memory) → processes (distributed memory)
- Bandwidth hierarchy: NVLink (900 GB/s) > PCIe (~128–256 GB/s) > InfiniBand (~50 GB/s)
- Traditionally one programming model per level – Kokkos abstracts the on-node levels; MPI handles the rest
