85
edits
Changes
GPU
,→The Implicitly Parallel SIMD Model
=The Implicitly Parallel SIMD Model=
CUDA, OpenCL, HIP, and even other GPU languages like GLSL, HLSL, C++AMP and even non-GPU languages like Intel [ISPC](https://ispc.github.io/) ISPC] all have the same model of implicitly parallel programming. Gangs of threads called Warps in CUDA or Wavefronts in OpenCL execute on a SIMD unit concurrently. The GPU executes a warp (NVidia) or wavefront (AMD) at a time, all 32 threads stepping with the same program counter / instruction pointer. This causes issues with if-statements or while-loops: in the GPU hardware, threads will disable themselves if the rest of the gang needs to execute an if-statement. This is called thread divergence and is a common source of GPU inefficiency.
Even at the lowest machine level: threads are ganged in warps or wavefronts. There is no way to have anything smaller than 32-threads at a time on NVidia Turing hardware. As such, the programmer must imagine this group of 32 (NVidia Turing, AMD RDNA) or 64 (AMD GCN) threads working throughout their code.
