0:00 IntroWhy can one chip multiply in eight bit, and another do it in four bit? Hopper, the chip behind the H one hundred and the H two hundred, has no four bit tensor cores. Blackwell does.
People often call that a software problem. It is not. This film separates what is carved into silicon from what software can still change.
We go through how a GPU runs work, what fixed silicon means, the real numbers for NVIDIA and AMD, and then a worked example: four bit on Hopper.
Every hardware number comes from a datasheet, a whitepaper, or the instruction manuals. Tensor numbers are dense, not the doubled sparse figures in marketing. Anything we could not confirm is left out.
0:51 How a GPU runs workStart with how a GPU runs work. The chip is made of many small processors. On an NVIDIA H one hundred they are called streaming multiprocessors, and there are one hundred thirty two. On AMD's MI three hundred X they are compute units, and there are three hundred and four.
Each one runs threads in groups. On NVIDIA a group is thirty two threads, called a warp. On AMD it is sixty four, called a wavefront. All threads in a group execute the same instruction together.
Inside an H one hundred processor sits a register file of two hundred fifty six kilobytes, and up to two hundred twenty eight kilobytes of shared memory. AMD calls its version the local data share, sixty four kilobytes on MI three hundred.
Each H one hundred processor has four tensor cores, five hundred twenty eight on the whole chip. Outside them sits a fifty megabyte L two cache, and then HBM, the stacked memory. Eighty gigabytes, at three point three five terabytes per second.
Data moves down this ladder: HBM, L two, shared memory, registers, then the math units. Each step is smaller and much faster. The whole game is keeping the math units fed.
2:15 Compute bound or memory boundThat gives two kinds of limit. Compute bound means the math units are the bottleneck. Memory bound means waiting for bytes is the bottleneck. The roofline model says speed is the smaller of two numbers: peak math, and bandwidth times operations per byte.
Divide the H one hundred's dense BF sixteen peak, nine hundred eighty nine point five teraflops, by its three point three five terabytes per second. You need about two hundred ninety five operations per byte moved, just to become compute bound. That is our arithmetic from the datasheet.
Generating text one token at a time is nowhere near that. ABF sixteen weight is two bytes and gives two operations, so about one operation per byte. That is why decoding is memory bound, as NVIDIA and the Marlin paper both say. Rollout generation lives here.
Training with big batches is the opposite. It is compute bound. So there are two separate questions: how fast can the chip multiply, and how fast can it read. A number format changes them in very different ways.
3:29 What fixed silicon meansNow fixed silicon. A tensor core is a physical block of multipliers and adders, wired to do one small matrix multiply and accumulate per step. On Volta, NVIDIA's whitepaper says each tensor core performs sixty four fused multiply adds per clock.
The wiring decides the number format. Multiplier arrays are built for particular bit widths: how wide the mantissa multiplier is, how exponents are added, how wide the accumulator is. Change the format, and you need different gates. That means a different chip.
Software reaches a tensor core only through instructions. On Hopper the instruction is WGMMA. We read its section of the PTX manual. Its dense input types are sixteen bit float, BF sixteen, TF thirty two, two eight bit float formats, eight bit integers, and single bits. No four bit float. No four bit integer.
It also has no operands for block scales. Its only scale controls are plus one or minus one. Blackwell's instruction, TC gen zero five MMA, has kinds called MXF four and MXF four NVF four, with scale factors read from a new on chip Tensor Memory.
A compiler cannot invent a kind that the silicon does not have. The instruction set is the contract with the hardware, and the formats listed in it are the formats that run at tensor core speed.
5:06 Why tricks fall shortWhy can't tricks recover the speed? First, packing. You can pack two four bit numbers into one byte, but the tensor core still wants its own formats as input. Packing saves memory. It does not save multiplier work.
Second, emulate on the ordinary cores. The H one hundred's thirty two bit cores peak at sixty seven teraflops. The tensor cores do nine hundred eighty nine point five in BF sixteen, fourteen point eight times more, and nineteen hundred seventy nine in eight bit float, twenty nine point five times more. Computed from the datasheet. And emulating four bit math adds extra operations on top.
Third, lookup tables. A table lookup is a memory access. It trades math for the very traffic we were short of. It cannot beat a unit that does the multiply and add in one step.
What does work is to unpack the four bit values into a format the tensor core supports, and then use it. Storage gets smaller. The arithmetic stays at sixteen bit or eight bit speed.
6:16 NVIDIA generationsHere are NVIDIA's tensor formats, generation by generation, all dense. Volta, the V one hundred: sixteen bit float only, one hundred twenty five teraflops.
Ampere's A one hundred adds BF sixteen, TF thirty two, and integer eight and four. Dense BF sixteen is three hundred twelve. There is no eight bit float. Marketing numbers double these figures by counting sparsity.
Hopper adds eight bit float. Dense: BF sixteen nine hundred eighty nine point five, eight bit float nineteen hundred seventy nine. The H two hundred keeps the same compute, and raises memory from eighty gigabytes at three point three five terabytes per second, to one hundred forty one gigabytes at four point eight.
Blackwell's B two hundred adds six bit and four bit formats, with block scaling. Per GPU, dense: four bit ten petaflops, eight bit five, BF sixteen two and a half. Memory up to one hundred ninety two gigabytes.
Blackwell Ultra, the B three hundred, raises dense four bit to fifteen petaflops, with eight bit and BF sixteen unchanged. So four bit is two times eight bit on the B two hundred, and three times on the B three hundred. It also cut float sixty four sharply. NVIDIA gives no reason.
7:44 AMD generationsNow AMD. The MI two fifty X, CDNA two, has matrix cores for sixty four, thirty two, and sixteen bit floats, and eight bit integers. Three hundred eighty three teraflops dense at sixteen bit. No eight bit float, and no TF thirty two.
The MI three hundred X, CDNA three, adds eight bit float and TF thirty two. Dense sixteen bit is thirteen hundred seven, eight bit twenty six hundred fifteen. One hundred ninety two gigabytes at five point three terabytes per second. The MI three twenty five X has the same compute, with two hundred fifty six gigabytes at six.
One detail. The MI three hundred's eight bit float is the FNUZ variant, not the standard OCP one. The instruction manual gives a different exponent bias and a maximum of two hundred forty. CDNA four switches to the standard format.
The MI three fifty five X, CDNA four, adds block scaled micro scaling formats: MXF eight, MXF six, and MXF four, in blocks of thirty two. Dense: sixteen bit two and a half petaflops, eight bit five, and six bit and four bit ten. Two hundred eighty eight gigabytes at eight terabytes per second.
To get there, AMD halved its matrix float sixty four rate, and moved TF thirty two to emulation. Both vendors made the same trade: spend silicon on low precision, and take it from somewhere else.
9:25 Hardware limits and softwareNow separate the two layers. Hard hardware limits first. Which operand formats the matrix unit accepts. Peak operations per second in each format. Memory capacity and bandwidth. Interconnect speed. And whether the instruction can take block scales. Hopper's cannot.
Even supported formats have limits. For eight bit float, the PTX manual says accumulation precision sits above half and below single. DeepSeek's team measured about fourteen bits retained on the H eight hundred, and worked around it by adding partial sums in full precision every one hundred twenty eight products. That is a third party measurement.
Now what software can change. How weights are stored. Whether they are unpacked in registers just before the multiply. Fused kernels that avoid writing intermediate results to memory. Scheduling, overlapping copies with math, and the quantization scheme itself.
A rule of thumb. Software can change how many bytes move, and how close you get to the roofline. It cannot raise the roofline.
10:37 4-bit weights on HopperFour bit on Hopper. Store weights as four bit values. Each weight takes a quarter of the bytes of BF sixteen, so memory traffic per weight drops four times.
A kernel such as Marlin loads the four bit weights from HBM, expands them to sixteen bit in registers, and feeds the tensor core in sixteen bit. The Marlin paper reports close to ideal speedup, about three point nine times, for batch sizes up to sixteen to thirty two, and up to two point eight times end to end in v LLM.
As the batch grows, the work stops being memory bound, the compute roof takes over, and the gain fades. About one point five times at batch one twenty eight, in the same paper. Machete is a newer kernel built on WGMMA for Hopper. Its authors report a thirty percent to forty percent throughput gain over other four bit kernels on Llama models.
For reinforcement learning this matters in rollout generation: long outputs, small effective batches, memory bound. Smaller weights mean faster tokens. It does not speed up the training step, which is compute bound. And v LLM's own code warns that its four bit fallback may degrade compute heavy workloads.
So on Hopper, four bit is a storage and bandwidth format. It is not a compute format. The multiply still happens in sixteen bits.
12:09 Native NVFP4 trainingWhat does native NVFP four training look like? NVIDIA's paper trains a twelve billion parameter model on ten trillion tokens. Values are E two M one, blocks are sixteen values, each block has an E four M three scale, and each tensor has a thirty two bit scale.
The recipe: a random Had-a-mard transform on the weight gradient inputs, two dimensional sixteen by sixteen scaling for weights, stochastic rounding on gradients, and the first two and last eight blocks kept in BF sixteen. It matched eight bit float on MMLU Pro, sixty two point five eight against sixty two point six two.
Transformer Engine lists NVFP four training only on compute capability ten point zero and ten point three. Those are Blackwell data center chips. Hopper is nine point zero. The two times and three times four bit speed over eight bit exists only there.
13:13 Block scales and FP8Why can't NVFP four block scales just fold into eight bit float on Hopper? A four bit E two M one value has a sign, two exponent bits and one mantissa bit. The eight magnitudes are zero, one half, one, one and a half, two, three, four, and six.
Sixteen values share one scale in E four M three, which has three mantissa bits. Counting the hidden leading bit, a four bit value carries two significant bits, and the scale carries four.
Multiply them, and the product can need up to six significant bits. E four M three holds four. We checked all one thousand sixteen products of the non negative values. Of the eight hundred eighty two nonzero ones, two hundred seventy five are not exact in E four M three. Computed, assuming the standard E four M three.
BF sixteen has eight significant bits and a wide exponent. All one thousand sixteen products fit exactly. So on Hopper you can unpack NVFP four into BF sixteen without loss, but you cannot hand the product to the eight bit tensor core and stay exact.
People do bring four bit ideas to Hopper in software. A March twenty twenty six paper trains mixture of experts models with MXF four activations and communication, while the matrix multiplies stay in eight bit float. It cuts peak activation memory by about fifteen percent at six hundred seventy one billion parameters. The multiplies keep eight bit speed. We read only its abstract.
14:55 The software layerNow the software layer, labelled as software. CUDA and ROCm are software stacks over different silicon. They are not the same kind of limit as a tensor core.
AMD's own documentation says HIP is not intended to be a drop in replacement for CUDA, and that its converter does not convert all CUDA code automatically. PyTorch on ROCm reuses the same Python interface, with fewer distributed back ends.
Library coverage differs. AMD's port of Transformer Engine lists matrix multiplication as partially supported, and attention as partial. SG Lang on AMD says its faster Marlin path is not available, and uses a Triton dequantization path for AWQ instead.
New hardware formats reach software later. ROCm seven point zero added support for MX four, six, and eight in its kernel libraries, alongside the MI three fifty series. The silicon came first, then the kernels.
These gaps can close with engineering. A missing kernel is a software limit. A missing multiplier is a hardware limit. Keep them apart when you read a spec sheet.
16:17 RecapTo recap. A tensor core is fixed silicon: multipliers wired for specific formats, reached only through instructions that list them.
Hopper's list ends at eight bit float, with no four bit and no block scales. Blackwell and CDNA four add them. The ratios are real. Four bit is two times eight bit on the B two hundred, and three times on the B three hundred.
Software changes how many bytes move, and how well the chip is used. Four bit weights on Hopper cut memory traffic four times, and help memory bound work like rollout generation. They do not make the math faster.
Whether your job is compute bound or memory bound tells you which limit you are touching. For Hopper reinforcement learning: expect gains from four bit weights in rollouts, and none in training compute.
Sources, and what we could not verify, are on the next screen.
17:20 SourcesHardware numbers come from NVIDIA and AMD datasheets, the Volta, Ampere and Blackwell whitepapers, the PTX instruction manual, the Transformer Engine docs, and the AMD CDNA whitepapers and instruction set manuals. Software claims come from the Marlin paper, the Machete write up, and the v LLM and ROCm documentation.
Left out because we could not confirm them: a per partition warp scheduler count on Hopper, the B two hundred processor count, the twenty petaflop four bit press figure for MI three fifty five X, and any reason for the Blackwell Ultra float sixty four cut.