Programming
performance assembly x86 cpu-architecture micro-optimization
Updated Wed, 24 Aug 2022 12:14:02 GMT

# Assembly - How to score a CPU instruction by latency and throughput

I'm looking for a type of a formula / way to measure how fast an instruction is, or more specific to give a "score" each of the instruction by CPU cycles.

Let's take the follow assembly program for an example,

``````nop
mov         eax,dword ptr [rbp+34h]
inc         eax
mov         dword ptr [rbp+34h],eax
``````

and the following Intel Skylake information:

mov r,m : Throughput=0.5 Latency=2

mov m,r : Throughput=1 Latency=2

nop : Throughput=0.25 Latency=non

inc : Throughput=0.25 Latency=1

I know that the order of the instructions in the program are matter in here but I'm looking to create something general that not need to be "accurate to the single cycle"

any one have any idea how can I do that?

## Solution

There is no formula you can apply; you have to measure.

The same instruction on different versions of the same uarch family can have different performance. e.g. `mulps`:

• Sandybridge 1c / 5c throughput/latency.
• HSW 0.5 / 5. BDW 0.5 / 3 (faster multiply path in the FMA unit? FMA is still 5c).
• SKL 0.5 / 4 (lower latency FMA, too). SKL runs `addps` on the FMA unit as well, dropping the dedicated FP multiply unit so add latency is higher, but throughput is higher.

There's no way you could predict any of this without measuring, or knowing some microarchitectural details. We expect FP math ops won't be single-cycle latency, because they're much more complicated than integer ops. (So if they were single cycle, the clock speed is set too low for integer ops.)

You measure by repeating the instruction many times in an unrolled loop. Or fully unrolled with no looping, but then you defeat the uop-cache and can get front-end bottlenecks. (e.g. for decoding 10-byte `mov r64, imm64`)

https://uops.info/ has already automated this testing for every form of every (unprivileged) instruction, and you can even click on any table entry to see what test loops they used. e.g. Skylake `xchg r32, eax` latency testing (https://uops.info/html-lat/SKL/XCHG_R32_EAX-Measurements.html) from each input operand to each output. (2 cycle latency from EAX -> R8D, but 1 cycle latency from R8D -> EAX.) So we can guess that the 3 uops include copying EAX to an internal temporary, but moving directly from the other operand to EAX.

https://uops.info/ is the current best source of test data; when it and Agner's tables disagree, my own measurements and/or other sources have always confirmed uops.info's testing was accurate. And they don't try to make up a latency number for 2 halves of a round-trip like movd xmm0,eax and back, they show you the range of possible latencies assuming the rest of the chain was the minimum plausible.

Agner Fog creates his instruction tables (which you appear to be reading) by timing large non-looping blocks of code that repeat an instruction. https://agner.org/optimize/. The intro section of his instruction-tables explains briefly how he measures, and his microarch guide explains more details of how different x86 microarchitectures work internally. Unfortunately there are occasional typos or copy/paste errors in his hand-edited tables.

http://instlatx64.atw.hu/ also has results of experimental measurements. I think they use a similar technique of a large block of the same instruction repeated, maybe small enough to fit in the uop cache. But they don't use perf counters to measure what execution port each instruction needs, so their throughput numbers don't help you figure out which instructions compete with which other instructions.

These latter two sources have been around for longer than uops.info, and cover some older CPUs, especially older AMD.

To measure latency yourself, you make the output of each instruction an input for the next.

`````` mov  ecx, 10000000
inc_latency:
inc eax
inc eax
inc eax
inc eax
inc eax
inc eax
sub ecx,1          ; avoid partial-flag false dep for P4
jnz inc_latency    ; dec or sub/jnz macro-fuses into 1 uop on Intel SnB-family
``````

This dependency chain of 7 `inc` instructions will bottleneck the loop at 1 iteration per `7 * inc_latency` cycles. Using perf counters for core clock cycles (not RDTSC cycles), you can easily measure the time for all the iterations to 1 part in 10k, and with more care probably even more precisely than that. The repeat count of 10000000 hides start/stop overhead of whatever timing you use.

I normally put a loop like this in a Linux static executable that just makes a `sys_exit(0)` system call directly (with a `syscall`) instruction, and time the whole executable with `perf stat ./testloop` to get time and a cycle count. (See Can x86's MOV really be "free"? Why can't I reproduce this at all? for an example).

Another example is Understanding the impact of lfence on a loop with two long dependency chains, for increasing lengths, with the added complication of using `lfence` to drain the out-of-order execution window for two dep chains.

To measure throughput, you use separate registers, and/or include an xor-zeroing occasionally to break dep chains and let out-of-order exec overlap things. Don't forget to also use perf counters to see which ports it can run on, so you can tell which other instructions it will compete with. (e.g. FMA (p01) and shuffles (p5) don't compete at all for back-end resources on Haswell/Skylake, only for front-end throughput.) Don't forget to measure front-end uop counts, too: some instructions decode to multiply uops.

How many different dependency chains do we need to avoid a bottleneck? Well we know the latency (measure it first), and we know the max possible throughput (number of execution ports, or front-end throughput.)

For example, if FP multiply had 0.25c throughput (4 per clock), we could keep 20 in flight at once on Haswell (5c latency). That's more than we have registers, so we could just use all 16 and discover that in fact the throughput is only 0.5c. But if it had turned out that 16 registers was a bottleneck, we could add `xorps xmm0,xmm0` occasionally and let out-of-order execution overlap some blocks.

More is normally better; having just barely enough to hide latency can slow down with imperfect scheduling. If we wanted to go nuts measuring `inc`, we'd do this:

`````` mov  ecx, 10000000
inc_latency:
%rep 10          ;; source-level repeat of a block, no runtime branching
inc eax
inc ebx
; not ecx, we're using it as a loop counter
inc edx
inc esi
inc edi
inc ebp
inc r8d
inc r9d
inc r10d
inc r11d
inc r12d
inc r13d
inc r14d
inc r15d
%endrep
sub ecx,1          ; break partial-flag false dep for P4
jnz inc_latency    ; dec/jnz macro-fuses into 1 uop on Intel SnB-family
``````

If we were worried about partial-flag false dependencies or flag-merging effects, we might experiment with mixing in an `xor eax,eax` somewhere to let OoO exec overlap more than just when `sub` wrote all flags. (See INC instruction vs ADD 1: Does it matter?)

There's a similar problem for measuring throughput and latency of `shl r32, cl` on Sandybridge-family: the flag dependency chain isn't normally relevant for a computation, but putting `shl` back-to-back creates a dependency through FLAGS as well as through the register. (Or for throughput, there isn't even a register dep).

I posted about this on Agner Fog's blog: https://www.agner.org/optimize/blog/read.php?i=415#860. I mixed `shl edx,cl` in with four `add edx,1` instructions, to see what incremental slowdown adding one more instruction had, where the FLAGS dependency was a non-issue. On SKL, it only slows down by an extra 1.23 cycles on average, so the true latency cost of that `shl` was only ~1.23 cycles, not 2. (It's not a whole number or just 1 because of resource conflicts to run the flag-merging uops of the `shl`, I guess. BMI2 `shlx edx, edx, ecx` would be exactly 1c because it's only a single uop.)

Related: for static performance analysis of whole blocks of code (containing different instructions), see What considerations go into predicting latency for operations on modern superscalar processors and how can I calculate them by hand?. (It's using the word "latency" for the end-to-end latency of a whole computation, but actually asking about things small enough for OoO exec to overlap different parts, so instruction latency and throughput both matter.)

The `Latency=2` numbers for load/store appear to be from Agner Fog's instruction tables (https://agner.org/optimize/). They unfortunately aren't accurate for a chain of `mov rax, [rax]`. You'll find that's 4c latency if you measure it by putting that in a loop.

Agner splits up load/store latency into something that makes the total store/reload latency come out correct, but for some reason he doesn't make the load part equal to the L1d load-use latency when it comes from cache instead of the store buffer. (But also note that if the load feeds an ALU instruction instead of another load, the latency is 5c. So the simple addressing-mode fast-path only helps for pure pointer-chasing.)