Vasilios Syrakis - performanceA simple blog made with Zola and DuckquillZola2026-07-07T00:00:00+00:00https://cetanu.github.io/tags/performance/atom.xmlProfit or Poverty: False Sharing2026-07-07T00:00:00+00:002026-07-07T00:00:00+00:00
Vasilios Syrakis
https://cetanu.github.io/blog/false-sharing/<p>In previous blogs I talked a little bit about
<a href="https://cetanu.github.io/blog/why-numa-is-important-in-trading/">NUMA</a> and the
<a href="https://cetanu.github.io/blog/translation-lookaside-buffer/">TLB</a>, if you haven’t read those, I
recommend starting there before reading this.</p>
<p>You should already understand that accessing main memory is relatively slow for
the CPU, especially in a low-latency environment, and you may be familiar with
the fact that the CPU has small caches of varying sizes on the chip, known as
the L1, L2, and L3 caches.</p>
<p>As their name implies, there are levels, or a hierarchy, from fastest (and
smallest) to slowest (and largest). The L1 cache is also typically split into
two caches, one being an instruction cache and the other being a data cache.</p>
<p>Fun fact: these caches are usually made of SRAM and usually they are the
largest part on the chip.</p>
<img class="no-hover"alt="relative speed of hardware"src="/img/relative-hw-speed.png"/><h2 id="requesting-data-from-the-cache"><a class="zola-anchor" href="#requesting-data-from-the-cache" aria-label="Anchor link for: requesting-data-from-the-cache"><i class="icon"></i></a>
Requesting data from the cache</h2>
<p>You may have heard of “cache lines” before (also called cache blocks), maybe on
some article about performance on hackernews.</p>
<p>A cache line can be thought of a single unit of data which is transferred from
RAM to the cache. Usually one cache line is 64 bytes, but this can differ
depending on your CPU architecture.</p>
<p>This introduces the concept of cache locality. When the CPU needs to access the
same memory several times in a row, after it’s already been pulled into the L1
cache, this is referred to as <strong>temporal locality</strong>. When the CPU pulls in a
cache line and all the data it needs is within that block, it’s referred to as
spatial locality. A good example of <strong>spatial locality</strong> is when you have an
array that fits into one cache line, resulting in traversal being extremely
fast. An example of potentially terrible cache performance might be something
like a linked-list.</p>
<p>On systems with multiple CPU cores, they might share the same chunk of memory
in their local caches. Most processors use what is known as the MESI protocol
to prevent overwriting each others data. Jon Gjengset has a <a href="https://cetanu.github.io/blog/false-sharing/www.youtube.com/watch?v=tND-wBBZ8RY">great
talk</a> about Mutexes which also covers the
MESI protocol.</p>
<h2 id="where-it-can-go-wrong"><a class="zola-anchor" href="#where-it-can-go-wrong" aria-label="Anchor link for: where-it-can-go-wrong"><i class="icon"></i></a>
Where it can go wrong</h2>
<p>CPUs can enter what is called a coherence livelock or cache-line pingponging,
where multiple cores fight over the same memory, or the same cache line.</p>
<p>The first way this can happen is called <strong>False Sharing</strong>, where your data may
reside in a cache line, but it may not be occupying that entire cache line by
itself. If your data sits in the first half but some other data sits in the
latter half, it’s possible that two CPUs may continuously write to the same
cache line, requiring them to communicate with each other to gain ownership in
order to modify the cache line. If you watched Jon’s video above you would know
that this can eat up dozens of precious nanoseconds, just for the comms between
CPUs. As you increase the number of cores on a system this problem can be
exacerbated.</p>
<img class="no-hover"alt="diagram of false sharing"src="/img/false-sharing.png"/>
<p>The second way is <strong>True Sharing</strong>, where cores may write to the same variable or
data structure in memory. This could happen during atomic operations or when
using synchronization primitives such as mutexes, semaphores, barriers,
spinlocks, and so on.</p>
<h2 id="how-to-protect-against-it"><a class="zola-anchor" href="#how-to-protect-against-it" aria-label="Anchor link for: how-to-protect-against-it"><i class="icon"></i></a>
How to protect against it</h2>
<p>At least for false sharing, the solution seems simple: just pad your data
structure or variable so that it occupies an entire cache line. Obviously it
would be silly to do this with every variable in your program, so you have to
pick the time-critical parts of your application here.</p>
<pre class="giallo" style="color: #A9B1D6; background-color: #1A1B26;"><code data-lang="rust"><span class="giallo-l"><span style="color: #89DDFF;">use</span><span style="color: #0DB9D7;"> std</span><span style="color: #89DDFF;">::</span><span style="color: #0DB9D7;">sync</span><span style="color: #89DDFF;">::</span><span style="color: #0DB9D7;">atomic</span><span style="color: #89DDFF;">::</span><span style="color: #C0CAF5;">AtomicU64</span><span style="color: #89DDFF;">;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #89DDFF;">#[</span><span>repr</span><span style="color: #89DDFF;">(</span><span>align</span><span style="color: #89DDFF;">(</span><span>64</span><span style="color: #89DDFF;">))]</span></span>
<span class="giallo-l"><span style="color: #89DDFF;">pub</span><span style="color: #BB9AF7;"> struct</span><span style="color: #C0CAF5;"> PaddedAtomic</span><span style="color: #89DDFF;"> {</span></span>
<span class="giallo-l"><span style="color: #89DDFF;"> pub</span><span style="color: #C0CAF5;"> value</span><span style="color: #89DDFF;">:</span><span style="color: #C0CAF5;"> AtomicU64</span><span style="color: #89DDFF;">,</span></span>
<span class="giallo-l"><span style="color: #89DDFF;">}</span></span></code></pre>
<p>For true sharing, it seems quite a bit more complicated to get right. Sometimes
you really do need atomic operations on global memory.</p>
<p>You could pull data into thread local storage, only merging back at the end of
expensive or time critical execution.</p>
<p>You could reorganize your program to do batching to reduce how often it needs
to write back to the shared memory.</p>
<p>Maybe a lock-free data structure could be used.</p>
<p>The solution most likely depends on exactly what you are willing to accept in
terms of trade-offs.</p>
<h2 id="conclusion"><a class="zola-anchor" href="#conclusion" aria-label="Anchor link for: conclusion"><i class="icon"></i></a>
Conclusion</h2>
<p>Like I usually say, for the most part things like this are unnoticeable and
work just fine for the majority of situations. If you’re running a flask webapp
you probably don’t and should not care about this.</p>
<p>When you require low-latency, then you need to understand how to write your
programs such that they exploit cache-lines for your architecture because it
makes the difference between a 1ns L1 cache hit, or falling off a cliff with a
100x slower execution.</p>
Profit or Poverty: TLB2026-04-28T00:00:00+00:002026-04-28T00:00:00+00:00
Vasilios Syrakis
https://cetanu.github.io/blog/translation-lookaside-buffer/<h2 id="the-illusion-of-memory"><a class="zola-anchor" href="#the-illusion-of-memory" aria-label="Anchor link for: the-illusion-of-memory"><i class="icon"></i></a>
The Illusion of Memory</h2>
<p>Applications do not allocate memory directly to physical addresses on the RAM.
They are presented with virtual memory by the operating system, so that it can
pretend that the application has a contiguous block of memory when in reality
its memory is scattered across the physical RAM in fixed-size chunks of memory
called pages, kept track of in a page table.</p>
<p>When a process starts, the operating system allocates a range of virtual
addresses to it in the form of pages, but leaves the entries in the page table
blank until they’re accessed (i.e. they are lazy-loaded)</p>
<p>If the process tries to access memory, the operating system follows the virtual
memory address to the physical memory address and hands back the data. The
actual translation from virtual address to physical address is not performed by
the OS but by a dedicated hardware component on the CPU called a memory
management unit (MMU).</p>
<p>Sometimes, if RAM is running low or hasn’t been accessed in a while, data is
taken out of RAM and swapped to disk. If the process then tries to access that
memory again, a <strong>major</strong> page fault occurs, the operating system has to retrieve the
data from disk, load it into RAM, and then update the page table before the
process can proceed.</p>
<p>A minor page fault is when the data is still in RAM, but the page table for the
process has not been mapped to the physical page yet. By default the operating
system is lazy about mapping page table entries, but there are strategies for
ensuring that all pages are mapped.</p>
<h2 id="the-bottleneck"><a class="zola-anchor" href="#the-bottleneck" aria-label="Anchor link for: the-bottleneck"><i class="icon"></i></a>
The Bottleneck</h2>
<p>Every time you’re reading variables, or loading instructions, or saving data,
the system is translating virtual memory addresses to physical addresses.</p>
<h3 id="a-hardware-page-table-walk"><a class="zola-anchor" href="#a-hardware-page-table-walk" aria-label="Anchor link for: a-hardware-page-table-walk"><i class="icon"></i></a>
A hardware page table walk</h3>
<p>This translation can be quite involved if we dig really deep into the detail.
The MMU looks at a register to find the physical address of the root page
table. Then it takes some bits from the virtual address, goes to RAM, and reads
the page table entry at that index. Then it takes the result from that lookup,
goes back to RAM again, and finds the entry point to another intermediate
table, and another intermediate table… until it gets the physical page frame
number. It then combines the address with some of the bits from the original
virtual address to find the exact byte in memory.</p>
<h3 id="introducing-the-tlb"><a class="zola-anchor" href="#introducing-the-tlb" aria-label="Anchor link for: introducing-the-tlb"><i class="icon"></i></a>
Introducing the TLB</h3>
<p>The translation look-aside buffer (TLB) is a specialized cache within the MMU
which remembers recent mappings to avoid the CPU having to check the full page
table (in RAM) on every single memory access (billions of times per second)</p>
<p>The consequence of a TLB miss is that the operation takes 100-200x longer. In
reality this equates to about 100 nanoseconds, which is miniscule, but the
volume of these operations causes that time to add up rapidly. Not to mention
that 1 nanosecond would be much preferred.</p>
<h2 id="how-to-make-money"><a class="zola-anchor" href="#how-to-make-money" aria-label="Anchor link for: how-to-make-money"><i class="icon"></i></a>
How to make 💵 MONEY 💵</h2>
<p>Since the consequence of TLB cache misses are so dire, it makes sense to avoid
them. Working around the cache is out of the question, so the way forward is to
enhance the effectiveness of the cache by making it have to cache less things,
thereby improving its hit rate.</p>
<p>The TLB has a limit on the number of page table entries it can hold. A page is
typically 4KB but this may differ based on your hardware. So if your system has
32GB of memory, you need 8 million entries to map all the available memory on
the system, which is not possible.</p>
<h3 id="hugepages"><a class="zola-anchor" href="#hugepages" aria-label="Anchor link for: hugepages"><i class="icon"></i></a>
Hugepages</h3>
<p>This is where “Hugepages” come in.</p>
<p>Hugepages is a feature in the linux kernel that allows a program to use much
larger page sizes, usually either 2MB or 1GB. By using this option, you can
effectively map all of the memory that you need such that it can fit in the
limited number of entries in the TLB easily.</p>
<p>For example, with the 32GB example from before, using 1GB hugepages, you would
only need 32 entries, rather than 8,000,000.</p>
<h4 id="enabling-hugepages-for-your-application"><a class="zola-anchor" href="#enabling-hugepages-for-your-application" aria-label="Anchor link for: enabling-hugepages-for-your-application"><i class="icon"></i></a>
Enabling hugepages for your application</h4>
<p>You could set a sysctl (<code>vm.nr_hugepages</code>), or you could control it from your program, for example…</p>
<p>In C++, when calling <code>mmap</code>, you can pass the <code>MAP_HUGETLB</code> bitflag or <code>MAP_HUGE_1GB</code> for example.</p>
<p>In Rust, there are some crates like <code>memmap2</code> or <code>hugepage-rs</code>.</p>
<h3 id="memory-pinning"><a class="zola-anchor" href="#memory-pinning" aria-label="Anchor link for: memory-pinning"><i class="icon"></i></a>
Memory Pinning</h3>
<p>I mentioned earlier that when a process starts, the OS allocates virtual
memory, but it lazily maps the page table entries. So you may have hugepages
enabled, but you can still encounter a page fault if you access memory that
hasn’t been mapped yet.</p>
<p>There is a linux system call, <code>mlockall</code> which can be used to eagerly map all
the virtual memory for your process. It’s advised to call this before entering
the critical part of the program, such as any part of the program which needs
to be real-time (requires deterministic timing).</p>
<p>This system call not only “pre-faults” the pages so that they’re ready, it also
pins them so that the operating system cannot swap them back to disk.</p>
<h4 id="how-to-pin-memory-from-within-your-program"><a class="zola-anchor" href="#how-to-pin-memory-from-within-your-program" aria-label="Anchor link for: how-to-pin-memory-from-within-your-program"><i class="icon"></i></a>
How to pin memory, from within your program</h4>
<p>In C++ you would include the <code>sys/mman.h</code> header, allowing you to call mlockall directly.</p>
<pre class="giallo" style="color: #A9B1D6; background-color: #1A1B26;"><code data-lang="c"><span class="giallo-l"><span style="color: #89DDFF;">#</span><span style="color: #BB9AF7;">include</span><span style="color: #89DDFF;"> <</span><span style="color: #9ECE6A;">sys/mman.h</span><span style="color: #89DDFF;">></span></span>
<span class="giallo-l"><span style="color: #89DDFF;">#</span><span style="color: #BB9AF7;">include</span><span style="color: #89DDFF;"> <</span><span style="color: #9ECE6A;">iostream</span><span style="color: #89DDFF;">></span></span>
<span class="giallo-l"><span style="color: #89DDFF;">#</span><span style="color: #BB9AF7;">include</span><span style="color: #89DDFF;"> <</span><span style="color: #9ECE6A;">cstring</span><span style="color: #89DDFF;">></span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #BB9AF7;">int</span><span style="color: #7AA2F7;"> main</span><span style="color: #9ABDF5;">() {</span></span>
<span class="giallo-l"><span style="color: #BB9AF7;"> if</span><span style="color: #9ABDF5;"> (</span><span style="color: #7AA2F7;">mlockall</span><span style="color: #9ABDF5;">(</span><span style="color: #7AA2F7;">MCL_CURRENT </span><span style="color: #89DDFF;">|</span><span style="color: #7AA2F7;"> MCL_FUTURE</span><span style="color: #9ABDF5;">)</span><span style="color: #BB9AF7;"> !=</span><span style="color: #FF9E64;"> 0</span><span style="color: #9ABDF5;">) {</span></span>
<span class="giallo-l"><span style="color: #9ABDF5;"> std::cerr </span><span style="color: #89DDFF;"><< "</span><span style="color: #9ECE6A;">Failed to lock memory: </span><span style="color: #89DDFF;">" <<</span><span style="color: #7AA2F7;"> std::strerror</span><span style="color: #9ABDF5;">(</span><span style="color: #7AA2F7;">errno</span><span style="color: #9ABDF5;">)</span><span style="color: #89DDFF;"> <<</span><span style="color: #9ABDF5;"> std::endl</span><span style="color: #89DDFF;">;</span></span>
<span class="giallo-l"><span style="color: #BB9AF7;"> return</span><span style="color: #FF9E64;"> 1</span><span style="color: #89DDFF;">;</span></span>
<span class="giallo-l"><span style="color: #9ABDF5;"> }</span></span>
<span class="giallo-l"><span style="color: #7AA2F7;"> munlockall</span><span style="color: #9ABDF5;">()</span><span style="color: #89DDFF;">;</span><span style="color: #9ABDF5;"> </span></span>
<span class="giallo-l"><span style="color: #BB9AF7;"> return</span><span style="color: #FF9E64;"> 0</span><span style="color: #89DDFF;">;</span></span>
<span class="giallo-l"><span style="color: #9ABDF5;">}</span></span>
<span class="giallo-l"></span></code></pre>
<p>In Rust, you would use the <code>libc</code> crate and call it similarly.</p>
<pre class="giallo" style="color: #A9B1D6; background-color: #1A1B26;"><code data-lang="rust"><span class="giallo-l"><span style="color: #89DDFF;">use</span><span style="color: #0DB9D7;"> libc</span><span style="color: #89DDFF;">::{</span><span>mlockall</span><span style="color: #89DDFF;">,</span><span style="color: #C0CAF5;"> MCL_CURRENT</span><span style="color: #89DDFF;">,</span><span style="color: #C0CAF5;"> MCL_FUTURE</span><span style="color: #89DDFF;">};</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #89DDFF;">fn</span><span style="color: #7AA2F7;"> main</span><span style="color: #89DDFF;">() {</span></span>
<span class="giallo-l"><span style="color: #BB9AF7;"> let</span><span style="color: #C0CAF5;"> locked</span><span style="color: #89DDFF;"> = unsafe {</span></span>
<span class="giallo-l"><span style="color: #7AA2F7;"> mlockall</span><span style="color: #89DDFF;">(</span><span style="color: #FF9E64;">MCL_CURRENT</span><span style="color: #BB9AF7;"> |</span><span style="color: #FF9E64;"> MCL_FUTURE</span><span style="color: #89DDFF;">)</span></span>
<span class="giallo-l"><span style="color: #89DDFF;"> };</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #BB9AF7;"> if</span><span style="color: #C0CAF5;"> locked</span><span style="color: #BB9AF7;"> !=</span><span style="color: #FF9E64;"> 0</span><span style="color: #89DDFF;"> {</span></span>
<span class="giallo-l"><span style="color: #BB9AF7;"> let</span><span style="color: #C0CAF5;"> err</span><span style="color: #89DDFF;"> =</span><span style="color: #0DB9D7;"> std</span><span style="color: #89DDFF;">::</span><span style="color: #0DB9D7;">io</span><span style="color: #89DDFF;">::</span><span style="color: #C0CAF5;">Error</span><span style="color: #89DDFF;">::</span><span style="color: #7AA2F7;">last_os_error</span><span style="color: #89DDFF;">();</span></span>
<span class="giallo-l"><span style="color: #7AA2F7;"> panic!</span><span style="color: #89DDFF;">("</span><span style="color: #9ECE6A;">Failed to lock memory: </span><span style="color: #89DDFF;">{}",</span><span style="color: #C0CAF5;"> err</span><span style="color: #89DDFF;">);</span></span>
<span class="giallo-l"><span style="color: #89DDFF;"> }</span></span>
<span class="giallo-l"><span style="color: #89DDFF;">}</span></span></code></pre><h2 id="conclusion"><a class="zola-anchor" href="#conclusion" aria-label="Anchor link for: conclusion"><i class="icon"></i></a>
Conclusion</h2>
<p>Ultimately, the illusion of memory that modern systems provide is a
double-edged sword. While virtual memory addressing solves major problems
around process isolation for most systems, the introduction of page tables and
the need to perform multi-level page table walks introduces latency that is
unnoticable to humans but harmful to low-latency and real-time applications.</p>
<p>With this small bit of operating systems and linux knowledge you can use
hugepages and memory pinning to reduce memory latency by up to 100 to 200 times
in certain scenarios.</p>
Profit or Poverty: Realtime kernel patch2026-04-20T00:00:00+00:002026-04-20T00:00:00+00:00
Vasilios Syrakis
https://cetanu.github.io/blog/linux-is-not-a-realtime-system/<p><strong>Linux is not a realtime operating system.</strong></p>
<p>This statement confused me at first, but as I looked further, it began to make
sense.</p>
<p>Linux provides an environment where multitasking is possible, facilitated by a
scheduler which has goals such as minimizing wait times and latency, maximizing
throughput, and maximizing fairness.</p>
<p>These goals can sometimes pull in different directions. If you maximize
throughput for one task, you increase latency for others. If you minimize
latency by switching tasks more often, overheads start to pile up. If every
process gets an equal slice of time for fairness, heavy tasks are starved while
light tasks sit idle.</p>
<p>Taking the general approach is the best decision for most usecases like servers
and desktop computing, but it means that Linux is a “soft realtime” system.</p>
<h2 id="defining-realtime"><a class="zola-anchor" href="#defining-realtime" aria-label="Anchor link for: defining-realtime"><i class="icon"></i></a>
Defining “realtime”</h2>
<p>What is evoked when you hear someone talk about something happening in
<strong>realtime?</strong> We often think that it is happening immediately, like we’re
describing something in the present.</p>
<p>In computing, realtime refers to something more specific. From Wikipedia:</p>
<blockquote>
<p>A system is said to be realtime if the total correctness of an operation
depends not only upon its logical correctness, but also upon the time in
which it is performed.</p>
</blockquote>
<p>That is to say that it does not mean “fast” necessarily, it’s more to do with
deterministic behavior in relation to time.</p>
<p>Linux is a <strong>soft</strong> realtime system, meaning that if an operation misses its
deadline, it isn’t a hard failure. Your process might wait a few hundred
microseconds before the scheduler gives it a time-slice, and for the vast
majority of cases, this is acceptable and nobody notices. If a deadline is
missed frequently, the user experiences a degradation of service. For example,
a web page might load slower, but still eventually load.</p>
<p>A <strong>hard</strong> realtime system would be something like an assembly line, where if a
part isn’t processed before its deadline, that part is no longer reachable by
the robot, and the next robot won’t be able to process it correctly.</p>
<h2 id="the-scheduler"><a class="zola-anchor" href="#the-scheduler" aria-label="Anchor link for: the-scheduler"><i class="icon"></i></a>
The Scheduler</h2>
<p>I mentioned earlier that Linux provides multitasking. It really provides the
illusion of multitasking, at least on a single core. The task scheduler is what
creates this illusion, by allocating slices of time to each task, interleaving
their execution with one another to create the illusion of simultaneous
progression. This is referred to as concurrency, as opposed to parallelism.</p>
<small>
Concurrency comes from the Latin "con" (together) and "currere" (to run,
literally, to move faster than walking).
</small>
<p>Concurrency is great, especially when tasks have to wait for something to
become available - the scheduler can pause that task until it’s ready to be
executed, and give time-slices to other tasks in the meanwhile.</p>
<p>The task itself can communicate to the scheduler that it should be paused to
allow other tasks to progress. Often, a task will do this implicitly by
executing a particular system call, such as <code>read()</code> or <code>sleep()</code>.</p>
<p>However, the <em>scheduler</em> can also pause a task <em>without any cooperation from the
task.</em></p>
<img class="no-hover"alt="Scheduler standing behind the
application, about to ruin its day"src="/img/preemption_meme_1.png"/><h2 id="preemption"><a class="zola-anchor" href="#preemption" aria-label="Anchor link for: preemption"><i class="icon"></i></a>
Preemption</h2>
<p>This particular style of multitasking on Linux is called preemptive
multitasking. Tasks can be “preempted”.</p>
<p>Sounds interesting, but what does it mean? <em>What</em> is being preempted?</p>
<p>The current task’s execution time is being preempted - cut short - so that a
more important task can be run.</p>
<p>During preemption, the process state is saved, and another’s is loaded. This
mechanism is what we know to be a context switch.</p>
<p>An example where this is good: you run an application, and that application
chews up all your compute, and you want to cancel it. If it couldn’t be
preempted, you wouldn’t be able to move your cursor to close or cancel it.</p>
<p>Where preemption becomes undesirable is when you’re in a low-latency
environment such as high-frequency trading, and your trading application gets
preempted and has to wait <strong>hundreds of microseconds</strong> to execute.</p>
<h2 id="switch-it-off"><a class="zola-anchor" href="#switch-it-off" aria-label="Anchor link for: switch-it-off"><i class="icon"></i></a>
Switch it off!!</h2>
<p>You can use a tool like <code>chrt</code> to change how your running process is treated.
For example, maybe you set its priority very high, in FIFO mode where it runs
until it voluntarily yields, like so:</p>
<pre class="giallo" style="color: #A9B1D6; background-color: #1A1B26;"><code data-lang="plain"><span class="giallo-l"><span>chrt --pid --fifo 99 <PID></span></span></code></pre>
<p><small>You can also check the current scheduling attributes with <code>chrt -p <PID></code></small></p>
<p>This will reduce the worst-case kernel latency but <strong>doesn’t get rid of all
sources of jitter.</strong></p>
<p>There are still things that can preempt the task. Here’s a few:</p>
<h3 id="hard-interrupts-irqs"><a class="zola-anchor" href="#hard-interrupts-irqs" aria-label="Anchor link for: hard-interrupts-irqs"><i class="icon"></i></a>
Hard interrupts (IRQs)</h3>
<p>These are signals from hardware - network cards, disk controllers, or system
timers. If your application runs on the same core that is handling interrupts,
the CPU will pause your app to run the handler.</p>
<p>You can examine the interrupts on an existing machine by inspecting
<code>/proc/interrupts</code>. This will show you devices and their IRQ number.</p>
<p>By default, all cores can service interrupts.</p>
<blockquote class="tip">
<p class="alert-title">
<i class="icon"></i>Tip</p>
<p>Non-essential hardware interrupts can be moved to another core or cores with
interrupt affinity (<code>/proc/irq/<IRQ_NUMBER>/smp_affinity</code>). See <a rel="external" href="https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/6/html/performance_tuning_guide/s-cpu-irq">Interrupts and IRQ
Tuning</a>
in the Red Hat tuning guide</p>
</blockquote>
<h3 id="non-maskable-interrupts-nmi"><a class="zola-anchor" href="#non-maskable-interrupts-nmi" aria-label="Anchor link for: non-maskable-interrupts-nmi"><i class="icon"></i></a>
Non-maskable interrupts (NMI)</h3>
<p>An NMI is another hardware interrupt which the CPU cannot ignore or disable.
These come from critical events like fatal hardware problems, memory errors,
power failures, and watchdog timer timeouts.</p>
<p>The watchdog is a daemon that keeps track of /dev/watchdog, which contains a
heartbeat from a hardware timer on the motherboard. This is to detect CPU lockups.</p>
<p>This can be and usually is disabled in HFT environments as far as I’m aware.
This supposedly the most frequent cause of this type of interrupt. The other
NMIs cannot be disabled and usually result in a kernel panic or system restart.</p>
<blockquote class="tip">
<p class="alert-title">
<i class="icon"></i>Tip</p>
<p>You can disable the watchdog while the system is running by setting the sysctl
<code>kernel.nmi_watchdog</code> to <code>0</code>, or you can edit your bootloader such as GRUB with
<code>nmi_watchdog=0</code> at the end of the line that starts with
<code>GRUB_CMDLINE_LINUX_DEFAULT</code></p>
<p>Occasionally it can show up in BIOS as “IPMI Watchdog Timer” which can be disabled.</p>
</blockquote>
<h3 id="system-management-interrupts-smi"><a class="zola-anchor" href="#system-management-interrupts-smi" aria-label="Anchor link for: system-management-interrupts-smi"><i class="icon"></i></a>
System Management Interrupts (SMI)</h3>
<p>This is also technically a non-maskable hardware interrupt, but it originates
from the BIOS or EFI and pauses the entire OS. They’re particularly insidious
because they cannot be detected via any standard linux observability.</p>
<p>These interrupts occur when the motherboard needs to perform some kind of
thermal throttling or power management, and can take in excess of 100
microseconds to complete.</p>
<blockquote class="tip">
<p class="alert-title">
<i class="icon"></i>Tip</p>
<p>There is nothing you can configure in the OS to prevent this, but there may be
BIOS options like disabling “Global SMI” or “C-States”, and these options may
only be present in server hardware that has been specifically optimized for HFT
by the vendor.</p>
</blockquote>
<h3 id="software-faults"><a class="zola-anchor" href="#software-faults" aria-label="Anchor link for: software-faults"><i class="icon"></i></a>
Software faults</h3>
<p>If the trading app tries to access memory and encounters a page fault or a TLB
miss, the kernel needs to take over and fetch the page.</p>
<blockquote class="tip">
<p class="alert-title">
<i class="icon"></i>Tip</p>
<p>Use huge-pages and pre-fault all pages. I’ll cover this more in a future
article about the TLB specifically.</p>
</blockquote>
<h2 id="validation"><a class="zola-anchor" href="#validation" aria-label="Anchor link for: validation"><i class="icon"></i></a>
Validation</h2>
<p>There are some tools out there which can be used to run tasks and measure interference, such as:</p>
<ul>
<li><code>cyclictest</code> can be used to measure kernel scheduling latency</li>
<li><code>hwlatdetect</code> can be used to measure latency caused by NMIs</li>
</ul>
<h2 id="conclusion"><a class="zola-anchor" href="#conclusion" aria-label="Anchor link for: conclusion"><i class="icon"></i></a>
Conclusion</h2>
<p>The systems we use daily work seamlessly for us, but under the surface they’re
executing sophisticated multitasking routines to give every process a fair
chance at execution, with low-level hardware events stepping in and taking
priority to avoid abrupt system failures.</p>
<p>In a trading environment, without knowledge of these interactions, your
application would be subject to all kinds of jitter, causing you to miss a
“fill” or to get sniped on a fast-moving order book…</p>
<p>… and that can be the difference between profit and poverty.</p>
Profit or Poverty: NUMA2026-04-17T00:00:00+00:002026-04-17T00:00:00+00:00
Vasilios Syrakis
https://cetanu.github.io/blog/why-numa-is-important-in-trading/<p>While modern software tries to hide hardware complexity behind abstractions,
staff working in HFT must dig down to ensure that every nanosecond is spent
executing trading logic, not chewed up by operating system overheads or noisy
neighbouring processes.</p>
<p>Learning the ins and outs of NUMA is not a micro-optimization, it is a requirement.</p>
<h3 id="short-history-of-numa"><a class="zola-anchor" href="#short-history-of-numa" aria-label="Anchor link for: short-history-of-numa"><i class="icon"></i></a>
Short history of NUMA</h3>
<p>Back in the day, CPUs were slower than memory. In the 1960s, processors began
to overtake memory in speed and as a result found themselves stuck waiting for
data to arrive from a memory access.</p>
<p>This was solved by either avoiding memory access, or by adding cache memory
some of which you may have heard of like L1 and L2 cache.</p>
<p>However, as operating systems and applications have grown, these caches are no
longer as effective. Additionally, on modern servers with multiple CPU sockets, only
one processor can access the memory at one time, causing the other to wait.</p>
<p>Non-uniform memory access (NUMA) was created to solve this problem by providing
separate memory to each socket.</p>
<h3 id="why-numa-must-be-considered-in-a-high-frequency-trading-hft-environment"><a class="zola-anchor" href="#why-numa-must-be-considered-in-a-high-frequency-trading-hft-environment" aria-label="Anchor link for: why-numa-must-be-considered-in-a-high-frequency-trading-hft-environment"><i class="icon"></i></a>
Why NUMA must be considered in a high-frequency trading (HFT) environment</h3>
<p>In HFT, nanoseconds matter. If your application has to wait an extra amount of
nanoseconds every time it needs to access some memory, it quickly adds up into
microseconds and perhaps even milliseconds.</p>
<p>On hardware with NUMA, data could reside in non-local memory aka the memory of
the other socket on the server board. For the local core to access this memory
it has to traverse an interconnect which can take 100-200 nanoseconds.</p>
<h3 id="how-to-protect-your-application-from-the-downsides-of-numa"><a class="zola-anchor" href="#how-to-protect-your-application-from-the-downsides-of-numa" aria-label="Anchor link for: how-to-protect-your-application-from-the-downsides-of-numa"><i class="icon"></i></a>
How to protect your application from the downsides of NUMA</h3>
<p>Firstly, on Linux, NUMA is modelled as nodes. For example, Node 0 is Socket 0
plus its attached memory, for however many sockets your system has.</p>
<p>You can configure different policies per-thread or per-allocation.</p>
<ul>
<li><code>local</code> (default) - allocate on the node where the thread is running.</li>
<li><code>bind</code> - allocate <em>only</em> on specified nodes.</li>
<li><code>preferred</code> - try a particular node first but fallback to others.</li>
<li><code>interleave</code> - round-robin across nodes.</li>
</ul>
<p>In HFT you probably want to use <code>bind</code>.</p>
<h4 id="discover-your-topology"><a class="zola-anchor" href="#discover-your-topology" aria-label="Anchor link for: discover-your-topology"><i class="icon"></i></a>
Discover your topology</h4>
<p>Even on your laptop or desktop linux computer, you should be able to install
<code>numactl</code>, <code>lscpu</code>, or <code>lstopo</code> and look at the nodes.</p>
<p>This is my desktop for example, which has one socket with eight cores, and 32
GB of memory:</p>
<img class="no-hover"alt="CPU topography"src="/img/lstopo.png"/>
<p>On a server system with multiple cores you should see additional separate NUMA nodes.</p>
<h4 id="disable-automatic-numa-balancing"><a class="zola-anchor" href="#disable-automatic-numa-balancing" aria-label="Anchor link for: disable-automatic-numa-balancing"><i class="icon"></i></a>
Disable automatic NUMA balancing</h4>
<p>Automatic balancing is done by <code>autonuma</code> which scans processes’ memory and
migrates pages to try to optimise locality, which is fine for most cases but
can cause latency spikes that are unacceptable in a trading environment.</p>
<p>There are two things that must be done to disable balancing.</p>
<p>Adjust the following sysctl:</p>
<pre class="giallo" style="color: #A9B1D6; background-color: #1A1B26;"><code data-lang="shellscript"><span class="giallo-l"><span style="color: #0DB9D7;">echo</span><span style="color: #FF9E64;"> 0</span><span style="color: #89DDFF;"> ></span><span style="color: #9ECE6A;"> /proc/sys/kernel/numa_balancing</span></span></code></pre>
<p>and then add <code>numa=off</code> to the boot-loader configuration, or similar, depending
on your boot-loader.</p>
<h4 id="pin-your-application"><a class="zola-anchor" href="#pin-your-application" aria-label="Anchor link for: pin-your-application"><i class="icon"></i></a>
Pin your application</h4>
<p>Use <code>numactl --cpunodebind=<node> --membind=<node> <the_program></code></p>
<p>It should be noted that the scheduler can still move threads between cores on
that node, so in addition to this you’ll want to pin your application to the
same cores using <code>taskset</code> or similar, like <code>taskset -acp <cores> <pid></code></p>
<h4 id="validate-the-setup"><a class="zola-anchor" href="#validate-the-setup" aria-label="Anchor link for: validate-the-setup"><i class="icon"></i></a>
Validate the setup</h4>
<p>Check that your process is actually bound correctly with <code>numastat -p <pid></code>.
This will print out a table detailing its memory usage for each node, where the
node should only be the one you’ve bound the process to.</p>
<h3 id="what-if-my-systems-engineers-misconfigure-numa"><a class="zola-anchor" href="#what-if-my-systems-engineers-misconfigure-numa" aria-label="Anchor link for: what-if-my-systems-engineers-misconfigure-numa"><i class="icon"></i></a>
What if my systems engineers misconfigure NUMA?</h3>
<p>If you’re a developer in a trading firm and you’ve deployed your application,
you can still check NUMA at runtime and ensure your allocations land on the
correct node. The library <code>libnuma</code> has an API that allows you to check if NUMA
is available, how many nodes there are, which node a given core belongs to, and
the ability to allocate on a particular node.</p>
<p>In reality, any competent linux/production/systems/sre team in a HFT should
already be working with their developers to ensure everything it set up
correctly.</p>
<h3 id="conclusion"><a class="zola-anchor" href="#conclusion" aria-label="Anchor link for: conclusion"><i class="icon"></i></a>
Conclusion</h3>
<p>In the overwhelming majority of environments NUMA goes by unnoticed, with the
defaults being suitable for most workloads. However in trading, if you ignore
NUMA, the results can be catastrophic. Microseconds of latency across millions
of memory accesses can be the difference between profit and poverty.</p>