In this article, we explore what it takes to push FIX message encoding and send latency below one microsecond at the P90 percentile.

We achieved 685 nanoseconds P90 for full FIX message encoding, file system persistence, and NIC enqueue on an AMD Solarflare using llfix, a low latency FIX engine.

Achieving this level of latency is not purely a software problem. It requires a combination of hardware selection and tuning, operating system level tuning, and careful software design:

The following sections focus primarily on software-level techniques, while the benchmarks section outlines the applied hardware and OS tunings.

The benchmarks and techniques presented in this article are based on llfix, a low-latency C++ FIX engine, available as both an open-source edition (https://github.com/CorewareLtd/llfix) and a commercial edition (www.llfix.net).

Basics of FIX and what we are measuring

FIX protocol is a TCP based messaging protocol used in electronic trading. A typical use case is “order entry” which is sending orders to venues. A FIX message is simply a group of ASCII encoded tag and value pairs. Those pairs are split by SOH (Ascii 0x1) character.

As an example, the message we used in the benchmarks (0x1s replaced by pipes) :

8=FIXT.1.1|9=218|35=D|34=2|49=CLIENT1|52=20251231-18:21:36.457245600|

56=EXECUTOR|50=SNDR_SUB|57=SRVR_SUB|11=1|55=NOKIA.HE|54=1|38=10| 44=10000|40=2|59=0|453=2|448=PARTY1|447=D|452=1|448=PARTY2|447=D|452=3| 60=20251231-18:21:36.457245600|10=221|

In low-latency trading, exchanges also provide native binary order entry protocols—such as OUCH, BOE, and ETI—typically at higher cost. These protocols are not ASCII-encoded but operate on compact binary representations, and their messages are often fixed in size.

This provides a structural advantage: fixed-size binary messages are simpler to parse, more cache-friendly, and avoid the overhead of string handling.

In contrast, the FIX protocol is string-based and inherently variable in length. As a result, achieving comparable latency with FIX is more challenging, requiring careful optimisation of encoding, parsing, and memory access patterns.

As for benchmarks, we will be measuring combined latency of the followings :

a. FIX message encoding : In a FIX engine, users interact with the engine by calling methods to set tag-value pairs. Encoding means to compute message body length for tag9 and checksum for tag10 and then concatenate all tag value pairs including equals signs and SOH characters into a contiguous stream.

b. Message serialisation to file system : Many financial systems are required by law to keep records of all transactions, including the exact data sent and received.

c. Enqueuing to NIC : Refers to the process of placing the encoded message into a network interface card (NIC) queue for transmission over the network. Note that the measurements in this article doesn’t include wire to wire latencies.

Benchmarks

The benchmark server details are as below :

As mentioned in the introduction , the major tunings applied are as below :

- Benchmark threads have been pinned to isolated CPU cores.

- CPU frequency was maximised and hyperthreading was disabled.

All benchmarks were conducted by initially sending 1000 FIX messages for warmups and then by sending 1 million FIX messages. Latency measurements are based on CPU timestamp counters using RDTSCP.

The compiler flags used for all builds are:
-DNDEBUG -O3 -march=native -mtune=native

For comparison, we include the well-known open-source FIX engines QuickFIX and FIX8, all benchmarked under identical conditions.

All benchmark source code is available at:

https://github.com/CorewareLtd/llfix/tree/main/benchmarks/networked_client_tx

https://github.com/CorewareLtd/llfix/tree/main/benchmarks/networked_client_tx_fix8

https://github.com/CorewareLtd/llfix/tree/main/benchmarks/networked_client_tx_quickfix

1. Cache friendly design

The memory wall is one of the most critical constraints in low-latency systems. It arises when CPU execution speed significantly outpaces the bandwidth and latency of the memory hierarchy :

(The memory hierarchy diagram is taken from the Microarchitecture cheatsheet https://github.com/akhin/microarchitecture-cheatsheet )

The latency degradation becomes worse as code moves further away from L1 cache residency and starts incurring L2, L3, and ultimately main memory accesses. For this reason, data layout must be treated as a first-class design concern from the very beginning, particularly in low-latency trading systems.

a. Separation of incoming and outgoing messages : One of the key design decisions in llfix is separating data structures for incoming and outgoing FIX messages.

Incoming messages require flexible, tag-by-tag access, which naturally leads to the use of hash based dictionaries.

Outgoing messages, however, do not have the same requirements. They are constructed once and encoded sequentially. Therefore, llfix represents outgoing messages using a simple std::vector of fields.

This avoids pointer chasing and unnecessary indirections, improving cache locality and reducing latency during encoding.

b. Using pools and reusable message instances per FIX session: It is important to use contigious memory for FIX message’s data to avoid indirections and potential memory access penalties.

Also heap allocations are a well-known source of latency spikes due to allocator slow paths and potential system calls. Beyond that, they also may negatively impact cache locality due to fragmentation and non-contiguous memory layouts.

To address both issues, llfix uses preallocated memory pools for both incoming and outgoing messages. This ensures contiguous memory usage improving cache locality.

Additionally, each FIX session maintains a single reusable instance for both incoming and outgoing messages. This further eliminates allocation overhead and keeps hot data consistently resident in cache.

c. No internal message queuing: Message queuing introduces additional memory traversal, pointer indirection, and cache misses. Unlike general-purpose FIX engines, llfix does not introduce internal message queues between the application and the engine.

Instead, llfix operates in immediate mode, where messages are processed and encoded directly upon invocation.

2. Memory mapped files for message serialisation

Most trading systems have the requirement of storing all messages due to regulatory rules in their region. Persisting FIX messages is also critical for automated testing, replay scenarios, and comparing against drop copies for reconciliation and record-keeping.

As a result, efficient message persistence is a core requirement of any FIX engine.

For this purpose, llfix uses operating systems’ memory mapped file feature. Memory-mapped I/O maps a file directly into the process’s virtual address space, allowing writes to be performed as simple memory stores rather than explicit file I/O operations.

The advantages compared to traditional buffered file IO are :

- Reduces required data copies between a buffer and virtual memory by mapping the file to our process’s address space

- Reduces syscall overhead

The latency impact of serialising messages via memory-mapped files is shown below:

You can find this benchmark’s source code on : https://github.com/CorewareLtd/llfix/tree/main/benchmarks/encoder

One important consideration when using memory-mapped I/O is that mappings are managed in fixed-size blocks. This requires preallocating files and designing a rotation strategy. In practice, mapping sizes should be aligned to the system’s virtual memory page size (typically 4096 bytes).

3. Kernel bypass & BSD socket interface bypass

One of the well known topics in low latency trading is the complexities in Linux traditional network IO :

- Additional data copies between kernel space and user space

- Deep callstacks

- The cost of system calls

The diagram below illustrates the lifecycle of an I/O packet under the standard Linux networking stack and highlights these overheads :

A widely adopted solution is kernel bypass, where the traditional kernel networking path is avoided. This is typically achieved through a combination of user-space TCP/IP stacks and specialised NIC hardware APIs.

Several vendors provide such solutions with their own user-space TCP stacks, including AMD’s Solarflare and NVIDIA’s Mellanox. These approaches eliminate kernel involvement in the data path, significantly reducing latency.

The diagram below illustrates the difference between the traditional kernel-based stack and user-space networking stacks :

The table below compares latency in the same benchmark using standard Linux BSD sockets versus Solarflare Onload:

These solutions require no application changes. Libraries such as Onload can transparently override BSD socket calls via LD_PRELOAD, allowing existing applications to benefit from reduced latency without recompilation.

In addition, vendors often provide proprietary APIs that support reduced features for even lower latency. llfix commercial edition uses Solarflare TCPDirect API to shave latency further. The table below compares Solarflare Onload with TCPDirect:

4. Others

a. Syscall avoidance : The primary overhead during syscalls comes from the context switch between kernel and user space. Typically, a 0x80 trap is used to switch the CPU into kernel mode. During this switch, the CPU registers must be saved and restored, which adds latency.

To avoid syscalls, llfix uses the methods below :

- Using memory pool instead of heap allocations ( as discussed previously )

- Kernel bypass ( as discussed previously )

- vDSO functions usage for timestamps

vDSO refers to a set of syscalls exposed by recent Linux kernels, allowing userspace programs to access certain system functions without entering kernel mode, thus avoiding the context switch and improving performance.

You can view the vDSO functionalities in the following link: https://github.com/CorewareLtd/llfix/blob/main/include/llfix/core/os/vdso.h

b. SIMD : Using SIMD (Single Instruction, Multiple Data) intrinsics allows parallelism within individual instructions. In llfix, we utilise SIMD AVX2 for checksum computations and validations.

Below is a benchmark comparison between 1 million iterations of checksum computation with and without SIMD (AVX2):

You can explore the SIMD-enabled methods in the following link: https://github.com/CorewareLtd/llfix/blob/main/include/llfix/fix_utilities.h

Conclusion

Achieving sub-microsecond latency at P90 in a FIX engine is not solely the result of software optimization, but rather the collective impact of several factors, including hardware and OS tuning.

On the software side, cache locality and the elimination of memory allocations along critical paths are fundamental to minimizing latency by avoiding the high cost of cache misses. Additionally, techniques like memory-mapped files for message serialisation and syscall avoidance further reduce overhead.

Ultimately, the network stack optimisation plays a pivotal role in achieving these latencies. By employing kernel bypass techniques and bypassing the BSD sockets interface (e.g., using Solarflare TCPDirect), we can eliminate the remaining latency introduced by the OS. These combined efforts make it possible to consistently maintain sub-microsecond latency, even at higher percentiles like P90.