Multicore Packet Processing Forum

Alexander Bachmutsky – Chief Architect at Nokia Siemens Networks

This post is the continuation of the discussion started by Kin-Yip Liu from Cavium Networks.

In general, I do agree that the best solution for the Fast Path data plane processing is to use a bare metal environment, which provides the most optimized way to process the traffic. At the same time, there is also a possibility of the Fast Path processing implemented in standard Linux, but as rightfully stated in the abovementioned post, “This is because synchronization and scheduling processing among multiple cores fundamentally takes significant overhead.”

However, I would like to argue that this problem can have a relatively “easy” solution. Already today Linux supports the concept of the thread affinity, when a particular thread can be assigned to a specific core. If we assume that only a single thread had been assigned to the core, and that this thread is implemented as a run-till-completion module, and the thread has the required access to the hardware, the final processing will be very similar to a bare metal implementation. The remaining problem is that Linux still tries to schedule some interrupts or other threads to this core, and this is what I called above “easy” solution: remove that particular core from Linux thread scheduling and timer scheduling. With this change there is no reason why Linux-based Fast Path will not show the performance level very similar to the bare metal case.

There are a number of things to remember in this approach:

1. There should be at least one core running “regular” Linux with its full scheduling and interrupt handling. This is one of disadvantages compared to a full AMP based bare metal approach.
2. There could be any number of such special threads limited to a total number of cores minus one “generic” core.
3. The desired thread has to have an access to the HW blocks. It can be done either through kernel threads or if the processor manufacturer provides a library for a user level thread to access the hardware directly. Such libraries exist from a number of the latest generation multicore processors. Full implementation would allow the HW access to be performed through the low overhead virtualization layer (hypervisor).
4. It is highly recommended to use very strict memory protection mechanisms to make sure that this special thread cannot harm other tasks in the system and vice versa.
5. There should be a communication library (in SW or HW) to exchange information with standard Linux cores.

So what are other pros and cons? The main problem is that current Linux does not support this functionality. We’ve discussed the feature with a number of Linux distribution companies for probably last 2 years, but the only announced similar capability had come recently from Tilera in a form of Zero Overhead Linux (ZOL).

The major advantage is much shorter software porting for the environment and the availability of the toolchain, libraries and utilities, debugging and other “goodies” from the Linux.

There is no doubt that bare hardware can still provide even more optimized implementation, but the simplicity and time-to-market of the proposed Linux-based concept can make it a popular development path. The biggest shift would happen if and when the concept is accepted into the kernel.org and appear eventually with every Linux distribution.

One Response to “Linux-based Fast Path”