Why IP Forwarding Performance is Important
August 26th, 2011
By: Ola Liljedahl
Company / Organisation Name: Enea Software AB
Job Position: Systems Engineer at CTO Office
IP forwarding figures are often shown as benchmarks between processor and software vendors, but is this a relevant benchmark? Will not IP forwarding soon be implemented in hardware anyway? Why is software-based IP forwarding a good benchmark even if your processor can do the basic forwarding operations in hardware?
IP forwarding is already today performed in hardware; the technology exists and is used where it makes sense. E.g. in core routers which have to handle extreme bandwidths and where forwarding is the only function performed. But such hardware is expensive (e.g. many megabytes of deterministic low-latency memory is needed for the FIB), power hungry and inflexible and is thus not used in all routers. In edge routers where bandwidth requirements are less extreme and where more functionality and flexibility is needed, pure hardware-based forwarding implementations make less sense. Software is flexible and easy to update (fix bugs, extend functionality) so has its advantages.
With multi-core/multi-threading and modern SoC architectures, general-purpose processors are getting better at handling multigigabit packet processing such as forwarding. As an example, assume an OCTEON2 with 32 cores @ 1.5GHz and 700 cycles per packet for forwarding, such a device can theoretically handle 46Gbit/s traffic (Ethernet with 64 bytes/packet). An example from real life is 6WIND’s 6WINDGate which can do 60Mpkt/s of IP forwarding on NetLogic XLP832.
If forwarding can be done at sufficient speeds using such general-purpose processors, the concept of software-defined function can be achieved. The same piece of hardware can be dynamically reallocated and used for different purposes as the need varies. Larger amounts of uniform hardware decreases cost in different ways and it decreases R&D risks because software can be reused on many different processors.
Now let us analyse and understand IP forwarding by breaking it down into its constituents:
1. Driver RX processing (retrieve a packet from the hardware).
2. Basic health checks (checksums etc.) and demultiplexing.
3. Route lookup and MAC address lookup (often combined).
4. Packet layer-2 and IP header updates.
5. Driver TX processing (pass the packet to the hardware for transmission).
(Beware of cheated benchmarks! A vendor can cheat by skipping basic health checks and demultiplexing (i.e. hard-wire IP forwarding) and perform non-scalable lookups (e.g. using a hash table). Such cheats may not be detected by testing equipment. If cheats are used in a specific benchmark, the real performance will not be reliably indicated).
Nothing strange here, indeed these are the basic operations of a networking/IP stack, performed for every IP packet that is received or transmitted! I.e. IP forwarding exercises the essential operations of an IP stack, operations which contribute to the per-packet overhead for every packet that is received and/or transmitted, whether forwarding or termination or any other function is realised. If IP forwarding has poor performance, then all other functions of the IP stack will suffer accordingly.
Thus software IP forwarding performance is an indication (litmus test) of the performance of the IP stack in general.
Posted in 
Tags:
I doubt it if the software solution is really more preferred than hw IP forwarding solutions. Considering its reliability, stability and performance…
Maxim,
Hardware solutions are expensive to design (partly since bugs are difficult to fix, you must spend a lot of effort on verification), may add significantly to the bill of material (the chip itself, board space, connections etc) and power consumption and they are not very flexible compared to software solutions (my original point). And also hardware contain bugs, if you have worked with any early silicon, you know that modern system-on-a-chip designs sometimes have errata lists with many dozens of items. A chip re-spin may cost in excess of $1million and incurring months of delay so many hardware bugs never get fixed, they are either worked around if possible or the specification is simply changed.
Hardware forwarding has its places though but I was not discussing core routers. Instead I argued that edge routers need the flexibility provided by software and may have less strict performance requirements, making the trade-off sensible.
– Ola
Ola,thank you for your explanation. Just for personal interests, do you have any documents,reports or articles regarding the design issues
which I could read or quote. Thank you!
Maxim, I am not a hardware designer but I have been a recipient of a number of early silicons and studied many errata descriptions.
I googled for “cost of re-spin” and found for instance this page: http://yosemite-bauernhof.com/respin.aspx
This is about FPGA’s and not custom ASIC’s but I can imagine the opportunities for problems are even bigger with full custom chip designs.
– Ola
Maxim,
I do not understand your comment.
Let’s analyse my understanding of your comment: HW forwarding means that it is made of ASIC (for instance Broadcom StratXGS, Marvell Prestera, Fulcrum/Intel Focalpoint). ASICs are designed like any software process, it can be made of bugs (any HDL bugs)
so due to bugs, it is not 100% reliable and it is not 100% stable.
ASICs can be faster than software based packet processing because they provide more execution parallelism without any overhead, but there is a major drawback: it does not allow customizing the execution flow very easily, excepted for the pre-defined use cases.
For instance, how to use gcc in order to customize the configurable logics of the ASICs?
Ola’s comment is about having high performance IP forwarding using well designed software for multicore processors. Of course, ultra high performance IP forwarding can be achieved using some FPGAs or some ASICs.
Please, could you explain a bit more your comparison?
Thank you,
Vincent
Hi Vincent,
thank you for your reply. First of all, I think no man-made systems, neither sw nor hw-based, could be 100% stable and reliable. Under this premises, if we compare the failure rate of a hardware based with software based systems, I believe that a SW-based system may be more failure-prone than a HW-based system, since the execution cycle of such system is more complicated thus allows error to creep in.
I do agree that software based solution will be more flexible, however, I wondering how often a backbone router would change its execution flow and if such flexibility ganna pay-off. (Would very much like to know if you have any statisitcs
regarding this).
cheers,
Maxim
Good observations Ola. I agree that the IP forwarding in SW is a valuable benchmark, but the raw number that can be reached should not be interpreted as the full application. But as with all benchmarks the raw number is the full story. The reason to go to SW is what one wants to do on top of the IP processing. The way to interpret the 46Gbit/s benchmark is that at 4.6Gbit/s one should have 90% of the cycles left for the actual application.
Pekka