FAQ

OpenDataPlane (ODP) is an open source API defined for networking data plane applications programming. The primary goal of ODP is to provide a common set of APIs for application portability across a diverse range of networking platforms (SoCs and servers) that offer various types of hardware acceleration. As an abstract API specification, ODP permits applications to run on and exploit the hardware offload capabilities of various platforms without requiring expertise in the nuances of any target platform.

At the same time, ODP is also a set of implementations of these APIs that are optimized for each platform that supports ODP. Implementations of ODP currently exist for a wide range of platforms spanning diverse instruction set architectures including ARM, MIPS, Power, x86, as well as proprietary SoC architectures, and include both general-purpose servers as well as specialized networking SoCs.

By decoupling the API definition from its implementation, ODP achieves two goals:

  1. APIs can be defined to address data plane application needs rather than to expose specific platform capabilities. This leads to easy application portability across any platform that supports a conforming ODP implementation. The result is that applications written to the ODP APIs can run on any platform without developers needing expertise in the nuances of that platform:
  2. [![aboutODP1](data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==){:class="img-responsive lazyload " data-src="/assets/aboutODP1.png"}](/assets/aboutODP1.png){: data-lightbox="aboutODP1.png" data-title="aboutODP1"}
  3. At the same time, by keeping APIs abstract, ODP allows vendors full freedom to implement these APIs in a manner optimized to the capabilities of each platform. This means that platforms can compete for any socket:
  4. [![aboutODP2](data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==){:class="img-responsive lazyload " data-src="/assets/aboutODP2.png"}](/assets/aboutODP2.png){: data-lightbox="aboutODP2.png" data-title="aboutODP2"}

This freedom is further enhanced by ODP’s use of 3-clause BSD licensing. ODP APIs are fully open source and open contribution, however individual implementors of these APIs may choose to make their code open or closed source as business needs determine. As part of the main ODP distribution, several reference implementations of the ODP APIs are made available and these implementations are themselves open source and open contribution. These reference implementations are designed to offer a good starting point for those wishing to develop their own implementations of ODP tailored to their platform, or to gain experiencing developing ODP applications without needing anything other than a standard Linux platform.
Also included as part of ODP is a validation test suite that permits applications and vendors to confirm that a given ODP implementation conforms to the ODP API specification, thus ensuring consistency and portability across various implementations of ODP.

The data plane is the part of a network that carries user traffic. The data plane, the control plane and the management plane are the three basic components of a telecommunications architecture. The control plane and management plane serve the data plane, which bears the traffic that the network exists to carry. ODP is only concerned with the data plane.

ODP is sponsored by the Linaro Networking Group (LNG) and its 13 member companies. These companies include network system vendors, silicon vendors, and software solution providers who are working to promote a truly cross platform solution for data plane applications that is portable across a wide range of network silicon yet can take full advantage of hardware acceleration and offload capabilities offered by these platforms.

  • To support Software-defined networking (SDN) in which control is decoupled from the physical infrastructure, allowing network administrators to support a network fabric across multi-vendor equipment.
  • To support Network function virtualization (NFV) which is an initiative to virtualize the network services that are now being carried out by proprietary hardware.
  • To enable a platform agnostic open source community to develop hardware accelerated software that is very portable.

The ODP project was launched in 2013. The first full-feature release of ODP occurred in March of 2015 and a production-ready release is targeted for year end 2015.

The history can be seen in the git stats for the the first implementation, the linux-generic reference [2].

aboutODP3

Although ODP applications are independent of available network speeds, at present the benefits of ODP are best seen on networks operating at 10Gb/s and above. This allows ODP applications to transition seamlessly from software-based acceleration found on general purpose servers to hardware acceleration found on specialized networking SoCs. At present this spans the “sweet spot” of speeds from 10Gb/s to 100Gb/s. Beyond 100Gb/s ODP abstractions have not matured enough to completely describe full offload processing, although this is a long term goal.

An application written as a clean room implementation will differ in structure from one ported from a legacy application allowing it to benefit from more of the ODP capabilities, a typical structure for a new application is shown below:

aboutODP4

Using the scheduler and an event driven model, packets are distributed to available workers maximizing the capacity of the cores. Where possible, the function will be performed in hardware (red). Migrating to a device where more of the functionality is in hardware will result in greater throughput without rewriting the application. In addition, migration to a device with more cores will automatically spread the load achieving greater throughput.

In summary:

  • The classifier might be fixed-function or programmable (for flexibility as network protocols evolve)
  • The scheduler is similar to a traffic manager
  • Processing cores can be added and removed dynamically (elasticity)
  • The scheduler knows which core is associated with which queue at every moment, which enables hardware synchronisation
  • Packets and other types of work (e.g. timers) are scheduled together

ODP_Arch_v3

An application written to the ODP API will be linked to the ODP implementation for the platform on which it is executing. This implementation will have been optimized for that specific hardware; it will often call the native SDK via an inline call, which also allows the application to simultaneously take advantage of vendor extensions that have not yet been standardized.

To date, ODP is running on seven different network platforms that span five different processing architectures (ARMv7, ARMv8, MIPS64, Power, and x86), offering both application portability and accelerated performance tailored to each platform. Other implementations are under development by both LNG member companies and other companies participating in the project.

ODP applications usually run as Linux user space applications, but there are also a number of “bare metal” environments in use. A typical deployment will be using Linux as the control node on at least one CPU and then using the NO_HZ and isolation features of Linux to essentially run the application’s fast-path packet processing code as if it were using “bare metal” on the remaining cores.

ODP is a true open source, open contribution project that is distributed under a 3-clause BSD license, meaning that anyone is free to use, modify and distribute it for commercial or other purposes without restriction.
ODP has a public mailing list (lng-odp@lists.linaro.org), and discussion on this list shows the wide base of participation in the ODP project. There are also a number of independent externally hosted ODP implementations [6].

The ODP linux-generic implementation is a functional reference targeting simplicity over performance if there is marked difference. The higher-performance implementations of ODP come directly from the vendors. Linaro also maintains implementations for some other important platforms that do not yet have direct vendor support [3].

Yes. ODP abstracts hardware capabilities for dataplane processing so that applications may be written more portably Application developers may make use of important hardware capabilities such as crypto hardware acceleration without needing deep knowledge of the hardware or the vendor-specific SDK associated with it. This will make it much easier for them to write portable applications that work well across multiple hardware implementations.

No. the ODP API does not allow for software extensions. However, ODP does not preclude calling a vendor’s SDK in parallel with ODP but the expectation is that over time any features that become common to multiple platforms will be supported in future versions of the portable ODP APIs.

There is no explicit initialization support for altering the image in an FPGA at boot time, but several major players have looked at ODP and we hope they will help define the support they require.

So far, ODP has been vigorously taken up by vendors who supply much more functionality in their hardware than a plain NIC can provide. One of those vendors package this capability as a NIC + ODP library. The ODP project also supports its own ODP-DPDK [1] implementation to help ensure that polling mode applications can continue to work even without picking up any of the new capabilities.

ODP does not dictate a model, although the majority of current contributors see greater value in an event-driven model which which it is felt will scale better than a polling mode driver.

No, ODP is just an API designed by the contributors. The implementation is developed by the hardware vendor to be optimal for their platform.

ODP is an API, DPDK is a specific implementation of an API.

ODP is an abstraction that is just at a high-enough level to allow platform abstraction without imposing strict models and overheads.

aboutODP6

ODP uses abstractions for the structures that are defined by the implementing vendor so that they map closely to the hardware and are very efficient. The application may then access this data though inline functions so that platform specific data is never exposed, for example odp_packet_len() [5] to determine a packet length.

ODP can be used to implement hardware acceleration for interfaces for sockets, or polling mode drivers.

No. ODP is defining the lowest level abstractions for hardware acceleration and it is expected that layers of software using these primitives will be able to add deeper application support. In addition ODP does not try to add Operating System abstractions.

They are traditionally the software in routers, switches, gateways, set top boxes, Evolved Node B, etc. Increasingly they are datacenter applications that can make use of the acceleration features available in servers, such as Open vSwitch [4] and SNORT.

Build

Basically you need to pass -O0 to gcc, You can bake this into the generated makefile with the ./configure step, that is probably your simplest step.

Also see the gcc manual and possibly ask on stackoverflow for more details on using gnu tools.

To build this change into the makefile at the configure step, do this

./configure CFLAGS="-O0"
make
        

look for the following in the configure output:

        
cc: gcc
cppflags:
am_cppflags: -I/tmp/check-odp/installed/x86_64/cunit-2.1-3/include
am_cxxflags: -std=c++11
cflags: -O0
                
            

This is a known issue in many cases, it is related to the use of some tricks to get strong typing out of C99 source code.

                
strong_types.h
….
/** Use strong typing for ODP types */
#define odp_handle_t struct { uint8_t unused_dummy_var; } *

                
            

If anyone knows how to inform valgrind that this is in fact correctly initialized it would be most welcome?

Question:
When I compile I get errors with _CU_TEST_INFO
make[3]: Entering directory `/home/<me>/odp/test/ validation/buffer’
CC buffer.lo
buffer.c:146:2: error: initialization discards ‘const’ qualifier from pointer target type [-Werror]
_CU_TEST_INFO(buffer_test_ pool_alloc),

Answer:
Check the DEPENDENCIES file most likely with any CU macro error you are not using the correct version of cunit.

Question:
When I compile I get errors with _CU_TEST_INFO
make[3]: Entering directory `/home/<me>/odp/test/ validation/buffer’
CC buffer.lo
buffer.c:146:2: error: initialization discards ‘const’ qualifier from pointer target type [-Werror]
_CU_TEST_INFO(buffer_test_ pool_alloc),

Answer:
Check the DEPENDENCIES file most likely with any CU macro error you are not using the correct version of cunit.

Quality Control - Testing, Verification and Validation

OpenDataPlane testing information from GCOV (using LCOV extension). Includes performance results from gcov using code test suites.

  • how often each line of code executes
  • what lines of code are actually executed
  • how much computing time each section of code uses

How to replicate:

./bootstrap
./configure --enable-cunit
make check CFLAGS="-fprofile-arcs -ftest-coverage"
lcov -c -d ${TOPDIR}/odp/platform/${PLATFORM} --output-file ${TOPDIR}/${PLATFORM}-coverage.info
genhtml ${TOPDIR}/${PLATFORM}-coverage.info --output-directory ${TOPDIR}/${PLATFORM}-gcov-html
                

Upstream

Read the odp/CONTRIBUTING file as a starting point, and then google for the many write ups about contributing to the linux kernel, ODP attempts to follow that kernel style as closely as reasonable.

This write up may be usefull: http://www.tuxradar.com/content/newbies-guide-hacking-linux-kernel

To check your patches will not be immediately rejected see the FAQ “How do I check my local patches are ready to be submitted “

And dont forget you can ask on the mailing list lng-odp@lists.linaro.org

You can re-use the CI scripts that run all ODP [regression testing](/testing/)
Assuming that you have your odp repo at ~/git/odp and it contains your committed changes that you then created patches for with git format-patch


git clone http://git.linaro.org/lng/check-odp.git
cd check-odp/
#point at the repo to test, not you can also specify a branch, see the help
PATCH_DIR=~/git/odp ./apply-and-build.sh

                    

You can get information on many more checks via:


./apply-and-build.sh -h

                        

At the time of writing:


./apply-and-build.sh makes use of the following environment variables,
GIT_ODP:	which ODP git repo to use, default: git://git.linaro.org/lng/odp.git
GIT_BRANCH:	which branch to checkout and test, default: master
PATCH_DIR:	where to look for patches, default: /home/mike/incoming
CLEANUP:	set to 0 to save *.log files, default: 1
CHECKPATCH:	set to 0 to disable checkpatch test, default: 1
CHECKFORMAT:	set to 0 to not verify that patch is ASCII-encoded, default: 1
ENABLE_CUNIT:	set to 0 to disable cunit test, default: 1
PLATFORM:	set platform, default: linux-generic
ENABLE_DRYRUN:	set to 1 to enable a dryrun, default: 0
ENABLE_NETMAP:	set to 1 to enable netmap build, default: 0
NUM_CPUS:	add parallel make, default: _NPROCESSORS_ONLN
FILE_EXT:	supported extensions in patch file names, default: mbox patch
M32_ON_64:	enable 32 bit builds on a 64 bit host, default: 0
CPP_TEST:	enable cpp test, default: 0
TOOLCHAIN_RELEASE: set toolchain release, default: 14.09
Only used when ARCH is given!
Linaros toolchain releases are located here:
http://releases.linaro.org/14.09/components/toolchain/binaries/
CUNIT_VERSION:	 Change version from sourceforge.net, default: 2.1-3

You can re-use the CI scripts that run all ODP [regression testing](/testing/)
Assuming that you have your odp repo at ~/git/odp and it contains your committed changes.

git clone http://git.linaro.org/lng/check-odp.git
cd check-odp/
#point at the repo to test, not you can also specify a branch, see the help
GIT_ODP=~/git/odp ./build.sh
                                

You can get information on many more checks via:

./build.sh -h
                                    

At the time of writing:

ARCH:		 which arch to build (arm, armeb, arm64, arm64be), default: not set
GIT_ODP:	 which ODP git repo to use, default: git://git.linaro.org/lng/odp.git
GIT_BRANCH:	 which branch to checkout and test, default: master
TOOLCHAIN_RELEASE: set toolchain release, default: 14.09
Only used when ARCH is given!
Linaros toolchain releases are located here:
http://releases.linaro.org/14.09/components/toolchain/binaries/
CUNIT_VERSION:	 Change version from sourceforge.net, default: 2.1-3
OPENSSL_BUILD:	 build custom openssl, default: 0
OPENSSL_GIT:	 git url to OpenSSL, default git://git.openssl.org/openssl.git
OPENSSL_BRANCH:	 git branch to OpenSSL, default OpenSSL_1_0_1h
COV_DIR:	 path to coverty, default: not set
LCOV:		 to genrate lcov set LCOV=1, default: 0
TEST_LIST:	 Overides the test list for 'make check' TEST_LIST=""
NUM_CPUS:	 add parallel make, default: _NPROCESSORS_ONLN
CLEANUP:	 to save workspace set CLEANUP=0, default: 1
M32_ON_64:	 enable 32 bit builds on a 64 bit host, default: 0
DOXYGEN_HTML:	 build doxygen-html, default: 0
CLANG:		 build with clang, default: 0
EXIT_ON_ERROR:	 bail out on error, default: 1
CPP_TEST:	 enable cpp test, default: 0
                                        

This is often OK, check to see if you created a cover letter with git format-patch in the PATCH_DIR you specified, the cover letter is not a patch. In this case you will probably see something like


Using patch: 0000-cover-letter.patch
Trying to apply patch
Patch applied
ERROR: Does not appear to be a unified-diff format patch

total: 1 errors, 0 warnings, 0 checks, 0 lines checked

NOTE: Ignored message types: DEPRECATED_VARIABLE NEW_TYPEDEFS

/home/mike/git/odp/0000-cover-letter.patch has style problems, please review.
                                            

There may be many lines indicating a problem deleting files that fit this pattern:

‘./check-odp/build/odp-apply/opendataplane-/’: Permission denied”

                                                

This occurs is the distcheck part of the regression testing was interrupted and you no longer have permission to clean up the files.
As root you need to delete the //check-odp/build directory.

A checkpatch warning unlike a checkpatch error may be accepted if it can be shown that overall the code would be better by ignoring it.

A common case is the line length warning in conjunction with printing debug output and a reviewer will ask why the patch was sent containing warnings.

To possibly save a round of questions, send a note in a cover letter explaining why you think the explanation should be ignored.

Miscellaneous

Specifics on a Makefile might be better answered on a forum like stack overflow, but more specifically for ODP you need to do the same things you would for any library starting with installing it, something like this:

1.) Install locally in your own dir rather than as root in the system

2.) Move to the ODP root dir

./configure --prefix=/home/<your user>/odp
make install
                                                    

That will provide the installed libraries and header files in the usual places in your home dir

/home/<your user>/odp/bin
/home/<your user>/odp/include
/home/<your user>/odp/lib
/home/<your user>/odp/share
                                                        

3.) Now just use -I and LDPATH to point to them as normal from a make file – see http://www.gnu.org/software/make/manual/make.html or Google gnu makefiles

But basically it will be like this in your shell

export LD_LIBRARY_PATH=/home/<your user>/odp/lib
                                                            

4.) In your makefile

CFLAGS=-I/home/<your user>/odp/include
LDFLAGS=-L/home/<your user>/odp/lib -lodp
CC=gcc
all: example
                                                                

example:

$(CC) $(CFLAGS) example.c -o $@ $(LDFLAGS)
                                                                    

The differences are quite important here.

The public odp/atomic.h uses a relaxed memory model so only suitable for things like statistics and shared sequence numbers etc. The internal atomics support supports a memory ordering parameter which make these functions usable for designing lock-less multithreaded data structures (and the implementation of the locks themselves). As it is not supposed that the user will design their own lock-less data structures, this API is internal to linux-generic (and thus may not exist on all platforms).