sudo mkdir /var/run/odp/ start ODP application that reads/writes to the interface of interest start the ODP application sudo dd if=/var/run/odp/<pid>-<ifname>-flow-<queue#> of=~/test.pcap cntrl^c to end capture wireshark ~/test.pcap to view capture
ODP v220.127.116.11 is a refresh of ODP, incorporating significant configurability and performance improvements as well as new APIs and API restructures.
odp_shm_t has the value
ODP_SHM_INVALID, consistent with other ODP types. The legacy synonym
ODP_SHM_NULL is now removed for consistency.
New support for 3GPP crypto algorithms is added by defining symbols for
In addition new authentication algorithm symbols are defined for
These values are returned as ODP capabilities as well as being accepted in crypto session creation for implementations that indicate support for them.
bit_mode capability Boolean is added to the
odp_crypto_cipher_capability_t struct to indicate that an implementation supports operating in bit mode. When operating in bit mode, field offsets and lengths are expressed in terms of bits rather than bytes. However, such lengths must always be specified in multiples of 8.
The ODP crypto API specification is tightened to specify default values for cipher and authentication algorithms. Also documented when key and IV parameters need to be set.
IPsec requires "salt" (extra keying material) when the GMAC authentication algorithm is used. To accommodate this the
auth_key_extra field is added to the
odp_ipsec_crypto_param_t struct and documentation is added clarifying when this field is needed and how it should be used.
odp_pmr_t type name for an invalid value is renamed from
ODP_PMR_INVALID for consistency with the rest of ODP type names. The old symbol is still available when ODP is configured with
odp_packet_subtype() API is added that returns the subtype of a packet event directly.
odp_packet_parse_result() API is added that returns the result of packet parsing as a single
odp_packet_parse_result_t struct. This can offer efficiency improvements for applications that need all parse results rather than making individual parse result calls.
PktIO interfaces support multiple input queues to enable increased parallelism in I/O processing. Previously, all of these input queues were required to belong to the same scheduler group. The
odp_pktin_queue_param_t struct is now extended with an optional
odp_pktin_queue_param_ovr_t struct that permits individual pktin queues to be assigned to separate scheduler groups. This may permit improved performance for advanced application use cases.
odp_timer_capability_t struct is extended to return three additional pieces of information:
The total number of timer pools that can be created combining different clock sources
The maximum number of timer pools for a given clock source.
The maximum number of timers in a single pool. A zero value means number is limited only by available memory.
Three new APIs:
odp_schedule_default_prio() are added that return the min, max, and default values specified for the
prio field in the
With the introduction of these scheduling priority functions the previously defined macros (
ODP_SCHED_PRIO_LOWEST) are now deprecated and should no longer be used.
odp_schedule_prio_t type definition was left to each implementation. With the addition of explicit schedule priority ranges, this type is now specified to be an
int to permit efficient implementation (including inlining) of these functions.
The new scheduler APIs
odp_schedule_multi_no_wait() are added to provide more efficiently implementable versions of these functions. The existing scheduler APIs remain unchanged. These new APIs can simply provide a fastpath for some applications/implementations as an alternative to specifying a parameter on
odp_init_t parameter passed to
odp_init_global() is extended to add the
mem_model field. This field is defined by the new
odp_mem_model_t struct and is used to specify whether the application will be using a thread (
ODP_MEM_MODEL_THREAD) or process (
ODP_MEM_MODEL_PROCESS) memory model. The default is a thread model is used for compatibility with previous levels of ODP.
A number of changes to the ODP ABI have also been made in this release to improve application binary portability.
odp_timer_pool_t is now strongly typed.
The values of the various
ODP_xxx_INVALID symbols for ODP abstract types in the
odp-linux reference implementation are now consistently zeros. This reduces errors and improves portability.
A new configuration file mechanism is introduced that makes use of libconfig to enable various runtime ODP parameters to be specified dynamically.
Default configuration values for the
odp-linux reference implementation are contained in the
config/odp-linux-generic.conf file. Users may override these default values by supplying their own configuration file. At
odp_init_global() time, if the
ODP_CONFIG_FILE environment variable is set, this is used to locate the path to the override configuration file.
odp-linux reference implementation now supports applications that run in process mode (
ODP_MEM_MODEL_PROCESS) as well as the default thread mode. This support only applies within a single ODP instance, so any
fork() calls must be done only after
odp_init_global() has been called to initialize ODP on a root process.
iQuery scheduler is removed from the
odp-linux reference implementation, as it offers no performance advantages and has not seen application use.
odp-linux reference implementation now supports up to 256 CPUs by default (increased from 128).
odp-linux reference implementation now supports large burst sizes for both I/O and non-I/O scheduled events. Large bursts (when available) are received directly to the application without any stashing for improved throughput. Burst sizes are configurable via the new configuration file mechanism, as described above.
--without-openssl a warning will be issued cautioning that strong cryptography will not be available.
The various enq/deq APIs are now subject to inlining when
odp-linux is built with
Inline timers are now controlled via a config file option. Timer polling frequency is similarly controlled via the config file.
The config file is now used to specify the huge page usage limit.
The queue implementation in
odp-linux now automatically makes use of optimized single and multi-consumer/producer rings to significantly speed up enq/deq processing.
The output from
odp_shm_print_all() is reworked to provide more useful and comprehensive shared memory usage information in
SA lifetime checking is now more scalable to multiple threads. This significantly reduces overhead for multithreaded IPsec applications.
When running in non-ABI compatibility mode,
odp-linux now enables native machine-specific optimizations for the CPU architecture of the local machine.
SCTP test packets are now used in parser testing. SCTP headers are added to ODP and ODP helpers and SCTP checksums are now inserted and verified as part of validation testing.
The packet validation test suite now properly tests
In support of process mode, ODP helper functions have been changed to better match these new capabilities
odph_linux_thread_type_t enum has been replaced with the new
odph_options() getter function is added that returns applicable options in effect via the new
odph_helper_options_t struct. This currently includes the memory model (thread or process) that is in use.
The new helper APIs
odph_sctp_chksum_verify() are added to facilitate working with SCTP packet checksums.
odp_pool_perf test has been added that stress-tests ODP pool functions in a multithreaded environment to generate performance statistics.
odp_sched_perf test has been added that stress-tests the scheduler in a multithreaded environment.
odp_cpu_bench performance test has been added that runs compute-intensive packet operations in a multithreaded environment and prints the observed maximum throughput for each thread.
odp_classifier example program now uses a reduced number of threads by default to reduce elapsed run time.
ODP_THREAD_COUNT_MAX is also now used as the max worker count.
odp_generator example has numerous cleanups and performance improvements.
odp_ipsec example now properly stops and closes pktio devices on exit.
odp_packet_dump example is added that prints received packets to the terminal. This is useful for debugging packet I/O interfaces.
odp_sysinfo example is provided that prints system information. Useful for checking the ODP environment during debugging. This includes providing detailed information about the various crypto facilities supported, as well as the feature flags used at build time (e.g., if the binary was built with ARMv8.0 or ARMv8.1 instructions).
The traffic manager example now properly destroys all TM queues it creates for improved reliability. It also now always prints a proper termination summary message.
Compile fails on OpenSuSE 42.2 Leap with error: negative width in bit field '__error_if_negative'
IPsec antireplay check drops packets when sequence number jumps.
IPsec SA creation must fail for ESN-enabled SAs
Per-SA IPv4 ID allocation may cause duplicate IDs.
Unexpected IP ID causes IPsec API validation to fail
rte_mempool_ops_alloc() is not dpdk api
Fixed enq/deq issues encountered on architectures with weak memory ordering.
Return 0 from
odp_sys_huge_page_size_all() if hugepages are not supported/detected. Tests modified to not treat this as an error.
ODP_CACHE_LINE_SIZE to 128 on ppc64le systems.
iplookuptable fix putting values into table
DPDK pktio support now works properly across multiple ODP instances.
Zero timer pool memory on reserve (fixes timer failures due to uninitialized variables).
-march=native disabled for
clang. This fixes a known issue with recent levels of clang.
ODP v18.104.22.168 is the second service update for the Tiger Moth release. It incorporates a number of corrections and enhancements that further improve the quality and testability of ODP.
There are no API changes in this release.
ODP is now paired with DPDK 17.11.3 for its DPDK-related support. This is the current service level for the DPDK 17.11 LTS package used by ODP Tiger Moth.
This release incorporates several improvements in the
odp-linux reference implementation of ODP.
ODP supports inlining of functions in embedded environments when ABI compatibility is not needed. ODP itself now makes use of inlined functions for all relevant internal use of its APIs, leading to improved performance.
odp_hash_crc_gen64() API is now properly implemented and appropriate validation tests added to support it.
In addition, a streamlined table-based implementation of the basic CRC functions eliminates the previous dependency on
To better integrate with DPDK parsing capabilities, ODP packet parsing has been restructured to operate at the PktIO level. This permits DPDK PktIO types to exploit the native DPDK packet parser and checksum facilities, leading to better integration.
The PktIO functions have been streamlined and refactored in a number of ways to provide better long-term maintainability of these functions. This includes moving per-PktIO data into individual files rather than sharing a common file, as well as better placement for I/O statistics.
Loop PktIO interfaces now add the capability to validate packet L3 and L4 checksums as part of receive processing. The existing
odp_pktio_capability() API now reports that checksum validation is available for these interfaces.
When defining lock free queues that have only a single producer and consumer, a streamlined implementation offers significant speedup.
Fast pcap capture is now provided in
odp-linux to capture packets on any interface. This is enabled via the
--enable-pcapng-support configuration option. Once enabled, packets can be captured using a sequence such as:
sudo mkdir /var/run/odp/ start ODP application that reads/writes to the interface of interest start the ODP application sudo dd if=/var/run/odp/<pid>-<ifname>-flow-<queue#> of=~/test.pcap cntrl^c to end capture wireshark ~/test.pcap to view capture
Interfaces of interest are identified by a string consisting of the application process ID, the interface name, and the queue number of interest, if the interface supports multiple queues.
A number of autotools/autoconf M4 macros used in configuring
odp-linux have been rewritten to avoid potential GPL licensing concerns. These macros are used only during ODP configuration processing and have no role in ODP or ODP application use.
The validation test suite for queue API testing is enhanced to now test operation on queue pairs properly. This enables the various enqueue/dequeue modes defined by the ODP specification to be more fully exercised, leading to improved API conformance.
The scheduling validation tests now better use the various capability APIs to ensure that implementations are only tested for advertised capabilities.
The crypto validation tests now better use the various capability APIs to ensure that implementations are tested across advertised crypto capabilities.
odp_queue_perf test has been added to test plain (non-scheduled) queue performance in various modes.
getopt library calls are no longer used to avoid packaging conflicts that can arise with this use. There are no changes to helper functionality. This change simply improves packaging.
odp_generator example adds UDP port range support.
Numerous changes to Travis integration are added to improve the quality and reliability of Continuous Integration (CI) testing.
Timeout accuracy breaks down with odd resolution requests
timer validation test fails when using 1GB huge pages
example/l2fwd_simple fails on some systems when using 1GB huge pages
Corrected the handling of timeout events in the scalable scheduler.
Fixed IPsec link order to streamline builds.
Fixed scaling issues with scheduler support for large core count systems.
Shmem validation test runs indefinitely with 1GB huge pages
There are several issues that seem to arise when running ODP applications on platforms with more than 64 cores. While the most critical of these, which affected the scheduler, have been fixed in this release, there are others that are still under investigation. These will be addressed in the next release of ODP.
ODP v22.214.171.124 is the first service update for the Tiger Moth release. It incorporates a number of corrections and enhancements that improve the quality and testability of ODP.
There are no API changes in this release.
ODP is now paired with DPDK 17.11.2 for it’s DPDK-related support. This is the current service level for the DPDK 17.11 LTS package used by ODP Tiger Moth and incorporates a number of important bug fixes.
The ODP reference implementation has been improved in a number of areas.
The GCC 8 series of compilers provides additional warnings about possible string truncation. A few ODP modules were changed to avoid misleading warnings when compiling with this level of GCC.
libnuma library is used for DPDK pktios to provide proper memory allocation on NUMA-aware systems. Linking with this library is improved to avoid extraneous error messages at build time.
Packet metadata is reorganized to reduce the cache footprint used by the ODP reference implementation, resulting in performance improvements.
odp_random_xxx() family of APIs was moved to a separate module (
odp_random.c) for modularity and better isolation from planned crypto enhancements.
Unnecessary locking is removed from the
odp_fdserver module, streamlining operations on shared memory.
The default number of timer pools supported by the ODP reference implementation has been reduced from 255 to 32. This lower number remains generously adequate for most applications and meaningfully reduces memory footprint, giving better performance.
During initialization ODP normally measures timer resolution to set the reported
highest_res_ns. When such measurement is not able to be performed, this is not limited to 500ns to avoid bounds errors with overly precise resolutions.
The ODP validation test suite has been improved in a number of areas.
The validation test now correctly handles corner cases when the implementation under test fails to process any test packets. It also includes the previously missing
Additionally, since individual implementations indicate which crypto/hash algorithms are supported via the
odp_crypto_capability() API, the crypto validation test now properly uses this information and only tests those algorithms that the implementation reports as supported. The list of unsupported algorithms is also reported as part of the test results.
The number of input/output queues used by this test can now be specified, providing additional controls for test flexibility. In addition, pktout queues are now selected based on input queue rather than worker id, thus ensuring packet order flow is maintained.
Finally, an inactivity timer is added that allows the test to report when packets were handled due to timeout rather than I/O activity.
The validation test for the ODP timer APIs has been reorganized to better characterize an implementation’s conformance to the ODP Timer API specification. Since implementations can have widely differing timer accuracy levels, particularly when running in virtualized environments, the test also relaxes its bounds checking and enforcement somewhat to better avoid false negative test results.
Additionally, a timer pool create/destroy test was added as this area was not adequately covered previously.
EXTRA_ASCIIDOC_FLAGS environment variable may now be used to supply additional build flags for Asciidoctor, which can be used to override icons and/or fonts for distribution or other needs.
PktIO does not work with Mellanox Interfaces
RX UDP checksum offload drops valid UDP packets with Niantic
IP header checksum not inserted if L4 offset not set
fdserver process interferes with signal handling
tests should fail if odp_crypto_op/op_enq process 0 packets
IPsec code can occasionally damage packets
Timer segfaults when creating and destroying multiple timer pools
linux-gen: ipc test fails to reserve memory
ODP v126.96.36.199 is the official Tiger Moth final release. It incorporates final implementation changes and bug fixes and completes the Tiger Moth ODP development cycle.
No functional changes for this release. The Tiger Moth API was frozen in ODP v188.8.131.52.
The specification for the
odp_packet_l4_chksum_status() API has been clarified to reflect that in IPv4 UDP checksums are optional. As a result, a zero (nonexistent) checksum will be reported as
The C++ test included in the ODP validation suite now uses
cout instead of
printf() to ensure that C++ is being used to compile it.
For the ODP Reference Implementation, The
config/odp-linux-generic.conf file is extended with sections to control the default and maximum sizes for basic queues, and the priority spread used by the scheduler for scheduled queues.
The configuration file is a template named
platform/odp-$platform.conf so this can be easily inherited by other ODP implementations.
configFile Read Order Improvements
For the ODP Reference Implementation, the default values of the application-provided
config file (if used) override the values provided by the built-in
odp-linux reference implementation is improved in a number of areas:
PktIO netmap support now uses the ODP config file to allow rings used for VALE processing to be specified. The supplied defaults provide optimal performance in typical settings.
These crypto/authentication algorithms are now implemented.
Proper packet checksum validation and insertion, in conformance with the relevant ODP APIs, is now provided.
The Tiger Moth LTS release is synchronized with the most recent DPDK LTS release for DPDK pktio support.
This dependency is removed. The Reference Implementation build tools now use the standard
od tool rather than the optional
A new test has been added to test the performance of PktIO operations in scheduled mode. Scheduled PktIO is inherently more scalable and simpler from an application standpoint than direct (polled) I/O, but depending on the efficiency of the scheduler implementation can incur additional levels of overhead. This test can give insight into a given platform’s scheduler efficiency. For the
odp-linux reference implementation, this test has shown scheduled I/O to be within 10% of rates achievable via direct I/O, meaning that for many applications the simplicity and scalability of the event model is preferable.
A new test has been added that measures outbound (TX) IPsec performance with a variety of cipher and authentication algorithms.
README file associated with this example has been clarified to explain that this example is a throughput test and as a result does not preserve packet order under all conditions.
ODP linux-generic fails on AArch64 in non-ABI-compat mode.
PktIO does not work with Mellanox Interfaces
RX UDP checksum offload drops valid UDP packets with Niantic
IP header checksum not inserted if L4 offset not set
fdserver process interferes with signal handling
return value not checked for some fdserver interface functions
ODP exposes symbols outside of odp*/_odp* namespace
ODP v184.108.40.206 is a fix level for Tiger Moth Release Candidate 2 (RC 2). It incorporates fixes and performance / serviceability enhancements but no API changes.
No changes for this release.
This release corrects a merge issue with crypto functionality that resulted in incorrect crypto operation for certain cipher suites.
Introduces a runtime configuration file that can be used by applications to set selected ODP configuration parameters dynamically at runtime rather than at
configure time. At present this is used to configure parameters needed by DPDK PMDs when using PktIO interfaces in the DPDK class. The intention is to expand the use of this file over time to allow more dynamic control of other aspect of ODP runtime configuration.
For the ODP Reference Implementation, a template configuration file is provided in
config/odp-linux.conf. Introduction of this support generates an additional dependency on the
libconfig package, which must be present to build ODP.
A few missing implementation features associated with TFC packet generation have been added in this fix level. This support is now functionally complete in the ODP Reference Implementation.
The information provided in debug builds of the Reference Implementation is improved to print the interface name on open, start, stop, and close calls. The driver name and number of queues are also printed to ease verification of correct configuration.
The performance of the default scheduler in the Reference Implementation is significantly improved in providing scheduled access to PktIO queues. Scheduled I/O now operates within 10% of the performance achievable using Direct I/O, while providing incomparably better scalability in multicore environments.
In preparation for the Tiger Moth official release, ODP has adopted a simplified
.so naming scheme, which is introduced here. ODP
.so numbers are now tied to the ODP release number since ODP does not promise backward compatibility across release boundaries.
ODP v220.127.116.11 is Tiger Moth Release Candidate 2 (RC 2). It completes the new APIs that are part of the Tiger Moth Long Term Support (LTS) release of ODP as well as various performance refinements and bug fixes. As of RC2 the ODP API is now frozen for the Tiger Moth development series.
The following new and changed APIs are included in this release:
odp_init_t struct used as the argument to the
odp_init_global() API has been expanded to include a
max_memory field that specifies the maximum amount of shared memory (shm) that the application will use. This is to better enable ODP implementations to optimize their use of shared memory in support of the application. If left as (or defaulted) to 0, the implementation may choose a default limit for the application.
A number of crypto refinements are included in this release:
The single initialization vector (
iv) in the
odp_crypto_session_param_t is replaced by a separate
The single initialization vector (
override_iv_ptr) in the
odp_crypto_op_param_t is replaced by a separate
The special nature of GCM and GMAC authenticated encryption modes is clarified in that these ciphers always combine ciphering with authentication and hence require both to be specified when used. This is simply a documentation change as this requirement has always existed.
Enumerations for AES_CCM (
ODP_AUTH_ALG_AES_CCM) authenticated encryption modes are added.
Enumeration for the AES_CMAC authenticated encryption mode (
ODP_AUTH_ALG_AES_CMAC) is added.
Enumerations for the ChaCha20-Poly1305 (
ODP_AUTH_ALG_CHACHA20_POLY1305) authenticated encryption modes are added.
Enumeration for the SHA-384 authentication algorithm (
ODP_AUTH_ALG_SHA384_HMAC) is added.
Enumeration for the AES-XCBC-MAC authentication algorithm (
ODP_AUTH_ALG_AES_XCBC_MAC) is added.
odp_nonblocking_t enums introduced in ODP v18.104.22.168 are now returned as separate
odp_queue_capability() limits for plain and scheduled queues. The ODP reference implementations now support
The specification for
odp_packet_user_ptr() is clarified that unless overridden by
odp_packet_user_ptr_set() the value of NULL will be returned.
ODP_PKTIN_WAIT option on
odp_pktin_recv_mq_tmo() is removed. Timeout options now consist of
ODP_PKTIN_NO_WAIT and a user-supplied timeout value. Since this timeout value can be specified to be arbitrarily long, there is no need for an indefinite wait capability as provision of such a capability proved problematic for some ODP implementations.
odp_packet_l4_type() are added to return the Layer 2, 3, and 4 protocols, respectively, associated with packets that have been parsed to the corresponding layer. If the packet was not parsed to the associated layer these return
The documentation of
odp_packet_data() is clarified to indicated when this shortcut may be used safely and a new API,
odp_packet_data_seg_len(), is added that returns both the address of the start of packet data as well as the number of bytes addressable from that pointer.
Two new APIs,
odp_schedule_order_lock_wait() are added to allow for asynchronous ordered lock acquisition in addition to the existing synchronous
odp_schedule_order_lock() API. In some implementations and applications, there may be a performance advantage to indicating the intent to acquire an ordered lock to allow the implementation to prepare for this while the application continues parallel processing and then enter the critical section protected by the ordered lock at a later time. In this case ordered lock protection is not guaranteed until the
odp_schedule_order_lock_wait() call returns.
IPsec support is further enhanced with the following:
odp_ipsec_ipv6_param_t structures are added to formalize the specification of IPv4 and IPv6 options in the
mode field of the
odp_ipsec_out_t is renamed to
frag_mode for better clarity. In addition the
flag.frag-mode option bit in the
odp_ipsec_out_opt_t struct is defined to hold per-operation options for the
odp_ipsec_capability_t struct returned by the
odp_ipsec_capability() API is expanded to include the
odp_proto_chksums_t available on inbound IPsec traffic. This indicates whether and how inbound packet checksums may be validated for decrypted IPsec traffic.
Traffic Flow Confidentiality (TFC) allows applications to defend against traffic analysis attacks by inserting dummy packets as well as add pad bytes to packets traversing IPsec flows.
Dummy packets have an L3 type of
ODP_PROTO_L3_TYPE_NONE in tunnel mode and
ODP_PROTO_L4_TYPE_NO_NEXT in transport mode. Padded packets have additional data suffixed to them that extends beyond the L3 or L4 packet lengths.
For RX processing, inline dummy packets may or may not be dropped from the inbound packet stream. For lookaside processing they are always visible. For TX processing, the
odp_ipsec_out_opt_t struct specifies the
tfc_pad bit if the packet is to be padded or the
tfc_dummy bit if a dummy packet is to be inserted. The dummy packet length is specified by the
ABI support has been reorganized to make it more modular and to omit headers and related ABI files when configure to disable this support.
The ODP Reference Implementation corporates a number of improvements that result in better code organization as well as improved processing efficiency.
In the LNG Reference Implemenations of ODP, the
odp_pktio_open() API may now specify devices of class
null to indicate the PktIO is treated as a dummy device. Null devices behave just like Linux
/dev/null in that they never receive packets and simply discard any packets sent to them.
Note that not all ODP implementations may support this device class. The specific device classes supported by each ODP implementation may vary and are listed in their own documentation.
The ODP Reference Implementation offers both a default and a number of alternate scheduler implementations. Previously these were selectable only at
configure time. They can now be dynamically selected at runtime by the use of the
ODP_SCHEDULER environment variable. If this environment variable is not set, the default (basic) scheduler is used. It can be set to select alternate schedulers:
ODP_SCHEDULER=basic Explicitly selects the default scheduler
ODP_SCHEDULER=sp Selects the strict priority scheduler
ODP_SCHEDULER=iquery Selects the iQuery scheduler
ODP_SCHEDULER=scalable Selects the scalable scheduler
The ODP Reference Implementation now uses a ring model for implementing ODP queues. This results in greatly improved efficiency for queue operations. The default maximum queue depth used is 4096 elements, and this information is returned via the
The tests that are part of the validation test suite are reorganized and simplified by having a single test file for each API rather than separate CUnit driver files and test files.
crypto validation test suite now offers better information on which crypto algorithms were skipped because they are not available. Testing of full HMAC lengths is now added
odp_generator example now offers configurable RX burst size, selectable packet handling (Direct I/O or Scheduled I/O), as well as streamlined packet processing.
l2fwd example offers improved efficiency via better cache usage.
timer test might fail
time_main test fails if run under heavy load
classification: CoS queues in invalid table index
classification: multiqueue CoS will always fail
classification: requested number of queues is ignored in multiqueue CoS
cls: capability to return max hash queues
classification: invalid memory access in RSS hash
classification: incorrect IPv6 RSS hash
IPsec SA may be used before fully initialized
IPsec SA lookup may leave extra SAs locked
new generator test assumes that null:0 pktio is always present
packet_main test can fail
DPDK pktio stops receiving packets if all configured RX queues are not used
Another timer_main failure
Creating a pool with total size over 4.29GB (UINT32_MAX) leads to undefined behavior
ODP v22.214.171.124 is Tiger Moth Release Candidate 1 (RC 1). It introduces significant new API support as well as functional refinements that expand ODP offload support to cover IPsec, as well as other improvements.
The following new and changed APIs are included in this release:
The ODP event model has been expanded to include new types as well as event subtypes. Subtypes are used to qualify an event by indicating that the event carries additional metadata relating to specific usage or operations.
ODP_EVENT_IPSEC_STATUS (to be discussed below) is added. The initial subtypes defined are:
ODP_EVENT_NO_SUBTYPE, which are also discussed below.
Associated with this support, new event APIs are added:
odp_event_subtype() extracts the
odp_event_subtype_t from an
odp_event_types() returns the
odp_event_subtype_t of an
odp_event_t in a single call.
odp_event_type_multi() scans a list of
odp_event_t objects and returns the number that share the same indicated
odp_event_type_t. This allows multiple events to be processed by groups of the same event type.
odp_event_filter_packet() scans a list of
odp_event_t objects and partitions them into a returned array of
odp_packet_t objects and a remaining array of non-packet events.
odp_event_free_multi() frees multiple events in a single call.
odp_event_free_sp() frees multiple events originating from the same pool in a single call. The caller must assure that the input events are from the same pool.
ODP Tiger Moth introduces comprehensive protocol offload support for IPsec, allowing applications to leverage the IPsec acceleration capabilities of many SoCs. Support includes APIs for Security Association (SA) creation and lifecycle management, as well as IPsec packet operations for inbound (RX) and outbound (TX) processing. Packet operations are further divided into lookaside and inline support.
Lookaside processing enables IPsec packets to be decrypted into plain packets or plain packets to be encrypted into IPsec packets in a single operation under explicit application control. This is useful for packets that need pre- or post-processing, or to better fit with existing application design.
Two forms of lookaside processing are provided: the
odp_ipsec_out() APIs provide synchronous decrypt and encrypt support, respectively. The corresponding
odp_ipsec_out_enq() APIs provide these same services in asynchronous form where operations can be launched and completed later.
In contrast to lookaside support, IPsec inline support permits applications to fully leverage the offload capabilities found in many SoCs by allowing inbound IPsec packets to be recognized and decrypted automatically before they are presented to the application for processing. This is done by configuring a Security Association (SA) and its associated PktIO to operate in inline mode.
Similarly, following output SA and PktIO configuration, the
odp_ipsec_out_inline() API permits a packet to be encrypted into an IPsec packet and automatically scheduled for TX processing in a single operation. Such "fire and forget" processing enables applications to leverage IPsec HW support for such processing in a portable manner.
Applications using IPsec inline support need only "touch" a packet once compared to three times when using lookaside processing, leading to greater processing efficiency.
New event types and subtypes are introduced to provide support for IPsec processing. The
ODP_EVENT_PACKET type has a new subtype:
ODP_EVENT_PACKET_IPSEC that provides extended metadata associated with IPsec packets that have been processed. The new
ODP_EVENT_IPSEC_STATUS event, in turn, is used to report IPsec status events such as completion notifications associated with
odp_ipsec_result() API is used to obtain IPsec result metadata from a packet that has event subtype
ODP_EVENT_PACKET_IPSEC, while the
odp_ipsec_status() API is used to obtain IPsec status metadata from an
Packet parsing has been overhauled with the introduction of two new APIs:
These use an
odp_packet_parse_param_t struct to control the type and depth of parsing to be performed. These routines are intended to be used to process packets that have been decapsulated following IPsec decryption or other tunneling or on IP fragments after they have been reassembled.
Associated with this improved parse support, the
odp_parser_layer_t struct is deprecated and replaced with a more general
odp_proto_layer_t struct that is used both in PktIO configuration as well as the new parser APIs.
The ODP crypto APIs are extended to provide support for AES-CTR cipher and AES-GMAC authentication modes, reflecting the growing availability of accelerated support for these.
DES-CBC is no longer considered secure and support for it is removed in ODP.
The Additional Authentication Data (AAD) length is now part of the
odp_crypto_session_t rather than individual crypto operations. This provides better compatibility with DPDK, which made a similar API adjustment in it’s 17.08 release.
odp_crypto_operation() API is retained for compatibility, new packet-oriented variants are introduced that provide additional capabilities and flexibility. These APIs are:
odp_crypto_op() Performs synchronous crypto operations on one or more input packets under the control of an associated
odp_crypto_op_enq() Performs asynchronous crypto operations on or or more input packets under the control of an associated
odp_crypto_operation() calls result in
ODP_EVENT_CRYPTO_COMPL events for compatibility, the new packet-oriented APIs result in
ODP_EVENT_PACKET events that carry the new event subtype
ODP_EVENT_PACKET_CRYPTO. These packets contain additional metadata associated with the crypto operation. New APIs added for manipulating this metadata include:
odp_crypto_packet_from_event() converts an
odp_event_t of type
ODP_EVENT_PACKET and subtype
ODP_EVENT_PACKET_CRYPTO to an
odp_crypto_packet_to_event() converts an
odp_packet_t crypto packet back into an
odp_crypto_result() extracts the
odp_crypto_packet_result_t struct that contains the crypto metadata associated with an
odp_packet_t of event subtype
ODP_EVENT_PACKET_CRYPTO. This struct provides a summary bit that says whether the operation completed successfully as well as
odp_crypto_op_status_t fields for the
auth_status if a problem was detected.
Random Early Detection (RED) provides a means for input HW to ensure that traffic is treated fairly under conditions of temporary resource overload due to excessive inbound traffic. ODP RED support provides the ability to measure and respond to resource pressure on either pools or queues, and only provides flexibility in how such conditions are to be processed. They can result in input packet drops or backpressure being indicated by transmitting pause frames, depending on the underlying platform capabilities.
odp_cls_capability_t struct returned by the
odp_cls_capability() API has been expanded to cover this support.
odp_time_diff_ns() API permits the delta between two
odp_time_t values to be computed in a single call.
odp_pktio_mtu() API is deprecated and replaced by two new APIs:
odp_pktout_maxlen(). These return the maximum sized packets that are supported for RX and TX processing, respectively, on a given
odp_pktio_mac_addr_set() API is added to allow setting of the MAC address associated with an
odp_pktio_set_op_t field of the
odp_pktio_capability_t returned by the
odp_pktio_capability() API now includes the
mac_addr` field to indicate that this
odp_ptkio_t supports setting its MAC address.
The reserved device name
loop is now extended to
loopX where X == integer (e.g.,
loop2, etc.). For compatibility,
loop is a synonym for
odp_pool_info() API is extended to return the
max_data_addr fields. These provide information about the minimum and maximum application-visible addresses that may be seen in objects allocated from a particular
odp_pool_t. Some applications use this information to enable them to store buffer addresses in compressed format. For example, if the "span" of valid addresses is less than 4GB this allows a 64-bit buffer address to be stored as a 32-bit offset.
Since this is purely informational, ODP implementations are under no constraint as to what addresses may be returned for these fields. 0 and
UNINTPTR_MAX may be used if there are no limits on pool extents.
odp_pool_param_t structure has been expanded to provide more flexibility to support platforms that are able to offer multiple segment sizes within a single pool. This can lead to greater storage efficiency. These are called subparameters and implementations supporting up to 7 of these are accommodated with these extensions.
odp_pool_capability_t structure is expanded to return the number of subparameters supported by this implementation. The application, in turn, specifies its expected packet size and number distribution in the
odp_pool_pkt_subparam_t structure that is part of the
odp_pool_param_t used to define the characteristics of
This is fully compatible with previous packet pool support since ODP implementations are under no obligation to support pool subparameters and these, when present, are advisory in nature. They simply serve to allow the application to better communicate its expected packet distribution within a pool so that the ODP implementation may better optimize storage use in the pool.
Checksum processing support has been formalized with the addition of APIs for determining packet checksum status, controlling packet checksum processing, retrieving partially computed checksums on packets, and computing checksum partial sums for memory areas.
odp_packet_l4_status() are added to allow the results of packet input checksum processing to be queried. These APIs return an
odp_packet_chksum_status_t enum that indicates whether checksum validation processing was performed and if so whether the layer 3 or 4 checksum was found to be valid. This is applicable to both normal packet input as well as those processed via IPsec.
PktIOs output checksum processing is configured as part of the
odp_pktout_config_opt_t struct used as input to
odp_pktio_config() API. These control whether L3 and/or L4 checksums are to be inserted by default as part of packet TX processing.
Individual packets may override these defaults via the new
odp_packet_l4_chksum_insert() APIs. These take precedence over the PktIO default, allowing checksums to be inserted when the PktIO default is to not insert checksums or to suppress checksum insertion if when the PktIO default is to insert checksums.
Two new APIs:
odp_chksum_ones_comp16() are added to assist in application-managed checksum processing. If an implementation has computed a partial checksum as part of the receive processing for an IP fragment, for example, then
odp_packet_ones_comp() can be used to retrieve this computed value, as well as the byte range over which it was computed. The
odp_chksum_ones_comp16() API, by contrast, allows the application to perform a 16-bit ones-complement sum over a range of in-memory bytes. Together these can be used to calculate IPv4, TCP, and UDP checksums.
New packet APIs have been added to streamline packet processing:
odp_packet_free_sp() is the same as
odp_packet_free_multi() except that the application guarantees that all packets come from the same pool.
odp_packet_from_event_multi() allows multiple events to be converted from
odp_packet_t objects in a single call. The caller guarantees that all input
odp_event_t objects are of type
odp_packet_to_event_multi() converts multiple
odp_packet_t objects to corresponding
odp_event_t objects in a single call.
Several changes have been made to the ODP shared memory APIs:
name field used as input to
odp_shm_reserve() is now optional. If specified as
NULL the shared memory area is anonymous and cannot be looked up with
odp_shm_lookup(). There is also no requirement that names be unique. Duplicate names result in indeterminate output from
odp_shm_info_t now includes the
page_size of the shared memory block and it’s (optional) name.
odp_shm_print() API is added to print implementation-defined information associated with the
odp_shm_t to the ODP log for diagnostic purposes.
New queue attributes are introduced to characterize queue behavior as blocking or non-blocking. A blocking queue may stall other threads if a thread is interrupted or suspending during an enqueue or dequeue operation. Nonblocking queues may be either lock free or wait free and provide progress and fairness guarantees to all threads regardless of interruptions or stalls on the part of threads performing queue operations.
odp_nonblocking_t attributes available are returned by the
odp_queue_capability() API for both plain and scheduled queues and are in turn requested as part of the
odp_queue_param_t struct passed to the
odp_queue_create() API. The intent is to allow applications that have realtime response requirements to better express these needs and utilize platform-specific capabilities in this area.
The following changes have been made to the scheduler APIs:
Documentation clarifies that an ordered context may only hold one ordered lock at a time. Results are undefined if a second ordered lock is attempted to be acquired while already holding one.
odp_schedule_order_unlock_lock() API is added to permit an ordered context to switch from one ordered lock to another in a single operation.
odp_timer_capability() API is added to return an
odp_timer_capability_t struct that details platform-specific timer capabilities for application use. The only capability currently defined is
highest_res_ns, which indicates the highest supported resolution supported by a timer. This is the minimum valid value for the
res_ns timer pool parameter.
odp-linux reference implementation adds a new scalable scheduler to the existing default, strict priority, and iquery schedulers. This is enabled by:
The scalable scheduler is designed to offer superior scalability in many core environments, especially on AArch64 platforms.
The following miscellaneous improvements have been made to the
linux-generic reference implementation of ODP.
When compiling with
--enable-abi-compat=no the following additional packet functions are inlined:
The ODP test suite now automatically skips C tests if no C compiler is available.
The odp_pktio_ordered tests are only performed if PCAP is available.
The DEPENDENCIES file has been updated to reflect build/test requirements for running under Red Hat Linux distributions.
PktIO DPDK support has been upgraded to DPDK 17.08.
A verbose option is added to provide more detail on test runs.
Numerous performance improvements have been made that results in significantly better I/O rates. This includes a configuration option to control checksum usage.
CID 1461688: odp_pool_create: Dereference before null check
CID 1396968: buggy check
l2fwd segfaults on api-next with dpdk checksum insertion override
ODP v126.96.36.199 is the final preview release before the official release of Tiger Moth. It introduces new APIs and extensions, as well as bug fixes and functional improvements.
The following new and changed APIs are included in this release:
odp_feature_t type is introduced that defines various feature bits for ODP components. This is used in an expanded
odp_init_t argument to
odp_init_global() to specify which ODP features are unused by the application. For example, if the application knows it will not be making use of crypto features or the classifier, this may permit the ODP implementation to configure itself more efficiently. Results are undefined if an application asserts that it will not be using a feature and it attempts to do so afterwards.
Associated with this new support the
odp_init_param_init() API is added to initialize the
odp_init_t struct to default values.
odp_packet_unshared_len() API is removed. Testing showed that this API was non-essential and conflicted with the goal of implementation efficiency.
odp_print_packet_data() API is added. This permits packet data to be logged along with platform-defined metadata for debugging or diagnostic purposes.
loop_supported attribute of the
odp_pktio_capability_t struct is deprecated since this is redundant. The
enable_loop field of the
odp_pktio_config_t struct (which is returned as part of the
odp_packet_capability_t struct) is the proper way to determine whether a PktIO supports loopback mode.
The documentation for the
odp_sys_huge_page_size() API is updated to clarify that a 0 return code indicates that huge pages are not supported by this platform.
odp_sys_huge_page_size_all() API is added to return all huge page sizes supported by this platform.
The documentation for the various parameters contained in the
odp_timer_pool_param_t struct are expanded and clarified.
The default packet headroom in
odp-linux has been increased from 66 to 128 bytes for better compatibility with
odp-linux reference implementation now fully supports zero-copy packet references. Previously these APIs were implemented via packet copies, which while functionally correct, were not how these APIs are intended to operate.
configure option is added to allow DPDK PktIO devices to avoid data copies, leading to improved performance.
DPDK PktIO now makes use of RX and TX IP/UDP/TCP checksum offload.
odp-linux reference implementation, shared memory is now backed to
/dev/shm rather than
/tmp for better reliability and robustness. This may be overridden as part of ODP build-time customization if desired.
PktIO IPC support has received improvements in both performance and reliability and is now suitable for development use.
The thread ID is now used to create unique vdev MAC addresses to avoid conflicts with multiple ODP processes running on the same host system.
include/odp/drv directory and related files have been removed. Driver support is moved to a follow-on ODP release, so removing these files avoids confusion as they are still incomplete.
Since some build environments do not supply autoconf-archive, this dependency is removed.
The ODP DPDK Packet I/O support has been upgraded to work with the DPDK 17.08 release.
ODP use of OpenSSL for crypto processing has been upgraded to allow use of OpenSSL 1.1.x.
The ODP build system has been overhauled to support more comprehensive and efficient automated testing under Travis CI. Greater use of Autoconf is now made to control ODP configuration and build options, permitting greater environmental flexibility. This includes an expanded range of test coverage, including cross-compilation support for ARMv8, MIPS,and Power architectures, as well as testing under the latest levels of GCC and clang.
ARMv8 generic timer support is now included
Improved time efficiency and accuracy by using native ARMv8 time/clock instructions.
test directory has been reorganized and streamlined. Platform-specific tests are moved from
platform/linux-generic/test/. As a result, the
test/common_plat directory is deleted so that
test/performance, etc. are now used for all platform-independent tests.
ipfragreass example program has been added to demonstrate IPv4 fragment reassembly.
odp_generator example program now uses packet references for improved performance in UDP and ICMP traffic. The program also now makes use of HW checksum offload support, when available.
The ODP Users Guide has clarified usage information about the ODP time APIs for better portability.
A section has been added to the ODP Users Guide on API specification principles. This clarifies expected behavior of ODP applications and implementations and makes explicit what the specification means by "undefined behavior".
All Doxygen used in ODP is upgraded to conform to the stricter documentation requirements introduced by Doxygen 1.8.13.
check-odp: valgrind generates "No rule to make target"
test odp_l2fwd_run.sh contains todo items
Helper/test/process fails on a single core system
Remove redundant loop_support parameter in pktio capability
Make file deps failure
IP headers checksum functions are incorrect
odp_traffic_mngr example is broken
pktio_ipc_run test can fail due to segfault
User guide error (packet diagram fix)
api-next out of tree build broken
Cross compile broken for ARMv8
IPC pktio test fails with taskset -c 1-2
Test case for classification enable
Memory allocation checks (in traffic manager)
Adding --enable-helper-linux configure flag breaks build
Doxygen warnings on helper header files
codecov: _odp_packet_cmp_data is not covered
setup_pktio_entry missing unlock
odp_cpu_hz() does not work on all Linux distros
Missing doxygen detected by Travis
'num_queues' isn’t ignored when "classifier_enable" is enabled
Validation tests cannot be disabled after commit b4d17b1
wrong openssl_lock pointer type
Cannot run l2fwd application on Cavium ThunderX platform
ODP v188.8.131.52 continues the preview of Tiger Moth, introducing new APIs and extensions, as well as numerous bug fixes and functional improvements.
To permit smoother evolution of the ODP API specification, a deprecation framework is introduced to permit controlled deprecation.
When an ODP API or defined struct parameter is deprecated, ODP validation tests will be updated to no longer use that API and instead use the replacement API. By default, attempts to compile with the older API/feature will fail and applications wishing to move to the new ODP release should be updated to use the replacement API. To permit evaluation of new ODP releases in advance of such updating, however, ODP supports the
--enable-deprecated, which makes the obsolete APIs visible again.
This feature will be used on a case-by-case basis and documented in the release notes for each release that introduces replacement API(s). In general the deprecated forms will not be maintained for more than a single release cycle. After that they will no longer be present in the API specification and the replacement forms must be used to compile with that level of the API specification.
A number of new and refined APIs are introduced in crypto, packet parsing, and queue configuration:
The ODP crypto APIs receive several enhancements in this release:
Additional enumerations added for HMAC-SHA-1 and HMAC-SHA-512 authentication.
The following cipher/authentication algorithms have been deprecated in favor of newer replacement algorithms:
ODP_CIPHER_ALG_AES128_CBC is replaced by
ODP_CIPHER_ALG_AES128_GCM is replaced by
ODP_AUTH_ALG_MD5_96 is replaced by
ODP_AUTH_ALG_SHA256_128 is replaced by
ODP_AUTH_ALG_AES128_GCM1 is replaced by `ODP_AUTH_ALG_AES_GCM
odp_crypto_op_params_t is deprecated in favor of
odp_crypto_op_param_t for consistency with other ODP
odp_crypto_session_param_t adds the field
auth_digest_len to permit specification of the authentication digest length to be used for this session. The
odp_crypto_auth_capability() API returns the list of supported digest lengths.
odp_crypto_op_param_t struct adds an optional pointer and length for AAD information. This allows applications to specify AAD information from the list of supported lengths provided by
odp_crypto_data_range_t type is deprecated and renamed to
odp_packet_data_range_t since it can be used to specify ranges for other than crypto purposes.
Applications may now specify the maximum packet layer of interest. For example, a router may not care about anything beyond ISO Layer 3 in packets. This can be used by ODP implementations to control the depth of packet parsing needed by the application and may allow greater efficiency, especially in software implementations.
This is controlled by a new
odp_pktio_parser_layer_t enum that is part of the new
odp_pktio_parser_config_t struct that is added to the
odp_pktio_config_t struct used by the
odp_pktio_config() API. The supported parser layers are also returned in this struct as part of the
odp_pktio_capability_t struct returned by the
odp_queue_capability_t struct returned by the
odp_queue_capability() API is enhanced as follows: * The
max_queues field is now defined to return the maximum number of event queues of any type. * New sub-structs (
sched) are added that detail the
max_num of queues of that type supported as well as the new field
max_size that specifies the maximum number of elements that each queue of this type can store. A value of zero for
max_num indicates no fixed limit.
In addition, the
odp_queue_param_t passed to
odp_queue_create() now adds a
size field to allow applications to specify the minimum number of events that this queue should be able to store. A value of zero specified requests that the implementation default limits be used.
The ODP examples have been updated to show this configuration, for example, the
l2fwd_simple example does Layer 2 forwarding and hence doesn’t need packets to be parsed beyond Layer 2. So prior to opening an interface via
odp_pktio_open() the configuration processing now includes code of the form:
odp_pktio_config_init(&config); config.parser.layer = ODP_PKTIO_PARSER_LAYER_L2; odp_pktio_config(pktio, &config);
This serves as a hint to the ODP implementation that parsing beyond Layer 2 is not required on this interface.
odp_time_to_u64() API is removed without deprecation in this release. This is redundant since the
odp_time_to_ns() API already returns a
uint64_t value for
odp_time_t and can be used for the same purpose.
For debugging / diagnostic use, the
odp_sys_info_print() API is added. This prints implementation defined information about the system and ODP environment and may contain information about CPUs, memory, and other hardware configuration.
ODP helper functions fall into two categories: system-independent, and those that rely on Linux. For backwards compatibility, these have been reorganized into a new
helper/linux directory that is installed only when the
configure option is specified.
The default scheduler is streamlined by using precomputed weight tables and separated priority queues for different scheduling groups. This improves both latency and scalability.
Also included in this change, the default number of scheduling groups is lowered from 256 to 32, which experience has shown is sufficient for most applications.
The scheduler fairness checks are also enhanced to avoid low priority event starvation if
CONFIG_BURST_SIZE is set to 1.
Errors in implicit packet copy operations performed during crypto operation are now handled properly.
odp_crypto_session_create() fails, proper cleanup is now performed to avoid memory leaks.
The auth digest len is now used when calculating HMACs, and full-length digests are now added to supported SHA capabilities.
Numerous crypto functions have been rewritten to use the OpenSSL
EVP_ functions for improved efficiency and accuracy.
The packet parser now recognizes ICMPv6 packets in addition to ICMPv4.
socket drivers add additional initializations to avoid valgrind warnings.
dpdk driver is upgraded to DPDK 17.02 and is now also built with the
--whole-archive option, which means that PMDs no longer need to be linked individually but are automatically included in builds that make use of DPDK pktio support.
The ODP Time APIs are now more efficiently handled by replacing the previous Linux timespec with simple nanoseconds (ns) and using native hardware time counters in x86 and ARM environments, when available.
odp-linuxTraffic Manager Improvements
Weighting is now handled properly when weight=1 is specified. Previously this caused an overflow that distorted packet priorities.
ARMv8 compilation now works properly when using the clang compiler.
odp_scheduling performance test now handles dequeueing from potentially concurrent queues properly.
The traffic manager and time tests are now invoked via scripts to better account for load-sensitivity when running regressions in CI environments.
The l2fwd test now supports an additional parameter to specify the number of scheduling groups to use in the test.
odp_sched_latency performance test now handles queue draining in a robust manner.
The crypto validation tests now properly check and validate message digests, and include negative tests to verify that incorrect digests are properly flagged. Note that this may expose omissions in other ODP implementations that had previously passed this validation test.
The crypto validation test now adds an explicit test for the NULL cipher, as well as HMAC-SHA-1 and HMAC-SHA-512.
The pktio_ipc test now properly cleans up only its own temp files.
odp_generator example now properly honors stop criteria based on number of packets sent.
odp_generator example now properly flushes output messages
odp_generator example now supports port configuration.
example/generator/odp_generator.c contains todo items
Error handling issues (CHECKED_RETURN)
Unchecked return in pool.c
Unchecked return in mmap_unmap_sock()
ODP_STATIC_ASSERT() fails when used by C++
odp_pktio_ordered_run.sh: line 34: integer expression expected
load sensitive tests fail on CI
odp_crypto_operation() routine does not work with multi-segment packets.
Packet validation test fails if implementation does not limit packet length
odph_iplookup_table_put_value() uses overlapping pointer addresses
Miss to call unlock if there are some errors happen in loopback_send() function
odp_packet_seg_t fails ABI compatibility between odp-linux and odp-dpdk
Compilation fails using clang 4.0.0
doxygen errors and travis does not catch them
TM validation test does find traffic_mngr_main
odp_rwlock_read_trylock() fails if lock is already held for read access
Packet order lost when enqueuing to pktout queue
AES-GCM returns 'valid' tag when checking invalid tag
Various failures if CONFIG_PACKET_MAX_SEGS is set to 1
Travis: time main test out of boundaries
Compilation failures using GCC 7 series
Socket pktio recv fails on large number of packet
ODP v184.108.40.206 represents another preview of the Tiger Moth release series and introduces new APIs and extensions.
Packet references are a lightweight mechanism to allow applications to create and manipulate different "views" of packets. These views consist of shared bytes common to all references created on the same
odp_packet_t, possibly prefixed with unique headers that are private to each reference. Five new APIs are introduced to enable this feature:
Static references allow a single packet to have multiple "aliases", all of which must be treated as read only. This is useful, for example, to retain a reference to a packet being transmitted to support retransmit processing, since the actual storage represented by a packet is not released until all references to it have been freed via
Dynamic references differ from static references is that they permit the individual
odp_packet_t handles to have unique prefixes created via
odp_packet_extend_head() calls. This can be used, for example, to support multicasting packets to different destinations by creating packets that consist of unique headers followed by a common shared payload.
odp_packet_ref(), but creates a dynamic reference by prepending a preexisting header onto another packet.
Returns a boolean indicator of whether a given
odp_packet_t has other references that share bytes with this packet.
Returns the number of unshared bytes accessible through a given
odp_packet_t handle. These are the only bytes that should be changed. All other bytes in the packet should be treated as read only as they are shared with other
Note that while the packet reference APIs and associated validation tests are present in this release, the
odp-linux reference implementation currently implements references via packet copying. A zero-copy implementation of packet references is planned to be part of the ODP v220.127.116.11 release.
ODP has supported an Application Binary Interface (ABI) that permits applications to be generated that are binary portable between select ODP implementations. This is now formalized with the addition of structures that permit ABI specifications to be defined that are shared between ODP implementations. ODP provides a default ABI specification that is the same as was provided before. This change means that additional ABIs may be defined that are optimized to collections of platforms that share an Instruction Set Architecture (ISA) and subscribe to them.
ABI specifications live in the
Adding to the range of scheduler implementations supported by
odp-linux, a new scheduler, the instance query scheduler, is available by specifying
This scheduler uses sparse bitmaps and offers scalability advantages when dealing with large numbers of schedule queues. It otherwise offers performance comparable to the default ODP scheduler implementation.
Helpers have been reorganized to provide better independence from odp-linux and to make them more useful with other ODP implementations. These reorganizations are functionally transparent to ODP users but should simplifying packaging and porting to other ODP implementations. This includes adding the "umbrella" include file
odph_api.h which can be used to include all helper API definitions in an application.
odp-linux reference implementation now supports Pattern Matching Rules (PMRs) for IPv6 addresses, inner and outer VLAN IDs, and inner and outer Ethernet types.
odp-linux reference implementation now offers improved inlining support for ODP APIs for applications compiled against ODP configured with the
--enable-abi-compat=no option, meaning they wish to forego binary compatibility in exchange for improved performance. These applications remain source portable to other ODP implementations.
odp_cpu_pause()for ARM processors
A native implementation of the
odp_cpu_pause() API has been added for ARMv7 and ARMv8 processors.
odp_packet_bench microbenchmark application has been added to the test performance directory to allow implementations to measure and calibrate the performance of individual ODP packet APIs.
odp_pktio_ordered application has been added to the test performance directory to provide stress-testing of packet ordering features of ODP.
In addition to expanded documentation related to the new packet reference APIs, a section on applicatin portability has been added that discusses the types of portability offered by ODP and the tradeoffs that application writers should consider when using ODP.
hello.c application can now run properly if Core 0 is not available (any available core will be used).
Provide proper return code checking within _ishm.c
Remove unused variables in iplookuptable helper routine.
Avoid memory leaks on error paths in the cuckoo table helper functions.
Fixes a race condition in shared memory allocation.
Provide proper fallback for shared memory when hugepages are not available.
Avoid null pointer dereference in the iplookuptable helper routine.
Missing doxygen documentation for helper table functions are added.
ODP v18.104.22.168 represents the initial preview of the Tiger Moth release series and as such introduces new APIs and extensions that will be built on as this release matures.
Many ODP APIs take parameter structs of type
odp_xxx_param_t. The crypto APIs, for historical reasons, failed to follow this convention, using instead structs of type
odp_crypto_params_t, etc. These pluralized names are now deprecated and their singular forms are introduced as the preferred names for these structs. The old names are retained (for now) to ease migration, however applications should convert to the new forms at their convenience as these deprecated forms will be removed in the final Tiger Moth release.
The changes introduced for this include:
To provide a more flexible programming for handling all possible key/digest/iv length combinations, the enums used for crypto specification are split to decouple lengths from algorithm names. The only exception is the SHA-2 family of hash routines, which have well-established naming conventions that incorporate digest lengths (SHA-224, SHA-256, etc.)
Changes included with this restructure include:
odp_crypto_capability_t structure returned by the
odp_crypto_capability() API contains streamlined
odp_crypto_cipher_capability() API is added to return detailed information about available cipher capabilities.
odp_crypto_auth_capability() API is added to return detailed information about available authentication capabilities.
For completeness the
odp_crypto_session_param_init() API is provided to enable platform-independent initialization of the
odp_crypto_session_param_t structure used as input to
ODP_BITFIELD_ORDER define is added to the ODP specification to parallel the existing
ODP_BYTEFIELD_ORDER define. This will be set to the value
ODP_LITTLE_ENDIAN_BITFIELD. This also addresses Bug 2402, however since fixing this required a small API change this was deferred until an API release boundary.
The use of name arguments to the various resource creation APIs has been standardized and documentation improved to clarify that unique names are not required and that these may be specified as NULL for anonymous resource creation. When non-unique resource names are used, it is unspecified which of these are returned by a corresponding lookup API.
odp_pool_param_t structure adds the new field
max_len to be used in packet pools to advise the implementation of the maximum sized packet that the application will allocate with a single
odp_packet_alloc() call. This may enable storage pool optimization.
API documentation for
odp_packet_copy_from_pkt() is clarified to specify that the source and destination packets supplied to these APIs should not refer to the same packet.
API documentation for
odp_packet_alloc() is clarified to specify that the supplied length for requested packet allocation should be greater than zero.
The APIs provided for working with random data have been revised. The third parameter to
odp_random_data() is now of type
odp_random_kind_t, which is an
enum that is used to specify the quality of random data required. The kinds of random data defined are:
No specific quality guarantees. This is assumed to be pseudo-random data generated by software where speed of generation is more important than the quality of the results.This is the lowest kind of random.
Random data suitable for use in cryptographic operations.
True random data generated from a hardware entropy source. This is the highest kind of random.
odp_random_max_kind() API is provided that returns the highest kind of data available on this implementation. Requests for higher kinds than can be supplied will fail.
For testing purposes it is often desirable to generate "random" sequences that are repeatable. To support this use case, the
odp_random_test_data() API is introduced. This always returns BASIC random data but uses a user-supplied 64-byte seed parameter that is update for each call and can be used to repeat the same sequence as needed.
odp_shm_reserve() API adds two new additional flags to support external memory.
ODP_SHM_SINGLE_VA guarantees that all ODP threads sharing this memory will see the block at the same virtual address regardless of whether threads are implemented as pthreads or processes and when
fork() calls are made to create them.
ODP_SHM_EXPORT allows the memory block to be visible to other ODP instances. Other ODP instances can retrieve this block via the new
The relationship between classification and queue configuration in the
odp_pktin_queue_param_t structure is clarified to emphasize that classification subsumes queue-based I/O processing. This is an API documentation change only.
New helper APIs are introduced for IP table lookup support for longest prefix matching as well as cuckoo hash tables. These are designed to provide analogs to functions available in DPDK to assist applications being ported to ODP.
The odp-linux reference implementation includes a number of improvements to make it more suitable for production use on platforms that rely on software implementations of key ODP APIs.
Storage pools used for buffers and packets are now implemented via lockless ring structures that support burst mode read/writes to local caches for significantly improved multi-core scalability
The layout of the structs used to support buffers and packets has been optimized to reduce cache footprint and minimize path lengths in packet manipulation.
The implementation of ordered queues has been streamlined and made more scaleable in multi-core systems.
The more advance segmentation capabilities present in the new ODP packet implementation are used to streamline the implementation of APIs like
odp_packet_concat() and the head/tail extend/trunc APIs.
A number of "todos" were removed from the packet validation test suite.
The CPU affinity is now correctly read from the cpuset.
The PktIO validation test no longer uses invalid MAC addresses.
The TCP checksum is now properly calculated for generated packets.
The odp-linux reference implementation makes use of the OpenSSL library to support the
odp_random_xxx() APIs and some crypto operations. To support OpenSSL versions prior to 1.1.0, which are not thread safe, support is added for OpenSSL locking callbacks that use ODP ticketlocks to provide OpenSSL thread safety.
Make check fails on a single core VM in the process mode helper test.
ODP v22.214.171.124 has no API changes from previous v1.11.0 Monarch LTS. Version is increased in current development release to make room for Monarch updates numbers.
Support is added to enable ODP applications to be binary compatible across different implementations of ODP sharing the same Instruction Set Architecture (ISA). This support introduces a new
no abi disable option
This is the default and specifies that the ODP library is to be built to support ABI compatibility mode. In this mode ODP APIs are never inlined. ABI compatibility ensures maximum application portability in cloud environments.
Specify this option to enable the inlining of ODP APIs. This may result in improved performance at the cost of ABI compatibility and is suitable for applications running in embedded environments.
Note that ODP applications retain source code portability between ODP implementations regardless of the ABI mode chosen. To move to a different ODP application running on a different ISA, code need simply be recompiled against that target ODP implementation.
The ODP classifier adds support for recognizing Stream Control Transmission Protocol (SCTP) packets. The APIs for this were previously not implemented.
ODP no longer has a dependency on Linux headers. This will help make the odp-linux reference implementation more easily portable to non-Linux environments.
The odp-linux implementation has been made independent of the helper library to avoid circular dependency issues with packaging. Helper functions may use ODP APIs, however ODP implementations should not use helper functions.
test directory has been reorganized to better support a unified approach to ODP component testing. API tests now live in
test/common_plat/validation/api instead of the former
test/validation. With this change performance and validation tests, as well as common and platform-specific tests can all be part of a unified test hierarchy.
The resulting test tree now looks like:
test ├── common_plat │ ├── common │ ├── m4 │ ├── miscellaneous │ ├── performance │ └── validation │ └── api │ ├── atomic │ ├── barrier │ ├── buffer │ ├── classification │ ├── cpumask │ ├── crypto │ ├── errno │ ├── hash │ ├── init │ ├── lock │ ├── packet │ ├── pktio │ ├── pool │ ├── queue │ ├── random │ ├── scheduler │ ├── shmem │ ├── std_clib │ ├── system │ ├── thread │ ├── time │ ├── timer │ └── traffic_mngr ├── linux-generic │ ├── m4 │ ├── mmap_vlan_ins │ ├── performance │ ├── pktio_ipc │ ├── ring │ └── validation │ └── api │ ├── pktio │ └── shmem └── m4
The maximum number of pools that may be created in the odp-linux reference implementation has been raised from 16 to 64.
The DPDK pktio support in odp-linux has been upgraded to work with DPDK 16.07. A number of miscellaneous fixes and performance improvements in this support are also present.
Packet I/O interfaces operating in TAP mode now can feed packets to the ODP classifier the same as other pktio modes can do.
The scheduler and pktio components have been reworked to use burst-mode buffer allocation/deallocation, yielding a measurable performance gain in almost all cases.
ODP queues internally now attempt to use burst-mode enq/deq operations to accelerate performance where applicable.
The ODP scheduler has been enhanced to use ring-based priority queues, resulting in significantly better scalability in many core environments.
ODP now supports the Travis framework needed to trigger CI automation in conjunction with GitHub. This support is considered experimental for now.
A new example application is provided that illustrates use of ODP for simple Layer 3 forwarding across multiple interfaces.
ODP now generates "pure" (implementation independent) doxygen documentation in addition to the specific documentation for the odp-linux implementation. The pure version is applicable to any ODP implementation as it simply describes the ODP API specification. Implementation-specific versions of this documentation describe both the APIs as well as specifics concerning typedefs, enums, etc. This should help clarify what aspects of ODP are platform independent.
The definition of an ODP thread in odp-linux has been clarified to specify that for this implementation ODP threads are Linux pthreads or Linux processes. Currently threads as processes is considered experimental in odp-linux.
Numerous refinements have been incorporated to make validation tests more robust in API coverage as well as correcting corner cases in the implementation of many ODP APIs. Notable fixes include:
ODP_TIMEOUT_INVALID is now defined consistently with other pool elements so the restriction on using Pool 0 as a timer pool is removed.
odp_packet_copy() API now correctly handles user areas between pools that are configured with different sized per-packet user area definitions.
Corrects the implementation of AES GCM decryption in the ODP crypto API.
The Timer validation test fails sporadically in environments with high core counts.
ODP v1.11.0 is the base tag and branch for the Monarch LTS (Long Term Support) release of ODP.
As part of the final Monarch LTS API set, a minor change to the
odp_tm_egress_t struct was made to better reflect the capabilities of SoC platforms targeting Monarch support. This change adds the boolean
egress_fcn_supported that indicates whether TM systems support an egress function. In addition, each TM system is now associated with a PktIO rather than a PktOUT queue. This struct is input to the
odp_tm_create() API and output from the
Building on the modular scheduler framework introduced in v126.96.36.199, An alternate Strict Priority (SP) scheduler is now available. The SP scheduler is selected when ODP is configured with the
The SP scheduler is designed to favor low-latency processing of high priority work at the expense of throughput. This is considered experimental code at this point and should be treated as such by those wishing to use it.
The scheduler now properly cleans up PktIOs operating in SCHEDULE mode following
odp_pktio_close() calls. This resolves Bug 2301.
The odph_chksum() helper routine is now C++ conformant. This resolves Bug 2302.
odp_packet_copy() API now copies the packet user area as part of its processing and the packet validation test has been enhanced to verify that this is done properly. This resolves Bug 2310.
The internal definition of
ODP_TIMEOUT_INVALID was changed in
odp-linux to enable the use of Pool 0 as a timeout pool. This resolves Bug 2316.
To facilitate API tracking,
libodphelper.so has been split out from
To avoid circular dependencies that cause issues when packaging ODP as a shared library, the
odp-linux implementation no longer makes and use of the ODP helper library.
Performance enhancements in the
odp-linux reference implementation have been made in PktIO routines and the packet classifier.
General cleanup throughout has resulted in adding proper termination calls to ODP examples as well as miscellaneous Doxygen corrections. Additionally, the
odp-linux code base has been cleaned up by removing "To Dos" as well as extraneous
#includes for extra C headers and dependencies on linux headers for improved portability.
The Queue validation test now properly exercises the
odp_queue_capability() API and attempts to create the maximum reported number of supported queues.
The platform-specific tests have been moved from
test/platform/linux-generc so that all tests are now found under a single
examples directory has been cleaned up and numerous corrections to existing examples have been made to add proper termination paths, etc. In addition, a new "hello world" example (
odp_hello.c) has been added that illustrates the basic startup and shutdown steps needed for all ODP applications. As a simple stand-alone application is is also a good initial build tests for new ODP implementations.
The Doxygen API guide has been split so that both a "pure" version that is platform-independent and retains unresolved ODP abstract types as well as an implementation-specific version (for
odp-linux) that shows platform-specific
The ODP Implementer’s Guide has been enhanced with a section on implementation considerations, including issues surrounding the definition of Application Binary Interfaces (ABIs).
The timer validation tests have seen occasional failures when run on systems containing a larger number of CPUs (typically 24 or more).
ODP cannot be compiled using the -m32 option to generate 32-bit code on a 64-bit platform when using GCC 4.9 (the default GCC found in Ubuntu 15.04). This is due to a known bug in GCC 4.9 and is closed as a permanent restriction.
Doxygen generates a warning for
ODP_BIG_ENDIAN_BITFIELD when compiling on little endian systems. A similar warning for
ODP_LITTLE_ENDIAN_BITFIELD is issued when compiling on big endian systems.
The traffic manager cannot be used with two PktIOs simultaneously. That is,
odp-linux does not properly support applications that create more than one TM system. This is a bug in the
odp-linux implementation that will be corrected. It is not a limit of the architecture or other ODP implementations.
ODP v188.8.131.52 is a maintenance release and as such introduces no new APIs or other changes that might result in application source incompatibility with v184.108.40.206.
The ODP scheduler included as part of the
odp-linux reference implementation has been modularized to support a "pluggable" architecture that will permit the easy introduction of alternate scheduler implementations that permit both easy experimentation as well as alternate scheduling models that are tuned to specific application workloads (i.e., a strict priority scheduler for latency-sensitive workloads). Included in this release is the just the foundation and alternate schedulers are expected to be added in future ODP releases.
While the platform name within the ODP git repository remains
linux-generic, the package name of this reference implementation has been changed to
odp-linux to better reflect that this is an implementation of ODP that is dependent only on the Linux kernel and is intended to be runnable on any Linux distribution. All ODP documentation has been changed to reflect this name change.
The current helper APIs for linux pthreads (
odph_linux_pthread_create(), etc.) and processes (
odph_linux_process_fork(), etc.) have been augmented with a new "agnostic" set of helper routines for general ODP thread management.
odph_odpthreads_join() are now the preferred helpers to use for creating and joining threads that may be implemented as either pthreads or Linux processes. Although currently ODP threads in Monarch are assumed to share the same (single) address space, this change is to permit evolution towards multi-address space (i.e., process oriented) threading models without requiring application and/or validation test changes.
The various ODP examples and validation tests that make use of threading have all been changed to new these new helper APIs.
As part of the new "agnostic" threading support, two additional helper APIs (
odph_odpthread_setaffinity()) have been added to permit easy management of thread affinity.
The CUnit framework used by the ODP validation suite has been changed to set the affinity of the initial thread to the first available control CPU. This avoids inconsistencies in a number of timing tests.
As a maintenance release, numerous bugs have been fixed in the
odp-linux reference implementation in areas such as platform support (particularly on 32-bit systems), compiler support (clang), C++ compatibility, and other cleanup items. Highlights include:
configure utility now correctly identifies and sets variables needed for proper compilation on 32-bit x86 systems. In addition the Traffic Manager now executes properly on 32-bit systems.
odp_timer_t abstract type is now strongly typed for consistency with other ODP types, and the
odp_timer_pool_to_u64() APIs are now properly implemented. In addition, an issue with compiling the timer routines on 32-bit systems using clang was resolved.
The default scheduler has been improved to avoid starvation and increase fairness when running with a limited number of threads or CPUs.
-mcx16 compiler option is now properly identified and used to support 128-bit atomics on systems that support these operations. This leads to increased efficiency in the implementation of both timers as well as other atomic operations.
In systems that support multiple hugepage sizes, the default hugepage size is now reported properly by the
Numerous issues identified by the Coverity scan tool have been cleaned up and resolved, leading to improved robustness in the
odp-linux reference implementation. In addition, several memory leaks identified by the valgrind tool have been fixed.
The Traffic Manager validation test has certain sub-tests that require a minimum of 2 CPUs to operate properly. These tests are now skipped with a warning when run on uniprocessor systems.
Performance enhancements in the
odp-linux reference implementation have been made in packet processing, classification, and pktio, leading to increased observed throughput and packet handling rates.
l2fwd_simple example has been added that does simple Layer 2 forwarding.
To get better formatting capabilities and future extensibility the ODP user documentation system has been switched from AsciiDoc to AsciiDoctor. There are a handful of notable changes as a result of this. Those submitting documentation patches should be aware of the following:
monospace font is now indicated by demarking the text with backticks (`) rather than plus signs (+).
Callouts are now indicated by C++-style double slash comments (// <1>) instead of C-style comments (/* <1> */). So, for example writing
[source,c] ---- ...some great code deserving a callout // <1> ---- <1> This is an example of a callout
...some great code deserving a callout (1)
This is an example of a callout
Note that this change only affects user documentation. Code documentation still uses Doxygen.
The images in the user guide have been trimmed to fit the page better. In addition, many new images have been added to better illustrate ODP API usage.
The ODP User’s Guide has been updated to include new sections documenting the use of the full Monarch packet APIs as well as the Timer APIs. In addition, the crypto, classification, and traffic manager API documentation has been improved to cover all of the latest Monarch features, and an introductory section on overall ODP packet flow architecture has been added.
The ODP Helper library now has its own document that describes these additional functions of use to ODP applications and validation tests.
The Process Guide, FAQ, CONTRIBUTING, and CHANGELOG files have been reworked for the new AsciiDoctor structure, leading to a more uniform method of presenting ODP support documentation. In addition, an option for generating both a "pure" ODP API document that defines the APIs in their implementation-independent forms as well as their implementation-dependent forms for the
odp-linux reference implementation is now available.
Move to capabilite structures: The classification, crypto, pktio, pool, queue, shared memory, and traffic manager areas all add new APIs to query implementation-specific capability limits for enhanced program portability.
Addtion of a structure that defines system level classification capability
Addition of range PMRs to complement the existing value PMRs to permit application to specifiy classification rules over a range of values.
Provides a way to get the available cipher and authentication algorithms.
Adds the ODP_STATIC_ASSERT() API to permit compile-time assertions.
Rename and add new packet copy and move functionality.
Add packet prefetch API for performance optimization.
New APIs to permit packet headroom and tailroom to be extended to additional segments or truncated as needed.
Align an area of packet data (e.g. IP/UDP headers) so that it is contiguous in memory and starts in a specified minimum alignment.
Add packet time stamp metadata support.
Add packet input index metadata support to allow application context to be anchored from the associated interface that received the packet.
Deprecation and removal of segment-level buffer APIs that are now superseded by additional packet-level manipulation APIs discussed above.
Replaced config definition for maximum pktio entries with maximum packet IO index call
Added the classify_enable bit to the odp_pktin_queue_param_t that allows applications to explicity control which pktin queues are subject to full classifier support.
Addition of pktin configuration options to control packet timestamping and checksum validation processing.
Addition of pktout configuration options to control packet checksum offload processing.
Add the ability to query (via capabilties) whether a pktio interface supports operating in loopback mode and if so to enable/disable this mode of operation.
Round out the polling APIs be adding the ability to receive packets in poll mode from one or more pktin queues with timeout.
Add the ability to set the pktio interface index for ease of setting up application context areas associated with each pktio.
Add the ability to inqure the timestamp resolution associated with pktio interfaces that support packet timestamping.
Add a queue capability API to query system-wide queue capabilities and limits.
Extend queue context with an explicit data length parameter that serves as a performance hint for the amount of context to prefetch when the scheduler selects an event from a scheduled queue.
Add the ability to query information about a defined scheduler group
Add a capability inquiry API to obtain implementation shared memory limits.
Add a capability inquiry API to obtain implementation traffic mngr limits.
Split TOS marking into two separate calls.
Add new APIs to support VLAN, ECN, and drop precedence marking.
Add destroy APIs for shapers, scheduler profiles, threshholds, WRED profiles, TM queues, and TM nodes for symmetry and completeness to allow applictions to terminate gracefully without resource leakage.
Add the ability to disconnect TM queues from their fanouts.
Add TM node contexts to permit applications to anchor user context areas to TM nodes.
Add the ability to query info about TM node fanins and TM queues.
Deprecate and remove the odp_tm_periodic_update() API
More complete and correct checksum implementation
A focus has been made on improving the packaging support of ODP for distributions such as Debian.
Improved creation of source tar ball and packaging scenarios moved to https://git.linaro.org/lng/odp-packaging.git
Renamed libodp to libodp-linux
Removed all internal functions from leaking through the libodp-linux
Time and Traffic Manager validation tests sometimes fail on heavy loaded systems. This behaviour depends on time accuracy measurements, which depends on system load.