malloc: add option --match-allocations

SPDK uses the rte_mem_event_callback_register API to
create RDMA memory regions (MRs) for newly allocated regions
of memory. This is used in both the SPDK NVMe-oF target
and the NVMe-oF host driver.

DPDK creates internal malloc_elem structures for these
allocated regions. As users malloc and free memory, DPDK
will sometimes merge malloc_elems that originated from
different allocations that were notified through the
registered mem_event callback routine. This results
in subsequent allocations that can span across multiple
RDMA MRs. This requires SPDK to check each DPDK buffer to
see if it crosses an MR boundary, and if so, would have to
add considerable logic and complexity to describe that
buffer before it can be accessed by the RNIC. It is somewhat
analagous to rte_malloc returning a buffer that is not
IOVA-contiguous.

As a malloc_elem gets split and some of these elements
get freed, it can also result in DPDK sending an
RTE_MEM_EVENT_FREE notification for a subset of the
original RTE_MEM_EVENT_ALLOC notification. This is also
problematic for RDMA memory regions, since unregistering
the memory region is all-or-nothing. It is not possible
to unregister part of a memory region.

To support these types of applications, this patch adds
a new --match-allocations EAL init flag. When this
flag is specified, malloc elements from different
hugepage allocations will never be merged. Memory will
also only be freed back to the system (with the requisite
memory event callback) exactly as it was originally
allocated.

Since part of this patch is extending the size of struct
malloc_elem, we also fix up the malloc autotests so they
do not assume its size exactly fits in one cacheline.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>

diff --git a/doc/guides/linux_gsg/linux_eal_parameters.rst b/doc/guides/linux_gsg/linux_eal_parameters.rst
index 28aebfbda2093968d441164d3ff98df1f3d6103f..c63f0f49a0e5b49c63e2fcb26b9e81e2b50e505d 100644 (file)
--- a/doc/guides/linux_gsg/linux_eal_parameters.rst
+++ b/doc/guides/linux_gsg/linux_eal_parameters.rst
@@ -90,6 +90,10 @@ Memory-related options
     Unlink hugepage files after creating them (implies no secondary process
     support).
 
+*   ``--match-allocations``
+
+    Free hugepages back to system exactly as they were originally allocated.
+
 Other options
 ~~~~~~~~~~~~~
 

diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
index 8b5d050c76cc14f7e4098d18d04580c7b8169ebe..19b470e278aad7b11ccbed6c732f9c948ab06f45 100644 (file)
--- a/doc/guides/prog_guide/env_abstraction_layer.rst
+++ b/doc/guides/prog_guide/env_abstraction_layer.rst
@@ -169,6 +169,20 @@ not allow acquiring or releasing hugepages from the system at runtime.
 If neither ``-m`` nor ``--socket-mem`` were specified, the entire available
 hugepage memory will be preallocated.
 
++ Hugepage allocation matching
+
+This behavior is enabled by specifying the ``--match-allocations`` command-line
+switch to the EAL. This switch is Linux-only and not supported with
+``--legacy-mem`` nor ``--no-huge``.
+
+Some applications using memory event callbacks may require that hugepages be
+freed exactly as they were allocated. These applications may also require
+that any allocation from the malloc heap not span across allocations
+associated with two different memory event callbacks. Hugepage allocation
+matching can be used by these types of applications to satisfy both of these
+requirements. This can result in some increased memory usage which is
+very dependent on the memory allocation patterns of the application.
+
 + 32-bit support
 
 Additional restrictions are present when running in 32-bit mode. In dynamic

diff --git a/doc/guides/rel_notes/release_19_02.rst b/doc/guides/rel_notes/release_19_02.rst
index 069f429a7ba78b3f2d50ab2a0b370da5d80c8ef0..aee93953ec157f29375ca1d48154a1b712cf5387 100644 (file)
--- a/doc/guides/rel_notes/release_19_02.rst
+++ b/doc/guides/rel_notes/release_19_02.rst
@@ -54,6 +54,15 @@ New Features
      Also, make sure to start the actual text at the margin.
      =========================================================
 
+* **Added support to free hugepages exactly as originally allocated.**
+
+  Some applications using memory event callbacks (especially for managing
+  RDMA memory regions) require that memory be freed back to the system
+  exactly as it was originally allocated. These applications typically
+  also require that a malloc allocation not span across two separate
+  hugepage allocations.  A new ``--match-allocations`` EAL init flag has
+  been added to fulfill both of these requirements.
+
 * **Updated the enic driver.**
 
   * Added support for ``RTE_ETH_DEV_CLOSE_REMOVE`` flag.

diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c
index e31eca5c034d0806763d0f47c106a0fd26a4f183..6e3a83b987e66c85dd0a1283e9d624e726de1cb5 100644 (file)
--- a/lib/librte_eal/common/eal_common_options.c
+++ b/lib/librte_eal/common/eal_common_options.c
@@ -79,6 +79,7 @@ eal_long_options[] = {
        {OPT_VMWARE_TSC_MAP,    0, NULL, OPT_VMWARE_TSC_MAP_NUM   },
        {OPT_LEGACY_MEM,        0, NULL, OPT_LEGACY_MEM_NUM       },
        {OPT_SINGLE_FILE_SEGMENTS, 0, NULL, OPT_SINGLE_FILE_SEGMENTS_NUM},
+       {OPT_MATCH_ALLOCATIONS, 0, NULL, OPT_MATCH_ALLOCATIONS_NUM},
        {0,                     0, NULL, 0                        }
 };
 
@@ -1424,6 +1425,16 @@ eal_check_common_options(struct internal_config *internal_cfg)
                                "with --"OPT_IN_MEMORY"\n");
                return -1;
        }
+       if (internal_cfg->legacy_mem && internal_cfg->match_allocations) {
+               RTE_LOG(ERR, EAL, "Option --"OPT_LEGACY_MEM" is not compatible "
+                               "with --"OPT_MATCH_ALLOCATIONS"\n");
+               return -1;
+       }
+       if (internal_cfg->no_hugetlbfs && internal_cfg->match_allocations) {
+               RTE_LOG(ERR, EAL, "Option --"OPT_NO_HUGE" is not compatible "
+                               "with --"OPT_MATCH_ALLOCATIONS"\n");
+               return -1;
+       }
 
        return 0;
 }

diff --git a/lib/librte_eal/common/eal_internal_cfg.h b/lib/librte_eal/common/eal_internal_cfg.h
index 737f17e3537699cfd8c78ae0bef7c0bde298649f..98e314fef42584f253ac6a0b335c1f44ee6af3cc 100644 (file)
--- a/lib/librte_eal/common/eal_internal_cfg.h
+++ b/lib/librte_eal/common/eal_internal_cfg.h
@@ -57,6 +57,8 @@ struct internal_config {
        /**< true to enable legacy memory behavior (no dynamic allocation,
         * IOVA-contiguous segments).
         */
+       volatile unsigned match_allocations;
+       /**< true to free hugepages exactly as allocated */
        volatile unsigned single_file_segments;
        /**< true if storing all pages within single files (per-page-size,
         * per-node) non-legacy mode only.

diff --git a/lib/librte_eal/common/eal_options.h b/lib/librte_eal/common/eal_options.h
index 5271f944934ba9f7dcdf2f3a2ed64871fa9a04f9..1480c5d77e38a67fa6e2032c531f2294b73bd3a3 100644 (file)
--- a/lib/librte_eal/common/eal_options.h
+++ b/lib/librte_eal/common/eal_options.h
@@ -65,6 +65,8 @@ enum {
        OPT_SINGLE_FILE_SEGMENTS_NUM,
 #define OPT_IOVA_MODE          "iova-mode"
        OPT_IOVA_MODE_NUM,
+#define OPT_MATCH_ALLOCATIONS  "match-allocations"
+       OPT_MATCH_ALLOCATIONS_NUM,
        OPT_LONG_MAX_NUM
 };
 

diff --git a/lib/librte_eal/common/malloc_elem.c b/lib/librte_eal/common/malloc_elem.c
index 9d3dcb6a9e85c757398ea2027025c63246e8d40e..fcdb18120adb6f11feaf2a0ad80d4ed41b4e6086 100644 (file)
--- a/lib/librte_eal/common/malloc_elem.c
+++ b/lib/librte_eal/common/malloc_elem.c
@@ -110,7 +110,8 @@ malloc_elem_find_max_iova_contig(struct malloc_elem *elem, size_t align)
  */
 void
 malloc_elem_init(struct malloc_elem *elem, struct malloc_heap *heap,
-               struct rte_memseg_list *msl, size_t size)
+               struct rte_memseg_list *msl, size_t size,
+               struct malloc_elem *orig_elem, size_t orig_size)
 {
        elem->heap = heap;
        elem->msl = msl;
@@ -120,6 +121,8 @@ malloc_elem_init(struct malloc_elem *elem, struct malloc_heap *heap,
        elem->state = ELEM_FREE;
        elem->size = size;
        elem->pad = 0;
+       elem->orig_elem = orig_elem;
+       elem->orig_size = orig_size;
        set_header(elem);
        set_trailer(elem);
 }
@@ -278,7 +281,8 @@ split_elem(struct malloc_elem *elem, struct malloc_elem *split_pt)
        const size_t old_elem_size = (uintptr_t)split_pt - (uintptr_t)elem;
        const size_t new_elem_size = elem->size - old_elem_size;
 
-       malloc_elem_init(split_pt, elem->heap, elem->msl, new_elem_size);
+       malloc_elem_init(split_pt, elem->heap, elem->msl, new_elem_size,
+                        elem->orig_elem, elem->orig_size);
        split_pt->prev = elem;
        split_pt->next = next_elem;
        if (next_elem)
@@ -317,14 +321,18 @@ static int
 next_elem_is_adjacent(struct malloc_elem *elem)
 {
        return elem->next == RTE_PTR_ADD(elem, elem->size) &&
-                       elem->next->msl == elem->msl;
+                       elem->next->msl == elem->msl &&
+                       (!internal_config.match_allocations ||
+                        elem->orig_elem == elem->next->orig_elem);
 }
 
 static int
 prev_elem_is_adjacent(struct malloc_elem *elem)
 {
        return elem == RTE_PTR_ADD(elem->prev, elem->prev->size) &&
-                       elem->prev->msl == elem->msl;
+                       elem->prev->msl == elem->msl &&
+                       (!internal_config.match_allocations ||
+                        elem->orig_elem == elem->prev->orig_elem);
 }
 
 /*

diff --git a/lib/librte_eal/common/malloc_elem.h b/lib/librte_eal/common/malloc_elem.h
index e2bda4c02755bf52624ae150da3a6644d2d716e4..207767c94568291ba9c5f8635ce2706c7b09fae0 100644 (file)
--- a/lib/librte_eal/common/malloc_elem.h
+++ b/lib/librte_eal/common/malloc_elem.h
@@ -32,6 +32,8 @@ struct malloc_elem {
        volatile enum elem_state state;
        uint32_t pad;
        size_t size;
+       struct malloc_elem *orig_elem;
+       size_t orig_size;
 #ifdef RTE_MALLOC_DEBUG
        uint64_t header_cookie;         /* Cookie marking start of data */
                                        /* trailer cookie at start + size */
@@ -116,7 +118,9 @@ void
 malloc_elem_init(struct malloc_elem *elem,
                struct malloc_heap *heap,
                struct rte_memseg_list *msl,
-               size_t size);
+               size_t size,
+               struct malloc_elem *orig_elem,
+               size_t orig_size);
 
 void
 malloc_elem_insert(struct malloc_elem *elem);

diff --git a/lib/librte_eal/common/malloc_heap.c b/lib/librte_eal/common/malloc_heap.c
index c6a6d4f6bdce1aafb2fabea6c66b6661fe3ad918..4c3632d02bdf9a310b4c4bbb9cddfd375c967212 100644 (file)
--- a/lib/librte_eal/common/malloc_heap.c
+++ b/lib/librte_eal/common/malloc_heap.c
@@ -94,7 +94,7 @@ malloc_heap_add_memory(struct malloc_heap *heap, struct rte_memseg_list *msl,
 {
        struct malloc_elem *elem = start;
 
-       malloc_elem_init(elem, heap, msl, len);
+       malloc_elem_init(elem, heap, msl, len, elem, len);
 
        malloc_elem_insert(elem);
 
@@ -857,6 +857,13 @@ malloc_heap_free(struct malloc_elem *elem)
        if (elem->size < page_sz)
                goto free_unlock;
 
+       /* if user requested to match allocations, the sizes must match - if not,
+        * we will defer freeing these hugepages until the entire original allocation
+        * can be freed
+        */
+       if (internal_config.match_allocations && elem->size != elem->orig_size)
+               goto free_unlock;
+
        /* probably, but let's make sure, as we may not be using up full page */
        start = elem;
        len = elem->size;
@@ -1259,6 +1266,10 @@ rte_eal_malloc_heap_init(void)
        struct rte_mem_config *mcfg = rte_eal_get_configuration()->mem_config;
        unsigned int i;
 
+       if (internal_config.match_allocations) {
+               RTE_LOG(DEBUG, EAL, "Hugepages will be freed exactly as allocated.\n");
+       }
+
        if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
                /* assign min socket ID to external heaps */
                mcfg->next_socket_id = EXTERNAL_HEAP_MIN_SOCKET_ID;

diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index d252c8591961d66758fa353d2467d960b2d594f0..dadba3385e2b574f66b292f20c98ea40205e787c 100644 (file)
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -518,6 +518,7 @@ eal_usage(const char *prgname)
               "  --"OPT_VFIO_INTR"         Interrupt mode for VFIO (legacy|msi|msix)\n"
               "  --"OPT_LEGACY_MEM"        Legacy memory mode (no dynamic allocation, contiguous segments)\n"
               "  --"OPT_SINGLE_FILE_SEGMENTS" Put all hugepage memory in single files\n"
+              "  --"OPT_MATCH_ALLOCATIONS" Free hugepages exactly as allocated\n"
               "\n");
        /* Allow the application to print its usage message too if hook is set */
        if ( rte_application_usage_hook ) {
@@ -786,6 +787,10 @@ eal_parse_args(int argc, char **argv)
                            strdup(optarg);
                        break;
 
+               case OPT_MATCH_ALLOCATIONS_NUM:
+                       internal_config.match_allocations = 1;
+                       break;
+
                default:
                        if (opt < OPT_LONG_MIN_NUM && isprint(opt)) {
                                RTE_LOG(ERR, EAL, "Option %c is not supported "

diff --git a/test/test/test_malloc.c b/test/test/test_malloc.c
index 5e52724194a07ddb0acea35fed7b141346aa0894..6b6c6fec1139ea2445277d588c8be45f400a4819 100644 (file)
--- a/test/test/test_malloc.c
+++ b/test/test/test_malloc.c
@@ -262,13 +262,26 @@ test_multi_alloc_statistics(void)
        struct rte_malloc_socket_stats pre_stats, post_stats ,first_stats, second_stats;
        size_t size = 2048;
        int align = 1024;
-#ifndef RTE_MALLOC_DEBUG
-       int trailer_size = 0;
-#else
-       int trailer_size = RTE_CACHE_LINE_SIZE;
-#endif
-       int overhead = RTE_CACHE_LINE_SIZE + trailer_size;
+       int overhead = 0;
+
+       /* Dynamically calculate the overhead by allocating one cacheline and
+        * then comparing what was allocated from the heap.
+        */
+       rte_malloc_get_socket_stats(socket, &pre_stats);
+
+       void *dummy = rte_malloc_socket(NULL, RTE_CACHE_LINE_SIZE, 0, socket);
+       if (dummy == NULL)
+               return -1;
+
+       rte_malloc_get_socket_stats(socket, &post_stats);
+
+       /* after subtracting cache line, remainder is overhead */
+       overhead = post_stats.heap_allocsz_bytes - pre_stats.heap_allocsz_bytes;
+       overhead -= RTE_CACHE_LINE_SIZE;
+
+       rte_free(dummy);
 
+       /* Now start the real tests */
        rte_malloc_get_socket_stats(socket, &pre_stats);
 
        void *p1 = rte_malloc_socket("stats", size , align, socket);