Merge branch 'feature/heap-trace-perf-improvment' into 'master'

heap: Allow for hashmap placement in external memory

Closes IDFGH-9845

See merge request espressif/esp-idf!23184

components/heap/Kconfig

@@ -84,10 +84,20 @@ menu "Heap memory debugging"
             the heap trace performances in adding retrieving and removing trace records will be
             enhanced.
 
+    config HEAP_TRACE_HASH_MAP_IN_EXT_RAM
+        bool "Place hash map in the bss section of the external RAM"
+        depends on HEAP_TRACE_HASH_MAP && SPIRAM_ALLOW_BSS_SEG_EXTERNAL_MEMORY
+        default n
+        help
+            When enabled this configuration forces the hash map to be placed in the bss section
+            of the external RAM.
+
+            Note that this functionality can only be enabled when CONFIG_SPIRAM_ALLOW_BSS_SEG_EXTERNAL_MEMORY
+            is enabled.
+
     config HEAP_TRACE_HASH_MAP_SIZE
         int "The number of entries in the hash map"
         depends on HEAP_TRACE_HASH_MAP
-        range 1 10000
         default 10
         help
             Defines the number of entries in the heap trace hashmap. The bigger this number is,

components/heap/heap_trace_standalone.c

@@ -84,7 +84,11 @@ static size_t r_get_idx;
 TAILQ_HEAD(heap_trace_hash_list_struct_t, heap_trace_record_t);
 typedef struct heap_trace_hash_list_struct_t heap_trace_hash_list_t;
 
-static heap_trace_hash_list_t hash_map[(size_t)CONFIG_HEAP_TRACE_HASH_MAP_SIZE]; // Buffer used for hashmap entries
+static
+#if CONFIG_HEAP_TRACE_HASH_MAP_IN_EXT_RAM
+EXT_RAM_BSS_ATTR
+#endif
+heap_trace_hash_list_t hash_map[(size_t)CONFIG_HEAP_TRACE_HASH_MAP_SIZE]; // Buffer used for hashmap entries
 static size_t total_hashmap_hits;
 static size_t total_hashmap_miss;
 

components/heap/test_apps/heap_tests/pytest_heap.py

@@ -80,7 +80,8 @@ def test_heap_8bit_access(dut: Dut) -> None:
 @pytest.mark.parametrize(
     'config',
     [
-        'heap_trace'
+        'heap_trace',
+        'heap_trace_hashmap'
     ]
 )
 def test_heap_trace_dump(dut: Dut) -> None:

components/heap/test_apps/heap_tests/sdkconfig.ci.heap_trace_hashmap

@@ -0,0 +1,5 @@
+CONFIG_SPIRAM=y
+CONFIG_SPIRAM_ALLOW_BSS_SEG_EXTERNAL_MEMORY=y
+CONFIG_HEAP_TRACING_STANDALONE=y
+CONFIG_HEAP_TRACE_HASH_MAP=y
+CONFIG_HEAP_TRACE_HASH_MAP_IN_EXT_RAM=y

docs/en/api-reference/system/heap_debug.rst

@@ -33,9 +33,7 @@ Heap allocation and free detection hooks allows you to be notified of every succ
 - Providing a definition of :cpp:func:`esp_heap_trace_alloc_hook` will allow you to be notified of every successful memory allocation operations
 - Providing a definition of :cpp:func:`esp_heap_trace_free_hook` will allow you to be notified of every memory free operations
 
-To activate the feature, navigate to ``Component config`` -> ``Heap Memory Debugging`` in the configuration menu and select ``Use allocation and free hooks`` option (see :ref:`CONFIG_HEAP_USE_HOOKS`).
-:cpp:func:`esp_heap_trace_alloc_hook` and :cpp:func:`esp_heap_trace_free_hook` have weak declarations, it is not necessary to provide a declarations for both hooks.
-Since allocating and freeing memory is allowed even though strongly recommended against, :cpp:func:`esp_heap_trace_alloc_hook` and :cpp:func:`esp_heap_trace_free_hook` can potentially be called from ISR.
+To activate the feature, navigate to ``Component config`` -> ``Heap Memory Debugging`` in the configuration menu and select ``Use allocation and free hooks`` option (see :ref:`CONFIG_HEAP_USE_HOOKS`). :cpp:func:`esp_heap_trace_alloc_hook` and :cpp:func:`esp_heap_trace_free_hook` have weak declarations, it is not necessary to provide a declarations for both hooks. Since allocating and freeing memory is allowed even though strongly recommended against, :cpp:func:`esp_heap_trace_alloc_hook` and :cpp:func:`esp_heap_trace_free_hook` can potentially be called from ISR.
 
 .. _heap-corruption:
 
@@ -84,6 +82,31 @@ The example below shows how to register an allocation failure callback::
       ...
   }
 
+Memory Allocation and Free Hooks
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+It is possible to implement two function hooks to get notified of every successful allocation and free operations by enabling :ref:`CONFIG_HEAP_USE_HOOKS`. With this configuration enabled, you will be able to provide the definition of :cpp:func:`esp_heap_trace_alloc_hook` and :cpp:func:`esp_heap_trace_free_hook`.
+
+Performing (or calling API functions performing) blocking operations or memory allocation / free operations in the hook functions is recommended against. In general, it is considered best practice to keep the implementation concise and leave the heavy computation outside of the hook functions.
+
+The example below shows how to define the allocation and free function hooks::
+
+  #include "esp_heap_caps.h"
+
+  void esp_heap_trace_alloc_hook(void* ptr, size_t size, uint32_t caps)
+  {
+    ...
+  }
+  void esp_heap_trace_free_hook(void* ptr)
+  {
+    ...
+  }
+
+  void app_main()
+  {
+      ...
+  }
+
 Finding Heap Corruption
 ^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -453,6 +476,12 @@ Enabling heap tracing in menuconfig increases the code size of your program, and
 
 When heap tracing is running, heap allocation/free operations are substantially slower than when heap tracing is stopped. Increasing the depth of stack frames recorded for each allocation (see above) will also increase this performance impact.
 
+To mitigate the performance loss when the heap tracing is enabled and active, enable :ref:`CONFIG_HEAP_TRACE_HASH_MAP`. With this configuration enable, a hash map mechanism will be used to handle the heap trace records thus considerably improving the heap allocation/free execution time. The size of the hash map can be modified by setting the value of :ref:`CONFIG_HEAP_TRACE_HASH_MAP_SIZE`.
+
+.. only:: SOC_SPIRAM_SUPPORTED
+
+  By default the hash map is placed into internal RAM. It can also be placed into external RAM if :ref:`CONFIG_HEAP_TRACE_HASH_MAP_IN_EXT_RAM` is enabled. In order to enable this configuration, make sure to enable :ref:`CONFIG_SPIRAM` and :ref:`CONFIG_SPIRAM_ALLOW_BSS_SEG_EXTERNAL_MEMORY`.
+
 False-Positive Memory Leaks
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^