esp-idf/components/freertos/FreeRTOS-Kernel-SMP/porting_notes.md

Overview

Terminology

The following terms will be used in this document to avoid confusion between the different FreeRTOS versions currently in ESP-IDF

• SMP FreeRTOS: The SMP branch of the FreeRTOS kernel found here
• IDF FreeRTOS: The version of FreeRTOS used in mainline ESP-IDF that contained custom modifications to support SMP features specific to the ESP chips.

Organization

This directory contains a copy of SMP FreeRTOS based off of upstream commit 483237711

• IDF FreeRTOS remains in components/freertos/FreeRTOS-Kernel
• SMP FreeRTOS is entirely contained in components/freertos/FreeRTOS-Kernel-SMP
  • The Xtensa port files are copied from IDF FreeRTOS with minimal modifications.
  • port.c and portmacro.h were mostly re-written from scratch
  • Some changes were made to SMP FreeRTOS tasks.c in order to be compatible with IDF.
    • All of these changes are wrapped in #ifdef ESP_PLATFORM
    • These additions will be removed after the compatibility issues are resolved in IDF.
• SMP FreeRTOS is only compiled if CONFIG_FREERTOS_SMP is set in menuconfig
• All changes made to the rest of ESP-IDF to support SMP FreeRTOS are wrapped in #ifdef CONFIG_FREERTOS_SMP. Thus SMP FreeRTOS changes should have no effect on mainline ESP-IDF.

Behavioral Differences and Porting

This section covers

• The major feature/behavior differences between SMP FreeRTOS vs IDF FreeRTOS
• The changes made to the port or SMP FreeRTOS sources to remedy these differences
• Future Todos

Scheduler Suspension

IDF FreeRTOS:

• Each core had its own suspendable scheduler (i.e., calling vTaskSuspendAll() on one core has no effect on the other)
• A core that calls vTaskSuspendAll() will only disable task switching on that core.

SMP FreeRTOS:

• vTaskSuspendAll() is global (no cores can switch tasks once the scheduler is disabled)
• While the scheduler is suspend, any other core that attempts to block, call FreeRTOS API, or also suspend the scheduler will result that the other core spinning with interrupts disabled (due to task lock contention)

Changes Made:

• Replaced all calls of vTaskSuspendAll()/xTaskResumeAll() in ESP-IDF with vTaskPreemptionDisable()/vTaskPreemptionEnable() when using SMP FreeRTOS

Critical Sections

IDF FreeRTOS:

• IDF FreeRTOS uses granualar spin locks for critical sections
  • Callers need to instantiate and provide their own spinlocks
  • But allows different cores to simultaneously enter separate critical sections using different locks without contesting
• portENTER_CRITICAL()/portEXIT_CRITICAL() modified to accept a spinlock argument
• Added variant portTRY_ENTER_CRITICAL() to allow for timeout to be specified when entering critical sections

SMP FreeRTOS:

• Uses a giant lock (task and ISR locks to be specific). Cores simultaneously entering a critical section will contest with each other.
• portENTER_CRITICAL()/portEXIT_CRITICAL() does not accept arguments. Maps directly to vTaskEnterCritical()/vTaskExitCritical()
• Has added capability of checking for state change, allowing critical section contester to yield

Changes Made:

• In order to be compatible with IDF style critical sections, portENTER_CRITICAL(...)/portEXIT_CRITICAL(...) now accepts VA_ARGS
  • If no argument exists, it calls SMP FreeRTOS critical sections vTaskEnterCritical()
  • If an argument exists, it calls IDF FreeRTOS critical sections (where the argument is the spinlock)

Todo:

• Assess how using a giant lock (i.e., SMP FreeRTOS critical sections) will affect performance on time sensitive components of ESP-IDF (e.g., BT, WiFi)
  • If performance is acceptable, replace all critical sections in ESP-IDF with SMP FreeRTOS critical sections
  • Else, refactor IDF FreeRTOS critical sections to a separate API, and have IDF components call those instead.

Idle Tasks

IDF FreeRTOS:

• Only uses a single idle task function prvIdleTask()
• An idle task is created for each core and pinned to that core
• prvIdleTask() calls a custom esp_vApplicationIdleHook(), mostly used to feed the task watchdog
• IDF adds a Thread Local Storage Pointer (TLSP) Deletion Callback that is called on task deletion (i.e., can be called from the idle task).

SMP FreeRTOS:

• There are now two idle task functions. prvIdleTask() and prvMinimalIdleTask()
  • prvMinimalIdleTask() simply calls the vApplicationMinimalIdleHook() and handles idle yielding
  • prvIdleTask() has the added responsibility of calling prvCheckTasksWaitingTermination() and vApplicationIdleHook().
  • One prvIdleTask() is created, the remaining idle tasks are prvMinimalIdleTask()
• The created idle tasks are all unpinned.

Changes Made:

• Idle tasks were now pinned in order to be compatible with ESP-IDF
• Currently, vApplicationIdleHook() and vApplicationMinimalIdleHook() both just call esp_vApplicationIdleHook() in order to feed the task watchdog.

Todo:

• Remove the pinning of idle tasks.
  • Find some way to handle how FPU register contexts are cleaned up on task deletion. Idle tasks are no longer pinned, thus we need to handle the case where FPU cleanup is triggered across cores.
  • Check if unpinned idle tasks will affect the feeding of the idle watchdog timer.
• Find some way to integrate TLSP Deletion Callbacks to the idle tasks.
• Check if tickless Idle (and automatic light sleep) works in SMP FreeRTOS
• Separate out esp_vApplicationIdleHook() from the SMP FreeRTOS Idle hooks for the following reasons:
  • Users may want to provide their own vApplicationIdleHook() and vApplicationMinimalIdleHook(), thus IDF should not use those identifiers
  • prvIdleTask() will call both vApplicationIdleHook() and vApplicationMinimalIdleHook(), thus the watchdog is fed twice.
• Add support for configIDLE_SHOULD_YIELD
• See if TLSP Deletion Callbacks can be upstreamed as a new feature

Tick Interrupt

IDF FreeRTOS:

• All cores have their own tick interrupts (same frequency but can be out of phase)
• Each core calls xTaskIncrementTick() independently. xTaskIncrementTick() has been modified as such:
  • Core 0 is responsible for time keeping (incrementing tick count and unblocking timed out tasks)
  • Each core independently checks for time slicing, and calls vApplicationTickHook()
  • There's also an additional esp_vApplicationTickHook() used to feed the interrupt watchdog.

SMP FreeRTOS:

• Only core 0 calls xTaskIncrementTick(). Thus core 0 is solely responsible for:
  • Time keeping (increment ticks and unblocking timed out tasks)
  • Time slicing (calls prvYieldCore() on each core that needs to yield)
  • Calls vApplicationTickHook()
  • Increments pended ticks when the scheduler is suspended

Changes Made:

• Each core still has a tick interrupt (to call esp_vApplicationTickHook()). But only core 0 will call xTaskIncrementTick()

Todo:

• See if we can find another way to implement the interrupt watchdog. Requiring each core to interrupt just to feed the watchdog loses the benefit of having core 0 handling the time slicing of all cores.

Additional API

IDF FreeRTOS added several APIs. These are copied over to SMP FreeRTOS to maintain compatibility. These API can either be:

• Upstreamed to SMP FreeRTOS officially
• Moved to freertos_tasks_c_additions.h
• Removed from IDF entirely

xTaskCreatePinnedToCore()/xTaskCreateStaticPinnedToCore()

• Used to create a task with a preset affinity on creation
• When a task can only run on a particular core, this function saves the need of adding logic to:
  • Disabling preemption on all cores
  • Setting the created task's affinity
  • Reenabling preemption on all cores.
• Check if this (or something similar) can be upstreamed

vTaskSetThreadLocalStoragePointerAndDelCallback()

• Used to set the TLSP deletion callbacks.

xTaskGetCurrentTaskHandleForCPU()

• Convenience function to the get current task of a particular CPU
• Check if this can be upstreamed

xTaskGetIdleTaskHandleForCPU()

• Convenience function to the get the idle task handle for a particular CPU
• Can be entirely replaced with xTaskGetIdleTaskHandle()

xTaskGetAffinity()

• Returns what core a task is pinned to, and not an affinity mask.
• Can be entirely replaced with vTaskCoreAffinityGet()