ESP-IDF FreeRTOS SMP Changes¶
The vanilla FreeRTOS is designed to run on a single core. However the ESP32 is dual core containing a Protocol CPU (known as CPU 0 or PRO_CPU) and an Application CPU (known as CPU 1 or APP_CPU). The two cores are identical in practice and share the same memory. This allows the two cores to run tasks interchangeably between them.
The ESP-IDF FreeRTOS is a modified version of vanilla FreeRTOS which supports symmetric multiprocessing (SMP). ESP-IDF FreeRTOS is based on the Xtensa port of FreeRTOS v8.2.0, however features such as static task creation and Thread Local Storage Pointers have been backported from later versions of FreeRTOS. This guide outlines the major differences between vanilla FreeRTOS and ESP-IDF FreeRTOS. The API reference for vanilla FreeRTOS can be found via http://www.freertos.org/a00106.html
Tasks and Task Creation: Use
xTaskCreateStaticPinnedToCore() to create tasks in ESP-IDF FreeRTOS. The
last parameter of the two functions is
xCoreID. This parameter specifies
which core the task is pinned to. Acceptable values are
0 for PRO_CPU,
1 for APP_CPU, or
tskNO_AFFINITY which allows the task to run on
Round Robin Scheduling: The ESP-IDF FreeRTOS scheduler will skip tasks when implementing Round-Robin scheduling between multiple tasks in the Ready state that are of the same priority. To avoid this behavior, ensure that those tasks either enter a blocked state, or are distributed across a wider range of priorities.
Scheduler Suspension: Suspending the scheduler in ESP-IDF FreeRTOS will only
affect the scheduler on the the calling core. In other words, calling
vTaskSuspendAll() on PRO_CPU will not prevent APP_CPU from scheduling, and
vice versa. Use critical sections or semaphores instead for simultaneous
Tick Interrupt Synchronicity: Tick interrupts of PRO_CPU and APP_CPU
are not synchronized. Do not expect to use
vTaskDelayUntil as an accurate method of synchronizing task execution
between the two cores. Use a counting semaphore instead as their context
switches are not tied to tick interrupts due to preemption.
Critical Sections & Disabling Interrupts: In ESP-IDF FreeRTOS, critical sections are implemented using mutexes. Entering critical sections involve taking a mutex, then disabling the scheduler and interrupts of the calling core. However the other core is left unaffected. If the other core attemps to take same mutex, it will spin until the calling core has released the mutex by exiting the critical section.
Thread Local Storage Pointers & Deletion Callbacks: ESP-IDF FreeRTOS has
backported the Thread Local Storage Pointers feature. However they have the
extra feature of deletion callbacks. Deletion callbacks are used to
automatically free memory used by Thread Local Storage Pointers during the task
to set Thread Local Storage Pointers and deletion callbacks.
FreeRTOS Hooks: Vanilla FreeRTOS Hooks were not designed for SMP. ESP-IDF provides its own Idle and Tick Hooks in addition to the Vanilla FreeRTOS hooks. For full details, see the ESP-IDF Hooks API Reference.
Configuring ESP-IDF FreeRTOS: Several aspects of ESP-IDF FreeRTOS can be
make meunconfig such as running ESP-IDF in Unicore Mode,
or configuring the number of Thread Local Storage Pointers each task will have.
Tasks and Task Creation¶
Tasks in ESP-IDF FreeRTOS are designed to run on a particular core, therefore
two new task creation functions have been added to ESP-IDF FreeRTOS by
PinnedToCore to the names of the task creation functions in
vanilla FreeRTOS. The vanilla FreeRTOS functions of
xTaskCreateStatic() have led to the addition of
For more details see freertos/task.c
The ESP-IDF FreeRTOS task creation functions are nearly identical to their
vanilla counterparts with the exception of the extra parameter known as
xCoreID. This parameter specifies the core on which the task should run on
and can be one of the following values.
0pins the task to PRO_CPU
1pins the task to APP_CPU
tskNO_AFFINITYallows the task to be run on both CPUs
xTaskCreatePinnedToCore(tsk_callback, “APP_CPU Task”, 1000, NULL, 10, NULL, 1)
creates a task of priority 10 that is pinned to APP_CPU with a stack size
of 1000 bytes. It should be noted that the
uxStackDepth parameter in
vanilla FreeRTOS specifies a task’s stack depth in terms of the number of
words, whereas ESP-IDF FreeRTOS specifies the stack depth in terms of bytes.
Note that the vanilla FreeRTOS functions
xTaskCreateStatic have been macro defined in ESP-IDF FreeRTOS to call
tskNO_AFFINITY as the
Each Task Control Block (TCB) in ESP-IDF stores the
xCoreID as a member.
Hence when each core calls the scheduler to select a task to run, the
xCoreID member will allow the scheduler to determine if a given task is
permitted to run on the core that called it.
The vanilla FreeRTOS implements scheduling in the
function. This function is responsible for selecting the highest priority task
to run from a list of tasks in the Ready state known as the Ready Tasks List
(described in the next section). In ESP-IDF FreeRTOS, each core will call
vTaskSwitchContext() independently to select a task to run from the
Ready Tasks List which is shared between both cores. There are several
differences in scheduling behavior between vanilla and ESP-IDF FreeRTOS such as
differences in Round Robin scheduling, scheduler suspension, and tick interrupt
Round Robin Scheduling¶
Given multiple tasks in the Ready state and of the same priority, vanilla FreeRTOS implements Round Robin scheduling between each task. This will result in running those tasks in turn each time the scheduler is called (e.g. every tick interrupt). On the other hand, the ESP-IDF FreeRTOS scheduler may skip tasks when Round Robin scheduling multiple Ready state tasks of the same priority.
The issue of skipping tasks during Round Robin scheduling arises from the way
the Ready Tasks List is implemented in FreeRTOS. In vanilla FreeRTOS,
pxReadyTasksList is used to store a list of tasks that are in the Ready
state. The list is implemented as an array of length
where each element of the array is a linked list. Each linked list is of type
List_t and contains TCBs of tasks of the same priority that are in the
Ready state. The following diagram illustrates the
Each linked list also contains a
pxIndex which points to the last TCB
returned when the list was queried. This index allows the
to start traversing the list at the TCB immediately after
implementing Round Robin Scheduling between tasks of the same priority.
In ESP-IDF FreeRTOS, the Ready Tasks List is shared between cores hence
pxReadyTasksList will contain tasks pinned to different cores. When a core
calls the scheduler, it is able to look at the
xCoreID member of each TCB
in the list to determine if a task is allowed to run on calling the core. The
pxReadyTasksList is illustrated below.
Therefore when PRO_CPU calls the scheduler, it will only consider the tasks in blue or purple. Whereas when APP_CPU calls the scheduler, it will only consider the tasks in orange or purple.
Although each TCB has an
xCoreID in ESP-IDF FreeRTOS, the linked list of
each priority only has a single
pxIndex. Therefore when the scheduler is
called from a particular core and traverses the linked list, it will skip all
TCBs pinned to the other core and point the pxIndex at the selected task. If
the other core then calls the scheduler, it will traverse the linked list
starting at the TCB immediately after
pxIndex. Therefore, TCBs skipped on
the previous scheduler call from the other core would not be considered on the
current scheduler call. This issue is demonstrated in the following
Referring to the illustration above, assume that priority 9 is the highest priority, and none of the tasks in priority 9 will block hence will always be either in the running or Ready state.
1) PRO_CPU calls the scheduler and selects Task A to run, hence moves
pxIndex to point to Task A
2) APP_CPU calls the scheduler and starts traversing from the task after
pxIndex which is Task B. However Task B is not selected to run as it is not
pinned to APP_CPU hence it is skipped and Task C is selected instead.
pxIndex now points to Task C
3) PRO_CPU calls the scheduler and starts traversing from Task D. It skips
Task D and selects Task E to run and points
pxIndex to Task E. Notice that
Task B isn’t traversed because it was skipped the last time APP_CPU called
the scheduler to traverse the list.
4) The same situation with Task D will occur if APP_CPU calls the
scheduler again as
pxIndex now points to Task E
One solution to the issue of task skipping is to ensure that every task will enter a blocked state so that they are removed from the Ready Task List. Another solution is to distribute tasks across multiple priorities such that a given priority will not be assigned multiple tasks that are pinned to different cores.
In vanilla FreeRTOS, suspending the scheduler via
prevent calls of
vTaskSwitchContext() from context switching until the
scheduler has been resumed with
vTaskResumeAll(). However servicing ISRs
are still permitted. Therefore any changes in task states as a result from the
current running task or ISRSs will not be executed until the scheduler is
resumed. Scheduler suspension in vanilla FreeRTOS is a common protection method
against simultaneous access of data shared between tasks, whilst still allowing
ISRs to be serviced.
In ESP-IDF FreeRTOS,
vTaskSuspendAll() will only prevent calls of
vTaskSwitchContext() from switching contexts on the core that called for the
suspension. Hence if PRO_CPU calls
vTaskSuspendAll(), APP_CPU will
still be able to switch contexts. If data is shared between tasks that are
pinned to different cores, scheduler suspension is NOT a valid method of
protection against simultaneous access. Consider using critical sections
(disables interrupts) or semaphores (does not disable interrupts) instead when
protecting shared resources in ESP-IDF FreeRTOS.
In general, it’s better to use other RTOS primitives like mutex semaphores to protect
against data shared between tasks, rather than
Tick Interrupt Synchronicity¶
In ESP-IDF FreeRTOS, tasks on different cores that unblock on the same tick count might not run at exactly the same time due to the scheduler calls from each core being independent, and the tick interrupts to each core being unsynchronized.
In vanilla FreeRTOS the tick interrupt triggers a call to
xTaskIncrementTick() which is responsible for incrementing the tick
counter, checking if tasks which have called
vTaskDelay() have fulfilled
their delay period, and moving those tasks from the Delayed Task List to the
Ready Task List. The tick interrupt will then call the scheduler if a context
switch is necessary.
In ESP-IDF FreeRTOS, delayed tasks are unblocked with reference to the tick interrupt on PRO_CPU as PRO_CPU is responsible for incrementing the shared tick count. However tick interrupts to each core might not be synchronized (same frequency but out of phase) hence when PRO_CPU receives a tick interrupt, APP_CPU might not have received it yet. Therefore if multiple tasks of the same priority are unblocked on the same tick count, the task pinned to PRO_CPU will run immediately whereas the task pinned to APP_CPU must wait until APP_CPU receives its out of sync tick interrupt. Upon receiving the tick interrupt, APP_CPU will then call for a context switch and finally switches contexts to the newly unblocked task.
Therefore, task delays should NOT be used as a method of synchronization between tasks in ESP-IDF FreeRTOS. Instead, consider using a counting semaphore to unblock multiple tasks at the same time.
Critical Sections & Disabling Interrupts¶
Vanilla FreeRTOS implements critical sections in
disables the scheduler and calls
portDISABLE_INTERRUPTS. This prevents
context switches and servicing of ISRs during a critical section. Therefore,
critical sections are used as a valid protection method against simultaneous
access in vanilla FreeRTOS.
On the other hand, the ESP32 has no hardware method for cores to disable each
other’s interrupts. Calling
portDISABLE_INTERRUPTS() will have no effect on
the interrupts of the other core. Therefore, disabling interrupts is NOT
a valid protection method against simultaneous access to shared data as it
leaves the other core free to access the data even if the current core has
disabled its own interrupts.
For this reason, ESP-IDF FreeRTOS implements critical sections using mutexes, and calls to enter or exit a critical must provide a mutex that is associated with a shared resource requiring access protection. When entering a critical section in ESP-IDF FreeRTOS, the calling core will disable its scheduler and interrupts similar to the vanilla FreeRTOS implementation. However, the calling core will also take the mutex whilst the other core is left unaffected during the critical section. If the other core attempts to take the same mutex, it will spin until the mutex is released. Therefore, the ESP-IDF FreeRTOS implementation of critical sections allows a core to have protected access to a shared resource without disabling the other core. The other core will only be affected if it tries to concurrently access the same resource.
The ESP-IDF FreeRTOS critical section functions have been modified as follows…
portENTER_CRITICAL_ISR(mux)are all macro defined to call
portEXIT_CRITICAL_ISR(mux)are all macro defined to call
It should be noted that when modifying vanilla FreeRTOS code to be ESP-IDF FreeRTOS compatible, it is trivial to modify the type of critical section called as they are all defined to call the same function. As long as the same mutex is provided upon entering and exiting, the type of call should not matter.
Thread Local Storage Pointers & Deletion Callbacks¶
Thread Local Storage Pointers are pointers stored directly in the TCB which allows each task to have a pointer to a data structure containing that is specific to that task. However vanilla FreeRTOS provides no functionality to free the memory pointed to by the Thread Local Storage Pointers. Therefore if the memory pointed to by the Thread Local Storage Pointers is not explicitly freed by the user before a task is deleted, memory leak will occur.
ESP-IDF FreeRTOS provides the added feature of deletion callbacks. These deletion callbacks are used to automatically free the memory pointed to by the Thread Local Storage Pointers when a task is deleted. Each Thread Local Storage Pointer can have its own call back, and these call backs are called when the Idle tasks cleans up a deleted tasks.
Vanilla FreeRTOS sets a Thread Local Storage Pointers using
vTaskSetThreadLocalStoragePointer() whereas ESP-IDF FreeRTOS sets a Thread
Local Storage Pointers and Deletion Callbacks using
vTaskSetThreadLocalStoragePointerAndDelCallback() which accepts a pointer
to the deletion call back as an extra parameter of type
`TlsDeleteCallbackFunction_t. Calling the vanilla FreeRTOS API
vTaskSetThreadLocalStoragePointer() is still valid however it is internally
defined to call
vTaskSetThreadLocalStoragePointerAndDelCallback() with a
NULL pointer as the deletion call back. This results in the selected Thread
Local Storage Pointer to have no deletion call back.
In IDF the FreeRTOS thread local storage at index 0 is reserved and is used to implement the pthreads API thread local storage (pthread_getspecific() & pthread_setspecific()). Other indexes can be used for any purpose, provided FREERTOS_THREAD_LOCAL_STORAGE_POINTERS is set to a high enough value.
For more details see freertos/include/freertos/task.h
Configuring ESP-IDF FreeRTOS¶
The ESP-IDF FreeRTOS can be configured using
make menuconfig under
Component_Config/FreeRTOS. The following section highlights some of the
ESP-IDF FreeRTOS configuration options. For a full list of ESP-IDF
FreeRTOS configurations, see FreeRTOS
FREERTOS_UNICORE will run ESP-IDF FreeRTOS only
on PRO_CPU. Note that this is not equivalent to running vanilla
FreeRTOS. Behaviors of multiple components in ESP-IDF will be modified such
as esp32/cpu_start.c. For more details regarding the
effects of running ESP-IDF FreeRTOS on a single core, search for
CONFIG_FREERTOS_UNICORE in the ESP-IDF components.
FREERTOS_THREAD_LOCAL_STORAGE_POINTERS will define the number of Thread Local Storage Pointers each task will have in ESP-IDF FreeRTOS.
SUPPORT_STATIC_ALLOCATION will enable the backported
xTaskCreateStaticPinnedToCore() in ESP-IDF FreeRTOS
FREERTOS_ASSERT_ON_UNTESTED_FUNCTION will trigger a halt in particular functions in ESP-IDF FreeRTOS which have not been fully tested in an SMP context.