bugfix(nvs_flash) : fixed nvs_set functions behaviour when called sequentially with same key and different data type(s)
Closes IDFGH-9727
See merge request espressif/esp-idf!25581
Currently the component_requirements hint module does not work
as expected if the component list for a project is trimmed down.
With the new "all_component_info" dictionary info in project_description.json,
the module can produce hints even if cmake's COMPONENTS variable is
set.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Add new "all_component_info" dictionary into the
project_description.json file. It contains information about all
registered components presented in the __COMPONENT_TARGETS list.
Since components in this list are not fully evaluated, because only the first
stage of cmakefiles processing is done, it does not contain the same information
as the "build_component_info" dictionary. The "type", "file" and "sources" variables
are missing.
Most of the properties are already attached to the component target, so
this only adds INCLUDE_DIRS property to the target during the first cmakefiles
processing stage.
The "all_component_info" dict is generated in a separate function, even
though the original function for "build_component_info" could be
adjusted. This introduces a little bit of boilerplate, but keeps it
logically separated and probably easier if we want to extend it in the
future.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
1. move startup_stack attr from soc_memory_type_desc_t to soc_memory_region_t and
remove unused aliased_iram field
2. all of the last level of RAM is retention dma accessible on esp32c3
3. remove esp32c2 and later chips retention dma accessible memory caps
4. allow allocate memory from RTC_RAM with MALLOC_CAP_EXEC cap
If there is a component(child) within a component(parent), like for test_apps, the parent
component may be wrongly identified as source component for the failed include. This may
lead to a false bug report if the parent component has component, which provides the missing
header, in requirements.
Fix this by looking for the longest matching source component directory.
Suggested-by: Ivan Grokhotkov <ivan@espressif.com>
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
If LCP keepalive mechanism is enabled in menuconfig, it's statically
configured on creation of an interface and cannot be changed runtime. In
some cases it's useful to relax LCP criteria during runtime operation,
for example before initiating OTA. This config option allows for
disabling already enabled LCP echo (this setting becomes effective after
reconnecting, i.e. initializing a new session)
Closes https://github.com/espressif/esp-protocols/issues/287
Add a test app to verify the working of the application when
SECURE_SIGNED_ON_UPDATE_NO_SECURE_BOOT is selected in the menuconfig
and the application is not signed
This commit adds beta support for the FreeRTOS v10.5.1 kernel which can be
enabled by enabling the CONFIG_FREERTOS_USE_KERNEL_10_5_1 option.
The following changes have been made:
- Updated freertos/CMakeLists.txt to build v10.5.1 kernel with v10.4.3. ports
- Updated existing Xtensa and RISC-V ports to work with V10.5.1
- Modifications to other ESP-IDF components to work with v10.5.1
- Added some ESP-IDF specific tracing changes to v10.5.1 kernel
- Make CONFIG_FREERTOS_USE_KERNEL_10_5_1 a public option
Note: The beta release is missing some minor fixes, performance improvements,
and features. Using this beta release for production is not recommended.
Closes https://github.com/espressif/esp-idf/issues/7137
Merges https://github.com/espressif/esp-idf/pull/12093
fix(ll): remove FLAG_ATTR macro
Such kind of operator overload will not work because C++ thinks such overload is ambiguous and it still prefer the built-in one which accepts and returns integer. Manually force type conversion seems to be unavoidable.
Add changes to use fast_pbkdf2 as default for PMK calculations.
fast_pbkdf2 is significantly faster than current implementations
for esp chips.
Also removes unnecessary code for pbkdf-sha256 and pbkdf-sha512.
This commit moves/merges the IDF FreeRTOS "PinnedToCore" task creation
functions from tasks.c/task.h to idf_additions.h/freertos_task_c_additions.h.
Also updated FreeRTOS Mock component to provide mocks for "idf_additions.h"
headers for our mock tests.
It corrupts environment on macOS runner because it uses shell, not docker-executor,
and makes all tree read-only for a non-root user.
+ Replace examples with real files in example README.md
Currently idf.py reports just "Please use idf.py only in an ESP-IDF shell environment".
Sometimes it may be useful to know for which module the import failed.
Also the problem does not have to be related to shell environment only, but the
python venv can be corrupted. This adds a little bit more verbose error
message.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Users of the nvs API are likely to have `nvs_handle_t` in all cases, but
not all of them carry around the partition name and namespace name (as
they aren't needed after creating an `nvs_handle_t`).
Allow this common case to use nvs iteration without them tracking
additional data by using the `nvs_handle_t` to locate the namespace
index and the partition name by introducing an alterate to
`nvs_entry_find` called `nvs_entry_find_in_handle`, which operates
similarly except that it is given a `nvs_handle_t` instead of partition
and namespace strings.
This is somewhat more limited than the `nvs_entry_find` API as one
cannot examine all keys in a given partition.
Previously, TLSP deletion callbacks were...
- Stored in a seprate TCB member "pvThreadLocalStoragePointersDelCallback"
- Called separately via multipole prvDeleteTLS() insertions in tasks.c
This commit refactors how TLSP deletion callbacks are stored and called:
- TLSP deletion callbacks are now stored in "pvThreadLocalStoragePointers"
directly. configNUM_THREAD_LOCAL_STORAGE_POINTERS is doubled in size so that
the deletion callbacks are stored in the latter half of the array
- The callbacks are now called via "portCLEAN_UP_TCB()". As such, the
prvDeleteTLS() additions are no longer needed and the function can be removed
- Removed some legacy TLSP tests using the old method of storing the callback
pointers.
This commit reduces the source code diff between IDF FreeRTOS and upstream
vanilla FreeRTOS, in preparation for v10.5.1 upgrade.
Previously, if CONFIG_FREERTOS_ENABLE_STATIC_TASK_CLEAN_UP was enabled, users
would provide a definition for a vPortCleanUpTCB() hook function that is called
right before a task's memory is freed in prvDeleteTCB(). However,
vPortCleanUpTCB() will be reclaimed by ESP-IDF for internal use in v6.0.
This commit introduces the following changes...
Introduced a new CONFIG_FREERTOS_TASK_PRE_DELETION_HOOK option:
- Provides the same pre-deletion hook functionality. But users now define
vTaskPreDeletionHook() instead.
- CONFIG_FREERTOS_ENABLE_STATIC_TASK_CLEAN_UP still exists, but is marked as
deprecated. This is to maintain compatibility with existing applications
that already define vPortCleanUpTCB().
- Removed redundant --wl --wrap workaround with vPortCleanUpTCB()
- Added todo notes to remove support for user defined vPortCleanUpTCB()
completely in v6.0.
- Updated test cases to use new CONFIG_FREERTOS_TASK_PRE_DELETION_HOOK option
Freed up portCLEAN_UP_TCB() to call a new internal vPortTCBPreDeleteHook():
- vPortTCBPreDeleteHook() now replaces the previous "wrapped" implementation
of vPortCleanUpTCB().
- vPortTCBPreDeleteHook() is an internal task pre-delete hook for IDF FreeRTOS
ports to inject some pre-deletion operations.
- Internal pre-delete hook now invokes user provided vTaskPreDeletionHook()
if enabled.
- Relocated vPortTCBPreDeleteHook() to correct section in port.c
- a recent commit fixed the API deprecations,
and the BLEDevice object now needs to be accessed through a tuple.
- thus the changes were made in the code wherein the object was used,
but updating the usage in the condition when de
vname is not None got skipped.
GDB on Windows incorrectly reads EOL in the script files causing 'gdb'
action to fail.
(gdb) source .../build/gdbinit/py_extensions
(gdb) source .../build\gdbinit\symbols
add symbol table from file "...\build\bootloader\bootloader.elf"
.../build\gdbinit\symbols:6: Error in sourced command file:
Undefined command: "". Try "help".
Forcing line separator to '\n' resolved the issue
Signed-off-by: Michal Jenikovsky <jendo@jmsystems.sk>
1. The original test, before hint modules support was added, used
tempfile.NamedTemporaryFile in a way which is not supported on windows.
It was having the file open, which the hints tried to read it, leading
the EPERM exception. The docs[1] says this is not supported.
<quote>
Whether the name can be used to open the file a second time, while the
named temporary file is still open, varies across platforms (it can be
so used on Unix; it cannot on Windows)
</quote>
2. The hint module component_requirements test used the idf.py directly,
which is idf.py.exe on windows.Now it's starting idf.py through python.
We could probably used shell=True, but this approach is used in other
tests too.
Anyway the test are now passing on windows.
[1] https://docs.python.org/3/library/tempfile.html
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Following tests were added.
1. Test for missing header directory in component's INCLUDE_DIRS
2. Test for missing dependency in component's PRIV_REQUIRES
3. Test for missing dependency in component's REQUIRES
4. Test for dependency in PRIV_REQUIRES which should be in REQUIRES
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Implementation of hint module for component dependency. It can provide
hint about missing header directory in component's INCLUDE_DIRS or
about missing component dependency in REQUIRES or PRIV_REQUIRES.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently hints are supported based on hints.yml only, which may be
limiting for some use cases. This introduces a generic plugin approach,
which allows to implement hint module that doesn't require entry in hints.yml.
Such module has the full command output available and it is not limited to
a single regex in hints.yml.
Note that regex in hint.yml expects the output concatenated into a single line,
but hint modules are getting the output unchanged.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Introduces get_build_context() helper, which allows to get build context, e.g.
project description, at places where this info is not available. The
build context is set within ensure_build_directory.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
feat(idf.py): Allow adding arguments from file via @filename.txt (#11783) (GitHub PR)
Closes IDFGH-10584 and IDFGH-10539
See merge request espressif/esp-idf!24955
The ticket below reports some very strange behaviour, where
even a simple command like the following
python3 -c "import subprocess; subprocess.run([''])"
actually spawns some process and pass instead of raising
PermissionError. Even though the problem is most probably somewhere
else, not in idf_tools, we may just return early if there is no
command available for get_version().
Closes https://github.com/espressif/esp-idf/issues/11880
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
When interrupt allocation fails, esp_intr_alloc will now print a
message telling that no free interrupt was found. This message is
then checked in hints.yml, to give a link to the troubleshooting
guide.
Currently ESP_PYTHON is used in the install.sh script, but it's
overwritten in detect_python.sh. This allows to explicitly specify
python binary, which should be used to create venv.
$ ESP_PYTHON=/usr/bin/python3.7 ./install.{sh,fish}
$ ESP_PYTHON=/usr/bin/python3.7 ./export.{sh,fish}
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Now the soc interrupts definitions are scattered around in the esp-idf
which are out of sync. Put interrupts definitions in soc/periph_defs.h
(!ESP32) or soc/soc.h(ESP32) together in soc/interrupts.h.
idf.py spawns gdb process within a thread and uses Thread.join() to wait
for the gdb process to finish. As CTRL+C(SIGINT) is used by gdb to interrupt the
running program, we catch the SIGINT while waiting on the gdb to finish,
and try Thread.join() again.
With cpython's commit
commit a22be4943c119fecf5433d999227ff78fc2e5741
Author: Victor Stinner <vstinner@python.org>
Date: Mon Sep 27 14:20:31 2021 +0200
bpo-45274: Fix Thread._wait_for_tstate_lock() race condition (GH-28532)
this logic doesn't work anymore, because cpython internally marks the
thread as stopped when join() is interrupted with an exception. IMHO
this is broken in cpython and there is a bug report about this
https://github.com/python/cpython/issues/90882. Problem is that
waiting on a thread to finish is based on acquiring a lock. Meaning
join() is waiting on _tstate_lock. If this wait is interrupted, the
above referenced commit adds a logic that checks if the lock is help,
meaning the thread is done and marks the thread as stopped. But there is
no way to tell if the lock was acquired by us running join() or if it's
held by someone else e.g. still by the thread bootstrap code. Meaning
the thread is still running.
I may be missing something, but I don't see any reason why to spawn gdb
process within a thread. This change removes the thread and spawns gdb
directly. Instead waiting on a thread, we wait on the process to finish,
replacing join() with wait() and avoiding this problem.
Closes https://github.com/espressif/esp-idf/issues/11871
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Usage of `flushregs` in gdb will result in the following warning:
```
Warning: 'flushregs', an alias for the command 'maintenance flush register-cache', is deprecated.
Use 'maintenance flush register-cache'.
```
This patch updates to use the recommend command.
Signed-off-by: Daniel Mangum <georgedanielmangum@gmail.com>
Merges https://github.com/espressif/esp-idf/pull/11535
There are some register description errors in parts of rom/gpio.h
This commit update the incorrect comments in rom/gpio.h And now
esp_rom_gpio.h is recommend instead of rom/gpio.h. So this commit adds
macro SIG_GPIO_OUT_IDX in esp_rom_gpio_connect_out_signal
function and removes the reference to gpio.h in some source files.
Closes https://github.com/espressif/esp-idf/issues/11737
Verify that curses tool can be successfully imported on unix systems
When detected:
- installing esp-idf -> reinstall python environment
- using idf.py menuconfig -> raise error with hint message
Closes https://github.com/espressif/esp-idf/issues/11643
fix(hal/include): fix header violations in hal component
fix(hal/include): Move type definitions from `xx_hal.h` to `xx_types.h`
fix(hal/include): Move type definitions from `xx_hal.h` to `xx_types.h`
fix(hal/include): Add comment for a far away `#endif`
fix(hal/include): change scope for cpp guard
ci: Remove components/hal/ comment from public headers check exceptions
Add missing include macro sdkconfig.h for header files
Add missing include macro stdbool.h for header files
Add missing include macro stdint.h for header files
Add missing capability guard macro for header files
Add missing cpp guard macro for header files
Remove some useless include macros
Add some missing `inline` attribute for functions defined in header files
Remove components/hal/ from public headers check exceptions
fix(hal/include): fix invalid licenses
fix(hal/include): fix invalid licenses
fix(hal/include): add missing soc_caps.h
fix(hal): include soc_caps.h before cap macro is used
fix(hal): Remove unnecessary target check
fix(hal): fix header and macro problems
Add missing include macro
Remove loop dependency in hal
Add comment for far-away endif
fix(hal): Add missing soc_caps.h
ci: update check_copyright_ignore.txt
Change the sequence of `#include` macro, cpp guard macro
Change the wrap scope of capacity macro
fix(hal): Change position of C++ guard to pass test
- add hardware stack guard based on assist-debug module
- enable hardware stack guard by default
- disable hardware stack guard for freertos ci.release test
- refactor rtos_int_enter/rtos_int_exit to change SP register inside them
- fix panic_reason.h header for RISC-V
- update docs to include information about the new feature
Current version of the test is using "git-submodule foreach", which
requires submodules to be initialized. Non-initialized submodules are
ignored. Our CI is not performing submodule initialization, but instead
it only downloads the submodule content in tools/ci/ci_fetch_submodule.py
from cache and copies it into the submodule path.
Since we already know the submodule path from .gitconfig, we can use it
as argument to git-ls-tree and avoid calling git-submodule at all. This
allows to perform the test even if the submodules are not initialization
and also it makes the code simpler.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This uses the test from CI introduced in MR !23989. Even though the
original CI test is run with pytest, it doesn't use any pytest specific
code/features and it should never need them. So it make sense just to
re-use the code.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The script works the same way with httpd and softap mode, but it's a bit
confusing to provision the device over Ethernet or USB and call the
transport "softap". That's why we introduce an alias called httpd which
uses the same idea (http server with service name), but that service
runs on any interface (where the specified service is available)
As pointed out by Fu Hanxi, the pytest_build_system job is currently
using --parallel-index and --parallel-count, which are provided by
pytest-embedded, so we should not disable it. Moreover to properly
disable pytest-embedded we should use "no:pytest_embedded". Meaning
this probably was not working as indented anyway.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This adds SBOM information for submodules, which are not managed
by Espressif. Meaning there is no fork for them in the espressif
namespace. Other submodules should add sbom.yml manifest file to
the root of their git repository.
The SBOM information for submodules is stored in the .gitmodules file.
Each SBOM related variable has the "sbom-" prefix and the following
variables may be used:
sbom-version:
submodule version
sbom-cpe:
CPE record if available in NVD. This will be used by the SBOM
tool to check for possible submodule vulnerabilities. The
version in the CPE can be replaced with the "{}" placeholder,
which will be replaced by the "sbom-version" value from above.
sbom-supplier:
Person or organization who is providing the submodule.
It has to start with "Person:" or "Organization:" prefix
as required by the SPDX-2.2 standard.
sbom-url:
URL to the project if exists, e.g. github.
sbom-description:
Project description.
sbom-hash:
Submodule SHA as recorded in the git-tree. This field is used by
CI to check that the submodule checkout hash and info in .gitmodules
are in sync. IOW if submodule is updated and it has SBOM info in
.gitmodules, the .gitmodules has to be updated too. The test is
part of this commit. The checkout has of the submodule can be found
by using "git submodule status".
Example for micro-ecc submodule
---8<---
[submodule "components/bootloader/subproject/components/micro-ecc/micro-ecc"]
path = components/bootloader/subproject/components/micro-ecc/micro-ecc
url = ../../kmackay/micro-ecc.git
sbom-version = 1.0
sbom-cpe = cpe:2.3🅰️micro-ecc_project:micro-ecc:{}:*:*:*:*:*:*:*
sbom-supplier = Person: Ken MacKay
sbom-url = https://github.com/kmackay/micro-ecc
sbom-description = A small and fast ECDH and ECDSA implementation for 8-bit, 32-bit, and 64-bit processors
sbom-hash = d037ec89546fad14b5c4d5456c2e23a71e554966
---8<---
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This extends information provided in the project_description.json file.
Newly added information can be used in the SBOM generating tool and
also to improve hints regarding the the component dependency issues.
Added fields
version:
This adds versioning to the project_description.json file,
so it's easy to identify if it contains the required information.
project_version:
Can be used as a version for the resulting binary e.g. `hello_world.bin`.
idf_path:
This one is probably not necessary, but it allows tools to run even without
esp-idf environment exported(e.g. export.sh).
c_compiler:
The `CMAKE_C_COMPILER` value with full path to the compiler binary. This can
be used to get information about toolchain, which was used to build the project.
common_component_reqs:
List of common components as presented in cmake's __COMPONENT_REQUIRES_COMMON
and set in tools/cmake/build.cmake:__build_init().
build_component_info:
Detailed information about components used during build. It's a
dictionary with the component name as a key and each component has
a dictionary with detailed information. Following is an example for
the efuse component.
"efuse": {
"alias": "idf::efuse",
"target": "___idf_efuse",
"prefix": "idf",
"dir": "/home/fhrbata/work/esp-idf/components/efuse",
"type": "LIBRARY",
"lib": "__idf_efuse",
"reqs": [],
"priv_reqs": [ "bootloader_support", "soc", "spi_flash" ],
"managed_reqs": [],
"managed_priv_reqs": [],
"file": "/home/fhrbata/work/blink/build/esp-idf/efuse/libefuse.a",
"sources": [ "/home/fhrbata/work/esp-idf/components/efuse/esp32s3/esp_efuse_table.c", ... ],
"include_dirs": [ "include", "esp32s3/include" ]
}
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently make_json_list() returns '[ "" ]' for empty cmake list. Fix this
so empty json list is returned instead.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
- This features allows the NVS encryption keys to be derived and protected using
the HMAC peripheral. Since the encryption keys are derived at runtime, they
are not stored anywhere in the flash and hence this feature does not require
a separate `nvs_keys` partition.
This commit adds support for handling multiple requests simultaneously by introducing two new functions: `httpd_req_async_handler_begin()` and `httpd_req_async_handler_complete()`. These functions allow creating an asynchronous copy of a request that can be used on a separate thread and marking the asynchronous request as completed, respectively.
Additionally, a new flag `for_async_req` has been added to the `httpd_sess_t` struct to indicate if a socket is being used for an asynchronous request and should not be purged from the LRU cache.
An example have been added to demonstrate the usage of these new functions.
Closes https://github.com/espressif/esp-idf/issues/10594
Signed-off-by: Harshit Malpani <harshit.malpani@espressif.com>
esp_bootloader_format: Adds bootloader description structure to read bootloader version from app
Closes IDFGH-7206 and IDFGH-7573
See merge request espressif/esp-idf!21592
spi_flash: One more step for supporting flash suspend. 1. Support more esp chips 2. Improve real-time performance 3. Make timing more stable
See merge request espressif/esp-idf!22755
Currently hints are processed only once the process is finished and
exits with non-error exit code. In interactive mode, e.g. for monitor,
we want to process ouput lines for hints right away.
This adds a simple buffer, which keeps the last line and once EOL is
reached, it is processed for hints.
Since the original hints processing was file based, a new helper
function was added to allow processing hints in string.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
* reverted old faulty mechanism to set different
standard according to the toolchain
* Using -std=gnu++2b now for both gcc and clang
* Added a build test app to check the C++ standard in IDF
* Updated english docs to reflect the change to C++23
This commit refactors the "freertos" component's structure as follows:
- "FreeRTOSConfig.h" related files moved to "./config" directory
- Refactored CMakeLists.txt file in preparation for v10.5.1 upgrade
- Grouped list appends based on component organization
- Removed some unecessarily public "include_dirs"
- Removed FreeRTOS-openocd.c
- uxTopUsedPriority has been added back to tasks.c since v10.4.2
- Thus the workaround in FreeRTOS-openocd.c is no longer needed and can
be removed.
ADDITIONAL_MAKE_CLEAN_FILES is deprecated and only worked with make.
Replaced with the new ADDITIONAL_CLEAN_FILES (CMake 3.15) which also works with ninja.
Changed rv_utils_intr_edge_ack and esp_cpu_intr_edge_ack to
take uint32_t instead of int to avoid build errors.
The test is to test in particular that __builtin_ffsll, used in
xt_utils.h, which is included via esp_cpu.h, compiles fine
in C++20 with -Wsign-conversion enabled.
Closes https://github.com/espressif/esp-idf/pull/10895
* The unit tests are derived from the FreeRTOS test app
in components/freertos/test_apps/freertos. They are
quite incompatible with the main test application, which
is why they have been placed under
tools/test_apps/linux_compatible/linux_freertos for now.
This feature allows calling fsync even if the vfs component is not
used.
The second part of the commit adds an fsync call in the panic test app
enabling it to be used over usb-serial-jtag.
system: support USB_OTG CDC console on ESP32-S3
Closes IDF-2048, IDF-2987, IDFGH-7134, IDFGH-7291, and IDFGH-7835
See merge request espressif/esp-idf!19312
Used esp-rom tag esp32s3-20210327 and did manual cleanup.
Rename s_usb_osglue to rom_usb_osglue like it was done for esp32s2.
Some comments in esp32s2 headers are synced from esp32s3.
On Windows, when path is specified as absolute for cmdl argument, cmake can
interpret parts of the path as invalid escape chars. For example "C:\Users\..."
will result in "Invalid character escape '\U'." Externally specified
paths should be converted into cmake's representation, which uses '/'.
This can be done e.g. by using 'get_filename_component()'. Currently
there doesn't seem to be any problem with this, but let's add a test for
this.
Suggested-by: Ivan Grokhotkov <ivan@espressif.com>
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently the forced progress in RunTool is trying to fit the output
line into a terminal width, but it doesn't take into an account a
situation when the terminal width is reported as zero. This manifests
when running in docker image with redirected output and can be seen
in the github workflow output for esp-idf-ci-action.
docker run --rm -t my_ubuntu_esp python3 -c 'import os,sys; print(os.get_terminal_size(), sys.stdout.isatty())'
os.terminal_size(columns=238, lines=59) True
vs
docker run --rm -t my_ubuntu_esp python3 -c 'import os,sys; print(os.get_terminal_size(), sys.stdout.isatty())' | tee
os.terminal_size(columns=0, lines=0) True
Since the output is reported as tty and the terminal width as 0, the
fit_text_in_terminal() function returns empty string. I also verified this
by running idf.py build inside a testing docker image.
This fix adjusts the fit_text_in_terminal() function to return original
line if the terminal width is zero.
Also simplify the progress print and use same approach as ninja
https://github.com/ninja-build/ninja/blob/master/src/line_printer.cc#L66
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently if the IDF_TARGET env is set, and old sdkconfig exists with
different target value in CONFIG_IDF_TARGET, the set-target action fails
complaining about the IDF_TARGET env and value in sdkconfig being different.
We should ignore IDF_TARGET value from sdkconfig, because we are
actually setting new target and the old sdkconfig is renamed in cmake.
This can be easily reproduced with
---8<---
$ IDF_TARGET=esp32 idf.py set-target esp32
$ IDF_TARGET=esp32s3 idf.py set-target esp32s3
Project sdkconfig '/home/fhrbata/work/hello_world/sdkconfig' was generated
for target 'esp32s3', but environment variable IDF_TARGET is set to 'esp32'.
Run 'idf.py set-target esp32' to generate new sdkconfig file for target esp32.
---8<---
This also adds test for this use case to test_non_default_target.py.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This commit removes all idf_size.py files and references to them and adds esp-idf-size as a dependency and adequate wrappers to avoid breaking changes.
Two new tests are added.
1) test_check_python_dependencies
The test prepares artificial constraints file containing packages from
requirements.core.txt, which are also reported in pip-freeze output
for virtual env. The constraints file requires package versions higher
than currently installed in venv, so check_python_dependencies
should fail for all of them.
2) test_check_required_packages_only
Test for espressif/esp-idf/-/merge_requests/17917. After installing
core requirements, install additional foopackage, which is embedded.
Add version requirement for foopackage, which cannot be satisfied,
to constraints file. Since foopackage is not a direct requirement,
check-python-dependencies should not fail.
This also fixes existing TestCustomPythonPathInstall test, which sets
IDF_PYTHON_ENV_PATH, but does not restore it. Unittest seems to be
running tests in order based on class/test name. Meaning this test runs
before TestPythonInstall and all tests in TestPythonInstall are using
the latest tmpdir from TestCustomPythonPathInstall as IDF_PYTHON_ENV_PATH.
IOW TestPythonInstall is actually testing custom python env path, same as
TestCustomPythonPathInstall. This fixes it by restoring the
IDF_PYTHON_ENV_PATH. Note that since the actual IDF_PYTHON_ENV_PATH
was different(tmpdir) from PYTHON_DIR, the default PYTHON_DIR was never used,
so the tests were running with old python env(no
shutil.rmtree(PYTHON_DIR).
Since TestCustomPythonPathInstall is inheriting from TestPythonInstall
it also runs test_opt_argument and test_no_constraints for the second
time. This seems unnecessary, so this patch also skips these two tests
in TestCustomPythonPathInstall.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
esp_execute_shared_stack_function always restored the stack watchpoint
regardless of CONFIG_FREERTOS_WATCHPOINT_END_OF_STACK. This would lead
to an abondoned but active watchpoint on a former stack once the task
calling esp_execute_shared_stack_function is deleted, if
CONFIG_FREERTOS_WATCHPOINT_END_OF_STACK is inactive.
This has been fixed now.
Closes https://github.com/espressif/esp-idf/issues/10414
Update wifi lib with below -
1. Create NAN Discovery SM for beaconing & cluster formation
2. Create NAN interface for Tx/Rx of beacons & action frames
3. Add commands & events for NAN Services Publish/Subscribe/Followup
4. Add NAN Datapath definitions, Events, Peer structures
5. Support for forming and parsing of Datapath related attributes
6. Modules for NDP Req, Resp, Confirm, Term, Peer management
7. NAN Interface related additions in Datapath, Data Tx Q's
In addition include below changes -
1. Add netif and driver support for NAN Interface
2. Add simple examples for Publisher-Subscriber usecases
3. Add an advanced console example that supports commands
for NAN Discovery, Services & Datapath
4. Add wifi_apps for providing better NAN API's and Peer management
Co-authored-by: Shyamal Khachane <shyamal.khachane@espressif.com>
Currently the set-target has sdkconfig file name hardcoded to the
default one and doesn't honor custom config paths or names.
IMHO the only place where we can really now the config file name
is in cmake. But also the config should be really renamed only if
the set-target action is running.
This moves the config file renaming into cmake and it's performed only
when _IDF_PY_SET_TARGET_ACTION env. var. is set to 'ON'. This should
hopefully guarantee that it's really renamed only while set-target is
running.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Extend target checks in cmake, in case it's run directly and not via
idf.py or if idf.py misses something. This may happen
for example if cmake variables are set in project's CMakeLists.txt.
Some clean-ups are included along with the new checks and tests.
1. __target_check() function is removed. IIUC it should never fail,
because the selected target is explicitly passed as environmental
variable to kconfgen. Meaning the IDF_TARGET from environment variable may
not be actually used in kconfgen if IDF_TARGET is already set it cmake cache.
Note that the IDF_TARGET environment variable used for kconfgen is not
based on the actual IDF_TARGET environment variable set for idf.py, but
rather on the value set in __target_init() with
set(IDF_TARGET ${env_idf_target} CACHE STRING "IDF Build Target")
My understanding is that the original check was introduced to handle
situation, where IDF_TARGET was already set in cmake's cache and
the IDF_TARGET from environment variable was different. Since
the kconfgen would use the original environment variable(not
explicitly passed as it is now) the IDF_TARGET in cmake and in
sdkconfig could differ. IOW I think the original check was introduced
to cope with the following cmake behaviour
set(VARIABLE "value1" CACHE STRING "test variable")
set(VARIABLE "value2" CACHE STRING "test variable")
message("Variable value: ${VARIABLE}")
output: Variable value: value1
2. I scratched by head how it is possible that the following code
in __target_check()
if(NOT ${IDF_TARGET} STREQUAL ${env_idf_target})
could fail if IDF_TARGET is not set. For example in clean project
IDF_TARGET=esp32 idf.py reconfigure
Here env_idf_target==esp32 and IDF_TARGET is not set, so I would
expect that cmake will fail with error message that the cache
and env target do not match. The thing is that the variable
evaluation is done before the if command, so it actually
sees this
if(NOT STREQUAL esp32)
which is false and the error is not printed. It can be seen
with 'cmake --trace-expand' command. I don't know if this
was used on purpose or it worked just as a coincidence, but
I find it very confusing, so I added explicit check if the
IDF_TARGET is defined before the actual check. Same for
CMAKE_TOOLCHAIN_FILE.
3. Error messages are not formated(line-wrapped) by cmake's markup
so it's easier to check the output in tests.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Extend existing target consistency checks for the two following cases.
1. Target does not match currently used toolchain
$ IDF_TARGET=esp32s2 idf.py reconfigure
$ idf.py -DIDF_TARGET=esp32c3 build
2. Target is ambiguous, because it's specified also as env. var.
IDF_TARGET=esp32s3 idf.py set-target esp32c2
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The _guess_or_check_idf_target() function has sdkconfig and sdkconfig.defaults
file names hardcoded. Since config file names may be specified with SDKCONFIG
or SDKCONFIG_DEFAULTS cmake vars, directly in CMakeLists.txt or passed in with
the -D cmake option, they are not respected.
Problem is when SDKCONFIG or SDKCONFIG_DEFAULTS is set in
CMakeLists.txt. While idf can detect cmake vars passed through it
to cmake via the -D option, detecting SDKCONFIG and SDKCONFIG_DEFAULTS
vars settings in CMakeLists.txt would require to parse it. This seems
like error prone approach. Also if the vars defined by the -D option
are passed directly to cmake, not via idf, they will not be visible to idf.
It seems reasonable to move the logic into cmake, where we know the correct
SDKCONFIG and SDKCONFIG_DEFAULTS values. So the IDF_TARGET detection/guessing
is moved into targets.cmake, where the IDF_TARGET is actually set. The target
is guessed based on the following precendence.
1) $ENV{IDF_TARGET}
2) IDF_TARGET
3) SDKCONFIG
4) sdkconfig
5) SDKCONFIG_DEFAULTS if non-empty or
$ENV{SDKCONFIG_DEFAULTS} if non-empty or
sdkconfig.defaults
6) esp32
All config files referred in $ENV{SDKCONFIG_DEFAULTS} and SDKCONFIG_DEFAULTS
are searched, compared to the current behaviour. First target found in the
above chain is used.
The original _guess_or_check_idf_target() is renamed to _check_idf_target() and
used for the target consistency checks only.
The get_sdkconfig_filename() helper is now used to get the sdkconfig file
for consistency checks. It looks in SDKCONFIG specified with the -D
option and project_description.json.
With this change config full paths are reported in messages, so it's clear
e.g. from which config the target was guessed from or which config has
consistency problem. test_non_default_target.py was adjusted to this
change and also new test for testing the IDF_TARGET guessing was added.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Get project's current sdkconfig file name. It looks in SDKCONFIG cmake var
defined by the -D option and project_description.json. If not found return
default sdkconfig.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This parses cmakes cache vars defined on command line with -D options
into dictionary. It allows to simplify the check for new cache entries
and also can be re-used for other checks.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
1. Remove RTC_CLOCK_BBPLL_POWER_ON_WITH_USB Kconfig option
During sleep, BBPLL clock always gets disabled
esp_restart does not disable BBPLL clock, so that first stage bootloader log can be printed
2. Add a new Kconfig option PM_NO_AUTO_LS_ON_USJ_CONNECTED
When this option is selected, IDF will constantly monitor USB CDC port connection status.
As long as it gets connected to a HOST, automatic light-sleep will not happen.
Closes https://github.com/espressif/esp-idf/issues/8507
Pros:
- Using thread would face GIL issue and turns out very slow when running
with poor hardware.
Cons:
- Does not support windows anymore. For testing purpose, it's fine.
- Add support for esp32s2, esp32c3 and esp32c2 for the `memprot`-related tests
- Preliminary support for esp32s3 has also been added,
the test app will be enabled for esp32s3 later when
the memprot-related issues are fixed.
- Override panic handler to dump the violation intr status
- Dump the `memprot` violation registers before calling the
real panic handler
- Handle `Illegal Instruction` exception in case of memprot permission violation
* In esp32c3 with `memprot` enabled, if we try to execute arbitrary code
from RTC_FAST_MEM we get an `Illegal Instruction` exception from the panic
handler rather than a `Memory Protection Fault`.
* This is because the Illegal Instruction interrupt occurs earlier than the
memory protection interrupt due to a higher interrupt latency.
- Added minor improvements to `panic` test app
* Replaced existing API to disable flash cache which did not disabled cache always
(`esp_flash_default_chip->os_func->start(esp_flash_default_chip->os_func_data)`)
with `spi_flash_enable_interrupts_caches_and_other_cpu`
* Included some required headers explicitly (`esp_memory_utils.h` and `esp_heap_caps.h`)
IDF_PYTHON_ENV_PATH is the path where the Python environment is created
and used. By default it is inside IDF_TOOLS_PATH. IDF_PYTHON_ENV_PATH
was exported by idf_tools.py but was not imported back. This fixes the
issue and ESP-IDF will honor the value of IDF_PYTHON_ENV_PATH.
Closes https://github.com/espressif/esp-idf/issues/10489
This adds a new outdated option, which only lists outdated
packages installed in IDF_TOOLS_PATH. It searches for the
latest installed tool version in the IDF_TOOLS_PATH/tools path and
compares it against the latest available version in the tools.json
file. If the latest version of a tool installed in IDF_TOOLS_PATH/tools
is smaller, it's reported as outdated. Nothing is reported if the tool
is up to date.
Two new tests are added. First just checks if nothing is reported in
case there is no update available. The second artificially generates
new tools.json file called tools.outdated.json and sets XTENSA_ESP32_ELF
version to 'zzzzzz'. It then checks if the XTENSA_ESP32_ELF tool
is reported as outdated by the 'zzzzzz' version.
Description of the new outdated option is addedd to docs as well.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This moves one hint, which was hardcoded in debug_ext.py to
hints.yml and adds a new one when openocd process does not have
permissions to the USB JTAG/serial device.
Also hint replacing the original 'Please check JTAG connection!'
hardcoded message is added.
Suggested-by: Alexey Lapshin <alexey.lapshin@espressif.com>
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The debug targets are currently not utilizing hints, because they
are not using RunTool() helper from tools.py to spawn sub-processes.
Adjusting debug targets to use RunTool() would require some significant
changes to debug targets and RunTool() as well. Since debug targets
are already storing their output in logs, we can use these and process
them for hints.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
cpu retention: add riscv core sleep critical and non-critical register layout structure definition
cpu retention: add assembly subroutine for cpu critical register backup and restore
cpu retention: add cpu core critical register context backup and restore support
cpu retention: add cpu core non-critical register context backup and restore support
cpu retention: add interrupt priority register context backup and restore support
cpu retention: add cache config register context backup and restore support
cpu retention: add plic interrupt register context backup and restore support
cpu retention: add clint interrupt register context backup and restore support
cpu retention: wait icache state idle before pmu enter sleep
Examples for zigbee are using custom license, which is not present
on the SPDX license list. This was discussed on !16205 and a decision
was made that the examples will be put on check-copyright's ignore list.
SPDX has LicenseRef-[idstring] identifier for such cases, so let's try
to use it. In this particular case the LicenseRef-Included is used to
express that the full license text is included in the source file.
Note that the LicenseRef-Included is not part of SPDX. It's just
something I used. No change is needed on the check-copyright side.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently loading of esp32s3 ROM ELF symbols fails with
"Cannot access memory at address 0x3ff194ad". Let's perform
add-symbol-file after connection to the target.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This commit fixes an issue where paths on Windows are case insensitive, for instance when setting the build folder its name would be converted to lowercase.
The culprit is our realpath() function, that was calling os.path.normcase() internally, since we are removing that call it makes sense to just remove the function entirely and call os.path.realpath() wherever necessary.
Closes https://github.com/espressif/esp-idf/issues/10282
The main idf.py process has a handler for SIGINT, which actually just
ignores it. The get_default_serial_port() function is called within the
idf.py context to detect port for several tools(monitor,flash) and it's
not possible to terminate it because of this. Let's ignore SIGINT only
while running idf_monitor, which uses it to spawn gdb.
Fixes: c6e3eb0922 ("idf.py.exe changes to handle Ctrl+C in correct way. H..")
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This commit fixes two issues:
1. Part of the header containing annotations for curr, reference and diff values for archive diffs wasn't formatted correctly
2. Only one third of the values in the table were displayed because of the wrong line format
Since the introduction of PCAddressMatcher, the executable produced
by the build system is passed to elftools.elf.elffile.ELFFile.
However on macOS, native executables are not ELF files, so the
ELFFile class raises a rather unhelpful AssertionError exception.
Given that the rest of the idf_monitor.py doesn't have assumptions
that the "elf_file" argument is an ELF file (rather than just an
executable), check if the file is a real ELF file inside
PCAddressMatcher.
When multiple targets are specified, e.g. idf.py flash monitor, the
automatic port detection is performed twice. Keep the port value in
args.port and avoid multiple calls to get_default_serial_port().
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This commit migrates the esp_pm unit tests from the legacy unit test to a
stand alone test app. The following CI configurations are provided
- Default: Automatic light sleep with mostly default configurations
- Options: Enables all of the optional esp_pm features
- Limits: Limit tests esp_pm
Set CONFIG_NEWLIB_TIME_SYSCALL_USE_HRT to enable to force CONFIG_ESP_TIME_FUNCS_USE_RTC_TIMER to disable
and hence prevent the following tests from being executed:
- Timestamp after abort is correct in case RTC & High-res timer have + big error
- Timestamp after restart is correct in case RTC & High-res timer have + big error
- Timestamp after restart is correct in case RTC & High-res timer have - big error
This commit removes the dependency on portUSING_MPU_WRAPPERS on the Xtensa port
of IDF FreeRTOS. This dependency was added due to a hack implemented in the
upstream port that required the usage of the "xMPUSettings" member of the TCB.
The "xMPUSettings" would be used as a pointer to the task's coprocessor save
area on the stack, even though FreeRTOS MPU support was not available.
The hack has now been removed, and the CPSA pointer is now calculated using
a combination of constant offsets values and the pxEndOfStack member of the
TCB.
Note: This impelemtation was copied from the Xtensa port of Amazon SMP FreeRTOS.
Previously get_idf_build_env didn't include the default environment
(os.environ) in its output, so the environment only contained the
variables returned by "idf_tools.py export".
If PATH already contains all the right paths, "idf_tools.py export"
doesn't return the PATH variable.
Therefore the environment returned by get_idf_build_env was usually
incomplete. Fix by combining the output of "idf_tools.py export" with
os.environ.
esp_netif various fixes: dependency cleanup, default flags, CI rules
Closes IDF-5550, IDFGH-7023, IDF-1261, and IDF-2155
See merge request espressif/esp-idf!20301
It is required to define a mandatory dependency on lwip, so we
introduced esp_netif_stack component and made it require lwip, instead
of directly depending on lwip.
This enables building w-out lwip and support other TCP/IP stacks.
This commit refactors the OS startup functions as follows:
- Moved the OS/app startup functions listed below to "app_startup.c". Their
implementations are now common to all ports (RISC-V and Xtensa) of all
FreeRTOS implementations (IDF and Amazon SMP).
- esp_startup_start_app()
- esp_startup_start_app_other_cores()
- Removed esp_startup_start_app_common() as app startup functions are now
already common to all ports.
- Added extra logs to "main_task" to help with user debugging
Note: Increased startup delay on "unity_task". The "unity_run_menu()" is non
blocking, thus if the main task or other startup tasks have not been freed
by the time "unity_run_menu()" is run, those tasks will be freed the next time
"unity_task" blocks. This could cause some tests to have a memory leak, thus
the "unity_task" startup delay has increased.
