Currently the forced progress in RunTool is trying to fit the output
line into a terminal width, but it doesn't take into an account a
situation when the terminal width is reported as zero. This manifests
when running in docker image with redirected output and can be seen
in the github workflow output for esp-idf-ci-action.
docker run --rm -t my_ubuntu_esp python3 -c 'import os,sys; print(os.get_terminal_size(), sys.stdout.isatty())'
os.terminal_size(columns=238, lines=59) True
vs
docker run --rm -t my_ubuntu_esp python3 -c 'import os,sys; print(os.get_terminal_size(), sys.stdout.isatty())' | tee
os.terminal_size(columns=0, lines=0) True
Since the output is reported as tty and the terminal width as 0, the
fit_text_in_terminal() function returns empty string. I also verified this
by running idf.py build inside a testing docker image.
This fix adjusts the fit_text_in_terminal() function to return original
line if the terminal width is zero.
Also simplify the progress print and use same approach as ninja
https://github.com/ninja-build/ninja/blob/master/src/line_printer.cc#L66
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently if the IDF_TARGET env is set, and old sdkconfig exists with
different target value in CONFIG_IDF_TARGET, the set-target action fails
complaining about the IDF_TARGET env and value in sdkconfig being different.
We should ignore IDF_TARGET value from sdkconfig, because we are
actually setting new target and the old sdkconfig is renamed in cmake.
This can be easily reproduced with
---8<---
$ IDF_TARGET=esp32 idf.py set-target esp32
$ IDF_TARGET=esp32s3 idf.py set-target esp32s3
Project sdkconfig '/home/fhrbata/work/hello_world/sdkconfig' was generated
for target 'esp32s3', but environment variable IDF_TARGET is set to 'esp32'.
Run 'idf.py set-target esp32' to generate new sdkconfig file for target esp32.
---8<---
This also adds test for this use case to test_non_default_target.py.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently the set-target has sdkconfig file name hardcoded to the
default one and doesn't honor custom config paths or names.
IMHO the only place where we can really now the config file name
is in cmake. But also the config should be really renamed only if
the set-target action is running.
This moves the config file renaming into cmake and it's performed only
when _IDF_PY_SET_TARGET_ACTION env. var. is set to 'ON'. This should
hopefully guarantee that it's really renamed only while set-target is
running.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Extend existing target consistency checks for the two following cases.
1. Target does not match currently used toolchain
$ IDF_TARGET=esp32s2 idf.py reconfigure
$ idf.py -DIDF_TARGET=esp32c3 build
2. Target is ambiguous, because it's specified also as env. var.
IDF_TARGET=esp32s3 idf.py set-target esp32c2
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The _guess_or_check_idf_target() function has sdkconfig and sdkconfig.defaults
file names hardcoded. Since config file names may be specified with SDKCONFIG
or SDKCONFIG_DEFAULTS cmake vars, directly in CMakeLists.txt or passed in with
the -D cmake option, they are not respected.
Problem is when SDKCONFIG or SDKCONFIG_DEFAULTS is set in
CMakeLists.txt. While idf can detect cmake vars passed through it
to cmake via the -D option, detecting SDKCONFIG and SDKCONFIG_DEFAULTS
vars settings in CMakeLists.txt would require to parse it. This seems
like error prone approach. Also if the vars defined by the -D option
are passed directly to cmake, not via idf, they will not be visible to idf.
It seems reasonable to move the logic into cmake, where we know the correct
SDKCONFIG and SDKCONFIG_DEFAULTS values. So the IDF_TARGET detection/guessing
is moved into targets.cmake, where the IDF_TARGET is actually set. The target
is guessed based on the following precendence.
1) $ENV{IDF_TARGET}
2) IDF_TARGET
3) SDKCONFIG
4) sdkconfig
5) SDKCONFIG_DEFAULTS if non-empty or
$ENV{SDKCONFIG_DEFAULTS} if non-empty or
sdkconfig.defaults
6) esp32
All config files referred in $ENV{SDKCONFIG_DEFAULTS} and SDKCONFIG_DEFAULTS
are searched, compared to the current behaviour. First target found in the
above chain is used.
The original _guess_or_check_idf_target() is renamed to _check_idf_target() and
used for the target consistency checks only.
The get_sdkconfig_filename() helper is now used to get the sdkconfig file
for consistency checks. It looks in SDKCONFIG specified with the -D
option and project_description.json.
With this change config full paths are reported in messages, so it's clear
e.g. from which config the target was guessed from or which config has
consistency problem. test_non_default_target.py was adjusted to this
change and also new test for testing the IDF_TARGET guessing was added.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Get project's current sdkconfig file name. It looks in SDKCONFIG cmake var
defined by the -D option and project_description.json. If not found return
default sdkconfig.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This parses cmakes cache vars defined on command line with -D options
into dictionary. It allows to simplify the check for new cache entries
and also can be re-used for other checks.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This moves one hint, which was hardcoded in debug_ext.py to
hints.yml and adds a new one when openocd process does not have
permissions to the USB JTAG/serial device.
Also hint replacing the original 'Please check JTAG connection!'
hardcoded message is added.
Suggested-by: Alexey Lapshin <alexey.lapshin@espressif.com>
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The debug targets are currently not utilizing hints, because they
are not using RunTool() helper from tools.py to spawn sub-processes.
Adjusting debug targets to use RunTool() would require some significant
changes to debug targets and RunTool() as well. Since debug targets
are already storing their output in logs, we can use these and process
them for hints.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently loading of esp32s3 ROM ELF symbols fails with
"Cannot access memory at address 0x3ff194ad". Let's perform
add-symbol-file after connection to the target.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This commit fixes an issue where paths on Windows are case insensitive, for instance when setting the build folder its name would be converted to lowercase.
The culprit is our realpath() function, that was calling os.path.normcase() internally, since we are removing that call it makes sense to just remove the function entirely and call os.path.realpath() wherever necessary.
Closes https://github.com/espressif/esp-idf/issues/10282
The main idf.py process has a handler for SIGINT, which actually just
ignores it. The get_default_serial_port() function is called within the
idf.py context to detect port for several tools(monitor,flash) and it's
not possible to terminate it because of this. Let's ignore SIGINT only
while running idf_monitor, which uses it to spawn gdb.
Fixes: c6e3eb0922 ("idf.py.exe changes to handle Ctrl+C in correct way. H..")
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
When multiple targets are specified, e.g. idf.py flash monitor, the
automatic port detection is performed twice. Keep the port value in
args.port and avoid multiple calls to get_default_serial_port().
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
All the partition handling API functions and data-types were moved from the 'spi_flash' component to the new one named 'esp_partition'. See Storage 5.x migration guide for more details
This commit ads the corresponding --output-file option from idf_size.py to idf.py.
This is needed because piping the idf.py size output to a file would result in CMake output being present in the file as well.
The actual output from the build tool (CMake/Ninja) may or may not
contain color escape codes, depending on various factors. The output
written to the log file should never include color escape codes,
though. This is because color escape codes in files are usually not
rendered as "color" in editors, and complicate reading. Also escape
codes would break the regular expressions used to display hints for
compilation errors.
If stdout is a TTY (meaning that the output is not redirected), tell
the build tool (GNU Make or Ninja) to enable colorized output.
GNU Make and Ninja also check if their stdout is redirected and
strip color escape sequences in that case. CLICOLOR_FORCE environment
variable overrides this behavior.
With this change, if the compiler was launched with the
-fcolor-diagnostics flag and idf.py output is not redirected, the
final output in the terminal will be colorized.
(-fcolor-diagnostics is handled at CMake level by the previous commit)
This fixes the issue with build output not being colorized on Windows,
while the hints messages are colorized.
The issue occurred because sys.stdout and sys.stderr get overridden
by colorama.init() at runtime, but the default argument
output_stream=sys.stdout holds the reference to the"original"
sys.stdout.
colorama.init() (which, by the way, gets called via a curious chain
of imports, via idf_component_tools.manifest and tqdm package)
overrides standard streams, on Windows only. The overridden streams
contain logic to convert ANSI color codes into Windows Console API
calls to colorize the text.
Since read_and_write_stream function used the default value of
output_stream evaluated at module loading time, it was using the
original sys.stdout, not the one overridden by colorama.
One extra note is that while this does fix the coloring issue, the
solution is a bit fragile, as it relies on one of the following
(on Windows):
- colorama.init() is called (this can change if idf-component-manager
stops importing tqdm)
- Sufficiently new version of Windows 10 is used, and ANSI color codes
support is enabled in the Registry.
The actual output from the build tool (CMake/Ninja) may or may not
contain color escape codes, depending on various factors. The output
written to the log file should never include color escape codes,
though. This is because color escape codes in files are usually not
rendered as "color" in editors, and complicate reading. Also escape
codes would break the regular expressions used to display hints for
compilation errors.
If stdout is a TTY (meaning that the output is not redirected), tell
the build tool (GNU Make or Ninja) to enable colorized output.
GNU Make and Ninja also check if their stdout is redirected and
strip color escape sequences in that case. CLICOLOR_FORCE environment
variable overrides this behavior.
With this change, if the compiler was launched with the
-fcolor-diagnostics flag and idf.py output is not redirected, the
final output in the terminal will be colorized.
(-fcolor-diagnostics is handled at CMake level by the previous commit)
This adds CSV support to idf_size.py and idf.py size actions and using the --format argument which accepts 'text', 'json' or 'csv' as input.
idf_size.py --json argument is deprecated but left to avoid a breaking change.
For idf.py size actions OUTPUT_JSON environment variable set at configuration time is overriden at target build time if --format is used.
Additionally, this commit refactors big parts of code, unified usage of json_dict and manually generated dictionaries for textual output and improves code quality in many parts.
hints are added:
- to indicate that esp_cpu_ccount_t must be replaced by esp_cpu_cycle_count_t
- to indicate that esp_cpu_get/set_ccount must be replaced by esp_cpu_get/set_cycle_count
- to indicate that soc/cpu.h and compare_set.h were removed and user must include esp_cpu.h instead
- to indicate that esp_intr.h was removed and user must include esp_intr_alloc.h instead
- to indicate that esp_panic.h was made private and user must use the functionalities from esp_debug_helper.h instead
- to indicate that spilock.h, clk_ctrl_os.h and rtc_wdt.h must be included without the soc/
- to indicate that soc_log.h was renamed esp_hw_log.h and made private
Since both chips have built-in JTAG functionality, and there are no
official boards with FT2232H for these chips, use the built-in JTAG
by default.
To use them with esp-prog, set:
OPENOCD_COMMANDS="-f board/esp32c3-ftdi.cfg"
or pass this via the --openocd-commands argument to idf.py.
