idf.py spawns gdb process within a thread and uses Thread.join() to wait
for the gdb process to finish. As CTRL+C(SIGINT) is used by gdb to interrupt the
running program, we catch the SIGINT while waiting on the gdb to finish,
and try Thread.join() again.
With cpython's commit
commit a22be4943c119fecf5433d999227ff78fc2e5741
Author: Victor Stinner <vstinner@python.org>
Date: Mon Sep 27 14:20:31 2021 +0200
bpo-45274: Fix Thread._wait_for_tstate_lock() race condition (GH-28532)
this logic doesn't work anymore, because cpython internally marks the
thread as stopped when join() is interrupted with an exception. IMHO
this is broken in cpython and there is a bug report about this
https://github.com/python/cpython/issues/90882. Problem is that
waiting on a thread to finish is based on acquiring a lock. Meaning
join() is waiting on _tstate_lock. If this wait is interrupted, the
above referenced commit adds a logic that checks if the lock is help,
meaning the thread is done and marks the thread as stopped. But there is
no way to tell if the lock was acquired by us running join() or if it's
held by someone else e.g. still by the thread bootstrap code. Meaning
the thread is still running.
I may be missing something, but I don't see any reason why to spawn gdb
process within a thread. This change removes the thread and spawns gdb
directly. Instead waiting on a thread, we wait on the process to finish,
replacing join() with wait() and avoiding this problem.
Closes https://github.com/espressif/esp-idf/issues/11871
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently hints are processed only once the process is finished and
exits with non-error exit code. In interactive mode, e.g. for monitor,
we want to process ouput lines for hints right away.
This adds a simple buffer, which keeps the last line and once EOL is
reached, it is processed for hints.
Since the original hints processing was file based, a new helper
function was added to allow processing hints in string.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This is partial backport of following commit, which changes
print_hints to generate_hints.
commit 92ef2a4c83
Author: simon.chupin <simon.chupin@espressif.com>
Date: Tue Aug 9 15:39:23 2022 +0200
Tools: Add unit tests for idf.py hints
Only hunks for core_ext.py and tools.py are picked. It would be possible
to use the original print_hints approach, where the hints are directly
printed out and not returned, but it seems to make sense to keep it
closer to most recent version. It should make further backports easier
and it allows to cherry pick the iterative hints approach.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently the forced progress in RunTool is trying to fit the output
line into a terminal width, but it doesn't take into an account a
situation when the terminal width is reported as zero. This manifests
when running in docker image with redirected output and can be seen
in the github workflow output for esp-idf-ci-action.
docker run --rm -t my_ubuntu_esp python3 -c 'import os,sys; print(os.get_terminal_size(), sys.stdout.isatty())'
os.terminal_size(columns=238, lines=59) True
vs
docker run --rm -t my_ubuntu_esp python3 -c 'import os,sys; print(os.get_terminal_size(), sys.stdout.isatty())' | tee
os.terminal_size(columns=0, lines=0) True
Since the output is reported as tty and the terminal width as 0, the
fit_text_in_terminal() function returns empty string. I also verified this
by running idf.py build inside a testing docker image.
This fix adjusts the fit_text_in_terminal() function to return original
line if the terminal width is zero.
Also simplify the progress print and use same approach as ninja
https://github.com/ninja-build/ninja/blob/master/src/line_printer.cc#L66
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently loading of esp32s3 ROM ELF symbols fails with
"Cannot access memory at address 0x3ff194ad". Let's perform
add-symbol-file after connection to the target.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This commit fixes an issue where paths on Windows are case insensitive, for instance when setting the build folder its name would be converted to lowercase.
The culprit is our realpath() function, that was calling os.path.normcase() internally, since we are removing that call it makes sense to just remove the function entirely and call os.path.realpath() wherever necessary.
Closes https://github.com/espressif/esp-idf/issues/10282
All the partition handling API functions and data-types were moved from the 'spi_flash' component to the new one named 'esp_partition'. See Storage 5.x migration guide for more details
The actual output from the build tool (CMake/Ninja) may or may not
contain color escape codes, depending on various factors. The output
written to the log file should never include color escape codes,
though. This is because color escape codes in files are usually not
rendered as "color" in editors, and complicate reading. Also escape
codes would break the regular expressions used to display hints for
compilation errors.
If stdout is a TTY (meaning that the output is not redirected), tell
the build tool (GNU Make or Ninja) to enable colorized output.
GNU Make and Ninja also check if their stdout is redirected and
strip color escape sequences in that case. CLICOLOR_FORCE environment
variable overrides this behavior.
With this change, if the compiler was launched with the
-fcolor-diagnostics flag and idf.py output is not redirected, the
final output in the terminal will be colorized.
(-fcolor-diagnostics is handled at CMake level by the previous commit)
This fixes the issue with build output not being colorized on Windows,
while the hints messages are colorized.
The issue occurred because sys.stdout and sys.stderr get overridden
by colorama.init() at runtime, but the default argument
output_stream=sys.stdout holds the reference to the"original"
sys.stdout.
colorama.init() (which, by the way, gets called via a curious chain
of imports, via idf_component_tools.manifest and tqdm package)
overrides standard streams, on Windows only. The overridden streams
contain logic to convert ANSI color codes into Windows Console API
calls to colorize the text.
Since read_and_write_stream function used the default value of
output_stream evaluated at module loading time, it was using the
original sys.stdout, not the one overridden by colorama.
One extra note is that while this does fix the coloring issue, the
solution is a bit fragile, as it relies on one of the following
(on Windows):
- colorama.init() is called (this can change if idf-component-manager
stops importing tqdm)
- Sufficiently new version of Windows 10 is used, and ANSI color codes
support is enabled in the Registry.
Since both chips have built-in JTAG functionality, and there are no
official boards with FT2232H for these chips, use the built-in JTAG
by default.
To use them with esp-prog, set:
OPENOCD_COMMANDS="-f board/esp32c3-ftdi.cfg"
or pass this via the --openocd-commands argument to idf.py.
