Commit Graph

62 Commits

Author SHA1 Message Date
Ivan Grokhotkov
54d4662107 feat(tools): idf_monitor: support for loadable hint provider modules
Currently hints are supported based on hints.yml only, which may be
limiting for some use cases. This introduces a generic plugin approach,
which allows to implement hint module that doesn't require entry in hints.yml.
Such module has the full command output available and it is not limited to
a single regex in hints.yml.

Note that regex in hint.yml expects the output concatenated into a single line,
but hint modules are getting the output unchanged.

Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
2023-08-05 20:13:28 +02:00
Frantisek Hrbata
ed35c3a94a feat(tools): add global build context
Introduces get_build_context() helper, which allows to get build context, e.g.
project description, at places where this info is not available. The
build context is set within ensure_build_directory.

Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
2023-08-05 20:13:28 +02:00
Peter Dragun
341390f928 fix(tools/hints): add guide to fix error with unsupported asyncio
Closes https://github.com/espressif/esp-idf/issues/11732
2023-07-03 10:26:26 +02:00
Alexey Lapshin
07c7fe929b tools: process progression for flash action output 2023-06-12 13:56:59 +04:00
Alexey Lapshin
59206e39a0 tools: fix control characters print if hints enabled
Closes https://github.com/espressif/esp-idf/issues/11351
2023-05-22 20:18:02 +08:00
Frantisek Hrbata
a5d3c2d443 tools: fix hints processing in interactive mode
Currently hints are processed only once the process is finished and
exits with non-error exit code. In interactive mode, e.g. for monitor,
we want to process ouput lines for hints right away.

This adds a simple buffer, which keeps the last line and once EOL is
reached, it is processed for hints.

Since the original hints processing was file based, a new helper
function was added to allow processing hints in string.

Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
2023-05-09 14:35:04 +02:00
Peter Dragun
bebd2cec4a bug(idf_monitor): fix color on windows with hints
Closes https://github.com/espressif/esp-idf/issues/9610
2023-04-17 11:18:42 +02:00
Roland Dobai
05a8d0b58a Merge branch 'bugfix/asyncio_progress_term_width' into 'master'
tools: fix progress output in asyncio.run with zero terminal width

Closes IDF-6646

See merge request espressif/esp-idf!22986
2023-04-06 15:23:39 +08:00
Frantisek Hrbata
84deb34035 tools: fix progress output in asyncio.run with zero terminal width
Currently the forced progress in RunTool is trying to fit the output
line into a terminal width, but it doesn't take into an account a
situation when the terminal width is reported as zero. This manifests
when running in docker image with redirected output and can be seen
in the github workflow output for esp-idf-ci-action.

docker run --rm -t my_ubuntu_esp python3 -c 'import os,sys; print(os.get_terminal_size(), sys.stdout.isatty())'
os.terminal_size(columns=238, lines=59) True

vs

docker run --rm -t my_ubuntu_esp python3 -c 'import os,sys; print(os.get_terminal_size(), sys.stdout.isatty())' | tee
os.terminal_size(columns=0, lines=0) True

Since the output is reported as tty and the terminal width as 0, the
fit_text_in_terminal() function returns empty string. I also verified this
by running idf.py build inside a testing docker image.

This fix adjusts the fit_text_in_terminal() function to return original
line if the terminal width is zero.

Also simplify the progress print and use same approach as ninja
https://github.com/ninja-build/ninja/blob/master/src/line_printer.cc#L66

Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
2023-03-29 17:12:36 +02:00
Frantisek Hrbata
9b731e5e35 tools: fix set-target if IDF_TARGET env is set and stale sdkconfig exists
Currently if the IDF_TARGET env is set, and old sdkconfig exists with
different target value in CONFIG_IDF_TARGET, the set-target action fails
complaining about the IDF_TARGET env and value in sdkconfig being different.
We should ignore IDF_TARGET value from sdkconfig, because we are
actually setting new target and the old sdkconfig is renamed in cmake.

This can be easily reproduced with

---8<---
$ IDF_TARGET=esp32 idf.py set-target esp32
$ IDF_TARGET=esp32s3 idf.py set-target esp32s3

Project sdkconfig '/home/fhrbata/work/hello_world/sdkconfig' was generated
for target 'esp32s3', but environment variable IDF_TARGET is set to 'esp32'.
Run 'idf.py set-target esp32' to generate new sdkconfig file for target esp32.
---8<---

This also adds test for this use case to test_non_default_target.py.

Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
2023-03-28 11:18:04 +02:00
Frantisek Hrbata
a558ad506e tools: fix custom sdkconfig renaming in set-target
Currently the set-target has sdkconfig file name hardcoded to the
default one and doesn't honor custom config paths or names.
IMHO the only place where we can really now the config file name
is in cmake. But also the config should be really renamed only if
the set-target action is running.

This moves the config file renaming into cmake and it's performed only
when _IDF_PY_SET_TARGET_ACTION env. var. is set to 'ON'. This should
hopefully guarantee that it's really renamed only while set-target is
running.

Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
2023-03-08 16:45:55 +01:00
Frantisek Hrbata
8e912faad1 tools: add target consistency checks for target specified with -D option
Extend existing target consistency checks for the two following cases.
1. Target does not match currently used toolchain
   $ IDF_TARGET=esp32s2 idf.py reconfigure
   $ idf.py -DIDF_TARGET=esp32c3 build

2. Target is ambiguous, because it's specified also as env. var.
   IDF_TARGET=esp32s3 idf.py set-target esp32c2

Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
2023-03-08 16:43:42 +01:00
Frantisek Hrbata
0d859f2786 tools: move target guessing into cmake
The _guess_or_check_idf_target() function has sdkconfig and sdkconfig.defaults
file names hardcoded. Since config file names may be specified with SDKCONFIG
or SDKCONFIG_DEFAULTS cmake vars, directly in CMakeLists.txt or passed in with
the -D cmake option, they are not respected.

Problem is when SDKCONFIG or SDKCONFIG_DEFAULTS is set in
CMakeLists.txt. While idf can detect cmake vars passed through it
to cmake via the -D option, detecting SDKCONFIG and SDKCONFIG_DEFAULTS
vars settings in CMakeLists.txt would require to parse it. This seems
like error prone approach. Also if the vars defined by the -D option
are passed directly to cmake, not via idf, they will not be visible to idf.

It seems reasonable to move the logic into cmake, where we know the correct
SDKCONFIG and SDKCONFIG_DEFAULTS values. So the IDF_TARGET detection/guessing
is moved into targets.cmake, where the IDF_TARGET is actually set. The target
is guessed based on the following precendence.

1) $ENV{IDF_TARGET}
2) IDF_TARGET
3) SDKCONFIG
4) sdkconfig
5) SDKCONFIG_DEFAULTS if non-empty or
   $ENV{SDKCONFIG_DEFAULTS} if non-empty or
   sdkconfig.defaults
6) esp32

All config files referred in $ENV{SDKCONFIG_DEFAULTS} and SDKCONFIG_DEFAULTS
are searched, compared to the current behaviour. First target found in the
above chain is used.

The original _guess_or_check_idf_target() is renamed to _check_idf_target() and
used for the target consistency checks only.

The get_sdkconfig_filename() helper is now used to get the sdkconfig file
for consistency checks. It looks in SDKCONFIG specified with the -D
option and project_description.json.

With this change config full paths are reported in messages, so it's clear
e.g. from which config the target was guessed from or which config has
consistency problem. test_non_default_target.py was adjusted to this
change and also new test for testing the IDF_TARGET guessing was added.

Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
2023-03-08 16:37:58 +01:00
Frantisek Hrbata
a8a4d7c66d tools: add get_sdkconfig_filename() helper
Get project's current sdkconfig file name. It looks in SDKCONFIG cmake var
defined by the -D option and project_description.json. If not found return
default sdkconfig.

Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
2023-03-08 16:37:58 +01:00
Frantisek Hrbata
e89f980193 tools: add _parse_cmdl_cmakecache() helper
This parses cmakes cache vars defined on command line with -D options
into dictionary. It allows to simplify the check for new cache entries
and also can be re-used for other checks.

Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
2023-03-08 16:37:58 +01:00
Djordje Nedic
17d1e9ae36 bugfix: Fix windows path case sensitivity
This commit fixes an issue where paths on Windows are case insensitive, for instance when setting the build folder its name would be converted to lowercase.

The culprit is our realpath() function, that was calling os.path.normcase() internally, since we are removing that call it makes sense to just remove the function entirely and call os.path.realpath() wherever necessary.

Closes https://github.com/espressif/esp-idf/issues/10282
2023-01-17 20:51:46 +01:00
Aleksei Apaseev
06ac2166d6 tools: added idf.py coredump sub-command 2022-10-28 03:01:56 +00:00
Roland Dobai
2083e4d359 Tools: Detect if the project is configured for a different Python
Closes https://github.com/espressif/esp-idf/issues/9993
2022-10-18 15:56:43 +02:00
Alexey Lapshin
c17a7f446c tools: move cmake executable check into function uses it
Before this change idf.py could exit with reason cmake does not exist in PATH
even cmake will not be executed by idf.py (e.g., 'idf.py gdb').
2022-09-21 22:39:03 +04:00
Roland Dobai
9307a932ad Tools: Handle IO error in idf.py output capturing
Closes https://github.com/espressif/esp-idf/issues/9649
2022-09-02 10:49:29 +02:00
simon.chupin
92ef2a4c83 Tools: Add unit tests for idf.py hints 2022-09-01 17:09:46 +02:00
Ivan Grokhotkov
13881a3832
idf.py: ensure that build log is always sanitized from color sequences
The actual output from the build tool (CMake/Ninja) may or may not
contain color escape codes, depending on various factors. The output
written to the log file should never include color escape codes,
though. This is because color escape codes in files are usually not
rendered as "color" in editors, and complicate reading. Also escape
codes would break the regular expressions used to display hints for
compilation errors.
2022-08-30 16:38:14 +02:00
Ivan Grokhotkov
d04b554065
tools: idf.py: enable CLICOLOR_FORCE for interactive builds
If stdout is a TTY (meaning that the output is not redirected), tell
the build tool (GNU Make or Ninja) to enable colorized output.

GNU Make and Ninja also check if their stdout is redirected and
strip color escape sequences in that case. CLICOLOR_FORCE environment
variable overrides this behavior.

With this change, if the compiler was launched with the
-fcolor-diagnostics flag and idf.py output is not redirected, the
final output in the terminal will be colorized.

(-fcolor-diagnostics is handled at CMake level by the previous commit)
2022-08-30 16:38:13 +02:00
Roland Dobai
cb81d14892 Merge branch 'bugfix/build_color_output_windows' into 'master'
tools: idf.py: use actual sys.stdout instead of the memoized one

See merge request espressif/esp-idf!19847
2022-08-30 16:31:42 +08:00
Ivan Grokhotkov
b548d9345e Revert "Merge branch 'feature/build_color_output' into 'master'"
This reverts merge request !19330
2022-08-30 15:14:47 +08:00
Ivan Grokhotkov
2c5b62f10e
tools: idf.py: use actual sys.stdout instead of the memoized one
This fixes the issue with build output not being colorized on Windows,
while the hints messages are colorized.

The issue occurred because sys.stdout and sys.stderr get overridden
by colorama.init() at runtime, but the default argument
output_stream=sys.stdout holds the reference to the"original"
sys.stdout.

colorama.init() (which, by the way, gets called via a curious chain
of imports, via idf_component_tools.manifest and tqdm package)
overrides standard streams, on Windows only. The overridden streams
contain logic to convert ANSI color codes into Windows Console API
calls to colorize the text.

Since read_and_write_stream function used the default value of
output_stream evaluated at module loading time, it was using the
original sys.stdout, not the one overridden by colorama.

One extra note is that while this does fix the coloring issue, the
solution is a bit fragile, as it relies on one of the following
(on Windows):
- colorama.init() is called (this can change if idf-component-manager
  stops importing tqdm)
- Sufficiently new version of Windows 10 is used, and ANSI color codes
  support is enabled in the Registry.
2022-08-30 01:30:32 +02:00
Ivan Grokhotkov
712743d777
idf.py: ensure that build log is always sanitized from color sequences
The actual output from the build tool (CMake/Ninja) may or may not
contain color escape codes, depending on various factors. The output
written to the log file should never include color escape codes,
though. This is because color escape codes in files are usually not
rendered as "color" in editors, and complicate reading. Also escape
codes would break the regular expressions used to display hints for
compilation errors.
2022-08-29 11:15:16 +02:00
Ivan Grokhotkov
12abe1f316
tools: idf.py: enable CLICOLOR_FORCE for interactive builds
If stdout is a TTY (meaning that the output is not redirected), tell
the build tool (GNU Make or Ninja) to enable colorized output.

GNU Make and Ninja also check if their stdout is redirected and
strip color escape sequences in that case. CLICOLOR_FORCE environment
variable overrides this behavior.

With this change, if the compiler was launched with the
-fcolor-diagnostics flag and idf.py output is not redirected, the
final output in the terminal will be colorized.

(-fcolor-diagnostics is handled at CMake level by the previous commit)
2022-08-29 11:15:00 +02:00
Roland Dobai
94a2d9f16a Merge branch 'bugfix/fix_hints_templates' into 'master'
tools: fix hints templates

See merge request espressif/esp-idf!19789
2022-08-29 16:43:14 +08:00
simon.chupin
e635492c95 tools: fix hints templates 2022-08-25 18:00:23 +02:00
simon.chupin
1cb4f8fdf4 tools: fix multi-byte character appearance in idf.py monitor 2022-08-23 13:25:38 +00:00
simon.chupin
cc1eb81b2e tools: add support of templates for hints 2022-08-18 13:37:32 +02:00
Roland Dobai
152ce8884b Tools: Fix idf.py hints to be enabled all the time and being able to disable them 2022-08-15 13:01:39 +02:00
simon.chupin
5468d79382 tools: fix idf.py monitor consol problem
Closes https://github.com/espressif/esp-idf/issues/9432
2022-07-27 08:35:30 +00:00
simon.chupin
a141be4add tools: Improve the error message for handling NotImplementedError on Windows 2022-07-18 19:03:55 +02:00
simon.chupin
cd10c328e5 tools: fix confserver out of limit error
Closes https://github.com/espressif/esp-idf/issues/9335
2022-07-12 11:27:03 +02:00
simon.chupin
2bf6ed4f2b idf.py: Fix IOError that costs CI failure 2022-07-04 08:15:23 +00:00
simon.chupin
43c69f0910 idf.py: Add automated hints on how to resolve errors 2022-07-04 08:15:23 +00:00
Roland Dobai
8dddb8b596 Revert "Merge branch 'feature/clippy' into 'master'"
This reverts merge request !16998
2022-06-29 16:46:47 +08:00
simon.chupin
c6a6eaeb60 idf.py: Add automated hints on how to resolve errors 2022-06-23 14:09:34 +02:00
simon.chupin
44f3c19fa9 tools: Add python types hints 2022-06-15 14:33:29 +02:00
simon.chupin
1c5e596633 idf.py: Change copyright in tools dir 2022-05-24 14:01:50 +02:00
simon.chupin
844dd5075f Tools: make idf_size work with overflow
Closes https://github.com/espressif/esp-idf/issues/6914
closes https://github.com/espressif/esp-idf/issues/4234
2021-09-16 14:12:46 +02:00
Simon
0b9afa509f tools: Add docs subcommand docs which opens browser with documentation for current version of idf and model of your target
Add docs subcommand with options:
--no-browser - Prints url for current documentation instead of opens browser with documentation
--language - Select documentation language
--starting-page - Choice section of documentation to open
--version - Choice version of esp-idf
--target - Choice model of your target
2021-07-12 10:56:37 +02:00
Roland Dobai
48488f1683 tools: Use git describe --match for filtering out internal names 2021-02-23 06:55:21 +00:00
Fu Hanxi
0146f258d7 style: format python files with isort and double-quote-string-fixer 2021-01-26 10:49:01 +08:00
Angus Gratton
ad295037a8 idf.py: Disable CMake --warn-uninitialized option by default
Can still be enabled by passing --cmake-warn-uninitialized on the command line

Prevents CMake warnings printed by default if IDF_PATH is underneath the CMake
project directory.

The reason for this is that CMake --warn-uninitialized only enables checks
inside the project directory (ie top-level CMakeLists.txt directory and
subdirectories), it doesn't enable for files included from other directories.

(The only way to enable warnings in other directories is to pass
--check-system-dirs and this looks like it's only useful for CMake's own
developers as it prints a lot of warnings from inside CMake otherwise - see
https://gitlab.kitware.com/cmake/cmake/-/issues/19645 )

Plan to follow up with a later commit to clean up most of the warnings (which
aren't problems for CMake execution), but we'll also disable this option by
default to avoid this unexpected triggering of IDF warnings.
2020-08-26 09:53:56 +10:00
Marius Vikhammer
30f075f0bb idf monitor: fix crash when monitor baudrate argument set
Closes IDF-1902
2020-07-10 10:29:48 +08:00
Angus Gratton
f9552baf97 Merge branch 'feature/dfu_build_flash_s2' into 'master'
Add build system support for programming ESP32-S2 using DFU utils

Closes IDF-579

See merge request espressif/esp-idf!8294
2020-05-04 14:57:51 +08:00
Roland Dobai
ab9f714248 Add build system support for programming ESP32-S2 using DFU utils 2020-04-30 07:59:18 +02:00