docs: add API guide page about IDF Tools

This commit is contained in:
Ivan Grokhotkov 2020-01-17 19:38:50 +03:00
parent c1ab87b580
commit 1cd6f67679
9 changed files with 369 additions and 12 deletions

View File

@ -50,3 +50,7 @@ a.internal + code.descname::before {
a.internal + em::before {
content: ' ';
}
.tool-sha256 {
word-break: break-all;
}

View File

@ -151,6 +151,13 @@ toolchain_tmpdir = '{}/toolchain_inc'.format(builddir)
call_with_python('../gen-toolchain-links.py ../../tools/toolchain_versions.mk {} {}'.format(base_url, toolchain_tmpdir))
copy_if_modified(toolchain_tmpdir, '{}/inc'.format(builddir))
print("Generating IDF Tools list")
os.environ["IDF_MAINTAINER"] = "1"
tools_rst = os.path.join(builddir, 'idf-tools-inc.rst')
tools_rst_tmp = os.path.join(builddir, 'inc', 'idf-tools-inc.rst')
call_with_python("{}/tools/idf_tools.py gen-doc --output {}".format(idf_path, tools_rst_tmp))
copy_if_modified(tools_rst_tmp, tools_rst)
# http://stackoverflow.com/questions/12772927/specifying-an-online-image-in-sphinx-restructuredtext-format
#
suppress_warnings = ['image.nonlocal_uri']

View File

@ -0,0 +1,50 @@
.. This file gets included from auto-generated part of idf-tools.rst.
.. Comments "tool-NAME-notes" act as delimiters.
.. tool-xtensa-esp32-elf-notes
---
.. tool-xtensa-esp32s2-elf-notes
---
.. tool-esp32ulp-elf-notes
---
.. tool-esp32s2ulp-elf-notes
---
.. tool-openocd-esp32-notes
---
.. tool-cmake-notes
On Linux and macOS, it is recommended to install CMake using the OS-specific package manager (like apt, yum, brew, etc.). However, for convenience it is possible to install CMake using idf_tools.py along with the other tools.
---
.. tool-ninja-notes
On Linux and macOS, it is recommended to install ninja using the OS-specific package manager (like apt, yum, brew, etc.). However, for convenience it is possible to install ninja using idf_tools.py along with the other tools.
---
.. tool-idf-exe-notes
---
.. tool-ccache-notes
---

View File

@ -0,0 +1,141 @@
Downloadable Tools
==================
ESP-IDF build process relies on a number of tools: cross-compiler toolchains, CMake build system, and others.
Installing the tools using an OS-specific package manager (like apt, yum, brew, etc.) is the preferred method when the required version of the tool is available. This recommendation is reflected in the Getting Started guide. For example, on Linux and macOS it is recommended to install CMake using an OS package manager.
However, some of the tools are IDF-specific and are not available in OS package repositories. Furthermore, different versions of ESP-IDF require different versions of the tools to operate correctly. To solve these two problems, ESP-IDF provides a set of scripts for downloading and installing the correct versions of tools, and exposing them in the environment.
The rest of the document refers to these downloadable tools simply as "tools". Other kinds of tools used in ESP-IDF are:
* Python scripts bundled with ESP-IDF (such as ``idf.py``)
* Python packages installed from PyPI.
The following sections explain the installation method, and provide the list of tools installed on each platform.
.. note::
This document is provided for advanced users who need to customize their installation, users who wish to understand the installation process, and ESP-IDF developers.
If you are looking for instructions on how to install the tools, see the :doc:`Getting Started Guide <../../get-started/index>`.
Tools metadata file
-------------------
The list of tools and tool versions required for each platform is located in :idf_file:`tools/tools.json`. The schema of this file is defined by :idf_file:`tools/tools_schema.json`.
This file is used by :idf_file:`tools/idf_tools.py` script when installing the tools or setting up the environment variables.
.. _idf-tools-path:
Tools installation directory
----------------------------
``IDF_TOOLS_PATH`` environment variable specifies the location where the tools are to be downloaded and installed. If not set, ``IDF_TOOLS_PATH`` defaults to ``HOME/.espressif`` on Linux and macOS, and ``%USER_PROFILE%\.espressif`` on Windows.
Inside ``IDF_TOOLS_PATH``, the scripts performing tools installation create the following directories:
- ``dist`` — where the archives of the tools are downloaded.
- ``tools`` — where the tools are extracted. The tools are extracted into subdirectories: ``tools/TOOL_NAME/VERSION/``. This arrangement allows different versions of tools to be installed side by side.
``idf_tools.py`` script
-----------------------
:idf_file:`tools/idf_tools.py` script bundled with ESP-IDF performs several functions:
* ``install``: Download the tool into ``${IDF_TOOLS_PATH}/dist`` directory, extract it into ``${IDF_TOOLS_PATH}/tools/TOOL_NAME/VERSION``.
``install`` command accepts the list of tools to install, in ``TOOL_NAME`` or ``TOOL_NAME@VERSION`` format. If ``all`` is given, all the tools (required and optional ones) are installed. If no argument or ``required`` is given, only the required tools are installed.
* ``download``: Similar to ``install`` but doesn't extract the tools. An optional ``--platform`` argument may be used to download the tools for the specific platform.
* ``export``: Lists the environment variables which need to be set to use the installed tools. For most of the tools, setting ``PATH`` environment variable is sufficient, but some tools require extra environment variables.
The environment variables can be listed in either of ``shell`` or ``key-value`` formats, set by ``--format`` parameter:
- ``shell`` produces output suitable for evaluation in the shell. For example,
::
export PATH="/home/user/.espressif/tools/tool/v1.0.0/bin:$PATH"
on Linux and macOS, and
::
set "PATH=C:\Users\user\.espressif\tools\v1.0.0\bin;%PATH%"
on Windows.
.. note::
Exporting environment variables in Powershell format is not supported at the moment. ``key-value`` format may be used instead.
The output of this command may be used to update the environment variables, if the shell supports this. For example::
eval $($IDF_PATH/tools/idf_tools.py export)
- ``key-value`` produces output in `VARIABLE=VALUE` format, suitable for parsing by other scripts::
PATH=/home/user/.espressif/tools/tool/v1.0.0:$PATH
Note that the script consuming this output has to perform expansion of ``$VAR`` or ``%VAR%`` patterns found in the output.
* ``list``: Lists the known versions of the tools, and indicates which ones are installed.
* ``check``: For each tool, checks whether the tool is available in the system path and in ``IDF_TOOLS_PATH``.
.. _idf-tools-install:
Install scripts
---------------
Shell-specific user-facing scripts are provided in the root of ESP-IDF repository to facilitate tools installation. These are:
* ``install.bat`` for Windows Command Prompt
* ``install.ps1`` for Powershell
* ``install.sh`` for Bash
Aside from downloading and installing the tools into ``IDF_TOOLS_PATH``, these scripts prepare a Python virtual environment, and install the required packages into that environment.
.. _idf-tools-export:
Export scripts
--------------
Since the installed tools are not permanently added into the user or system ``PATH`` environment variable, an extra step is required to use them in the command line. The following scripts modify the environment variables in the current shell to make the correct versions of the tools available:
* ``export.bat`` for Windows Command Prompt
* ``export.ps1`` for Powershell
* ``export.sh`` for Bash
.. note::
To modify the shell environment in Bash, ``export.sh`` must be "sourced": ``. ./export.sh`` (note the leading dot and space).
``export.sh`` may be used with shells other than Bash (such as zsh). However in this case the ``IDF_PATH`` environment variable must be set before running the script. When used in Bash, the script will guess the ``IDF_PATH`` value from its own location.
In addition to calling ``idf_tools.py``, these scripts list the directories which have been added to the ``PATH``.
Other installation methods
--------------------------
Depending on the environment, more user-friendly wrappers for ``idf_tools.py`` are provided:
* :ref:`IDF Tools installer for Windows <get-started-windows-tools-installer>` can download and install the tools. Internally the installer uses ``idf_tools.py``.
* :doc:`Eclipse plugin for ESP-IDF <../../get-started/eclipse-setup>` includes a menu item to set up the tools. Internally the plugin calls ``idf_tools.py``.
* Visual Studio Code extension for ESP-IDF includes an onboarding flow. This flow helps setting up the tools. Although the extension does not rely on ``idf_tools.py``, the same installation method is used.
Custom installation
-------------------
Although the methods above are recommended for ESP-IDF users, they are not a must for building ESP-IDF applications. ESP-IDF build system expects that all the necessary tools are installed somewhere, and made available in the ``PATH``.
.. _idf-tools-list:
List of IDF Tools
-----------------
.. include:: /_build/inc/idf-tools-inc.rst

View File

@ -4,5 +4,6 @@ Tools
.. toctree::
:maxdepth: 1
IDF Tools <idf-tools>
IDF Monitor <idf-monitor>
IDF Docker image <idf-docker-image>

View File

@ -0,0 +1,52 @@
.. This file gets included from auto-generated part of idf-tools.rst.
.. Comments "tool-NAME-notes" act as delimiters.
.. This zh_CN version is a duplicate of en version, since includes are not expanded recursively
.. before processing :start-after: and :end-before: options.
.. tool-xtensa-esp32-elf-notes
---
.. tool-xtensa-esp32s2-elf-notes
---
.. tool-esp32ulp-elf-notes
---
.. tool-esp32s2ulp-elf-notes
---
.. tool-openocd-esp32-notes
---
.. tool-cmake-notes
On Linux and macOS, it is recommended to install CMake using the OS package manager. However, for convenience it is possible to install CMake using idf_tools.py along with the other tools.
---
.. tool-ninja-notes
On Linux and macOS, it is recommended to install ninja using the OS package manager. However, for convenience it is possible to install ninja using idf_tools.py along with the other tools.
---
.. tool-idf-exe-notes
---
.. tool-ccache-notes
---

View File

@ -0,0 +1 @@
.. include:: ../../../en/api-guides/tools/idf-tools.rst

View File

@ -4,5 +4,6 @@
.. toctree::
:maxdepth: 1
IDF Tools <idf-tools>
IDF 监视器 <idf-monitor>
IDF Docker image <idf-docker-image>

View File

@ -56,6 +56,11 @@ import functools
import copy
from collections import OrderedDict, namedtuple
try:
import typing # noqa: F401
except ImportError:
pass
try:
from urllib.request import urlretrieve
except ImportError:
@ -137,9 +142,9 @@ EXPORT_KEY_VALUE = 'key-value'
global_quiet = False
global_non_interactive = False
global_idf_path = None
global_idf_tools_path = None
global_tools_json = None
global_idf_path = None # type: typing.Optional[str]
global_idf_tools_path = None # type: typing.Optional[str]
global_tools_json = None # type: typing.Optional[str]
def fatal(text, *args):
@ -356,7 +361,7 @@ class IDFToolVersion(object):
def add_download(self, platform_name, url, size, sha256):
self.downloads[platform_name] = IDFToolDownload(platform_name, url, size, sha256)
def get_download_for_platform(self, platform_name):
def get_download_for_platform(self, platform_name): # type: (str) -> IDFToolDownload
if platform_name in PLATFORM_FROM_NAME.keys():
platform_name = PLATFORM_FROM_NAME[platform_name]
if platform_name in self.downloads.keys():
@ -368,6 +373,9 @@ class IDFToolVersion(object):
def compatible_with_platform(self, platform_name=PYTHON_PLATFORM):
return self.get_download_for_platform(platform_name) is not None
def get_supported_platforms(self): # type: () -> typing.Set[str]
return set(self.downloads.keys())
OPTIONS_LIST = ['version_cmd',
'version_regex',
@ -392,7 +400,7 @@ class IDFTool(object):
strip_container_dirs=0):
self.name = name
self.description = description
self.versions = OrderedDict()
self.versions = OrderedDict() # type: typing.Dict[str, IDFToolVersion]
self.version_in_path = None
self.versions_installed = []
if version_regex_replace is None:
@ -403,7 +411,7 @@ class IDFTool(object):
self._platform = CURRENT_PLATFORM
self._update_current_options()
def copy_for_platform(self, platform):
def copy_for_platform(self, platform): # type: (str) -> IDFTool
result = copy.deepcopy(self)
result._platform = platform
result._update_current_options()
@ -422,18 +430,18 @@ class IDFTool(object):
assert(type(version) is IDFToolVersion)
self.versions[version.version] = version
def get_path(self):
def get_path(self): # type: () -> str
return os.path.join(global_idf_tools_path, 'tools', self.name)
def get_path_for_version(self, version):
def get_path_for_version(self, version): # type: (str) -> str
assert(version in self.versions)
return os.path.join(self.get_path(), version)
def get_export_paths(self, version):
def get_export_paths(self, version): # type: (str) -> typing.List[str]
tool_path = self.get_path_for_version(version)
return [os.path.join(tool_path, *p) for p in self._current_options.export_paths]
def get_export_vars(self, version):
def get_export_vars(self, version): # type: (str) -> typing.Dict[str]
"""
Get the dictionary of environment variables to be exported, for the given version.
Expands:
@ -448,7 +456,7 @@ class IDFTool(object):
result[k] = v_repl
return result
def check_version(self, extra_paths=None):
def check_version(self, extra_paths=None): # type: (typing.Optional[typing.List[str]]) -> str
"""
Execute the tool, optionally prepending extra_paths to PATH,
extract the version string and return it as a result.
@ -481,6 +489,12 @@ class IDFTool(object):
def compatible_with_platform(self):
return any([v.compatible_with_platform() for v in self.versions.values()])
def get_supported_platforms(self): # type: () -> typing.Set[str]
result = set()
for v in self.versions.values():
result.update(v.get_supported_platforms())
return result
def get_recommended_version(self):
recommended_versions = [k for k, v in self.versions.items()
if v.status == IDFToolVersion.STATUS_RECOMMENDED
@ -792,7 +806,7 @@ class IDFTool(object):
return tool_json
def load_tools_info():
def load_tools_info(): # type: () -> typing.Dict[str, IDFTool]
"""
Load tools metadata from tools.json, return a dictionary: tool name - tool info
"""
@ -1267,6 +1281,87 @@ def action_validate(args):
# on failure, this will raise an exception with a fairly verbose diagnostic message
def action_gen_doc(args):
f = args.output
tools_info = load_tools_info()
def print_out(text):
f.write(text + '\n')
print_out(".. |zwsp| unicode:: U+200B")
print_out(" :trim:")
print_out("")
idf_gh_url = "https://github.com/espressif/esp-idf"
for tool_name, tool_obj in tools_info.items():
info_url = tool_obj.options.info_url
if idf_gh_url + "/tree" in info_url:
info_url = re.sub(idf_gh_url + r"/tree/\w+/(.*)", r":idf:`\1`", info_url)
license_url = "https://spdx.org/licenses/" + tool_obj.options.license
print_out("""
.. _tool-{name}:
{name}
{underline}
{description}
.. include:: idf-tools-notes.inc
:start-after: tool-{name}-notes
:end-before: ---
License: `{license} <{license_url}>`_
More info: {info_url}
.. list-table::
:widths: 10 10 80
:header-rows: 1
* - Platform
- Required
- Download
""".rstrip().format(name=tool_name,
underline=args.heading_underline_char * len(tool_name),
description=tool_obj.description,
license=tool_obj.options.license,
license_url=license_url,
info_url=info_url))
for platform_name in sorted(tool_obj.get_supported_platforms()):
platform_tool = tool_obj.copy_for_platform(platform_name)
install_type = platform_tool.get_install_type()
if install_type == IDFTool.INSTALL_NEVER:
continue
elif install_type == IDFTool.INSTALL_ALWAYS:
install_type_str = "required"
elif install_type == IDFTool.INSTALL_ON_REQUEST:
install_type_str = "optional"
else:
raise NotImplementedError()
version = platform_tool.get_recommended_version()
version_obj = platform_tool.versions[version]
download_obj = version_obj.get_download_for_platform(platform_name)
# Note: keep the list entries indented to the same number of columns
# as the list header above.
print_out("""
* - {}
- {}
- {}
.. rst-class:: tool-sha256
SHA256: {}
""".strip('\n').format(platform_name, install_type_str, download_obj.url, download_obj.sha256))
print_out('')
print_out('')
def main(argv):
parser = argparse.ArgumentParser()
@ -1328,6 +1423,11 @@ def main(argv):
subparsers.add_parser('validate', help='Validate tools.json against schema file')
gen_doc = subparsers.add_parser('gen-doc', help='Write the list of tools as a documentation page')
gen_doc.add_argument('--output', type=argparse.FileType('w'), default=sys.stdout,
help='Output file name')
gen_doc.add_argument('--heading-underline-char', help='Character to use when generating RST sections', default='~')
args = parser.parse_args(argv)
if args.action is None: