Merge branch 'feature/prefer_python3_in_installer' into 'master'

tools: Prefer python3 during install and export

Closes IDF-2682, IDFGH-4659, and IDFGH-4726

See merge request espressif/esp-idf!12250

docs/en/get-started/linux-setup.rst

@@ -15,7 +15,7 @@ To compile with ESP-IDF you need to get the following packages. The command to r
 
 - CentOS 7 & 8::
 
-    sudo yum -y update && sudo yum install git wget flex bison gperf python3 cmake ninja-build ccache dfu-util libusbx
+    sudo yum -y update && sudo yum install git wget flex bison gperf python3 python3-pip python3-setuptools cmake ninja-build ccache dfu-util libusbx
 
 CentOS 7 is still supported but CentOS version 8 is recommended for a better user experience.
 
@@ -35,33 +35,6 @@ Permission issues /dev/ttyUSB0
 
 With some Linux distributions you may get the ``Failed to open port /dev/ttyUSB0`` error message when flashing the {IDF_TARGET_NAME}. :ref:`This can be solved by adding the current user to the dialout group<linux-dialout-group>`.
 
-Setting up Python 3 as default for CentOS
------------------------------------------
-
-CentOS 7 and older is providing Python 2.7 as the default interpreter.
-Python 3 is recommended instead and can be installed in old distributions as follows, or please consult the documentation of your operating system for other recommended ways to achieve this::
-
-    sudo yum -y update && sudo yum install python3 python3-pip python3-setuptools
-
-Making Python 3 the default interpreter is possible by running::
-
-    sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 10 && alias pip=pip3
-
-Setting up Python 3 as default for Ubuntu and Debian
-----------------------------------------------------
-
-Ubuntu (version 18.04 and older) and Debian (version 9 and older) are still providing Python 2.7 as the default interpreter.
-Python 3 is recommended instead and can be installed in old distributions as follows, or please consult the documentation of your operating system for other recommended ways to achieve this::
-
-    sudo apt-get install python3 python3-pip python3-setuptools
-
-Making Python 3 the default interpreter is possible by running::
-
-    sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 10 && alias pip=pip3
-
-.. note::
-    This is system-wide change which may affect all of the applications.
-
 Fixing broken pip on Ubuntu 16.04
 =================================
 

docs/en/get-started/macos-setup.rst

@@ -34,8 +34,8 @@ ESP-IDF will use the version of Python installed by default on macOS.
 
    Then you will need to install the XCode command line tools to continue. You can install these by running ``xcode-select --install``.
 
-Installing and setting up Python 3 as default
----------------------------------------------
+Installing Python 3
+-------------------
 
 Basing on macOS `Catalina 10.15 release notes`_, use of Python 2.7 is not recommended and Python 2.7 will not be included by default in future versions of macOS. Check what Python you currently have::
 
@@ -47,30 +47,15 @@ If the output is like ``Python 2.7.17``, your default interpreter is Python 2.7.
 
 If above command returns an error, it means Python 3 is not installed. 
 
-Below is an overview of steps to install Python 3 and making it default interpreter. 
+Below is an overview of steps to install Python 3.
 
   - Installing with HomeBrew_ can be done as follows::
 
       brew install python3
-      ln -s /usr/local/bin/python3.8 /usr/local/bin/python
-
-    Adjust above directory name ``/usr/local/bin/python3.8`` to point where Python 3 has been installed. To check this directory you can run  ``which -a python3``.
 
   - If you have MacPorts_, you can run::
 
       sudo port install python38
-      sudo port select --set python python38
-
-Now open a new terminal session to verify what python is running by default::
-
-  python --version
-
-If the output is similar to ``Python 3.8.5``, your installation has been done successfully.
-
-
-.. note::
-
-    This is system-wide change which may affect all of the applications.
 
 Python 2 deprecation
 ====================

docs/zh_CN/get-started/linux-setup.rst

@@ -15,7 +15,7 @@ Linux 平台工具链的标准设置
 
 - CentOS 7 & 8::
 
-    sudo yum -y update && sudo yum install git wget flex bison gperf python3 cmake ninja-build ccache dfu-util libusbx
+    sudo yum -y update && sudo yum install git wget flex bison gperf python3 python3-pip python3-setuptools cmake ninja-build ccache dfu-util libusbx
 
 目前仍然支持 CentOS 7，但为了更好的用户体验，建议使用 CentOS 8。
 
@@ -35,33 +35,6 @@ Linux 平台工具链的标准设置
 
 使用某些 Linux 版本向 {IDF_TARGET_NAME} 烧录固件时，可能会出现 ``Failed to open port /dev/ttyUSB0`` 错误消息。此时可以将用户添加至 :ref:`Linux Dialout 组<linux-dialout-group>`。
 
-设置 Python 3 为 CentOS 默认 Python 版本
-----------------------------------------------------
-
-CentOS 7 及更早的版本提供 Python 2.7 作为默认解释器。但这里推荐使用 Python 3，您可以运行下方命令安装 Python 3。或者查看当前所用系统的相关文档，按照文档推荐的其它方法安装 Python 3::
-
-    sudo yum -y update && sudo yum install python3 python3-pip python3-setuptools
-
-设置 Python 3 为默认 Python 版本::
-
-    sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 10 && alias pip=pip3
-
-
-设置 Python 3 为 Ubuntu 和 Debian 默认 Python 版本
-----------------------------------------------------
-
-Ubuntu（v18.04 及之前的版本）和 Debian（v9 及之前的版本）的默认解释器为 Python 2.7，但这里推荐使用 Python 3，您可以运行下方命令安装 Python 3。或者查看当前所用系统的相关文档，按照文档推荐的其它方法安装 Python 3::
-
-    sudo apt-get install python3 python3-pip python3-setuptools
-
-设置 Python 3 为默认 Python 版本::
-
-    sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 10 && alias pip=pip3
-
-.. 注解::
-    上述设置为全局设置，同时会影响到其它应用。
-
-
 修复 Ubuntu 16.04 损坏的 pip 
 =================================
 

docs/zh_CN/get-started/macos-setup.rst

@@ -28,14 +28,13 @@ ESP-IDF 将使用 macOS 上默认安装的 Python 版本。
 - 强烈建议同时安装 ccache_ 以获得更快的编译速度。如有 HomeBrew_，可通过 MacPorts_ 上的 ``brew install ccache`` 或 ``sudo port install ccache`` 完成安装。
 
 .. note::   
-    
     如您在上述任何步骤中遇到以下错误::
 
     ``xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at:/Library/Developer/CommandLineTools/usr/bin/xcrun``
 
     则必须安装 XCode 命令行工具，具体可运行 ``xcode-select --install``。
 
-安装并设置 Python 3 为默认版本
+安装 Python 3
 ---------------------------------------------
 
 `Catalina 10.15 发布说明`_ 中表示不推荐使用 Python 2.7 版本，在未来的 macOS 版本中也不会默认包含 Python 2.7。执行以下命令来检查您当前使用的 Python 版本::
@@ -48,31 +47,15 @@ ESP-IDF 将使用 macOS 上默认安装的 Python 版本。
 
 如果运行上述命令出现错误，则代表电脑上没有安装 Python 3。
 
-请根据以下步骤安装 Python 3 并使其成为默认解释器：
+请根据以下步骤安装 Python 3：
 
   - 使用 HomeBrew_ 进行安装的方法如下::
 
       brew install python3
-      ln -s /usr/local/bin/python3.8 /usr/local/bin/python
-
-    将上述的目录名 ``/usr/local/bin/python3.8`` 修改为 Python 3 所在的目录。您可以运行 ``which -a python3`` 来查看 Python 3 所在的目录。
 
   - 使用 MacPorts_ 进行安装的方法如下::
 
       sudo port install python38
-      sudo port select --set python python38
-
-现在您可以打开终端窗口验证默认运行的 Python 版本::
-
-  python --version
-
-如果输出结果类似于 ``Python 3.8.5`` 则代表安装成功。
-
-.. 注解::
-
-    上述设置为全局设置，同时会影响到其它应用。
-
-
 
 停用 Python 2 
 ====================

export.fish

@@ -8,11 +8,14 @@ function idf_export_main
 
     set oldpath = $PATH
 
+    echo "Detecting the Python interpreter"
+    source "$IDF_PATH"/tools/detect_python.fish
+
     echo "Adding ESP-IDF tools to PATH..."
     # Call idf_tools.py to export tool paths
     set -x IDF_TOOLS_EXPORT_CMD "$IDF_PATH"/export.fish
     set -x IDF_TOOLS_INSTALL_CMD "$IDF_PATH"/install.fish
-    set idf_exports ("$IDF_PATH"/tools/idf_tools.py export) || return 1
+    set idf_exports ("$ESP_PYTHON" "$IDF_PATH"/tools/idf_tools.py export) || return 1
     eval "$idf_exports"
 
     echo "Checking if Python packages are up to date..."
@@ -50,6 +53,7 @@ function idf_export_main
     set -e path_entry
     set -e IDF_ADD_PATHS_EXTRAS
     set -e idf_exports
+    set -e ESP_PYTHON
 
     # Not unsetting IDF_PYTHON_ENV_PATH, it can be used by IDF build system
     # to check whether we are using a private Python environment

export.sh

@@ -79,11 +79,14 @@ idf_export_main() {
 
     old_path="$PATH"
 
+    echo "Detecting the Python interpreter"
+    . "${IDF_PATH}/tools/detect_python.sh"
+
     echo "Adding ESP-IDF tools to PATH..."
     # Call idf_tools.py to export tool paths
     export IDF_TOOLS_EXPORT_CMD=${IDF_PATH}/export.sh
     export IDF_TOOLS_INSTALL_CMD=${IDF_PATH}/install.sh
-    idf_exports=$("${IDF_PATH}/tools/idf_tools.py" export) || return 1
+    idf_exports=$("$ESP_PYTHON" "${IDF_PATH}/tools/idf_tools.py" export) || return 1
     eval "${idf_exports}"
 
     echo "Using Python interpreter in $(which python)"
@@ -125,6 +128,7 @@ idf_export_main() {
     unset path_entry
     unset IDF_ADD_PATHS_EXTRAS
     unset idf_exports
+    unset ESP_PYTHON
 
     # Not unsetting IDF_PYTHON_ENV_PATH, it can be used by IDF build system
     # to check whether we are using a private Python environment

install.fish

@@ -4,11 +4,14 @@ set basedir $PWD
 
 set -x IDF_PATH $basedir
 
+echo "Detecting the Python interpreter"
+source "$IDF_PATH"/tools/detect_python.fish
+
 echo "Installing ESP-IDF tools"
-"$IDF_PATH"/tools/idf_tools.py install
+"$ESP_PYTHON" "$IDF_PATH"/tools/idf_tools.py install
 
 echo "Installing Python environment and packages"
-"$IDF_PATH"/tools/idf_tools.py install-python-env
+"$ESP_PYTHON" "$IDF_PATH"/tools/idf_tools.py install-python-env
 
 echo "All done! You can now run:"
 echo ""

install.sh

@@ -5,11 +5,14 @@ set -u
 
 export IDF_PATH=$(cd $(dirname $0); pwd)
 
+echo "Detecting the Python interpreter"
+. ${IDF_PATH}/tools/detect_python.sh
+
 echo "Installing ESP-IDF tools"
-${IDF_PATH}/tools/idf_tools.py install
+${ESP_PYTHON} ${IDF_PATH}/tools/idf_tools.py install
 
 echo "Installing Python environment and packages"
-${IDF_PATH}/tools/idf_tools.py install-python-env
+${ESP_PYTHON} ${IDF_PATH}/tools/idf_tools.py install-python-env
 
 basedir="$(dirname $0)"
 echo "All done! You can now run:"

tools/ci/config/host-test.yml

@@ -344,3 +344,16 @@ test_autocomplete:
     expire_in: 1 week
   script:
     - ${IDF_PATH}/tools/ci/multirun_with_pyenv.sh ${IDF_PATH}/tools/ci/test_autocomplete.py
+
+test_detect_python:
+  extends: .host_test_template
+  image: $CI_DOCKER_REGISTRY/linux-shells:1
+  script:
+    - cd ${IDF_PATH}
+    - shellcheck -s sh tools/detect_python.sh
+    - shellcheck -s bash tools/detect_python.sh
+    - shellcheck -s dash tools/detect_python.sh
+    - "bash -c '. tools/detect_python.sh && echo Our Python: ${ESP_PYTHON?Python is not set}'"
+    - "dash -c '. tools/detect_python.sh && echo Our Python: ${ESP_PYTHON?Python is not set}'"
+    - "zsh -c '. tools/detect_python.sh && echo Our Python: ${ESP_PYTHON?Python is not set}'"
+    - "fish -c 'source tools/detect_python.fish && echo Our Python: $ESP_PYTHON'"

tools/detect_python.fish

@@ -0,0 +1,25 @@
+# This file should be sourced, not executed!
+#
+# This is a port of detect_python.sh. More information are provided there.
+
+set -x ESP_PYTHON python
+
+for p_cmd in python python3;
+    echo "Checking \"$p_cmd\" ..."
+
+    set res ($p_cmd -c "import sys; print(sys.version_info.major)")
+    if [ "$res" = "3" ]
+        set -x ESP_PYTHON $p_cmd
+        break
+    end
+end
+
+$ESP_PYTHON --version
+if [ $status -ne 0 ]
+    echo "\"$ESP_PYTHON\" is not installed! Please see the documentation for how to install it."
+    # The following exit skips the rest of this file but won't exit fish where the script was sourced. This is not a
+    # fatal issue.
+    exit 1
+end
+
+echo "\"$ESP_PYTHON\" has been detected"

tools/detect_python.sh

@@ -0,0 +1,24 @@
+# This file should be sourced, not executed!
+#
+# This is a helper script for detecting Python executables in the PATH. It is intended to be used for determining
+# which Python should be used with idf_tools.py for installing tools and exporting environment variables.
+#
+# 1. The script will set variable ESP_PYTHON to "python" if it is of version 3.
+# 2. Otherwise, "python3" will be exported if it exists.
+# 3. The script will fall-back to "python" as the last resort and fail if it doesn't exist.
+
+ESP_PYTHON=python
+
+for p_cmd in python python3
+do
+    echo "Checking \"$p_cmd\" ..."
+
+    if [ "$($p_cmd -c "import sys; print(sys.version_info.major)")" = 3 ]; then
+        ESP_PYTHON=$p_cmd
+        break
+    fi
+done
+
+$ESP_PYTHON --version || { echo "\"$ESP_PYTHON\" is not installed! Please see the documentation for how to install it."; exit 1; }
+
+echo "\"$ESP_PYTHON\" has been detected"