Merge branch 'bugfix/windows_setup' into 'master'

Windows setup improvements

* Tweak Makefile to significantly reduce startup time on Windows (also reduced on other platforms, but less obvious.)
* Revise windows setup installation script to fix various issues, use latest toolchain.
* Tweak Windows config docs

See merge request !597

docs/linux-setup.rst

@@ -142,6 +142,9 @@ The easiest way to start a project is to download the template project from GitH
 
 This will download ``esp-idf-template`` project into ``~/esp/myapp`` directory.
 
+**IMPORTANT:** The esp-idf build system does not support spaces in paths to esp-idf or to projects.
+
+You can also find a range of example projects under the "examples" directory in IDF. These example project directories can be copied to outside IDF in order to begin your own projects.
 
 Step 4: Building and flashing the application
 =============================================

docs/macos-setup.rst

@@ -130,6 +130,9 @@ This will download ``esp-idf-template`` project into ``~/esp/myapp`` directory.
 
 **IMPORTANT:** The esp-idf build system does not support spaces in paths to esp-idf or to projects.
 
+You can also find a range of example projects under the "examples" directory in IDF. These example project directories can be copied to outside IDF in order to begin your own projects.
+
+
 Step 4: Building and flashing the application
 =============================================
 

docs/windows-setup-scratch.rst

@@ -0,0 +1,65 @@
+Set up Windows Toolchain from Scratch
+*************************************
+
+This is an **optional alternative to the normal Windows Guide**.
+
+:doc:`See the main Windows setup page for the quickest and simplest Windows setup guide </windows-setup>`.
+
+Setting up the environment gives you some more control over the process, and also provides the information for advanced users to customise the install. The pre-built environment has been prepared by following these steps.
+
+Configure toolchain & environment from scratch
+==============================================
+
+This process involves installing MSYS2_, then installing the MSYS2_ and Python packages which ESP-IDF uses, and finally downloading and installing the Xtensa toolchain.
+
+* Navigate to the MSYS2_ installer page and download the ``msys2-i686-xxxxxxx.exe`` installer executable (we only support a 32-bit MSYS environment, it works on both 32-bit and 64-bit Windows.) At time of writing, the latest installer is ``msys2-i686-20161025.exe``.
+
+* Run through the installer steps. **Uncheck the "Run MSYS2 32-bit now" checkbox at the end.**
+
+* Once the installer exits, open Start Menu and find "MSYS2 MinGW 32-bit" to run the terminal.
+
+  *(Why launch this different terminal? MSYS2 has the concept of different kinds of environments. The default "MSYS" environment is Cygwin-like and uses a translation layer for all Windows API calls. We need the "MinGW" environment in order to have a native Python which supports COM ports.)*
+
+* The ESP-IDF repository on github contains a script in the tools directory titled ``windows_install_prerequisites.sh``. If you haven't got a local copy of the ESP-IDF yet, that's OK - you can just download that one file in Raw format from here: :idf_raw:`tools/windows/windows_install_prerequisites.sh`. Save it somewhere on your computer.
+
+* Type the path to the shell script into the MSYS2 terminal window. You can type it as a normal Windows path, but use forward-slashes instead of back-slashes. ie: ``C:/Users/myuser/Downloads/windows_install_prerequisites.sh``. You can read the script beforehand to check what it does.
+
+* The ``windows_install_prerequisites.sh`` script will download and install packages for ESP-IDF support, and the ESP32 toolchain.
+
+* During the initial update step, MSYS may update itself into a state where it can no longer operate. You may see errors like the following::
+
+     *** fatal error - cygheap base mismatch detected - 0x612E5408/0x612E4408. This problem is probably due to using incompatible versions of the cygwin DLL.
+
+  If you see errors like this, close the terminal window entirely (terminating the processes running there) and then re-open a new terminal. Re-run ``windows_install_prerequisites.sh`` (tip: use the up arrow key to see the last run command). The update process will resume after this step.
+
+
+MSYS2 Mirrors in China
+~~~~~~~~~~~~~~~~~~~~~~
+
+There are some (unofficial) MSYS2 mirrors inside China, which substantially improves download speeds inside China.
+
+To add these mirrors, edit the following two MSYS2 mirrorlist files before running the setup script. The mirrorlist files can be found in the ``/etc/pacman.d`` directory (ie ``c:\msys2\etc\pacman.d``).
+
+Add these lines at the top of ``mirrorlist.mingw32``::
+
+  Server = http://mirrors.ustc.edu.cn/msys2/mingw/i686/
+  Server = http://mirror.bit.edu.cn/msys2/REPOS/MINGW/i686
+
+Add these lines at the top of ``mirrorlist.msys``::
+
+  Server = http://mirrors.ustc.edu.cn/msys2/msys/$arch
+  Server = http://mirror.bit.edu.cn/msys2/REPOS/MSYS2/$arch
+
+
+HTTP Proxy
+~~~~~~~~~~
+
+You can enable an HTTP proxy for MSYS and PIP downloads by setting the ``http_proxy`` variable in the terminal before running the setup script::
+
+    export http_proxy='http://http.proxy.server:PORT'
+
+Or with credentials::
+
+    export http_proxy='http://user:password@http.proxy.server:PORT'
+
+Add this line to ``/etc/profile`` in the MSYS directory in order to permanently enable the proxy when using MSYS.

docs/windows-setup.rst

@@ -4,34 +4,18 @@ Set up of Toolchain for Windows
 Step 1: Quick Steps
 ===================
 
-Windows doesn't have a built-in "make" environment, so as well as installing the toolchain you will need a GNU-compatible environment. We use the MSYS2_ environment to provide.
- You don't need to use this environment all the time (you can use Eclipse_ or some other front-end), but it runs behind the scenes.
+Windows doesn't have a built-in "make" environment, so as well as installing the toolchain you will need a GNU-compatible environment. We use the MSYS2_ environment to provide this. You don't need to use this environment all the time (you can use Eclipse_ or some other front-end), but it runs behind the scenes.
 
 The quick setup is to download the Windows all-in-one toolchain & MSYS zip file from dl.espressif.com:
 
-https://dl.espressif.com/dl/esp32_win32_msys2_environment_and_toolchain-20170111.zip
-
-Unzip the zip file to C:\ and it will create an "msys32" directory with a pre-prepared environment.
+https://dl.espressif.com/dl/esp32_win32_msys2_environment_and_toolchain-20170321.zip
 
+Unzip the zip file to ``C:\`` (or some other location, but this guide assumes ``C:\``) and it will create an "msys32" directory with a pre-prepared environment.
 
 Alternative Step 1: Configure toolchain & environment from scratch
 ==================================================================
 
-As an alternative to getting a pre-prepared environment, you can set up the environment from scratch:
-
-* Navigate to the MSYS2_ installer page and download the ``msys2-i686-xxxxxxx.exe`` installer executable (we only support a 32-bit MSYS environment, it works on both 32-bit and 64-bit Windows.)
-
-* Run through the installer steps, and accept the "Run MSYS2 now" option at the end. A window will open with a MSYS2 terminal.
-
-* The ESP-IDF repository on github contains a script in the tools directory titled ``windows_install_prerequisites.sh``. If you haven't downloaded the ESP-IDF yet, that's OK - you can just download that one file in Raw format from here: :idf_raw:`tools/windows/windows_install_prerequisites.sh`. Save it somewhere on your computer.
-
-* Type the path to the shell script into the MSYS2 terminal window. You can type it as a normal Windows path, but use forward-slashes instead of back-slashes. ie: ``C:/Users/myuser/Downloads/windows_install_prerequisites.sh``. You can read the script beforehand to check what it does.
-
-* If you use the 201602 MSYS2 installer, the first time you run ``windows_install_prerequisites.sh`` it will update the MSYS2 core system. At the end of this update, you will be prompted to close the MSYS2 terminal and re-open. When you re-open after the update, re-run ``windows_install_prerequisites.sh``. The next version of MSYS2 (after 201602) will not need this interim step.
-
-* The ``windows_install_prerequisites.sh`` script will download and install packages for ESP-IDF support, and the ESP32 toolchain.
-
-Note: You may encounter a bug where svchost.exe uses 100% CPU in Windows after setup is finished, resulting in the ESP-IDF building very slowly. Terminating svchost.exe or restarting Windows will solve this problem.
+Rather than use the pre-prepared environment, you can :doc:`alternatively follow this guide to set up the MSYS2 environment from scratch <windows-setup-scratch>`.
 
 Another Alternative Step 1: Just download a toolchain
 =====================================================
@@ -40,14 +24,14 @@ If you already have an MSYS2 install or want to do things differently, you can d
 
 https://dl.espressif.com/dl/xtensa-esp32-elf-win32-1.22.0-61-gab8375a-5.2.0.zip
 
-If you followed one of the above options for Step 1, you won't need this download.
+**If you followed one of the above options for Step 1, you already have the toolchain and you won't need this download.**
 
-Important: Just having this toolchain is *not enough* to use ESP-IDF on Windows. You will need GNU make, bash, and sed at minimum. The above environments provide all this, plus a host compiler (required for menuconfig support).
+**Important**: Just having this toolchain is *not enough* to use ESP-IDF on Windows. You will need GNU make, bash, and sed at minimum. The above environments provide all this, plus a host compiler (required for menuconfig support).
 
 Step 2: Getting the esp-idf repository from github
 ==================================================
 
-Open an MSYS2 terminal window by running ``C:\msys32\msys2_shell.cmd``. The environment in this window is a bash shell.
+Open an MSYS2 terminal window by running ``C:\msys32\mingw32.exe``. The environment in this window is a bash shell.
 
 Change to the directory you want to clone the SDK into by typing a command like this one: ``cd "C:/path/to/dir"`` (note the forward-slashes in the path). Then type ``git clone --recursive https://github.com/espressif/esp-idf.git``
 
@@ -66,6 +50,9 @@ The process is the same as for checking out the ESP-IDF from github. Change to t
 
 **IMPORTANT:** The esp-idf build system does not support spaces in paths to esp-idf or to projects.
 
+You can also find a range of example projects under the "examples" directory in IDF. These example project directories can be copied to outside IDF in order to begin your own projects.
+
+
 Step 4: Configuring the project
 ===============================
 

make/project.mk

@@ -404,6 +404,9 @@ clean: config-clean
 # This only works for components inside IDF_PATH
 check-submodules:
 
+# Dump the git status for the whole working copy once, then grep it for each submodule. This saves a lot of time on Windows.
+GIT_STATUS := $(shell cd ${IDF_PATH} && git status --porcelain=v1 --ignore-submodules=dirty)
+
 # Generate a target to check this submodule
 # $(1) - submodule directory, relative to IDF_PATH
 define GenerateSubmoduleCheckTarget
@@ -415,9 +418,8 @@ $(IDF_PATH)/$(1)/.git:
 	@echo "Attempting 'git submodule update --init $(1)' in esp-idf root directory..."
 	cd ${IDF_PATH} && git submodule update --init $(1)
 
-# Parse 'git submodule status' output for out-of-date submodule.
-# Status output prefixes status line with '+' if the submodule commit doesn't match
-ifneq ("$(shell cd ${IDF_PATH} && git submodule status $(1) | grep '^+')","")
+# Parse 'git status' output to check if the submodule commit is different to expected
+ifneq ("$(filter $(1),$(GIT_STATUS))","")
 $$(info WARNING: esp-idf git submodule $(1) may be out of date. Run 'git submodule update' in IDF_PATH dir to update.)
 endif
 endef

tools/windows/windows_install_prerequisites.sh

@@ -1,7 +1,11 @@
 #!/bin/bash
 #
-# Setup script to configure an MSYS2 environment for Espressif IoT Development Framework (ESP-IDF).
-# See docs/windows-setup.rst for details.
+# Setup script to configure an MSYS2 environment for ESP-IDF.
+#
+# Use of this script is optional, there is also a prebuilt MSYS2 environment available
+# which can be downloaded and used as-is.
+#
+# See http://esp-idf.readthedocs.io/en/latest/windows-setup.html for full details.
 
 if [ "$OSTYPE" != "msys" ]; then
   echo "This setup script expects to be run from an MSYS2 environment on Windows."
@@ -11,6 +15,12 @@ if ! [ -x /bin/pacman ]; then
   echo "This setup script expects to use the pacman package manager from MSYS2."
   exit 1
 fi
+if [ "$MSYSTEM" != "MINGW32" ]; then
+  echo "This setup script must be started from the 'MSYS2 MinGW 32-bit' start menu shortcut"
+  echo "OR by running `cygpath -w /mingw32.exe`"
+  echo "(The current MSYSTEM mode is $MSYSTEM but it expects it to be MINGW32)"
+  exit 1
+fi
 
 # if update-core still exists, run it to get the latest core MSYS2 system
 # (which no longer needs or includes update-core!)
@@ -21,16 +31,16 @@ fi
 
 set -e
 
-pacman --noconfirm -Syu
+pacman --noconfirm -Syu # This step may require the terminal to be closed and restarted
 
-pacman --noconfirm -S gettext-devel gcc git make ncurses-devel flex bison gperf vim mingw-w64-i686-python2-pip unzip
+pacman --noconfirm -S --needed gettext-devel gcc git make ncurses-devel flex bison gperf vim mingw-w64-i686-python2-pip unzip winpty
 
 python -m pip install --upgrade pip
 
 pip install pyserial
 
-# TODO: automatically download precompiled toolchain, unpack at /opt/xtensa-esp32-elf/
-TOOLCHAIN_ZIP=xtensa-esp32-elf-win32-1.22.0-59.zip
+# Automatically download precompiled toolchain, unpack at /opt/xtensa-esp32-elf/
+TOOLCHAIN_ZIP=xtensa-esp32-elf-win32-1.22.0-61-gab8375a-5.2.0.zip
 echo "Downloading precompiled toolchain ${TOOLCHAIN_ZIP}..."
 cd ~
 curl -LO --retry 10 http://dl.espressif.com/dl/${TOOLCHAIN_ZIP}
@@ -39,6 +49,8 @@ unzip ~/${TOOLCHAIN_ZIP}
 rm ~/${TOOLCHAIN_ZIP}
 
 cat > /etc/profile.d/esp32_toolchain.sh << EOF
+# This file was created by ESP-IDF windows_install_prerequisites.sh
+# and will be overwritten if that script is run again.
 export PATH="$PATH:/opt/xtensa-esp32-elf/bin"
 EOF
 
@@ -46,15 +58,21 @@ EOF
 pacman --noconfirm -R unzip
 pacman --noconfirm -Scc
 
-echo "************************************************"
-echo "MSYS2 environment is now ready to use ESP-IDF."
-echo "Run 'source /etc/profile' to add the toolchain to"
-echo "your path. Execute 'msys_shell.cmd' to launch an"
-echo "MSYS terminal."
-echo
-echo "Once ESP-IDF is downloaded/checked out, set the"
-echo "environment variable IDF_PATH in /etc/profile to"
-echo "point to the directory."
-echo "************************************************"
-echo
+cat << EOF
+************************************************
+MSYS2 environment is now ready to use ESP-IDF.
 
+1) Run 'source /etc/profile' to add the toolchain to
+your path in this terminal. This command produces no output.
+You only need to do this once, future terminals do this
+automatically when opened.
+
+2) After ESP-IDF is set up (see setup guide), edit the file
+`cygpath -w /etc/profile`
+and add a line to set the variable IDF_PATH so it points to the
+IDF directory, ie:
+
+export IDF_PATH=/c/path/to/esp-idf/directory
+
+************************************************
+EOF