Eclipse: Process Windows paths correctly using cygpath

Includes splitting the Windows Eclipse setup doc into a separate page,
as it has so many additional steps.

Addresses github #17 and #166
https://github.com/espressif/esp-idf/issues/17
https://github.com/espressif/esp-idf/issues/166

docs/eclipse-setup-windows.rst

@@ -0,0 +1,77 @@
+Eclipse IDE on Windows
+**********************
+
+Configuring Eclipse on Windows requires some different steps. The full configuration steps for Windows are shown below.
+
+(For OS X and Linux instructions, see the :doc:`Eclipse IDE page <eclipse-setup>`.)
+
+Installing Eclipse IDE
+======================
+
+Follow the steps under :ref:`Installing Eclipse IDE <eclipse-install-steps>` for all platforms.
+
+.. _eclipse-windows-setup:
+
+Setting up Eclipse on Windows
+=============================
+
+Once your new Eclipse installation launches, follow these steps:
+
+Import New Project
+------------------
+
+* Eclipse makes use of the Makefile support in ESP-IDF. This means you need to start by creating an ESP-IDF project. You can use the idf-template project from github, or open one of the examples in the esp-idf examples subdirectory.
+
+* Once Eclipse is running, choose File -> Import...
+
+* In the dialog that pops up, choose "C/C++" -> "Existing Code as Makefile Project" and click Next.
+
+* On the next page, enter "Existing Code Location" to be the directory of your IDF project. Don't specify the path to the ESP-IDF directory itself (that comes later). The directory you specify should contain a file named "Makefile" (the project Makefile).
+
+* On the same page, under "Toolchain for Indexer Settings" uncheck "Show only available toolchains that support this platform".
+
+* On the extended list that appears, choose "Cygwin GCC". Then click Finish.
+
+*Note: you may see warnings in the UI that Cygwin GCC Toolchain could not be found. This is OK, we're going to reconfigure Eclipse to find our toolchain.*
+
+Project Properties
+------------------
+
+* The new project will appear under Project Explorer. Right-click the project and choose Properties from the context menu.
+
+* Click on the "C/C++ Build" properties page (top-level):
+
+  * Uncheck "Use default build command" and enter this for the custom build command: ``python ${IDF_PATH}/tools/windows/eclipse_make.py``.
+
+* Click on the "Environment" properties page under "C/C++ Build":
+
+  * Click "Add..." and enter name ``V`` and value ``1``.
+
+  * Click "Add..." again, and enter name ``IDF_PATH``. The value should be the full path where ESP-IDF is installed. The IDF_PATH directory should be specified using forwards slashes not backslashes, ie *C:/Users/MyUser/Development/esp-idf*.
+
+  * Edit the PATH environment variable. Delete the existing value and replace it with ``C:\msys32\usr\bin;C:\msys32\mingw32\bin;C:\msys32\opt\xtensa-esp32-elf\bin`` (If you installed msys32 to a different directory then you'll need to change these paths to match).
+
+* Click on "C/C++ General" -> "Preprocessor Include Paths, Macros,etc." property page:
+
+  * Click the "Providers" tab
+
+     * In the list of providers, click "CDT GCC Built-in Compiler Settings Cygwin". Under "Command to get compiler specs", replace the text ``${COMMAND}`` at the beginning of the line with ``xtensa-esp32-elf-gcc``. This means the full "Command to get compiler specs" should be ``xtensa-esp32-elf-gcc ${FLAGS} -E -P -v -dD "${INPUTS}"``.
+
+     * In the list of providers, click "CDT GCC Build Output Parser" and type ``xtensa-esp32-elf-`` at the beginning of the Compiler command pattern. This means the full Compiler command pattern should be ``xtensa-esp32-elf-(g?cc)|([gc]\+\+)|(clang)``
+
+
+Building in Eclipse
+-------------------
+
+Continue from :ref:`Building in Eclipse <eclipse-build-project>` for all platforms.
+
+Technical Details
+=================
+
+**Of interest to Windows gurus or very curious parties, only.**
+
+Explanations of the technical reasons for some of these steps. You don't need to know this in order to use esp-idf with Eclipse on Windows, but it may be helpful background knowledge if you plan to do dig into the Eclipse support:
+
+* The xtensa-esp32-elf-gcc cross-compiler is *not* a Cygwin toolchain, even though we tell Eclipse that it is one. This is because msys2 uses Cygwin and supports Cygwin paths (of the type ``/c/blah`` instead of ``c:/blah`` or ``c:\\blah``). In particular, xtensa-esp32-elf-gcc reports to the Eclipse "built-in compiler settings" function that its built-in include directories are all under ``/usr/``, which is a Unix/Cygwin-style path that Eclipse otherwise can't resolve. By telling Eclipse the compiler is Cygwin, it resolves these paths internally using the ``cygpath`` utility.
+
+* The same problem occurs when parsing make output from esp-idf. Eclipse parses this output to find header directories, but it can't resolve include directories of the form ``/c/blah`` without using ``cygpath``. There is a heuristic that Eclipse Build Output Parser uses to determine whether it should call ``cygpath``, but for currently unknown reasons the esp-idf configuration doesn't trigger it. For this reason the ``eclipse_make.py`` wrapper script is used to call ``make`` and then use ``cygpath`` to process the output for Eclipse.

docs/eclipse-setup.rst

@@ -1,6 +1,8 @@
 Build and Flash with Eclipse IDE
 ********************************
 
+.. _eclipse-install-steps:
+
 Installing Eclipse IDE
 ======================
 
@@ -8,10 +10,17 @@ The Eclipse IDE gives you a graphical integrated development environment for wri
 
 * Start by installing the esp-idf for your platform (see files in this directory with steps for Windows, OS X, Linux).
 
+* We suggest building a project from the command line first, to get a feel for how that process works. You also need to use the command line to configure your esp-idf project (via ``make menuconfig``), this is not currently supported inside Eclipse.
+
 * Download the Eclipse Installer for your platform from eclipse.org_.
 
 * When running the Eclipse Installer, choose "Eclipse for C/C++ Development" (in other places you'll see this referred to as CDT.)
 
+Windows Users
+=============
+
+Using ESP-IDF with Eclipse on Windows requires different configuration steps. :ref:`See the Eclipse IDE on Windows guide <eclipse-windows-setup>`.
+
 Setting up Eclipse
 ==================
 
@@ -20,13 +29,13 @@ Once your new Eclipse installation launches, follow these steps:
 Import New Project
 ------------------
 
-* Eclipse makes use of the Makefile support in ESP-IDF. This means you need to start by creating an ESP-IDF project. You can use the skeleton project from github.
+* Eclipse makes use of the Makefile support in ESP-IDF. This means you need to start by creating an ESP-IDF project. You can use the idf-template project from github, or open one of the examples in the esp-idf examples subdirectory.
 
 * Once Eclipse is running, choose File -> Import...
 
 * In the dialog that pops up, choose "C/C++" -> "Existing Code as Makefile Project" and click Next.
 
-* On the next page, enter "Existing Code Location" to be the directory of your IDF project. Don't specify the path to the ESP-IDF directory itself.
+* On the next page, enter "Existing Code Location" to be the directory of your IDF project. Don't specify the path to the ESP-IDF directory itself (that comes later). The directory you specify should contain a file named "Makefile" (the project Makefile).
 
 * On the same page, under "Toolchain for Indexer Settings" choose "Cross GCC". Then click Finish.
 
@@ -38,13 +47,7 @@ Project Properties
 
 * Click on the "Environment" properties page under "C/C++ Build". Click "Add..." and enter name ``V`` and value ``1``.
 
-* Click "Add..." again, and enter name ``IDF_PATH``. The value should be the full path where ESP-IDF is installed. *Windows users: Use forward-slashes not backslashes for this path, ie C:/Users/MyUser/Development/esp-idf*.
+* Click "Add..." again, and enter name ``IDF_PATH``. The value should be the full path where ESP-IDF is installed.
-
-*Windows users only, follow these two additional steps:*
-
-* On the same Environment property page, edit the PATH environment variable. Delete the existing value and replace it with ``C:\msys32\usr\bin;C:\msys32\mingw32\bin;C:\msys32\opt\xtensa-esp32-elf\bin`` (If you installed msys32 to a different directory then you'll need to change these paths to match).
-
-* Click on the "C/C++ Build" top-level properties page then uncheck "Use default build command" and enter this for the custom build command: ``bash ${IDF_PATH}/tools/windows/eclipse_make.sh``.
 
 *All users, continue with these steps:*
 
@@ -56,7 +59,22 @@ Navigate to "C/C++ General" -> "Preprocessor Include Paths" property page:
 
 * In the list of providers, click "CDT GCC Build Output Parser" and type ``xtensa-esp32-elf-`` at the beginning of the Compiler command pattern. This means the full Compiler command pattern should be ``xtensa-esp32-elf-(g?cc)|([gc]\+\+)|(clang)``
 
-* Click OK to close the Properties dialog, and choose Project -> Build to build your project.
+.. _eclipse-build-project:
+
+Building in Eclipse
+-------------------
+
+Before your project is first built, Eclipse may show a lot of errors and warnings about undefined values. This is because some source files are automatically generated as part of the esp-idf build process. These errors and warnings will go away after you build the project.
+
+* Click OK to close the Properties dialog in Eclipse.
+
+* Outside Eclipse, open a command line prompt. Navigate to your project directory, and run ``make menuconfig`` to configure your project's esp-idf settings. This step currently has to be run outside Eclipse.
+
+*If you try to build without running a configuration step first, esp-idf will prompt for configuration on the command line - but Eclipse is not able to deal with this, so the build will hang or fail.*
+
+* Back in Eclipse, choose Project -> Build to build your project.
+
+**TIP**: If your project had already been built outside Eclipse, you may need to do a Project -> Clean before chosing Project -> Build. This is so Eclipse can see the compiler arguments for all source files. It uses these to determine the header include paths.
 
 Flash from Eclipse
 ------------------
@@ -77,7 +95,3 @@ Follow the same steps to add ``bootloader`` and ``partition_table`` targets, if
 
 .. _eclipse.org: http://www.eclipse.org/
 
-Eclipse Troubleshooting
------------------------
-
-* ``*** Make was invoked from ... However please do not run make from the sdk or a component directory; ...`` - Eclipse will detect any directory with a Makefile in it as being a possible directory to run "make" in. All component directories also contain a Makefile (the wrong one), so it is important when using Project -> Make Target to always select the top-level project directory in Project Explorer.

tools/windows/eclipse_make.py

@@ -0,0 +1,36 @@
+#!/usr/bin/env python
+#
+# Wrapper to run make and preprocess any paths in the output from MSYS/Cygwin paths
+# to Windows paths, for Eclipse
+from __future__ import print_function, division
+import sys, subprocess, os.path, re
+
+UNIX_PATH_RE = re.compile(r'(/[^ \'"]+)+')
+
+paths = {}
+def check_path(path):
+    try:
+        return paths[path]
+    except KeyError:
+        pass
+    paths[path] = path  # cache as failed, replace with success if it works
+    try:
+        winpath = subprocess.check_output(["cygpath", "-w", path]).strip()
+    except subprocess.CalledProcessError:
+        return path  # something went wrong running cygpath, assume this is not a path!
+    if not os.path.exists(winpath):
+        return path  # not actually a valid path
+    winpath = winpath.replace("\\", "/")  # make consistent with forward-slashes used elsewhere
+    paths[path] = winpath
+    return winpath
+
+def main():
+    print("Running make in '%s'" % check_path(os.getcwd()))
+    make = subprocess.Popen(["make"] + sys.argv[1:] + ["V=1"], stdout=subprocess.PIPE)
+    for line in iter(make.stdout.readline, ''):
+         line = re.sub(UNIX_PATH_RE, lambda m: check_path(m.group(0)), line)
+         print(line.rstrip())
+    sys.exit(make.wait())
+
+if __name__ == "__main__":
+    main()

tools/windows/eclipse_make.sh

@@ -1,10 +1,4 @@
 #!/bin/bash
-# A wrapper for make on Windows with Eclipse
+echo "eclipse_make.sh has been replaced with eclipse_make.py. Check the Windows Eclipse docs for the new command."
-#
+echo "This shell script will continue to work until the next major release."
-# Eclipse's output parser expects to see output of the form C:/dir/dir/file but our Make
+python ${IDF_PATH}/tools/windows/eclipse_make.py $@
-# process uses MinGW paths of the form /c/dir/dir/file. So parse these out...
-#
-# A little hacky as it looks for any single character of form /X/something.
-#
-echo "Running make in $(pwd)"
-make $@ V=1 | sed -E "s@/([a-z])/([^/+])@\1:/\2@g"