alex/esp-idf
alex/esp-idf
1
0
Fork 0
mirror of https://github.com/espressif/esp-idf.git synced 2024-10-05 20:47:46 -04:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity

docs: add README about pytest in IDF

Browse Source
This commit is contained in:
Fu Hanxi 2022-02-23 15:48:05 +08:00
parent 8a15f1ce1e
commit 5ccb5eeebd
6 changed files with 457 additions and 63 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
CONTRIBUTING.rst
View File

@ -60,3 +60,4 @@ Related Documents
../api-reference/template
contributor-agreement
copyright-guide
esp-idf-tests-with-pytest

2
docs/en/contribute/creating-examples.rst
View File

@ -11,7 +11,7 @@ Structure
- If the example has additional functionality, split it logically into separate C or C++ source files under ``main`` and place a corresponding header file in the same directory.
- If the example has a lot of additional functionality, consider adding a ``components`` directory to the example project and make some example-specific components with library functionality. Only do this if the components are specific to the example, if they're generic or common functionality then they should be added to ESP-IDF itself.
- The example should have a ``README.md`` file. Use the :idf_file:`template example README <docs/TEMPLATE_EXAMPLE_README.md>` and adapt it for your particular example.
- Examples should have an ``example_test.py`` file for running an automated example test. If submitting a GitHub Pull Request which includes an example, it's OK not to include this file initially. The details can be discussed as part of the `Pull Request <https://help.github.com/articles/creating-a-pull-request/>`_.
- Examples should have a ``pytest_<example name>.py`` file for running an automated example test. If submitting a GitHub Pull Request which includes an example, it's OK not to include this file initially. The details can be discussed as part of the `Pull Request <https://help.github.com/articles/creating-a-pull-request/>`_. Please refer to :doc:`IDF Tests with Pytest Guide <esp-idf-tests-with-pytest>` for details.
General Guidelines
------------------

375
docs/en/contribute/esp-idf-tests-with-pytest.rst Normal file
View File

@ -0,0 +1,375 @@
ESP-IDF Tests with Pytest Guide
===============================
This documentation is a guide that introduces the following aspects:
1. The basic idea of different test types in ESP-IDF
2. How to apply the pytest framework to the test python scripts to make sure the apps are working as expected.
3. ESP-IDF CI target test process
4. Run ESP-IDF tests with pytest locally
5. Tips and tricks on pytest
Disclaimer
----------
In ESP-IDF, we use the following plugins by default:
- `pytest-embedded <https://github.com/espressif/pytest-embedded>`__ with default services ``esp,idf``
- `pytest-rerunfailures <https://github.com/pytest-dev/pytest-rerunfailures>`__
All the introduced concepts and usages are based on the default behavior in ESP-IDF. Not all of them are available in vanilla pytest.
Basic Concepts
--------------
Component-based Unit Tests
~~~~~~~~~~~~~~~~~~~~~~~~~~
Component-based unit tests are our recommended way to test your component. All the test apps should be located under ``${IDF_PATH}/components/<COMPONENT_NAME>/test_apps``.
For example:
.. code:: text
components/
└── my_component/
├── include/
│ └── ...
├── test_apps/
│ ├── test_app_1
│ │ ├── main/
│ │ │ └── ...
│ │ ├── CMakeLists.txt
│ │ └── pytest_my_component_app_1.py
│ ├── test_app_2
│ │ ├── ...
│ │ └── pytest_my_component_app_2.py
│ └── parent_folder
│ ├── test_app_3
│ │ ├── ...
│ │ └── pytest_my_component_app_3.py
│ └── ...
├── my_component.c
└── CMakeLists.txt
Example Tests
~~~~~~~~~~~~~
Example Tests are tests for examples that are intended to demonstrate parts of the ESP-IDF functionality to our customers.
All the test apps should be located under ``${IDF_PATH}/examples``. For more information please refer to the :idf_file:`Examples Readme <examples/README.md>`.
For example:
.. code:: text
examples/
└── parent_folder/
└── example_1/
├── main/
│ └── ...
├── CMakeLists.txt
└── pytest_example_1.py
Custom Tests
~~~~~~~~~~~~
Custom Tests are tests that aim to run some arbitrary test internally. They are not intended to demonstrate the ESP-IDF functionality to our customers in any way.
All the test apps should be located under ``${IDF_PATH}/tools/test_apps``. For more information please refer to the :idf_file:`Custom Test Readme <tools/test_apps/README.md>`.
Pytest in ESP-IDF
-----------------
Pytest Execution Process
~~~~~~~~~~~~~~~~~~~~~~~~
1. Bootstrapping Phase
Create session-scoped caches:
- port-target cache
- port-app cache
2. Collection Phase
1. Get all the python files with the prefix ``pytest_``
2. Get all the test functions with the prefix ``test_``
3. Apply the `params <https://docs.pytest.org/en/latest/how-to/parametrize.html>`__, duplicate the test functions.
4. Filter the test cases with CLI options. Introduced detail usages `here <#filter-the-test-cases>`__
3. Test Running Phase
1. Construct the `fixtures <https://docs.pytest.org/en/latest/how-to/fixtures.html>`__. In ESP-IDF, the common fixtures are initialized with this order:
1. ``pexpect_proc``: `pexpect <https://github.com/pexpect/pexpect>`__ instance
2. ``app``: `IdfApp <https://docs.espressif.com/projects/pytest-embedded/en/latest/references/pytest_embedded_idf/#pytest_embedded_idf.app.IdfApp>`__ instance
The information of the app, like sdkconfig, flash_files, partition_table, etc., would be parsed at this phase.
3. ``serial``: `IdfSerial <https://docs.espressif.com/projects/pytest-embedded/en/latest/references/pytest_embedded_idf/#pytest_embedded_idf.serial.IdfSerial>`__ instance
The port of the host which connected to the target type parsed from the app would be auto-detected. The flash files would be auto flashed.
4. ``dut``: `IdfDut <https://docs.espressif.com/projects/pytest-embedded/en/latest/references/pytest_embedded_idf/#pytest_embedded_idf.dut.IdfDut>`__ instance
2. Run the real test function
3. Deconstruct the fixtures with this order:
1. ``dut``
1. close the ``serial`` port
2. (Only for apps with `unity test framework <https://github.com/ThrowTheSwitch/Unity>`__) generate junit report of the unity test cases
2. ``serial``
3. ``app``
4. ``pexpect_proc``: Close the file descriptor
4. (Only for apps with `unity test framework <https://github.com/ThrowTheSwitch/Unity>`__)
Raise ``AssertionError`` when detected unity test failed if you call ``dut.expect_from_unity_output()`` in the test function.
4. Reporting Phase
1. Generate junit report of the test functions
2. Modify the junit report test case name into ESP-IDF test case ID format: ``<target>.<config>.<test function name>``
5. Finalizing Phase (Only for apps with `unity test framework <https://github.com/ThrowTheSwitch/Unity>`__)
Combine the junit reports if the junit reports of the unity test cases are generated.
Example Code
~~~~~~~~~~~~
This code example is taken from :idf_file:`pytest_console_basic.py <examples/system/console/basic/pytest_console_basic.py>`.
.. code:: python
@pytest.mark.esp32
@pytest.mark.esp32c3
@pytest.mark.generic
@pytest.mark.parametrize('config', [
'history',
'nohistory',
], indirect=True)
def test_console_advanced(config: str, dut: Dut) -> None:
if config == 'history':
dut.expect('Command history enabled')
elif config == 'nohistory':
dut.expect('Command history disabled')
.. note::
Using ``expect_exact`` is better here. For further reading about the different types of ``expect`` functions, please refer to the `pytest-embedded Expecting documentation <https://docs.espressif.com/projects/pytest-embedded/en/latest/expecting>`__.
Use Markers to Specify the Supported Targets
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
You can use markers to specify the supported targets and the test env in CI. You can run ``pytest --markers`` to get more details about different markers.
.. code:: python
@pytest.mark.esp32 # <-- support esp32
@pytest.mark.esp32c3 # <-- support esp32c3
@pytest.mark.generic # <-- test env `generic, would assign to runner with tag `generic`
Besides, if the test case supports all officially ESP-IDF-supported targets, like esp32, esp32s2, esp32s3, esp32c3 for now (2022.2), you can use a special marker ``supported_targets`` to apply them all in one line.
This code example is taken from :idf_file:`pytest_gptimer_example.py <examples/peripherals/timer_group/gptimer/pytest_gptimer_example.py>`.
.. code:: python
@pytest.mark.supported_targets
@pytest.mark.generic
def test_gptimer_example(dut: Dut) -> None:
...
Use Params to Specify the sdkconfig Files
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
You can use ``pytest.mark.parametrize`` with “config” to apply the same test to different apps with different sdkconfig files. For more information about ``sdkconfig.ci.xxx`` files, please refer to the Configuration Files section under :idf_file:`this readme <tools/test_apps/README.md>`.
.. code:: python
@pytest.mark.parametrize('config', [
'history', # <-- run with app built by sdkconfig.ci.history
'nohistory', # <-- run with app built by sdkconfig.ci.nohistory
], indirect=True) # <-- `indirect=True` is required
Overall, this test case would be duplicated to 4 test functions:
- esp32.history.test_console_advanced
- esp32.nohistory.test_console_advanced
- esp32c3.history.test_console_advanced
- esp32c3.nohistory.test_console_advanced
Advanced Examples
~~~~~~~~~~~~~~~~~
Support different targets with different sdkconfig files
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This code example is taken from :idf_file:`pytest_panic.py <tools/test_apps/system/panic/pytest_panic.py>` as an advanced example.
.. code:: python
CONFIGS = [
pytest.param('coredump_flash_bin_crc', marks=[pytest.mark.esp32, pytest.mark.esp32s2]),
pytest.param('coredump_flash_elf_sha', marks=[pytest.mark.esp32]), # sha256 only supported on esp32
pytest.param('coredump_uart_bin_crc', marks=[pytest.mark.esp32, pytest.mark.esp32s2]),
pytest.param('coredump_uart_elf_crc', marks=[pytest.mark.esp32, pytest.mark.esp32s2]),
pytest.param('gdbstub', marks=[pytest.mark.esp32, pytest.mark.esp32s2]),
pytest.param('panic', marks=[pytest.mark.esp32, pytest.mark.esp32s2]),
]
@pytest.mark.parametrize('config', CONFIGS, indirect=True)
...
Use Custom Class
^^^^^^^^^^^^^^^^
Usually, you can write a custom class in these conditions:
1. Add more reusable functions for a certain number of DUTs
2. Add custom setup and teardown functions in different phases described `here <#pytest-execution-process>`__
This code example is taken from :idf_file:`panic/conftest.py <tools/test_apps/system/panic/conftest.py>`
.. code:: python
class PanicTestDut(IdfDut):
...
@pytest.fixture(scope='module')
def monkeypatch_module(request: FixtureRequest) -> MonkeyPatch:
mp = MonkeyPatch()
request.addfinalizer(mp.undo)
return mp
@pytest.fixture(scope='module', autouse=True)
def replace_dut_class(monkeypatch_module: MonkeyPatch) -> None:
monkeypatch_module.setattr('pytest_embedded_idf.dut.IdfDut', PanicTestDut)
``monkeypatch_module`` provide a `module-scoped <https://docs.pytest.org/en/latest/how-to/fixtures.html#scope-sharing-fixtures-across-classes-modules-packages-or-session>`__ `monkeypatch <https://docs.pytest.org/en/latest/how-to/monkeypatch.html>`__ fixture.
``replace_dut_class`` is a `module-scoped <https://docs.pytest.org/en/latest/how-to/fixtures.html#scope-sharing-fixtures-across-classes-modules-packages-or-session>`__ `autouse <https://docs.pytest.org/en/latest/how-to/fixtures.html#autouse-fixtures-fixtures-you-don-t-have-to-request>`__ fixture. This function replaces the ``IdfDut`` class with your custom class.
Mark Flaky Tests
^^^^^^^^^^^^^^^^
Sometimes, our test is based on ethernet or wifi. The network may cause the test flaky. We could mark the single test case within the code repo.
This code example is taken from :idf_file:`pytest_esp_eth.py <components/esp_eth/test_apps/pytest_esp_eth.py>`
.. code:: python
@pytest.mark.flaky(reruns=3, reruns_delay=5)
def test_esp_eth_ip101(dut: Dut) -> None:
...
This flaky marker means that if the test function failed, the test case would rerun for a maximum of 3 times with 5 seconds delay.
Run the Tests in CI
-------------------
The workflow in CI is simple, build jobs -> target test jobs.
Build Jobs
~~~~~~~~~~
Build Job Names
^^^^^^^^^^^^^^^
- Component-based Unit Tests: ``build_pytest_components_<target>``
- Example Tests: ``build_pytest_examples_<target>``
- Custom Tests: ``build_pytest_test_apps_<target>``
Build Job Command
^^^^^^^^^^^^^^^^^
The command used by CI to build all the relevant tests is: ``python $IDF_PATH/tools/ci/build_pytest_apps.py <parent_dir> --target <target> -vv``
All apps which supported the specified target would be built with all supported sdkconfig files under ``build_<target>_<config>``.
For example, If you run ``python $IDF_PATH/tools/ci/build_pytest_apps.py $IDF_PATH/examples/system/console/basic --target esp32``, the folder structure would be like this:
.. code:: text
basic
├── build_esp32_history/
│ └── ...
├── build_esp32_nohistory/
│ └── ...
├── main/
├── CMakeLists.txt
├── pytest_console_basic.py
└── ...
All the binaries folders would be uploaded as artifacts under the same directories.
Target Test Jobs
~~~~~~~~~~~~~~~~
Target Test Job Names
^^^^^^^^^^^^^^^^^^^^^
- Component-based Unit Tests: ``component_ut_pytest_<target>_<test_env>``
- Example Tests: ``example_test_pytest_<target>_<test_env>``
- Custom Tests: ``test_app_test_pytest_<target>_<test_env>``
Target Test Job Command
^^^^^^^^^^^^^^^^^^^^^^^
The command used by CI to run all the relevant tests is: ``pytest <parent_dir> --target <target> -m <test_env_marker>``
All test cases with the specified target marker and the test env marker under the parent folder would be executed.
The binaries in the target test jobs are downloaded from build jobs, the artifacts would be placed under the same directories.
Run the Tests Locally
---------------------
The local executing process is the same as the CI process.
For example, if you want to run all the esp32 tests under the ``$IDF_PATH/examples/system/console/basic`` folder, you may:
.. code:: shell
$ pip install pytest-embedded-serial-esp pytest-embedded-idf
$ cd $IDF_PATH
$ . ./export.sh
$ cd examples/system/console/basic
$ python $IDF_PATH/tools/ci/build_pytest_apps.py . --target esp32 -vv
$ pytest --target esp32
Tips and Tricks
---------------
Filter the Test Cases
~~~~~~~~~~~~~~~~~~~~~
- filter by target with ``pytest --target <target>``
pytest would run all the test cases that support specified target.
- filter by sdkconfig file with ``pytest --sdkconfig <sdkconfig>``
if ``<sdkconfig>`` is ``default``, pytest would run all the test cases with the sdkconfig file ``sdkconfig.defaults``.
In other cases, pytest would run all the test cases with sdkconfig file ``sdkconfig.ci.<sdkconfig>``.
Add New Markers
~~~~~~~~~~~~~~~
We’re using two types of custom markers, target markers which indicate that the test cases should support this target, and env markers which indicate that the test case should be assigned to runners with these tags in CI.
You can add new markers by adding one line under the ``${IDF_PATH}/pytest.ini`` ``markers =`` section. The grammar should be: ``<marker_name>: <marker_description>``
Further Readings
----------------
- pytest documentation: https://docs.pytest.org/en/latest/contents.html
- pytest-embedded documentation: https://docs.espressif.com/projects/pytest-embedded/en/latest/

1
docs/zh_CN/contribute/esp-idf-tests-with-pytest.rst Normal file
View File

@ -0,0 +1 @@
.. include:: ../../en/contribute/esp-idf-tests-with-pytest.rst

70
examples/README.md
View File

@ -2,64 +2,70 @@
This directory contains a range of example ESP-IDF projects. These examples are intended to demonstrate parts of the ESP-IDF functionality, and to provide code that you can copy and adapt into your own projects.
# Example Layout
## Example Layout
The examples are grouped into subdirectories by category. Each category directory contains one or more example projects:
* `bluetooth/bluedroid` Classic BT, BLE and coex examples using default Bluedroid host stack.
* `bluetooth/nimble` BLE examples using NimBLE host stack.
* `bluetooth/esp_ble_mesh` ESP BLE Mesh examples.
* `bluetooth/hci` HCI transport (VHCI and HCI UART) examples.
* `build_system` Examples of build system features.
* `cxx` C++ language utilization examples and experimental components.
* `ethernet` Ethernet network examples.
* `get-started` Simple examples with minimal functionality. Good start point for beginners.
* `mesh` Wi-Fi Mesh examples.
* `network` Examples related to general network environment, test & analysis.
* `openthread` OpenThread examples.
* `peripherals` Examples showing driver functionality for the various onboard ESP32 peripherals.
* `protocols` Examples showing network protocol interactions.
* `provisioning` Wi-Fi provisioning examples.
* `security` Examples about security features.
* `storage` Examples showing data storage methods using SPI flash, external storage like the SD/MMC interface and flash partitioning.
* `system` Demonstrates some internal chip features, or debugging & development tools.
* `wifi` Advanced Wi-Fi features (For network protocol examples, see `protocols` instead.)
- `bluetooth/bluedroid` Classic BT, BLE and coex examples using default Bluedroid host stack.
- `bluetooth/nimble` BLE examples using NimBLE host stack.
- `bluetooth/esp_ble_mesh` ESP BLE Mesh examples.
- `bluetooth/hci` HCI transport (VHCI and HCI UART) examples.
- `build_system` Examples of build system features.
- `cxx` C++ language utilization examples and experimental components.
- `ethernet` Ethernet network examples.
- `get-started` Simple examples with minimal functionality. Good start point for beginners.
- `mesh` Wi-Fi Mesh examples.
- `network` Examples related to general network environment, test & analysis.
- `openthread` OpenThread examples.
- `peripherals` Examples showing driver functionality for the various onboard ESP32 peripherals.
- `protocols` Examples showing network protocol interactions.
- `provisioning` Wi-Fi provisioning examples.
- `security` Examples about security features.
- `storage` Examples showing data storage methods using SPI flash, external storage like the SD/MMC interface and flash partitioning.
- `system` Demonstrates some internal chip features, or debugging & development tools.
- `wifi` Advanced Wi-Fi features (For network protocol examples, see `protocols` instead.)
In addition to these examples, `commmon_components` directory contains code shared by several examples.
# Using Examples
## Using Examples
Before building an example, be sure to follow the [ESP-IDF Getting Started Guide](https://idf.espressif.com/) to ensure you have the required development environment.
Building an example is the same as building any other project:
* Change into the directory of the new example you'd like to build.
* Run `idf.py set-target TARGET` to select the correct chip target to build before opening the project configuration menu. By default the target is `esp32`. For all options see `idf.py set-target --help`
* Run `idf.py menuconfig` to open the project configuration menu. Most examples have a project-specific "Example Configuration" section here (for example, to set the WiFi SSID & password to use).
* `idf.py build` to build the example.
* Follow the printed instructions to flash, or run `idf.py -p PORT flash`.
- Change into the directory of the new example you'd like to build.
- Run `idf.py set-target TARGET` to select the correct chip target to build before opening the project configuration menu. By default the target is `esp32`. For all options see `idf.py set-target --help`
- Run `idf.py menuconfig` to open the project configuration menu. Most examples have a project-specific "Example Configuration" section here (for example, to set the WiFi SSID & password to use).
- `idf.py build` to build the example.
- Follow the printed instructions to flash, or run `idf.py -p PORT flash`.
## Running Test Python Script
## Running Test Python Script (ttfw)
Some of the examples have `..._test.py` scripts that are used to test that the example works as expected. These scripts run automatically in the internal test queue. They are not intended to be run by ESP-IDF users but sometimes you may want to run them locally. The following requirements must be met in the IDF python virtual environment.
* ttfw needs to be in the `PYTHONPATH`. Add it like this: `export PYTHONPATH=$PYTHONPATH:$IDF_PATH/tools/ci/python_packages`
* Install all requirements from `tools/ci/python_packages/ttfw_idf/requirements.txt`: `python -m pip install -r $IDF_PATH/tools/ci/python_packages/ttfw_idf/requirements.txt`
- ttfw needs to be in the `PYTHONPATH`. Add it like this: `export PYTHONPATH=$PYTHONPATH:$IDF_PATH/tools/ci/python_packages`
- Install all requirements from `tools/ci/python_packages/ttfw_idf/requirements.txt`: `python -m pip install -r $IDF_PATH/tools/ci/python_packages/ttfw_idf/requirements.txt`
These commands help solve the issue with `ModuleNotFoundError: No module named 'ttfw_idf'` and `ModuleNotFoundError: No module named 'tiny_test_fw'`.
Some examples might fail due to other missing packages. You might need to install them manually: `pip install websocket`.
# Copying Examples
## Running Test Python Script (pytest)
Each example is a standalone project. The examples *do not have to be inside the esp-idf directory*. You can copy an example directory to anywhere on your computer in order to make a copy that you can modify and work with.
Some of the examples have `pytest_....py` scripts that are using the `pytest` as the test framework. For detailed information, please refer to the "Run the Tests Locally" Section under [ESP-IDF tests in Pytest documentation](../docs/en/contribute/esp-idf-tests-with-pytest.rst)
Using `pytest` is the recommended way to write new tests. We will migrate all the example test scripts to this new framework soon.
## Copying Examples
Each example is a standalone project. The examples _do not have to be inside the esp-idf directory_. You can copy an example directory to anywhere on your computer in order to make a copy that you can modify and work with.
The `IDF_PATH` environment variable is the only thing that connects the example to the rest of ESP-IDF.
If you're looking for a more bare-bones project to start from, try [esp-idf-template](https://github.com/espressif/esp-idf-template).
# Contributing Examples
## Contributing Examples
If you have a new example you think we'd like, please consider sending it to us as a Pull Request.
In the ESP-IDF documentation, you can find a "Creating Examples" page which lays out the steps to creating a top quality example.
In the ESP-IDF documentation, you can find a ["Creating Examples" page](../docs/en/contribute/creating-examples.rst) which lays out the steps to creating a top quality example.

71
tools/test_apps/README.md
View File

@ -6,72 +6,83 @@ various conditions or combination with other components, including custom test f
The test apps are not intended to demonstrate the ESP-IDF functionality in any way.
# Test Apps projects
## Test Apps projects
Test applications are treated the same way as ESP-IDF examples, so each project directory shall contain
* Build recipe in cmake and the main component with app sources
* Configuration files, `sdkconfig.ci` and similar (see below)
* Test executor in `ttfw_idf` format if the project is intended to also run tests (otherwise the example is build only)
- test file in the project dir must end with `_test.py`, by should be named `app_test.py`
- Build recipe in cmake and the main component with app sources
- Configuration files, `sdkconfig.ci` and similar (see below)
- Test executor in `ttfw_idf` format if the project is intended to also run tests (otherwise the example is build only)
- test file in the project dir must end with `_test.py`, by should be named `app_test.py`
- test cases shall be decorated with `@ttfw_idf.idf_custom_test(env_tag="...")`
## Test Apps layout
# CI Behavior
The test apps should be grouped into subdirectories by category. Categories are:
## Configuration Files
- `protocols` contains test of protocol interactions.
- `network` contains system network tests
- `system` contains tests on the internal chip features, debugging and development tools.
- `security` contains tests on the chip security features.
## CI Behavior
### Configuration Files
For each project in test_apps (and also examples):
* If a file `sdkconfig.ci` exists then it's built as the `default` CI config.
* If any additional files `sdkconfig.ci.<CONFIG>` exist then these are built as alternative configs, with the specified `<CONFIG>` name.
- If a file `sdkconfig.ci` exists then it's built as the `default` CI config.
- If any additional files `sdkconfig.ci.<CONFIG>` exist then these are built as alternative configs, with the specified `<CONFIG>` name.
The CI system expects to see at least a "default" config, so add `sdkconfig.ci` before adding any `sdkconfig.ci.CONFIG` files.
* By default, every CI configurations is built for every target SoC (an `m * n` configuration matrix). However if any `sdkconfig.ci.*` file contains a line of the form `CONFIG_IDF_TARGET="targetname"` then that CI config is only built for that one target. This only works in `sdkconfig.ci.CONFIG`, not in the default `sdkconfig.ci`.
* Each configuration is also built with the contents of any `sdkconfig.defaults` file or a file named `sdkconfig.defaults.<TARGET>` appended. (Same as a normal ESP-IDF project build.)
- By default, every CI configurations is built for every target SoC (an `m * n` configuration matrix). However if any `sdkconfig.ci.*` file contains a line of the form `CONFIG_IDF_TARGET="targetname"` then that CI config is only built for that one target. This only works in `sdkconfig.ci.CONFIG`, not in the default `sdkconfig.ci`.
- Each configuration is also built with the contents of any `sdkconfig.defaults` file or a file named `sdkconfig.defaults.<TARGET>` appended. (Same as a normal ESP-IDF project build.)
## Test Execution
### Test Execution
If an example test or test app test supports more targets than just `ESP32`, then the `app_test.py` file needs to specify the list of supported targets in the test decorator. For example:
```
```python
@ttfw_idf.idf_example_test(env_tag='Example_GENERIC', target=['esp32', 'esp32s2'])
def test_app_xyz(env, extra_data):
```
If the app test supports multiple targets but you only want some of these targets to be run automatically in CI, the list can be further filtered down by adding the `ci_target` list:
```
```python
@ttfw_idf.idf_example_test(env_tag='Example_GENERIC', target=['esp32', 'esp32s2'], ci_target=['esp32'])
def test_app_xyz(env, extra_data):
```
(If no ci_target list is specified, all supported targets will be tested in CI.)
# Test Apps layout
## Test Apps local execution (ttfw)
The test apps should be grouped into subdirectories by category. Categories are:
* `protocols` contains test of protocol interactions.
* `network` contains system network tests
* `system` contains tests on the internal chip features, debugging and development tools.
* `security` contains tests on the chip security features.
# Test Apps local execution
All the following instructions are general. Part of them may be complemented by more particular instructions in the corresponding app's README.
## Requirements
### Requirements
The following requirements need to be satisfied in the IDF python virtual environment.
* ttfw needs to be in the `PYTHONPATH`. Add it like this: `export PYTHONPATH=$PYTHONPATH:$IDF_PATH/tools/ci/python_packages`
* Install all requirements from `tools/ci/python_packages/ttfw_idf/requirements.txt`: `pip install -r $IDF_PATH/tools/ci/python_packages/ttfw_idf/requirements.txt`
- ttfw needs to be in the `PYTHONPATH`. Add it like this: `export PYTHONPATH=$PYTHONPATH:$IDF_PATH/tools/ci/python_packages`
- Install all requirements from `tools/ci/python_packages/ttfw_idf/requirements.txt`: `pip install -r $IDF_PATH/tools/ci/python_packages/ttfw_idf/requirements.txt`
You should also set the port via the environment variable ESPPORT to prevent the tools from looking and iterating over all serial ports. The latter causes much trouble, currently:
```
export ESPPORT=/dev/ttyUSB<X>
```
## Execution
* Create an sdkconfig file from the relevant `sdkconfig.ci.<CONFIG>` and `sdkconfig.defaults`: `cat sdkconfig.defaults sdkconfig.ci.<CONFIG> > sdkconfig`
* Run `idf.py menuconfig` to configure local project attributes
* Run `idf.py build` to build the test app
* Run `python app_test.py` to run the test locally
## Test Apps local execution (pytest)
Some of the examples have `pytest_....py` scripts that are using the `pytest` as the test framework. For detailed information, please refer to the "Run the Tests Locally" Section under [ESP-IDF tests in Pytest documentation](../../docs/en/contribute/esp-idf-tests-with-pytest.rst)
Using `pytest` is the recommended way to write new tests. We will migrate all the test apps scripts to this new framework soon.
### Execution
- Create an sdkconfig file from the relevant `sdkconfig.ci.<CONFIG>` and `sdkconfig.defaults`: `cat sdkconfig.defaults sdkconfig.ci.<CONFIG> > sdkconfig`
- Run `idf.py menuconfig` to configure local project attributes
- Run `idf.py build` to build the test app
- Run `python app_test.py` to run the test locally