Component-based unit tests are our recommended way to test your component. All the test apps should be located under ``${IDF_PATH}/components/<COMPONENT_NAME>/test_apps``.
Example Tests are tests for examples that are intended to demonstrate parts of the ESP-IDF functionality to our customers.
All the test apps should be located under ``${IDF_PATH}/examples``. For more information please refer to the :idf_file:`Examples Readme <examples/README.md>`.
Custom Tests are tests that aim to run some arbitrary test internally. They are not intended to demonstrate the ESP-IDF functionality to our customers in any way.
All the test apps should be located under ``${IDF_PATH}/tools/test_apps``. For more information please refer to the :idf_file:`Custom Test Readme <tools/test_apps/README.md>`.
Using ``expect_exact`` is better here. For further reading about the different types of ``expect`` functions, please refer to the `pytest-embedded Expecting documentation <https://docs.espressif.com/projects/pytest-embedded/en/latest/expecting>`__.
Use Markers to Specify the Supported Targets
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
You can use markers to specify the supported targets and the test env in CI. You can run ``pytest --markers`` to get more details about different markers.
..code:: python
@pytest.mark.esp32 # <-- support esp32
@pytest.mark.esp32c3 # <-- support esp32c3
@pytest.mark.generic # <-- test env `generic, would assign to runner with tag `generic`
Besides, if the test case supports all officially ESP-IDF-supported targets, like esp32, esp32s2, esp32s3, esp32c3 for now (2022.2), you can use a special marker ``supported_targets`` to apply them all in one line.
This code example is taken from :idf_file:`pytest_gptimer_example.py <examples/peripherals/timer_group/gptimer/pytest_gptimer_example.py>`.
..code:: python
@pytest.mark.supported_targets
@pytest.mark.generic
def test_gptimer_example(dut: Dut) -> None:
...
Use Params to Specify the sdkconfig Files
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
You can use ``pytest.mark.parametrize`` with “config” to apply the same test to different apps with different sdkconfig files. For more information about ``sdkconfig.ci.xxx`` files, please refer to the Configuration Files section under :idf_file:`this readme <tools/test_apps/README.md>`.
..code:: python
@pytest.mark.parametrize('config', [
'history', # <-- run with app built by sdkconfig.ci.history
'nohistory', # <-- run with app built by sdkconfig.ci.nohistory
], indirect=True) # <-- `indirect=True` is required
Here the first dut was flashed with the app :idf_file:`softap <examples/wifi/getting_started/softAP/main/softap_example_main.c>`, and the second dut was flashed with the app :idf_file:`station <examples/wifi/getting_started/station/main/station_example_main.c>`.
..note::
Here the ``app_path`` should be set with absolute path. the ``__file__`` macro in python would return the absolute path of the test script itself.
This code example is taken from :idf_file:`pytest_wifi_getting_started.py <examples/wifi/getting_started/pytest_wifi_getting_started.py>`. As the comment says, for now it's not running in the ESP-IDF CI.
``monkeypatch_module`` provide a `module-scoped <https://docs.pytest.org/en/latest/how-to/fixtures.html#scope-sharing-fixtures-across-classes-modules-packages-or-session>`__`monkeypatch <https://docs.pytest.org/en/latest/how-to/monkeypatch.html>`__ fixture.
``replace_dut_class`` is a `module-scoped <https://docs.pytest.org/en/latest/how-to/fixtures.html#scope-sharing-fixtures-across-classes-modules-packages-or-session>`__`autouse <https://docs.pytest.org/en/latest/how-to/fixtures.html#autouse-fixtures-fixtures-you-don-t-have-to-request>`__ fixture. This function replaces the ``IdfDut`` class with your custom class.
Now you may mark this test case with marker `xfail <https://docs.pytest.org/en/latest/how-to/skipping.html#xfail-mark-test-functions-as-expected-to-fail>`__ with a user-friendly readable reason.
This code example is taken from :idf_file:`pytest_panic.py <tools/test_apps/system/panic/pytest_panic.py>`
The command used by CI to build all the relevant tests is: ``python $IDF_PATH/tools/ci/ci_build_apps.py <parent_dir> --target <target> -vv --pytest-apps``
For example, If you run ``python $IDF_PATH/tools/ci/ci_build_apps.py $IDF_PATH/examples/system/console/basic --target esp32 --pytest-apps``, the folder structure would be like this:
We’re using two types of custom markers, target markers which indicate that the test cases should support this target, and env markers which indicate that the test case should be assigned to runners with these tags in CI.
You can add new markers by adding one line under the ``${IDF_PATH}/pytest.ini````markers =`` section. The grammar should be: ``<marker_name>: <marker_description>``
You can call pytest with ``--junitxml <filepath>`` to generate the JUnit report. In ESP-IDF, the test case name would be unified as "<target>.<config>.<function_name>".
Sometimes you may need to record some statistics while running the tests, like the performance test statistics.
You can use `record_xml_attribute <https://docs.pytest.org/en/latest/how-to/output.html?highlight=junit#record-xml-attribute>`__ fixture in your test script, and the statistics would be recorded as attributes in the JUnit report.
``pytest-embedded`` is using some threads internally to help gather all stdout to the pexpect process. Due to the limitation of `Global Interpreter Lock <https://en.wikipedia.org/wiki/Global_interpreter_lock>`__, if you're using threads to do performance tests, these threads would block each other and there would be great performance loss.
**workaround**
Use `Process <https://docs.python.org/3/library/multiprocessing.html#the-process-class>`__ instead, the APIs should be almost the same as ``Thread``.
