From 3e1a6afb00eb665962bc09c2da1a85a6d26c671a Mon Sep 17 00:00:00 2001 From: Ren Pei Ying Date: Thu, 24 Aug 2023 05:32:11 +0800 Subject: [PATCH] docs: add CN translation for contribute/esp-idf-tests-with-pytest.rst --- docs/en/contribute/copyright-guide.rst | 2 - .../contribute/esp-idf-tests-with-pytest.rst | 509 +++++++------ docs/zh_CN/contribute/copyright-guide.rst | 6 +- .../contribute/esp-idf-tests-with-pytest.rst | 710 +++++++++++++++++- 4 files changed, 977 insertions(+), 250 deletions(-) diff --git a/docs/en/contribute/copyright-guide.rst b/docs/en/contribute/copyright-guide.rst index 26e6526fb8..a4ac059a43 100644 --- a/docs/en/contribute/copyright-guide.rst +++ b/docs/en/contribute/copyright-guide.rst @@ -3,8 +3,6 @@ Copyright Header Guide :link_to_translation:`zh_CN:[中文]` -.. highlight:: c - ESP-IDF is released under :project_file:`the Apache License 2.0 ` with some additional third-party copyrighted code released under various licenses. For further information please refer to :doc:`the list of copyrights and licenses <../../../COPYRIGHT>`. This page explains how the source code should be properly marked with a copyright header. ESP-IDF uses the `Software Package Data Exchange (SPDX) `_ format which is short and can be easily read by humans or processed by automated tools for copyright checks. diff --git a/docs/en/contribute/esp-idf-tests-with-pytest.rst b/docs/en/contribute/esp-idf-tests-with-pytest.rst index d0c2108a07..2ba0b276b8 100644 --- a/docs/en/contribute/esp-idf-tests-with-pytest.rst +++ b/docs/en/contribute/esp-idf-tests-with-pytest.rst @@ -1,29 +1,39 @@ -=============================== -ESP-IDF Tests with Pytest Guide -=============================== +================= +pytest in ESP-IDF +================= -This documentation is a guide that introduces the following aspects: +:link_to_translation:`zh_CN:[中文]` -1. The basic idea of different test types in ESP-IDF -2. How to apply the pytest framework to the test python scripts to make sure the apps are working as expected. -3. ESP-IDF CI target test process -4. Run ESP-IDF tests with pytest locally -5. Tips and tricks on pytest +ESP-IDF has numerous types of tests that are meant to be executed on an ESP chip (known as **on target testing**). Target tests are usually compiled as part of an IDF project used for testing (known as a **test app**), where test apps follows the same build, flash, and monitor process of any other standard IDF project. -Disclaimer -========== +Typically, on target testing will require a connected host (e.g., a PC) that is responsible for triggering a particular test case, providing test data, and inspecting test results. -In ESP-IDF, we use the following plugins by default: +ESP-IDF uses the pytest framework (and some pytest plugins) on the host side to automate on target testing. This guide introduces pytest in ESP-IDF and covers the following concepts: -- `pytest-embedded `__ with default services ``esp,idf`` -- `pytest-rerunfailures `__ +1. The different types of test apps in ESP-IDF. +2. Using the pytest framework in Python scripts to automate target testing. +3. ESP-IDF Continuous Integration (CI) target testing process. +4. How to run target tests locally with pytest. +5. pytest tips and tricks. -All the introduced concepts and usages are based on the default behavior in ESP-IDF. Not all of them are available in vanilla pytest. +.. note:: + + In ESP-IDF, we use the following pytest plugins by default: + + - `pytest-embedded `__ with default services ``esp,idf`` + - `pytest-rerunfailures `__ + + All the concepts and usages introduced in this guide are based on the default behavior of these plugins, thus may not be available in vanilla pytest. Installation ============ -All dependencies could be installed by running the install script with the ``--enable-pytest`` argument, e.g., ``$ install.sh --enable-pytest``. +All dependencies could be installed by running the install script with the ``--enable-pytest`` argument: + +.. code-block:: bash + + $ install.sh --enable-pytest + Common Issues During Installation --------------------------------- @@ -31,251 +41,260 @@ Common Issues During Installation No Package 'dbus-1' Found ^^^^^^^^^^^^^^^^^^^^^^^^^ -If you are facing an error message like: - .. code:: text - configure: error: Package requirements (dbus-1 >= 1.8) were not met: - - No package 'dbus-1' found - - Consider adjusting the PKG_CONFIG_PATH environment variable if you - installed software in a non-standard prefix. + configure: error: Package requirements (dbus-1 >= 1.8) were not met: -If you are running under ubuntu system, you may need to run: + No package 'dbus-1' found + + Consider adjusting the PKG_CONFIG_PATH environment variable if you + installed software in a non-standard prefix. + +If you encounter the error message above, you may need to install some missing packages. + +If you are using Ubuntu, you may need to run: .. code:: shell - sudo apt-get install libdbus-glib-1-dev + sudo apt-get install libdbus-glib-1-dev -or +or .. code:: shell - sudo apt-get install libdbus-1-dev + sudo apt-get install libdbus-1-dev -For other linux distros, you may Google the error message and find the solution. This issue could be solved by installing the related header files. +For other Linux distributions, please Google the error message above and find which missing packages need to be installed for your particular distribution. Invalid Command 'bdist_wheel' ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -If you are facing an error message like: - .. code:: text - error: invalid command 'bdist_wheel' + error: invalid command 'bdist_wheel' -You may need to run: +If you encounter the error message above, you may need to install some missing Python packages such as: .. code:: shell - python -m pip install -U pip + python -m pip install -U pip -Or +or .. code:: shell - python -m pip install wheel + python -m pip install wheel -Before running the pip commands, please make sure you are using the IDF python virtual environment. +.. note:: -Basic Concepts -============== + Before running the pip commands, please make sure you are using the IDF Python virtual environment. -Component-based Unit Tests --------------------------- -Component-based unit tests are our recommended way to test your component. All the test apps should be located under ``${IDF_PATH}/components//test_apps``. +Test Apps +========= -For example: +ESP-IDF contains different types of test apps that can be automated using pytest. + +Component Tests +--------------- + +ESP-IDF components typically contain component specific test apps that execute component specific unit tests. Component test apps are the recommended way to test components. All the test apps should be located under ``${IDF_PATH}/components//test_apps``, for example: .. code:: text - components/ - └── my_component/ - ├── include/ - │ └── ... - ├── test_apps/ - │ ├── test_app_1 - │ │ ├── main/ - │ │ │ └── ... - │ │ ├── CMakeLists.txt - │ │ └── pytest_my_component_app_1.py - │ ├── test_app_2 - │ │ ├── ... - │ │ └── pytest_my_component_app_2.py - │ └── parent_folder - │ ├── test_app_3 - │ │ ├── ... - │ │ └── pytest_my_component_app_3.py - │ └── ... - ├── my_component.c - └── CMakeLists.txt + components/ + └── my_component/ + ├── include/ + │ └── ... + ├── test_apps/ + │ ├── test_app_1 + │ │ ├── main/ + │ │ │ └── ... + │ │ ├── CMakeLists.txt + │ │ └── pytest_my_component_app_1.py + │ ├── test_app_2 + │ │ ├── ... + │ │ └── pytest_my_component_app_2.py + │ └── parent_folder + │ ├── test_app_3 + │ │ ├── ... + │ │ └── pytest_my_component_app_3.py + │ └── ... + ├── my_component.c + └── CMakeLists.txt Example Tests ------------- -Example Tests are tests for examples that are intended to demonstrate parts of the ESP-IDF functionality to our customers. +The purpose of ESP-IDF examples is to demonstrate parts of ESP-IDF functionality to users (refer to :idf_file:`Examples Readme ` for more information). -All the test apps should be located under ``${IDF_PATH}/examples``. For more information please refer to the :idf_file:`Examples Readme `. - -For example: +However, to ensure that these examples operate correctly, examples can be treated as test apps and executed automatically by using pytest. All examples should be located under ``${IDF_PATH}/examples``, with tested example including a Python test script, for example: .. code:: text - examples/ - └── parent_folder/ - └── example_1/ - ├── main/ - │ └── ... - ├── CMakeLists.txt - └── pytest_example_1.py + examples/ + └── parent_folder/ + └── example_1/ + ├── main/ + │ └── ... + ├── CMakeLists.txt + └── pytest_example_1.py Custom Tests ------------ -Custom Tests are tests that aim to run some arbitrary test internally. They are not intended to demonstrate the ESP-IDF functionality to our customers in any way. +Custom Tests are tests that aim to test some arbitrary functionality of ESP-IDF, thus are not intended to demonstrate IDF functionality to users in any way. -All the test apps should be located under ``${IDF_PATH}/tools/test_apps``. For more information please refer to the :idf_file:`Custom Test Readme `. +All custom test apps are located under ``${IDF_PATH}/tools/test_apps``. For more information please refer to the :idf_file:`Custom Test Readme `. -Pytest in ESP-IDF +pytest in ESP-IDF ================= -Pytest Execution Process +.. _pytest-execution-process: + +pytest Execution Process ------------------------ 1. Bootstrapping Phase - Create session-scoped caches: + Create session-scoped caches: - - port-target cache - - port-app cache + - port-target cache + - port-app cache 2. Collection Phase - 1. Get all the python files with the prefix ``pytest_`` - 2. Get all the test functions with the prefix ``test_`` - 3. Apply the `params `__, and duplicate the test functions. - 4. Filter the test cases with CLI options. Introduced detailed usages `here <#filter-the-test-cases>`__ + A. Gather all Python files with the prefix ``pytest_``. + B. Gather all test functions with the prefix ``test_``. + C. Apply the `params `__, and duplicate the test functions. + D. Filter the test cases with CLI options. For the detailed usages, see :ref:`filter-the-test-cases`. -3. Test Running Phase +3. Execution Phase - 1. Construct the `fixtures `__. In ESP-IDF, the common fixtures are initialized in this order: + A. Construct the `fixtures `__. In ESP-IDF, the common fixtures are initialized in this order: - 1. ``pexpect_proc``: `pexpect `__ instance + a. ``pexpect_proc``: `pexpect `__ instance - 2. ``app``: `IdfApp `__ instance + b. ``app``: `IdfApp `__ instance - The information of the app, like sdkconfig, flash_files, partition_table, etc., would be parsed at this phase. + The test app's information (e.g., sdkconfig, flash_files, partition_table, etc) would be parsed at this phase. - 3. ``serial``: `IdfSerial `__ instance + c. ``serial``: `IdfSerial `__ instance - The port of the host which connected to the target type parsed from the app would be auto-detected. The flash files would be auto flashed. + The port of the host to which the target is connected is auto-detected. In the case of multiple targets connected to the host, the test target's type is parsed from the app. The test app binary files are flashed to the test target automatically. - 4. ``dut``: `IdfDut `__ instance + d. ``dut``: `IdfDut `__ instance - 2. Run the real test function + B. Run the real test function. - 3. Deconstruct the fixtures in this order: + C. Deconstruct the fixtures in this order: - 1. ``dut`` + a. ``dut`` - 1. close the ``serial`` port - 2. (Only for apps with `unity test framework `__) generate junit report of the unity test cases + i. close the ``serial`` port. + ii. (Only for apps with `Unity test framework `__) generate JUnit report of the Unity test cases. - 2. ``serial`` - 3. ``app`` - 4. ``pexpect_proc``: Close the file descriptor + b. ``serial`` + c. ``app`` + d. ``pexpect_proc``: Close the file descriptor - 4. (Only for apps with `unity test framework `__) + D. (Only for apps with `Unity test framework `__) - Raise ``AssertionError`` when detected unity test failed if you call ``dut.expect_from_unity_output()`` in the test function. + If ``dut.expect_from_unity_output()`` is called, an ``AssertionError`` is raised upon detection of a Unity test failure. 4. Reporting Phase - 1. Generate junit report of the test functions - 2. Modify the junit report test case name into ESP-IDF test case ID format: ``..`` + A. Generate JUnit report of the test functions. + B. Modify the JUnit report test case name into ESP-IDF test case ID format: ``..``. -5. Finalizing Phase (Only for apps with `unity test framework `__) +5. Finalizatoin Phase (Only for apps with `Unity test framework `__) - Combine the junit reports if the junit reports of the unity test cases are generated. + Combine the JUnit reports if the JUnit reports of the Unity test cases are generated. -Getting Started Example ------------------------ +Basic Example +------------- -This code example is taken from :idf_file:`pytest_console_basic.py `. +This following Python test script example is taken from :idf_file:`pytest_console_basic.py `. .. code:: python - @pytest.mark.esp32 - @pytest.mark.esp32c3 - @pytest.mark.generic - @pytest.mark.parametrize('config', [ - 'history', - 'nohistory', - ], indirect=True) - def test_console_advanced(config: str, dut: IdfDut) -> None: - if config == 'history': - dut.expect('Command history enabled') - elif config == 'nohistory': - dut.expect('Command history disabled') + @pytest.mark.esp32 + @pytest.mark.esp32c3 + @pytest.mark.generic + @pytest.mark.parametrize('config', [ + 'history', + 'nohistory', + ], indirect=True) + def test_console_advanced(config: str, dut: IdfDut) -> None: + if config == 'history': + dut.expect('Command history enabled') + elif config == 'nohistory': + dut.expect('Command history disabled') -Let us go through this simple test case line by line in the following subsections. +To demonstrate how pytest is typically used in an ESP-IDF test script, let us go through this simple test script line by line in the following subsections. -Use Markers to Specify the Supported Targets -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Target Markers +^^^^^^^^^^^^^^ + +Pytest markers can be used to indicate which targets (i.e., which ESP chip) a particular test case should should run on. For example: .. code:: python - @pytest.mark.esp32 # <-- support esp32 - @pytest.mark.esp32c3 # <-- support esp32c3 - @pytest.mark.generic # <-- test env "generic" + @pytest.mark.esp32 # <-- support esp32 + @pytest.mark.esp32c3 # <-- support esp32c3 + @pytest.mark.generic # <-- test env "generic" -The above lines indicate that this test case supports target esp32 and esp32c3, the target board type should be "generic". If you want to know what is the "generic" type refers to, you may run ``pytest --markers`` to get the detailed information of all markers. +The example above indicates that a particular test case is supported on the ESP32 and ESP32-C3. Furthermore, the target's board type should be ``generic``. For more details regarding the ``generic`` type, you may run ``pytest --markers`` to get detailed information regarding all markers. .. note:: - If the test case supports all officially ESP-IDF supported targets (You may check the value via "idf.py --list-targets"), you can use a special marker ``supported_targets`` to apply all of them in one line. + If the test case can be run on all targets officially supported by ESP-IDF (call ``idf.py --list-targets`` for more details), you can use a special marker ``supported_targets`` to apply all of them in one line. -Use Params to Specify the ``sdkconfig`` Files -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Parameterized Markers +^^^^^^^^^^^^^^^^^^^^^ -You can use ``pytest.mark.parametrize`` with "config" to apply the same test to different apps with different sdkconfig files. For more information about ``sdkconfig.ci.xxx`` files, please refer to the Configuration Files section under :idf_file:`this readme `. +You can use ``pytest.mark.parametrize`` with ``config`` to apply the same test to different apps with different sdkconfig files. For more information about ``sdkconfig.ci.xxx`` files, please refer to the Configuration Files section under :idf_file:`this readme `. .. code:: python - @pytest.mark.parametrize('config', [ - 'history', # <-- run with app built by sdkconfig.ci.history - 'nohistory', # <-- run with app built by sdkconfig.ci.nohistory - ], indirect=True) # <-- `indirect=True` is required + @pytest.mark.parametrize('config', [ + 'history', # <-- run with app built by sdkconfig.ci.history + 'nohistory', # <-- run with app built by sdkconfig.ci.nohistory + ], indirect=True) # <-- `indirect=True` is required Overall, this test function would be replicated to 4 test cases: -- esp32.history.test_console_advanced -- esp32.nohistory.test_console_advanced -- esp32c3.history.test_console_advanced -- esp32c3.nohistory.test_console_advanced +- ``esp32.history.test_console_advanced`` +- ``esp32.nohistory.test_console_advanced`` +- ``esp32c3.history.test_console_advanced`` +- ``esp32c3.nohistory.test_console_advanced`` -Expect From the Serial Output -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Testing Serial Output +^^^^^^^^^^^^^^^^^^^^^ + +To ensure that test has executed successfully on target, the test script can test that serial output of the target using the ``dut.expect()`` function, for example: .. code:: python - - def test_console_advanced(config: str, dut: IdfDut) -> None: # The value of argument ``config`` is assigned by the parametrization. - if config == 'history': - dut.expect('Command history enabled') - elif config == 'nohistory': - dut.expect('Command history disabled') -When we are using ``dut.expect(...)``, the string would be compiled into regex at first, and then seeks through the serial output until the compiled regex is matched, or a timeout is exceeded. You may have to pay extra attention when the string contains regex keyword characters, like parentheses, or square brackets. + def test_console_advanced(config: str, dut: IdfDut) -> None: # The value of argument ``config`` is assigned by the parameterization. + if config == 'history': + dut.expect('Command history enabled') + elif config == 'nohistory': + dut.expect('Command history disabled') -Actually using ``dut.expect_exact(...)`` here is better, since it would seek until the string is matched. For further reading about the different types of ``expect`` functions, please refer to the `pytest-embedded Expecting documentation `__. +The ``dut.expect(...)`` will first compile the expected string into regex, which in turn is then used to seek through the serial output until the compiled regex is matched, or until a timeout occurs. + +Please pay extra attention to the expected string when it contains regex keyword characters (e.g., parentheses, square brackets). Alternatively, you may use ``dut.expect_exact(...)`` that will attempt to match the string without converting it into regex. + +For more information regarding the different types of ``expect`` functions, please refer to the `pytest-embedded Expecting documentation `__. Advanced Examples ----------------- -Multi Dut Tests with the Same App -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Multi-Target Tests with the Same App +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +In some cases a test may involve multiple targets running the same test app. In this case, multiple DUTs can be instantiated using ``parameterize``, for example: .. code:: python @@ -292,8 +311,10 @@ Multi Dut Tests with the Same App After setting the param ``count`` to 2, all these fixtures are changed into tuples. -Multi Dut Tests with Different Apps -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Multi-Target Tests with Different Apps +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +In some cases (in particular protocol tests), a test may involve multiple targets running different test apps (e.g., separate targets to act as master and slave). In this case, multiple DUTs with different test apps can be instantiated using ``parameterize``. This code example is taken from :idf_file:`pytest_wifi_getting_started.py `. @@ -312,14 +333,14 @@ This code example is taken from :idf_file:`pytest_wifi_getting_started.py `, and the second dut was flashed with the app :idf_file:`station `. +Here the first DUT was flashed with the app :idf_file:`softAP `, and the second DUT was flashed with the app :idf_file:`station `. .. note:: - Here the ``app_path`` should be set with absolute path. the ``__file__`` macro in python would return the absolute path of the test script itself. + Here the ``app_path`` should be set with absolute path. The ``__file__`` macro in Python would return the absolute path of the test script itself. -Multi Dut Tests with Different Apps, and Targets -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Multi-Target Tests with Different Apps and Targets +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This code example is taken from :idf_file:`pytest_wifi_getting_started.py `. As the comment says, for now it is not running in the ESP-IDF CI. @@ -343,11 +364,11 @@ This code example is taken from :idf_file:`pytest_wifi_getting_started.py ` as an advanced example. @@ -365,58 +386,58 @@ This code example is taken from :idf_file:`pytest_panic.py `__ +1. Add more reusable functions for a certain number of DUTs. +2. Add custom setup and teardown functions in different phases described in Section :ref:`pytest-execution-process`. -This code example is taken from :idf_file:`panic/conftest.py ` +This code example is taken from :idf_file:`panic/conftest.py `. .. code:: python - class PanicTestDut(IdfDut): - ... + class PanicTestDut(IdfDut): + ... - @pytest.fixture(scope='module') - def monkeypatch_module(request: FixtureRequest) -> MonkeyPatch: - mp = MonkeyPatch() - request.addfinalizer(mp.undo) - return mp + @pytest.fixture(scope='module') + def monkeypatch_module(request: FixtureRequest) -> MonkeyPatch: + mp = MonkeyPatch() + request.addfinalizer(mp.undo) + return mp - @pytest.fixture(scope='module', autouse=True) - def replace_dut_class(monkeypatch_module: MonkeyPatch) -> None: - monkeypatch_module.setattr('pytest_embedded_idf.dut.IdfDut', PanicTestDut) + @pytest.fixture(scope='module', autouse=True) + def replace_dut_class(monkeypatch_module: MonkeyPatch) -> None: + monkeypatch_module.setattr('pytest_embedded_idf.dut.IdfDut', PanicTestDut) -``monkeypatch_module`` provide a `module-scoped `__ `monkeypatch `__ fixture. +``monkeypatch_module`` provides a `module-scoped `__ `monkeypatch `__ fixture. ``replace_dut_class`` is a `module-scoped `__ `autouse `__ fixture. This function replaces the ``IdfDut`` class with your custom class. Mark Flaky Tests ^^^^^^^^^^^^^^^^ -Sometimes, our test is based on ethernet or wifi. The network may cause the test flaky. We could mark the single test case within the code repo. +Certain test cases are based on Ethernet or Wi-Fi. However, the test may be flaky due to networking issues. Thus, it is possible to mark a particular test case as flaky. -This code example is taken from :idf_file:`pytest_esp_eth.py ` +This code example is taken from :idf_file:`pytest_esp_eth.py `. .. code:: python - @pytest.mark.flaky(reruns=3, reruns_delay=5) - def test_esp_eth_ip101(dut: IdfDut) -> None: - ... + @pytest.mark.flaky(reruns=3, reruns_delay=5) + def test_esp_eth_ip101(dut: IdfDut) -> None: + ... This flaky marker means that if the test function failed, the test case would rerun for a maximum of 3 times with 5 seconds delay. -Mark Known Failure Cases -^^^^^^^^^^^^^^^^^^^^^^^^ +Mark Known Failures +^^^^^^^^^^^^^^^^^^^ -Sometimes a test could not pass for the following reasons: +Sometimes, a test can consistently fail for the following reasons: -- Has a bug -- The success ratio is too low because of environment issue, such as network issue. Retry could not help +- The feature under test (or the test itself) has a bug. +- The test environment is unstable (e.g., due to network issues) leading to a high failure ratio. Now you may mark this test case with marker `xfail `__ with a user-friendly readable reason. @@ -424,10 +445,10 @@ This code example is taken from :idf_file:`pytest_panic.py None: + @pytest.mark.xfail('config.getvalue("target") == "esp32s2"', reason='raised IllegalInstruction instead') + def test_cache_error(dut: PanicTestDut, config: str, test_func_name: str) -> None: -This marker means that if the test would be a known failure one on esp32s2. +This marker means that test is a known failure on the ESP32-S2. Mark Nightly Run Test Cases ^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -440,8 +461,8 @@ Some test cases are only triggered in nightly run pipelines due to a lack of run This marker means that the test case would only be run with env var ``NIGHTLY_RUN`` or ``INCLUDE_NIGHTLY_RUN``. -Mark Temp Disabled in CI -^^^^^^^^^^^^^^^^^^^^^^^^ +Mark Temporarily Disabled in CI +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Some test cases which can pass locally may need to be temporarily disabled in CI due to a lack of runners. @@ -454,36 +475,36 @@ This marker means that the test case could still be run locally with ``pytest -- Run Unity Test Cases ^^^^^^^^^^^^^^^^^^^^ -For component-based unit test apps, one line could do the trick to run all single-board test cases, including normal test cases and multi-stage test cases: +For component-based unit test apps, all single-board test cases (including normal test cases and multi-stage test cases) can be run using the following command: .. code:: python - def test_component_ut(dut: IdfDut): - dut.run_all_single_board_cases() + def test_component_ut(dut: IdfDut): + dut.run_all_single_board_cases() -It would also skip all the test cases with ``[ignore]`` mark. +Using this command will skip all the test cases containing the ``[ignore]`` tag. If you need to run a group of test cases, you may run: .. code:: python - def test_component_ut(dut: IdfDut): - dut.run_all_single_board_cases(group='psram') + def test_component_ut(dut: IdfDut): + dut.run_all_single_board_cases(group='psram') -It would trigger all test cases with module name ``[psram]``. +It would trigger all test cases with the ``[psram]`` tag. You may also see that there are some test scripts with the following statements, which are deprecated. Please use the suggested one as above. .. code:: python - def test_component_ut(dut: IdfDut): - dut.expect_exact('Press ENTER to see the list of tests') - dut.write('*') - dut.expect_unity_test_output() + def test_component_ut(dut: IdfDut): + dut.expect_exact('Press ENTER to see the list of tests') + dut.write('*') + dut.expect_unity_test_output() For further reading about our unit testing in ESP-IDF, please refer to :doc:`our unit testing guide <../api-guides/unit-tests>`. -Run the Tests in CI +Running Tests in CI =================== The workflow in CI is simple, build jobs > target test jobs. @@ -509,17 +530,17 @@ For example, If you run ``python $IDF_PATH/tools/ci/ci_build_apps.py $IDF_PATH/e .. code:: text - basic - ├── build_esp32_history/ - │ └── ... - ├── build_esp32_nohistory/ - │ └── ... - ├── main/ - ├── CMakeLists.txt - ├── pytest_console_basic.py - └── ... + basic + ├── build_esp32_history/ + │ └── ... + ├── build_esp32_nohistory/ + │ └── ... + ├── main/ + ├── CMakeLists.txt + ├── pytest_console_basic.py + └── ... -All the binaries folders would be uploaded as artifacts under the same directories. +All the build folders would be uploaded as artifacts under the same directories. Target Test Jobs ---------------- @@ -538,18 +559,18 @@ The command used by CI to run all the relevant tests is: ``pytest - All test cases with the specified target marker and the test env marker under the parent folder would be executed. -The binaries in the target test jobs are downloaded from build jobs, the artifacts would be placed under the same directories. +The binaries in the target test jobs are downloaded from build jobs. the artifacts would be placed under the same directories. -Run the Tests Locally +Running Tests Locally ===================== -First you need to install ESP-IDF with additional python requirements: +First you need to install ESP-IDF with additional Python requirements: .. code-block:: shell - $ cd $IDF_PATH - $ bash install.sh --enable-pytest - $ . ./export.sh + $ cd $IDF_PATH + $ bash install.sh --enable-pytest + $ . ./export.sh By default, the pytest script will look for the build directory in this order: @@ -564,9 +585,9 @@ For example, if you want to run all the esp32 tests under the ``$IDF_PATH/exampl .. code-block:: shell - $ cd examples/get-started/hello_world - $ idf.py build - $ pytest --target esp32 + $ cd examples/get-started/hello_world + $ idf.py build + $ pytest --target esp32 If you have multiple sdkconfig files in your test app, like those ``sdkconfig.ci.*`` files, the simple ``idf.py build`` won't apply the extra sdkconfig files. Let us take ``$IDF_PATH/examples/system/console/basic`` as an example. @@ -574,47 +595,49 @@ If you want to test this app with config ``history``, and build with ``idf.py bu .. code-block:: shell - $ cd examples/system/console/basic - $ idf.py -DSDKCONFIG_DEFAULTS="sdkconfig.defaults;sdkconfig.ci.history" build - $ pytest --target esp32 --sdkconfig history + $ cd examples/system/console/basic + $ idf.py -DSDKCONFIG_DEFAULTS="sdkconfig.defaults;sdkconfig.ci.history" build + $ pytest --target esp32 --sdkconfig history If you want to build and test with all sdkconfig files at the same time, you should use our CI script as an helper script: .. code-block:: shell - $ cd examples/system/console/basic - $ python $IDF_PATH/tools/ci/ci_build_apps.py . --target esp32 -vv --pytest-apps - $ pytest --target esp32 + $ cd examples/system/console/basic + $ python $IDF_PATH/tools/ci/ci_build_apps.py . --target esp32 -vv --pytest-apps + $ pytest --target esp32 The app with ``sdkconfig.ci.history`` will be built in ``build_esp32_history``, and the app with ``sdkconfig.ci.nohistory`` will be built in ``build_esp32_nohistory``. ``pytest --target esp32`` will run tests on both apps. Tips and Tricks =============== +.. _filter-the-test-cases: + Filter the Test Cases --------------------- -- filter by target with ``pytest --target `` +- Filter by target with ``pytest --target `` pytest would run all the test cases that support specified target. -- filter by sdkconfig file with ``pytest --sdkconfig `` +- Filter by sdkconfig file with ``pytest --sdkconfig `` - if ```` is ``default``, pytest would run all the test cases with the sdkconfig file ``sdkconfig.defaults``. + If ```` is ``default``, pytest would run all the test cases with the sdkconfig file ``sdkconfig.defaults``. In other cases, pytest would run all the test cases with sdkconfig file ``sdkconfig.ci.``. Add New Markers --------------- -We are using two types of custom markers, target markers which indicate that the test cases should support this target, and env markers which indicate that the test case should be assigned to runners with these tags in CI. +We are using two types of custom markers, target markers which indicate that the test cases should support this target, and env markers which indicate that the test cases should be assigned to runners with these tags in CI. -You can add new markers by adding one line under the ``${IDF_PATH}/conftest.py``. If it is a target marker, it should be added into ``TARGET_MARKERS``. If it is a marker that specifies a type of test environment, it should be added into ``ENV_MARKERS``. The grammar should be: ``: ``. +You can add new markers by adding one line under the ``${IDF_PATH}/conftest.py``. If it is a target marker, it should be added into ``TARGET_MARKERS``. If it is a marker that specifies a type of test environment, it should be added into ``ENV_MARKERS``. The syntax should be: ``: ``. Generate JUnit Report --------------------- -You can call pytest with ``--junitxml `` to generate the JUnit report. In ESP-IDF, the test case name would be unified as "..". +You can call pytest with ``--junitxml `` to generate the JUnit report. In ESP-IDF, the test case name would be unified as ``..`__ to achieve this. +You can use `Python logging module `__ to achieve this. Useful Logging Functions (as Fixture) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -651,7 +674,7 @@ Useful Logging Functions (as Fixture) ) -> None: log_performance('test', 1) -The above example would log the performance item with pre-defined format: "[performance][test]: 1" and record it under the ``properties`` tag in the junit report if ``--junitxml `` is specified. The junit test case node would look like: +The above example would log the performance item with pre-defined format: ``[performance][test]: 1`` and record it under the ``properties`` tag in the JUnit report if ``--junitxml `` is specified. The JUnit test case node would look like: .. code:: html @@ -664,7 +687,7 @@ The above example would log the performance item with pre-defined format: "[perf ``check_performance`` """"""""""""""""""""" -We provide C macros ``TEST_PERFORMANCE_LESS_THAN`` and ``TEST_PERFORMANCE_GREATER_THAN`` to log the performance item and check if the value is in the valid range. Sometimes the performance item value could not be measured in C code, so we also provide a python function for the same purpose. Please note that using C macros is the preferred approach, since the python function could not recognize the threshold values of the same performance item under different ifdef blocks well. +We provide C macros ``TEST_PERFORMANCE_LESS_THAN`` and ``TEST_PERFORMANCE_GREATER_THAN`` to log the performance item and check if the value is in the valid range. Sometimes the performance item value could not be measured in C code, so we also provide a Python function for the same purpose. Please note that using C macros is the preferred approach, since the Python function could not recognize the threshold values of the same performance item under different ifdef blocks well. .. code:: python @@ -677,10 +700,10 @@ We provide C macros ``TEST_PERFORMANCE_LESS_THAN`` and ``TEST_PERFORMANCE_GREATE The above example would first get the threshold values of the performance item ``RSA_2048KEY_PUBLIC_OP`` from :idf_file:`components/idf_test/include/idf_performance.h` and the target-specific one :idf_file:`components/idf_test/include/esp32/idf_performance_target.h`, then check if the value reached the minimum limit or exceeded the maximum limit. -Let us assume the value of ``IDF_PERFORMANCE_MAX_RSA_2048KEY_PUBLIC_OP`` is 19000. so the first ``check_performance`` line would pass and the second one would fail with warning: ``[Performance] RSA_2048KEY_PUBLIC_OP value is 19001, doesn\'t meet pass standard 19000.0`` +Let us assume the value of ``IDF_PERFORMANCE_MAX_RSA_2048KEY_PUBLIC_OP`` is 19000. so the first ``check_performance`` line would pass and the second one would fail with warning: ``[Performance] RSA_2048KEY_PUBLIC_OP value is 19001, doesn\'t meet pass standard 19000.0``. Further Readings ================ - pytest documentation: https://docs.pytest.org/en/latest/contents.html -- pytest-embedded documentation: https://docs.espressif.com/projects/pytest-embedded/en/latest/ +- pytest-embedded documentation: https://docs.espressif.com/projects/pytest-embedded/en/latest/ \ No newline at end of file diff --git a/docs/zh_CN/contribute/copyright-guide.rst b/docs/zh_CN/contribute/copyright-guide.rst index 9a879d9b64..1bb17c927a 100644 --- a/docs/zh_CN/contribute/copyright-guide.rst +++ b/docs/zh_CN/contribute/copyright-guide.rst @@ -3,9 +3,7 @@ :link_to_translation:`en:[English]` -.. highlight:: c - -ESP-IDF 基于 Apache License 2.0，并包含一些不同许可证下的第三方版权代码。要了解更多信息，请参考 :doc:`the list of copyrights and licenses <../../../COPYRIGHT>`。 +ESP-IDF 基于 :project_file:`the Apache License 2.0 `，并包含一些不同许可证下的第三方版权代码。要了解更多信息，请参考 :doc:`the list of copyrights and licenses <../../../COPYRIGHT>`。 本页面介绍了如何在源代码中正确标注版权标头。ESP-IDF 使用 `Software Package Data Exchange (SPDX) `_ 格式，简短易读，能够方便自动化工具处理及进行版权检查。 @@ -86,4 +84,4 @@ ESP-IDF 的某些部分特意采用了限制性较小的许可证，方便在商 * 可以永久禁用对选定文件集的检查。请谨慎使用该选项，并且仅在其他选项都不适用时，才可使用该选项。 .. _SPDX license list: https://spdx.org/licenses -.. _LicenseRef-[idString]: https://spdx.github.io/spdx-spec/v2.3/other-licensing-information-detected/#101-license-identifier-field \ No newline at end of file +.. _LicenseRef-[idString]: https://spdx.github.io/spdx-spec/v2.3/other-licensing-information-detected/#101-license-identifier-field diff --git a/docs/zh_CN/contribute/esp-idf-tests-with-pytest.rst b/docs/zh_CN/contribute/esp-idf-tests-with-pytest.rst index 50d587a152..4ec5c6673c 100644 --- a/docs/zh_CN/contribute/esp-idf-tests-with-pytest.rst +++ b/docs/zh_CN/contribute/esp-idf-tests-with-pytest.rst @@ -1 +1,709 @@ -.. include:: ../../en/contribute/esp-idf-tests-with-pytest.rst +======================== +ESP-IDF pytest 指南 +======================== + +:link_to_translation:`en:[English]` + +ESP-IDF 有多种类型的测试需在 ESP 芯片上执行（即 **目标测试**）。目标测试通常作为 IDF 测试项目（即 **测试应用程序**）的一部分进行编译，在这个过程中，测试应用程序和其他标准 IDF 项目遵循同样的构建、烧写和监控流程。 + +通常，目标测试需要连接一台主机（如个人电脑），负责触发特定的测试用例、提供测试数据、检查测试结果。 + +ESP-IDF 在主机端使用 pytest 框架（以及一些 pytest 插件）来自动进行目标测试。本文档介绍 ESP-IDF 中的 pytest，并介绍以下内容： + +1. ESP-IDF 中不同类型的测试应用程序。 +2. 将 pytest 框架应用于 Python 测试脚本，进行自动化目标测试。 +3. ESP-IDF CI (Continuous Integration) 板载测试流程。 +4. 使用 pytest 在本地执行目标测试。 +5. pytest 的使用技巧。 + +.. note:: + + ESP-IDF 默认使用以下插件： + + - `pytest-embedded `__ 和默认服务 ``esp,idf`` + - `pytest-rerunfailures `__ + + 本文档介绍的所有概念和用法都基于 ESP-IDF 的默认配置，并非都适用于原生 pytest。 + +安装 +============ + +所有依赖项都可以通过执行安装脚本的 ``--enable-pytest`` 进行安装: + +.. code-block:: bash + + $ install.sh --enable-pytest + + +安装过程常见问题 +---------------------------- + +No Package 'dbus-1' Found +^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. code:: text + + configure: error: Package requirements (dbus-1 >= 1.8) were not met: + + No package 'dbus-1' found + + Consider adjusting the PKG_CONFIG_PATH environment variable if you + installed software in a non-standard prefix. + +如果遇到上述错误信息，可能需要安装一些缺失的软件包。 + +如果使用 Ubuntu 系统，可能需要执行： + +.. code:: shell + + sudo apt-get install libdbus-glib-1-dev + +或 + +.. code:: shell + + sudo apt-get install libdbus-1-dev + +如使用 Linux 其他发行版本，请在搜索引擎中查询上述错误信息，并查找对应发行版需安装哪些缺失的软件包。 + +Invalid command 'bdist_wheel' +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. code:: text + + error: invalid command 'bdist_wheel' + +如果遇到上述错误信息，可能需要安装一些缺失的 Python 包，例如： + +.. code:: shell + + python -m pip install -U pip + +或 + +.. code:: shell + + python -m pip install wheel + +.. note:: + + 执行 pip 命令前，请确保使用的环境为 IDF Python 虚拟环境。 + + +测试应用程序 +================== + +ESP-IDF 包含不同类型的测试应用程序，可用 pytest 自动完成。 + +组件测试 +---------- + +ESP-IDF 组件通常包含针对特定组件的测试应用程序，执行针对特定组件的单元测试。推荐通过组件测试应用程序来测试组件。所有测试应用程序都应位于 ``${IDF_PATH}/components//test_apps`` 下，例如： + +.. code:: text + + components/ + └── my_component/ + ├── include/ + │ └── ... + ├── test_apps/ + │ ├── test_app_1 + │ │ ├── main/ + │ │ │ └── ... + │ │ ├── CMakeLists.txt + │ │ └── pytest_my_component_app_1.py + │ ├── test_app_2 + │ │ ├── ... + │ │ └── pytest_my_component_app_2.py + │ └── parent_folder + │ ├── test_app_3 + │ │ ├── ... + │ │ └── pytest_my_component_app_3.py + │ └── ... + ├── my_component.c + └── CMakeLists.txt + +例程测试 +------------- + +例程测试是为了向用户展示 ESP-IDF 的部分功能（要了解更多信息，请参考 :idf_file:`Examples Readme `）。 + +但是，要确保这些例程正确运行，可将例程看作测试应用，并用 pytest 自动执行。所有例程都应和已测试的例程，包括 Python 测试脚本一起放在 ``${IDF_PATH}/examples`` 路径下，例如： + +.. code:: text + + examples/ + └── parent_folder/ + └── example_1/ + ├── main/ + │ └── ... + ├── CMakeLists.txt + └── pytest_example_1.py + +自定义测试 +-------------- + +自定义测试是为了测试 ESP-IDF 的一些任意功能，这些测试不是为了向用户展示 ESP-IDF 的功能。 + +所有自定义测试应用都位于 ``${IDF_PATH}/tools/test_apps`` 路径下。要了解更多信息，请参考 :idf_file:`Custom Test Readme `。 + +在 ESP-IDF 中使用 pytest +============================ + +.. _pytest-execution-process: + +pytest 执行步骤 +--------------------- + +1. 引导阶段 + +创建会话缓存： + + - 端口目标缓存 + - 端口应用缓存 + +2. 数据获取阶段 + + A. 获取所有前缀为 ``pytest_`` 的 Python 文件。 + B. 获取所有前缀为 ``test_`` 的测试函数。 + C. 应用 `params `__，并复制测试函数。 + D. 利用 CLI 选项筛选测试用例。详细用法请参考 :ref:`filter-the-test-cases`。 + +3. 运行阶段 + + A. 创建 `fixture `__。在 ESP-IDF 中，常见 fixture 的初始化顺序如下： + + a. ``pexpect_proc``: `pexpect `__ 实例 + + b. ``app``: `IdfApp `__ 实例 + + 此阶段会解析测试应用程序的相关信息，如 sdkconfig、flash_files、partition_table 等。 + + c. ``serial``: `IdfSerial `__ 实例 + + 此阶段会自动检测目标芯片所连接的主机端口。考虑到主机可能连接了多个目标芯片，应用程序会解析测试目标芯片的类型。测试程序的二进制文件会自动烧写到测试目标芯片上。 + + d. ``dut``: `IdfDut `__ 实例 + + B. 运行测试函数。 + + C. 析构 fixture。析构顺序如下： + + a. ``dut`` + + i. 关闭 ``serial`` 端口。 + ii. （仅适用于使用了 `Unity 测试框架 `__ 的应用程序）生成 Unity 测试用例的 JUnit 报告。 + + b. ``serial`` + c. ``app`` + d. ``pexpect_proc``：关闭文件描述符 + + D.（仅适用于使用了 `Unity 测试框架 `__ 的应用程序） + + 如果调用了 ``dut.expect_from_unity_output()``，那么检测到 Unity 测试失败时会触发 ``AssertionError``。 + +4. 报告阶段 + + A. 为测试函数生成 Junit 报告。 + B. 将 JUnit 报告中的测试用例名修改为 ESP-IDF 测试用例 ID 格式： ``..``。 + +5. 完成阶段（仅适用于使用了 `Unity 测试框架 `__ 的应用程序） + + 如果生成了 Unity 测试用例的 JUnit 报告，这些报告会被合并。 + +入门示例 +------------- + +以下 Python 测试脚本示例来自 :idf_file:`pytest_console_basic.py `。 + +.. code:: python + + @pytest.mark.esp32 + @pytest.mark.esp32c3 + @pytest.mark.generic + @pytest.mark.parametrize('config', [ + 'history', + 'nohistory', + ], indirect=True) + def test_console_advanced(config: str, dut: IdfDut) -> None: + if config == 'history': + dut.expect('Command history enabled') + elif config == 'nohistory': + dut.expect('Command history disabled') + +下面的小节对这个简单的测试脚本进行了逐行讲解，以说明 pytest 在 ESP-IDF 测试脚本中的典型使用方法。 + +目标芯片 marker +^^^^^^^^^^^^^^^^^^ + +使用 Pytest marker 可以指出某个特定测试用例应在哪个目标芯片（即 ESP 芯片）上运行。例如： + +.. code:: python + + @pytest.mark.esp32 # <-- support esp32 + @pytest.mark.esp32c3 # <-- support esp32c3 + @pytest.mark.generic # <-- test env "generic" + +上例表明，某一测试用例可以在 ESP32 和 ESP32-C3 上运行。此外，目标芯片的类型应为 ``generic``。要了解有关 ``generic`` 类型，运行 ``pytest --markers`` 以获取所有 marker 的详细信息。 + +.. note:: + + 如果测试用例可以在 ESP-IDF 官方支持的所有目标芯片上运行（调用 ``idf.py --list-targets`` 了解详情），则可以使用特殊 marker ``supported_targets`` 指定所有目标芯片。 + +参数化 marker +^^^^^^^^^^^^^^^^ + +可使用 ``pytest.mark.parametrize`` 和 ``config`` 参数对包含不同 sdkconfig 文件的不同应用程序进行相同的测试。如需了解关于 ``sdkconfig.ci.xxx`` 文件的更多信息，请参考 :idf_file:`readme ` 下的 Configuration Files 章节。 + +.. code:: python + + @pytest.mark.parametrize('config', [ + 'history', # <-- run with app built by sdkconfig.ci.history + 'nohistory', # <-- run with app built by sdkconfig.ci.nohistory + ], indirect=True) # <-- `indirect=True` is required + +总体而言，这一测试函数会复制为 4 个测试用例： + +- ``esp32.history.test_console_advanced`` +- ``esp32.nohistory.test_console_advanced`` +- ``esp32c3.history.test_console_advanced`` +- ``esp32c3.nohistory.test_console_advanced`` + +测试串行输出 +^^^^^^^^^^^^^^^^ + +为确保测试在目标芯片上顺利执行，测试脚本可使用 ``dut.expect()`` 函数来测试目标芯片上的串行输出： + +.. code:: python + + def test_console_advanced(config: str, dut: IdfDut) -> None: # The value of argument ``config`` is assigned by the parameterization. + if config == 'history': + dut.expect('Command history enabled') + elif config == 'nohistory': + dut.expect('Command history disabled') + +在执行 ``dut.expect(...)`` 时，首先会将预期字符串编译成正则表达式用于搜索串行输出结果，直到找到与该编译后的正则表达式匹配的结果或运行超时。 + +如果预期字符串中包含正则表达式关键字（如括号或方括号），则需格外注意。或者，也可以使用 ``dut.expect_exact(...)``，它会尝试直接匹配字符串，而不将其转换为正则表达式。 + +如需了解关于 ``expect`` 函数类型的更多信息，请参考 `pytest-embedded Expecting documentation `__。 + +进阶示例 +---------------- + +用同一应用程序进行多个 DUT 测试 +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +有时，一个测试可能涉及多个运行同一测试程序的目标芯片。在这种情况下，可以使用 ``parameterize`` 将多个 DUT 实例化，例如： + +.. code:: python + + @pytest.mark.esp32s2 + @pytest.mark.esp32s3 + @pytest.mark.usb_host + @pytest.mark.parametrize('count', [ + 2, + ], indirect=True) + def test_usb_host(dut: Tuple[IdfDut, IdfDut]) -> None: + device = dut[0] # <-- assume the first dut is the device + host = dut[1] # <-- and the second dut is the host + ... + +将参数 ``count`` 设置为 2 后，所有 fixture 都会改为元组。 + +用不同应用程序进行多个 DUT 测试 +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +有时（特别是协议测试），一个测试可能涉及多个运行不同测试程序的目标芯片（例如不同目标芯片作为主机和从机）。在这种情况下，可以使用 ``parameterize`` 将针对不同测试应用程序的多个 DUT 实例化。 + +以下代码示例来自 :idf_file:`pytest_wifi_getting_started.py `。 + +.. code:: python + + @pytest.mark.esp32 + @pytest.mark.multi_dut_generic + @pytest.mark.parametrize( + 'count, app_path', [ + (2, + f'{os.path.join(os.path.dirname(__file__), "softAP")}|{os.path.join(os.path.dirname(__file__), "station")}'), + ], indirect=True + ) + def test_wifi_getting_started(dut: Tuple[IdfDut, IdfDut]) -> None: + softap = dut[0] + station = dut[1] + ... + +以上示例中，第一个 DUT 用 :idf_file:`softAP ` 应用程序烧录，第二个 DUT 用 :idf_file:`station ` 应用程序烧录。 + +.. note:: + + 这里的 ``app_path`` 应设置为绝对路径。 Python 中的 ``__file__`` 宏会返回测试脚本自身的绝对路径。 + +用不同应用程序和目标芯片进行多目标测试 +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +以下代码示例来自 :idf_file:`pytest_wifi_getting_started.py `。如注释所述，该代码目前尚未在 ESP-IDF CI 中运行。 + +.. code:: python + + @pytest.mark.parametrize( + 'count, app_path, target', [ + (2, + f'{os.path.join(os.path.dirname(__file__), "softAP")}|{os.path.join(os.path.dirname(__file__), "station")}', + 'esp32|esp32s2'), + (2, + f'{os.path.join(os.path.dirname(__file__), "softAP")}|{os.path.join(os.path.dirname(__file__), "station")}', + 'esp32s2|esp32'), + ], + indirect=True, + ) + def test_wifi_getting_started(dut: Tuple[IdfDut, IdfDut]) -> None: + softap = dut[0] + station = dut[1] + ... + +总体而言，此测试函数会被复制为 2 个测试用例： + +- 在 ESP32 上烧录 softAP，在 ESP32-S2 上烧录 station +- 在 ESP32-S2 上烧录 softAP，在 ESP32 上烧录 station + +支持对不同 sdkconfig 文件及目标芯片的组合测试 +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +以下进阶代码示例来自 :idf_file:`pytest_panic.py `。 + +.. code:: python + + CONFIGS = [ + pytest.param('coredump_flash_bin_crc', marks=[pytest.mark.esp32, pytest.mark.esp32s2]), + pytest.param('coredump_flash_elf_sha', marks=[pytest.mark.esp32]), # sha256 only supported on esp32 + pytest.param('coredump_uart_bin_crc', marks=[pytest.mark.esp32, pytest.mark.esp32s2]), + pytest.param('coredump_uart_elf_crc', marks=[pytest.mark.esp32, pytest.mark.esp32s2]), + pytest.param('gdbstub', marks=[pytest.mark.esp32, pytest.mark.esp32s2]), + pytest.param('panic', marks=[pytest.mark.esp32, pytest.mark.esp32s2]), + ] + + @pytest.mark.parametrize('config', CONFIGS, indirect=True) + ... + +自定义类 +^^^^^^^^^^^^ + +通常，可能会在下列情况下编写自定义类： + +1. 向一定数量的 DUT 添加更多可复用功能。 +2. 为不同阶段添加自定义的前置和后置函数，请参考章节 :ref:`pytest-execution-process`。 + +以下代码示例来自 :idf_file:`panic/conftest.py `。 + +.. code:: python + + class PanicTestDut(IdfDut): + ... + + @pytest.fixture(scope='module') + def monkeypatch_module(request: FixtureRequest) -> MonkeyPatch: + mp = MonkeyPatch() + request.addfinalizer(mp.undo) + return mp + + + @pytest.fixture(scope='module', autouse=True) + def replace_dut_class(monkeypatch_module: MonkeyPatch) -> None: + monkeypatch_module.setattr('pytest_embedded_idf.dut.IdfDut', PanicTestDut) + +``monkeypatch_module`` 提供了一个 `基于模块 `__ 的 `monkeypatch `__ fixture。 + +``replace_dut_class`` 是一个 `基于模块 `__ 的 `自动执行 `__ fixture。 该函数会用你的自定义类替换 ``IdfDut`` 类。 + +标记不稳定测试 +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +某些测试用例基于以太网或 Wi-Fi。然而由于网络问题，测试可能会不稳定。此时，可以将某个测试用例标记为不稳定的测试用例。 + +以下代码示例来自 :idf_file:`pytest_esp_eth.py `。 + +.. code:: python + + @pytest.mark.flaky(reruns=3, reruns_delay=5) + def test_esp_eth_ip101(dut: IdfDut) -> None: + ... + +这一 marker 表示，如果该测试函数失败，其测试用例会每隔 5 秒钟再运行一次，最多运行三次。 + +标记已知失败 +^^^^^^^^^^^^^^^^^^^^^^^^ + +有时，测试会因以下原因而持续失败： + +- 测试的功能（或测试本身）存在错误。 +- 测试环境不稳定（例如网络问题），导致失败率较高。 + +可使用 `xfail `__ marker 来标记此测试用例，并写出原因。 + +以下代码来自 :idf_file:`pytest_panic.py `。 + +.. code:: python + + @pytest.mark.xfail('config.getvalue("target") == "esp32s2"', reason='raised IllegalInstruction instead') + def test_cache_error(dut: PanicTestDut, config: str, test_func_name: str) -> None: + +这一 marker 表示该测试在 ESP32-S2 上是一个已知失败。 + +标记夜间运行的测试用例 +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +在缺少 runner 时，一些测试用例仅在夜间运行的管道中触发。 + +.. code:: python + + @pytest.mark.nightly_run + +这一 marker 表示，此测试用例仅在环境变量为 ``NIGHTLY_RUN`` 或 ``INCLUDE_NIGHTLY_RUN`` 时运行。 + +标记在 CI 中暂时禁用的测试用例 +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +在缺少 runner 时，可以在 CI 中禁用一些本地能够通过测试的测试用例。 + +.. code:: python + + @pytest.mark.temp_skip_ci(targets=['esp32', 'esp32s2'], reason='lack of runners') + +这一 marker 表明，此测试用例仍可以在本地用 ``pytest --target esp32`` 执行，但不会在 CI 中执行。 + +运行 Unity 测试用例 +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +对基于组件的单元测试应用程序，以下代码即可执行所有的单板测试用例，包括普通测试用例和多阶段测试用例： + +.. code:: python + + def test_component_ut(dut: IdfDut): + dut.run_all_single_board_cases() + +此代码还会跳过所有 tag 为 ``[ignore]`` 的测试用例。 + +如需按组执行测试用例，可运行： + +.. code:: python + + def test_component_ut(dut: IdfDut): + dut.run_all_single_board_cases(group='psram') + +此代码会触发模块包含 ``[psram]`` tag 的所有测试用例。 + +你可能还会看到一些包含以下语句的测试脚本，这些脚本已被弃用。请使用上述建议的方法。 + +.. code:: python + + def test_component_ut(dut: IdfDut): + dut.expect_exact('Press ENTER to see the list of tests') + dut.write('*') + dut.expect_unity_test_output() + +如需了解关于 ESP-IDF 单元测试的更多内容，请参考 :doc:`../api-guides/unit-tests`。 + +在 CI 中执行测试 +====================== + +CI 的工作流程很简单，即 编译任务 -> 板载测试任务。 + +编译任务 +-------------- + +编译任务命名 +^^^^^^^^^^^^^^^^^ + +- 基于组件的单元测试： ``build_pytest_components_`` +- 例程测试： ``build_pytest_examples_`` +- 自定义测试： ``build_pytest_test_apps_`` + +编译任务命令 +^^^^^^^^^^^^^^^^^ + +CI 用于创建所有相关测试的命令为： ``python $IDF_PATH/tools/ci/ci_build_apps.py --target -vv --pytest-apps`` + +所有支持指定目标芯片的应用程序都使用 ``build__`` 下支持的 sdkconfig 文件创建。 + +例如，如果运行 ``python $IDF_PATH/tools/ci/ci_build_apps.py $IDF_PATH/examples/system/console/basic --target esp32 --pytest-apps`` 指令，文件夹结构将如下所示： + +.. code:: text + + basic + ├── build_esp32_history/ + │ └── ... + ├── build_esp32_nohistory/ + │ └── ... + ├── main/ + ├── CMakeLists.txt + ├── pytest_console_basic.py + └── ... + +所有编译文件的文件夹都会上传到同一路径。 + +板载测试任务 +---------------- + +板载测试任务命名 +^^^^^^^^^^^^^^^^^^^ + +- 基于部件的单元测试： ``component_ut_pytest__`` +- 例程测试： ``example_test_pytest__`` +- 自定义测试： ``test_app_test_pytest__`` + +板载测试任务命令 +^^^^^^^^^^^^^^^^^^^^^^^^^ + +CI 用于执行所有相关测试的命令为： ``pytest --target -m `` + +这一命令将执行父文件夹下所有具有指定目标芯片 marker 和测试环境 marker 的测试用例。 + +板载测试任务中的二进制文件是从编译任务中下载的，相应文件会放在同一路径下。 + +本地测试 +========== + +首先，你需为 ESP-IDF 安装 Python 依赖： + +.. code-block:: shell + + $ cd $IDF_PATH + $ bash install.sh --enable-pytest + $ . ./export.sh + +默认情况下，pytest 脚本会按照以下顺序查找编译目录： + +- ``build__`` +- ``build_`` +- ``build_`` +- ``build`` + +因此，运行 pytest 最简单的方式是调用 ``idf.py build``。 + +例如，如果你要执行 ``$IDF_PATH/examples/get-started/hello_world`` 文件夹下的所有 ESP32 测试，你可执行： + +.. code-block:: shell + + $ cd examples/get-started/hello_world + $ idf.py build + $ pytest --target esp32 + +如果你的测试应用程序中有多个 sdkconfig 文件，例如那些 ``sdkconfig.ci.*`` 文件， 仅使用 ``idf.py build`` 命令并不能调用这些额外的 sdkconfig 文件。下文以 ``$IDF_PATH/examples/system/console/basic`` 为例进行说明。 + +如果要用 ``history`` 配置测试此应用程序，并用 ``idf.py build`` 进行编译，你需运行： + +.. code-block:: shell + + $ cd examples/system/console/basic + $ idf.py -DSDKCONFIG_DEFAULTS="sdkconfig.defaults;sdkconfig.ci.history" build + $ pytest --target esp32 --sdkconfig history + +如果你想同时编译测试所有 sdkconfig 文件，则需运行我们的 CI 脚本 (ci_build_apps.py) 作为辅助脚本： + +.. code-block:: shell + + $ cd examples/system/console/basic + $ python $IDF_PATH/tools/ci/ci_build_apps.py . --target esp32 -vv --pytest-apps + $ pytest --target esp32 + +包含 ``sdkconfig.ci.history`` 配置的应用程序会编译到 ``build_esp32_history`` 中，而包含 ``sdkconfig.ci.nohistory`` 配置的应用程序会编译到 ``build_esp32_nohistory`` 中。 ``pytest --target esp32`` 命令会在这两个应用程序上运行测试。 + +使用技巧 +============ + +.. _filter-the-test-cases: + +筛选测试用例 +--------------------- + +- 根据目标芯片筛选： ``pytest --target `` + + pytest 会执行所有支持指定目标芯片的测试用例。 + +- 根据 sdkconfig 文件筛选： ``pytest --sdkconfig `` + + 如果 ```` 为 ``default``，pytest 会执行所有 sdkconfig 文件包含 ``sdkconfig.defaults`` 的测试用例。 + + 如果是其他情况，pytest 会执行所有 sdkconfig 文件包含 ``sdkconfig.ci.`` 的测试用例。 + +添加新 marker +---------------- + +我们目前使用两种自定义 marker。target marker 是指测试用例支持此目标芯片，env marker 是指测试用例应分配到 CI 中具有相应 tag 的 runner 上。 + +你可以在 ``${IDF_PATH}/conftest.py`` 文件后添加一行新的 marker。如果该 marker 是 target marker，应将其添加到 ``TARGET_MARKERS`` 中。如果该 marker 指定了一类测试环境，应将其添加到 ``ENV_MARKERS`` 中。自定义 marker 格式： ``: ``。 + +生成 JUnit 报告 +--------------------- + +你可调用 pytest 执行 ``--junitxml `` 生成 JUnit 报告。在 ESP-IDF 中，测试用例命名会统一为 ``..``。 + +跳过自动烧录二进制文件 +------------------------------------- + +调试测试脚本时最好跳过自动烧录二进制文件。 + +调用 pytest 执行 ``--skip-autoflash y`` 即可实现。 + +记录数据 +-------------- + +在执行测试时，你有时需要记录一些数据，例如性能测试数据。 + +在测试脚本中使用 `record_xml_attribute `__ fixture，数据就会记录在 JUnit 报告的属性中。 + +日志系统 +------------ + +在执行测试用例时，你有时可能需要添加一些额外的日志行。 + +这可通过使用 `Python 日志模块 `__ 实现。 + +其他日志函数（作为 fixture） +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +``log_performance`` +""""""""""""""""""" + +.. code:: python + + def test_hello_world( + dut: IdfDut, + log_performance: Callable[[str, object], None], + ) -> None: + log_performance('test', 1) + +以上示例可实现用预定义格式 ``[performance][test]: 1`` 记录性能数据，并在指定 ``--junitxml `` 的情况下将其记录在 JUnit 报告的 ``properties`` tag 下。相应的 JUnit 测试用例节点如下所示： + +.. code:: html + + + + + + + +``check_performance`` +""""""""""""""""""""" + +我们提供了 ``TEST_PERFORMANCE_LESS_THAN`` 和 ``TEST_PERFORMANCE_GREATER_THAN`` 宏来记录性能项，并检测性能项的数值是否在有效范围内。有时 C 宏无法检测一些性能项的值，为此，我们提供了 Python 函数实现相同的目的。注意，由于该 Python 函数不能很好地识别不同的 ifdef 块下同一性能项的阈值，请尽量使用 C 宏。 + +.. code:: python + + def test_hello_world( + dut: IdfDut, + check_performance: Callable[[str, float, str], None], + ) -> None: + check_performance('RSA_2048KEY_PUBLIC_OP', 123, 'esp32') + check_performance('RSA_2048KEY_PUBLIC_OP', 19001, 'esp32') + +以上示例会首先从 :idf_file:`components/idf_test/include/idf_performance.h` 和指定目标芯片的 :idf_file:`components/idf_test/include/esp32/idf_performance_target.h` 头文件中获取性能项 ``RSA_2048KEY_PUBLIC_OP`` 的阈值，然后检查该值是否达到了最小值或超过了最大值。 + +例如，假设 ``IDF_PERFORMANCE_MAX_RSA_2048KEY_PUBLIC_OP`` 的值为 19000，则上例中第一行 ``check_performance`` 会通过测试，第二行会失败并警告： ``[Performance] RSA_2048KEY_PUBLIC_OP value is 19001, doesn\'t meet pass standard 19000.0``。 + +扩展阅读 +============= + +- pytest：https://docs.pytest.org/en/latest/contents.html +- pytest-embedded：https://docs.espressif.com/projects/pytest-embedded/en/latest/ \ No newline at end of file