docs: add multi dut chapter

This commit is contained in:
Fu Hanxi 2022-05-07 12:54:02 +08:00
parent 52b5a8348e
commit 3697cd0440

View File

@ -19,6 +19,11 @@ In ESP-IDF, we use the following plugins by default:
All the introduced concepts and usages are based on the default behavior in ESP-IDF. Not all of them are available in vanilla pytest. All the introduced concepts and usages are based on the default behavior in ESP-IDF. Not all of them are available in vanilla pytest.
Installation
------------
``$ pip install -U pytest-embedded-serial-esp~=0.7.0 pytest-embedded-idf~=0.7.0``
Basic Concepts Basic Concepts
-------------- --------------
@ -95,12 +100,12 @@ Pytest Execution Process
1. Get all the python files with the prefix ``pytest_`` 1. Get all the python files with the prefix ``pytest_``
2. Get all the test functions with the prefix ``test_`` 2. Get all the test functions with the prefix ``test_``
3. Apply the `params <https://docs.pytest.org/en/latest/how-to/parametrize.html>`__, duplicate the test functions. 3. Apply the `params <https://docs.pytest.org/en/latest/how-to/parametrize.html>`__, and duplicate the test functions.
4. Filter the test cases with CLI options. Introduced detail usages `here <#filter-the-test-cases>`__ 4. Filter the test cases with CLI options. Introduced detailed usages `here <#filter-the-test-cases>`__
3. Test Running Phase 3. Test Running Phase
1. Construct the `fixtures <https://docs.pytest.org/en/latest/how-to/fixtures.html>`__. In ESP-IDF, the common fixtures are initialized with this order: 1. Construct the `fixtures <https://docs.pytest.org/en/latest/how-to/fixtures.html>`__. In ESP-IDF, the common fixtures are initialized in this order:
1. ``pexpect_proc``: `pexpect <https://github.com/pexpect/pexpect>`__ instance 1. ``pexpect_proc``: `pexpect <https://github.com/pexpect/pexpect>`__ instance
@ -116,7 +121,7 @@ Pytest Execution Process
2. Run the real test function 2. Run the real test function
3. Deconstruct the fixtures with this order: 3. Deconstruct the fixtures in this order:
1. ``dut`` 1. ``dut``
@ -198,7 +203,7 @@ You can use ``pytest.mark.parametrize`` with “config” to apply the same test
'nohistory', # <-- run with app built by sdkconfig.ci.nohistory 'nohistory', # <-- run with app built by sdkconfig.ci.nohistory
], indirect=True) # <-- `indirect=True` is required ], indirect=True) # <-- `indirect=True` is required
Overall, this test case would be duplicated to 4 test functions: Overall, this test function would be replicated to 4 test cases:
- esp32.history.test_console_advanced - esp32.history.test_console_advanced
- esp32.nohistory.test_console_advanced - esp32.nohistory.test_console_advanced
@ -208,6 +213,80 @@ Overall, this test case would be duplicated to 4 test functions:
Advanced Examples Advanced Examples
~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~
Multi Dut Tests with the Same App
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This code example is taken from :idf_file:`pytest_usb_host.py <tools/test_apps/peripherals/usb/pytest_usb_host.py>`.
.. code:: python
@pytest.mark.esp32s2
@pytest.mark.esp32s3
@pytest.mark.usb_host
@pytest.mark.parametrize('count', [
2,
], indirect=True)
def test_usb_host(dut: Tuple[IdfDut, IdfDut]) -> None:
device = dut[0] # <-- assume the first dut is the device
host = dut[1] # <-- and the second dut is the host
...
After setting the param ``count`` to 2, all these fixtures are changed into tuples.
Multi Dut Tests with Different Apps
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This code example is taken from :idf_file:`pytest_wifi_getting_started.py <examples/wifi/getting_started/pytest_wifi_getting_started.py>`.
.. code:: python
@pytest.mark.esp32
@pytest.mark.multi_dut_generic
@pytest.mark.parametrize(
'count, app_path', [
(2,
f'{os.path.join(os.path.dirname(__file__), "softAP")}|{os.path.join(os.path.dirname(__file__), "station")}'),
], indirect=True
)
def test_wifi_getting_started(dut: Tuple[IdfDut, IdfDut]) -> None:
softap = dut[0]
station = dut[1]
...
Here the first dut was flashed with the app :idf_file:`softap <examples/wifi/getting_started/softAP/main/softap_example_main.c>`, and the second dut was flashed with the app :idf_file:`station <examples/wifi/getting_started/station/main/station_example_main.c>`.
.. note::
Here the ``app_path`` should be set with absolute path. the ``__file__`` macro in python would return the absolute path of the test script itself.
Multi Dut Tests with Different Apps, and Targets
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This code example is taken from :idf_file:`pytest_wifi_getting_started.py <examples/wifi/getting_started/pytest_wifi_getting_started.py>`. As the comment says, for now it's not running in the ESP-IDF CI.
.. code:: python
@pytest.mark.parametrize(
'count, app_path, target', [
(2,
f'{os.path.join(os.path.dirname(__file__), "softAP")}|{os.path.join(os.path.dirname(__file__), "station")}',
'esp32|esp32s2'),
(2,
f'{os.path.join(os.path.dirname(__file__), "softAP")}|{os.path.join(os.path.dirname(__file__), "station")}',
'esp32s2|esp32'),
],
indirect=True,
)
def test_wifi_getting_started(dut: Tuple[IdfDut, IdfDut]) -> None:
softap = dut[0]
station = dut[1]
...
Overall, this test function would be replicated to 2 test cases:
- softap with esp32 target, and station with esp32s2 target
- softap with esp32s2 target, and station with esp32 target
Support different targets with different sdkconfig files Support different targets with different sdkconfig files
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@ -278,7 +357,7 @@ Mark Known Failure Cases
Sometimes a test couldn't pass for the following reasons: Sometimes a test couldn't pass for the following reasons:
- Has a bug - Has a bug
- Success ratio too low because of environment issue, such as network issue. Retry couldn't help - The success ratio is too low because of environment issue, such as network issue. Retry couldn't help
Now you may mark this test case with marker `xfail <https://docs.pytest.org/en/latest/how-to/skipping.html#xfail-mark-test-functions-as-expected-to-fail>`__ with a user-friendly readable reason. Now you may mark this test case with marker `xfail <https://docs.pytest.org/en/latest/how-to/skipping.html#xfail-mark-test-functions-as-expected-to-fail>`__ with a user-friendly readable reason.