Commit Graph

168 Commits

Author SHA1 Message Date
He Yin Ling
201018da43 Merge branch 'ci/shorter_performance_test_log' into 'master'
ci: optimize performance test log

See merge request espressif/esp-idf!13607
2021-05-21 06:48:16 +00:00
He Yin Ling
79cd45fdee Merge branch 'test/support_multiple_targets_for_iperf_example_test' into 'master'
test: support multiple targets for iperf example test

Closes TCI-656

See merge request espressif/esp-idf!13527
2021-05-21 06:43:21 +00:00
He Yin Ling
016a37c8ad ci: save built binaries could be tested locally:
we have some test cases not executed in CI. we need to save those
binaries as artifacts so we can test locally.
2021-05-20 16:58:39 +08:00
He Yin Ling
8434c0c731 ttfw: fix DUT exception not added to junit report 2021-05-20 16:28:20 +08:00
Fu Hanxi
bccaab7e6c fix(ci): fix missing [Performance] prefix in junit report 2021-05-20 14:13:54 +08:00
Fu Hanxi
4304866780 ci: shorter performance test log 2021-05-20 14:13:54 +08:00
KonstantinKondrashov
2e4b625f59 simple_ota_example: Adds sha256 check for app images 2021-05-05 22:18:40 +08:00
Cao Sen Miao
2059990c94 ci: Add more information for flash performance test 2021-04-25 17:59:10 +08:00
Fu Hanxi
9b8d1e7826 fix(ci): return as failed when no test case result generated 2021-04-20 11:18:12 +08:00
Fu Hanxi
15b6389ca7 fix(ci): ci_fetch_submodule return full match name first 2021-04-19 19:34:11 +08:00
Fu Hanxi
7651e031cb fix: running ttfw tests locally hang forever while detecting ports 2021-04-14 11:43:13 +08:00
Fu Hanxi
92a9ad0b31 ci: allow wildcard in known failure cases 2021-04-01 19:29:04 +08:00
Fu Hanxi
4ffd458ddd ci: bugfix: known failure cases cannot read files 2021-02-09 15:33:49 +08:00
Ivan Grokhotkov
9a20283485 Merge branch 'bugfix/scan_test_missing_build_apps_without_tests' into 'master'
ci: bugfix: scan_test missing build apps without tests

See merge request espressif/esp-idf!12138
2021-01-28 17:48:24 +08:00
Fu Hanxi
85d4bca81a feat(tiny_test_fw): unify all junit report test case name
new format: <target>.<config>.<case_name>, the default value of
"config" is "default"
2021-01-28 12:25:14 +08:00
Fu Hanxi
d63069068f feat(tiny_test_fw): ignore known failure cases result
py
2021-01-28 12:25:14 +08:00
Fu Hanxi
be3d74efb2 fix(ci): scan_test missing build apps without tests 2021-01-27 12:35:49 +08:00
Fu Hanxi
0146f258d7 style: format python files with isort and double-quote-string-fixer 2021-01-26 10:49:01 +08:00
Aditya Patwardhan
b77c85c1e1 ttfw/IDFDUT: i) Added option to erase_flash in the IDFDUT class
ii) Renamed the method dump_flush to dump_flash
2021-01-20 11:12:14 +00:00
He Yin Ling
5befaac4be ttfw: fix get dut failed when dut port is set as env variable 2021-01-14 17:22:11 +08:00
Angus Gratton
6d3e06a9a3 ttfw: Move TestCaseFailed exception and handle differently to other exceptions
But also ensure the string form of this exception is never empty, if it ends up
somewhere else.
2021-01-12 11:42:03 +11:00
Angus Gratton
441d1306be ci ttfw: Consistently handle unexpected exceptions in test cases 2021-01-12 11:42:03 +11:00
He Yin Ling
97220af848 test: use pyecharts to replace matplotlib in iperf test 2021-01-07 17:10:41 +08:00
He Yin Ling
4e82540730 ttfw: save console log to file 2021-01-07 17:10:40 +08:00
He Yin Ling
16a488c405 test: fix iperf example errors:
1. fix TypeError when running with python3
2. fix throughput chart x/y axis label error
3. make test case compatibile with iperf bin on earlier release branches
2021-01-07 17:10:40 +08:00
Angus Gratton
ec75f923ef Merge branch 'ci/raise_esptool_errors' into 'master'
ci: Don't swallow exception details from flashing process

See merge request espressif/esp-idf!11803
2020-12-30 11:22:57 +08:00
Marius Vikhammer
7fe16bae25 tools: merge C3 changes into master 2020-12-30 07:04:12 +08:00
Angus Gratton
88d0d1fa02 ci: Don't swallow exception details from flashing process
If flashing fails, throw an exception that includes some information about the failure.
2020-12-29 15:41:22 +11:00
Omar Chebib
fcbc706a66 ci: fix flasher_args.json parser (iterate over dictionary)
Closes IDFCI-347
2020-12-29 12:27:51 +08:00
Ivan Grokhotkov
0b31f6a5fc Merge branch 'feature/ci_flasher_args_json_encrypted_flag' into 'master'
ci: use "encrypted" information in flasher_args.json

Closes IDF-2231

See merge request espressif/esp-idf!11551
2020-12-28 06:37:05 +08:00
Omar Chebib
05f97542b8 ci: use "encrypted" information in flasher_args.json
Take into account the new field "encrypted" that is part of the
partition entries in flasher_args.json file
Closes IDF-2231
2020-12-23 18:16:17 +08:00
Fu Hanxi
eda42035d2 scan test: bugfix: now can build multi target for one app 2020-12-15 17:56:53 +08:00
Fu Hanxi
3a88626ec3 search case: print target as well 2020-12-15 17:56:53 +08:00
Fu Hanxi
a5bcaea9cc use "additional_duts" to overwrite dut class for different targets 2020-12-15 17:55:54 +08:00
Fu Hanxi
7281a5a0cc Split component_ut tests based on targets
- add more label in CIScanTest.py
- fix the bug that will cause labels not iterate correctly
2020-11-30 14:08:54 +08:00
Angus Gratton
190aed41ee Merge branch 'bugfix/ttfw_example_test_fails_port_remains_open' into 'master'
ci: Fix CI issue if one example test fails, and if serial port returns some noise

Closes IDFCI-236

See merge request espressif/esp-idf!11273
2020-11-26 08:04:00 +08:00
Angus Gratton
26fb74bc97 ci: ttfw: Encode serial port data to whatever the console encoding is
This is a bit of a hack, but gives us a way to always log it
2020-11-24 18:32:47 +11:00
Angus Gratton
37838803db ci: Log failure to close any DUT 2020-11-24 18:32:47 +11:00
He Yin Ling
e568daf213 ttfw: fix idf performance check doesn't work 2020-11-12 09:49:44 +00:00
Mahavir Jain
0fb65d0bae Merge branch 'bugfix/ttfw_idf_find_artifacts_inaccurate_with_same_prefix_path' into 'master'
CI: fix the bug returns wrong artifact path when app_name is a subset of a longer one

Closes IDFCI-217

See merge request espressif/esp-idf!11187
2020-11-12 12:12:51 +08:00
Angus Gratton
66fb5a29bb Whitespace: Automated whitespace fixes (large commit)
Apply the pre-commit hook whitespace fixes to all files in the repo.

(Line endings, blank lines at end of file, trailing whitespace)
2020-11-11 07:36:35 +00:00
Fu Hanxi
5fa935e5ea CI: fix the bug returns wrong artifact path when app_name is a subset of a longer one 2020-11-11 12:31:39 +08:00
Fu Hanxi
75ab9223f9 scan test when weekend test. 2020-10-26 10:25:21 +08:00
Ivan Grokhotkov
2218ac91c2 tools/ci: fix import error when running ttfw tests locally 2020-10-13 18:09:22 +02:00
morris
9fa06719fa global: enable build uinit test for esp32-s3 2020-09-22 15:15:03 +08:00
Ivan Grokhotkov
d17a3c76d0 ci: fetch submodules from mirror, if enabled 2020-09-16 21:22:26 +02:00
Ivan Grokhotkov
f16d231197 Merge branch 'feature/improve_load_env_log' into 'master'
CI: Improve load env config log output

See merge request espressif/esp-idf!9587
2020-09-16 01:59:05 +08:00
Fu Hanxi
edc7cc9c00 add component_ut in assign-test and target-test stage.
Add one template_test python file to get test case
2020-09-07 12:12:11 +08:00
Fu Hanxi
e35328afd9 build component unit test apps
use idf_py_actions supported targets
2020-09-07 12:11:55 +08:00
Fu Hanxi
ef2813a068 CI: Improve load env config log output 2020-08-05 07:49:02 +00:00
Fu Hanxi
0478edc6ba CI: download only required bin for unit-tests. Refactor AssignTest related code 2020-08-04 13:27:16 +08:00
He Yin Ling
4c32c71db9 Merge branch 'feature/support_python3_autotest' into 'master'
test: run IT jobs with python3

See merge request espressif/esp-idf!9673
2020-07-23 16:07:43 +08:00
Chen Yudong
4088702077 test: auto test script supports python3 2020-07-22 16:14:21 +08:00
Alexey Gerenkov
0e3c7918e4 examples: Fixes 'apptrace_to_host' to work with updated OpenOCD tracing command 2020-07-22 06:49:27 +00:00
Fu Hanxi
6885421976 CI: add size info for binaries 2020-07-21 16:00:05 +08:00
Fu Hanxi
34f459c1e1 remove this line. it will block pipelines without unit test 2020-07-06 19:05:44 +08:00
Fu Hanxi
4087152305 remove build_or_not attr. 2020-07-06 19:05:44 +08:00
Fu Hanxi
270e7dec38 update after MR reviews 2020-07-06 19:05:44 +08:00
Fu Hanxi
010d7e9023 apply to make build system 2020-07-06 19:05:44 +08:00
Fu Hanxi
d7639d5cf8 add build-all flag and BUILD_ALL_APPS ci environment variable 2020-07-06 19:05:44 +08:00
Fu Hanxi
a238b266c2 fix after rebase 2020-07-06 19:05:44 +08:00
Fu Hanxi
53aacef29c code clean up 2020-07-06 19:05:44 +08:00
Fu Hanxi
b26d42afe3 Now scan_tests will generate 'scan_<target>.json`
also updated CI yaml and shell scripts
2020-07-06 19:05:44 +08:00
Angus Gratton
002a13a71a ci provisioning: Add some experiments to check the runner's STA state
Not clear why sometimes it associates without negotiating a new DHCP lease.
2020-06-28 22:52:45 +00:00
Angus Gratton
7934df3a6f ci wifi_tools: Log the wpa_supplicant interface state when trying to connect
Trigger reconnection if wpa_supplicant seems to have dropped the connection.
2020-06-28 22:52:45 +00:00
Angus Gratton
40d7ea68af Merge branch 'bugfix/ci_softap_example_test' into 'master'
Set sleep time to allow wlan0 to connect to softap

See merge request espressif/esp-idf!9258
2020-06-25 09:06:55 +08:00
Shivani Tipnis
76232fd263 ci: Set sleep time to allow wlan0 to connect to softap 2020-06-24 23:07:46 +05:30
Roland Dobai
50e817fc77 CI: Ignore error if py_debug_backend not installed 2020-06-23 13:45:12 +02:00
Ivan Grokhotkov
b3a76a9d83 Merge branch 'feature/idf_monitor_debug_ws' into 'master'
tools/idf_monitor: add WebSocket client for IDE integration

Closes IDF-1719

See merge request espressif/esp-idf!9032
2020-06-22 15:50:11 +08:00
Ivan Grokhotkov
cf155161c4 Merge branch 'feature/higher_level_com_gdb' into 'master'
CI: Use higher-level interaction with GDB in example tests and test apps

Closes IDF-1622

See merge request espressif/esp-idf!8840
2020-06-19 16:13:01 +08:00
Fu Hanxi
23822e2e3f ci: retry download if catched IOError/EOFError 2020-06-17 16:03:49 +08:00
Roland Dobai
e67314f646 tools/idf_monitor: add WebSocket client for IDE integration 2020-06-10 10:56:20 +00:00
Roland Dobai
493c852b73 CI: Use higher-level interaction with GDB in example tests and test apps 2020-06-05 14:58:34 +00:00
Fu Hanxi
b76df30915 don't overwrite special cases dut 2020-06-01 13:39:09 +08:00
Fu Hanxi
3b17b9e393 panic_test contains some reassign dut cases 2020-06-01 12:08:57 +08:00
Fu Hanxi
377d3eaaa5 Move local target detection to ttfw_idf
rename upper_list to upper_list_or_str
minor fix for `unit_test.py` `is 'name'` -> `== 'name`
2020-06-01 12:08:57 +08:00
Fu Hanxi
e99172fbac make upper_list do not make str into list 2020-05-29 12:42:57 +08:00
Fu Hanxi
d06d2a2b9f remove typing as a dependency.
return None if the input is None
2020-05-29 12:42:56 +08:00
Fu Hanxi
5c92d36078 Add list support for ttfw_idf test decorators. Only replicate supported keys 2020-05-29 12:42:56 +08:00
Fu Hanxi
e553092d62 move ci_target_check to ttfw_idf, move ci_target_filter to AssignTest
mark `supported_in_ci` for AssignTest filter.
2020-05-29 12:42:56 +08:00
Fu Hanxi
38c288bc0e Add BOT_TARGET_FILTER to _apply_bot_filter
fix the bug that will calculate twice if ci_target is a list.
2020-05-29 12:42:56 +08:00
Fu Hanxi
bc026133c5 add ci_target filter for target and local target check 2020-05-29 12:42:56 +08:00
Fu Hanxi
9f8b63da38 Pass current target to test_method **overwrite. 2020-05-29 12:42:56 +08:00
Fu Hanxi
6c98d7e4bd Add multi target support for search examples
drop keyword `dut`, use `target` instead to assign`dut_class` to `Env`
2020-05-29 12:42:56 +08:00
Ivan Grokhotkov
3fe4adc98d ttfw: allow relative imports in test scripts 2020-05-12 22:27:06 +02:00
Ivan Grokhotkov
7d74cc4255 ttfw_idf: IDFApp: convert partition offset to int 2020-05-12 22:27:06 +02:00
Ivan Grokhotkov
952830ab12 ttfw_idf: IDFApp: add method to get SHA256(elf file) 2020-05-12 22:27:06 +02:00
Ivan Grokhotkov
26f6136a7d ttfw: only send relevant DUT kwargs to serial.serial_for_url 2020-05-12 22:27:06 +02:00
Roland Dobai
79e58eff8f Add JTAG related example tests 2020-05-05 11:08:33 +02:00
Fu Hanxi
c131a18059 load all yml files while assigning tests 2020-04-30 06:47:49 +00:00
Fu Hanxi
7709d4ccfd Use IDF_TARGET to determine the build method and the output file name 2020-04-30 06:47:49 +00:00
Fu Hanxi
04342a73f2 Fix some python bugs left before 2020-04-30 06:47:49 +00:00
Roland Dobai
2d709c6384 Migrate "loadable elf" test from hello_world example to test_apps 2020-04-29 09:17:28 +02:00
Roland Dobai
819f7a4e57 examples: Add storage example tests 2020-04-29 09:16:18 +02:00
He Yin Ling
c2e068f0e4 Merge branch 'bugfix/ci_partition_table_detection' into 'master'
CI: Fix partition table detection if "partition" is in the project name

See merge request espressif/esp-idf!8223
2020-04-22 15:10:11 +08:00
Fu Hanxi
76d118577e test_config: fix the bug that will generate !!python/unicode
As an example, [7964999 example_test_002_](https://gitlab.espressif.cn:6688/espressif/esp-idf/-/jobs/7964999/artifacts/file/examples/test_configs/example_test_002_.yml)
2020-04-08 11:19:13 +08:00
David Cermak
a463e80d43 ci: fix ttfw ignoring test failure if exception in print_debug_info 2020-04-05 20:07:27 +02:00
Roland Dobai
512906fa32 CI: Fix partition table detection if "partition" is in the project name 2020-04-03 11:41:06 +02:00
Roland Dobai
15884eccf2 Add multi-target support for performance tests 2020-03-09 13:41:56 +01:00
David Cermak
692deac5ae CI: assign and target-test stages updated to run test-apps in the ci 2020-02-06 16:46:41 +00:00