File System Integration test fails on CY8CKIT_062_WIFI_BT

[maclobdell]

Description

Target: CY8CKIT-062-WIFI-BT (most likely other Cypress targets too)
Mbed OS branch: mbed-os-5.14
Toolchain: ARM compiler 6

Tests Command:

mbed test -n tests-integration-fs-single* -DINTEGRATION_TESTS -t ARM

Suspected Problem: The heap doesn't seem to be allocated correctly. Looking at the map file it is not filling the available memory like it should.

Result:

| target                    | platform_name       | test suite                      | result  | elapsed_time (sec) | copy_method |
|---------------------------|---------------------|---------------------------------|---------|--------------------|-------------|
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-fs-single     | TIMEOUT | 335.87             | default     |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-fs-threaded   | TIMEOUT | 635.28             | default     |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-net-single    | OK      | 337.32             | default     |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-net-threaded  | OK      | 122.74             | default     |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-stress-net-fs | TIMEOUT | 788.71             | default     |
mbedgt: test suite results: 3 TIMEOUT / 2 OK
mbedgt: test case report:
| target                    | platform_name       | test suite                      | test case                               | passed | failed | result  | elapsed_time (sec) |
|---------------------------|---------------------|---------------------------------|-----------------------------------------|--------|--------|---------|--------------------|
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-fs-single     | QSPIF+LFS 1 file, buff     1            | 1      | 0      | OK      | 4.46               |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-fs-single     | QSPIF+LFS 1 file, buff     4            | 0      | 0      | ERROR   | 0.0                |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-fs-single     | QSPIF+LFS 1 file, buff    16            | 0      | 0      | SKIPPED | 0.0                |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-fs-single     | QSPIF+LFS 1 file, buff    64            | 0      | 0      | SKIPPED | 0.0                |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-fs-single     | QSPIF+LFS 1 file, buff   256            | 0      | 0      | SKIPPED | 0.0                |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-fs-single     | QSPIF+LFS 1 file, buff  1024            | 0      | 0      | SKIPPED | 0.0                |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-fs-single     | QSPIF+LFS 1 file, buff  4096            | 0      | 0      | SKIPPED | 0.0                |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-fs-single     | QSPIF+LFS 1 file, buff 16384            | 0      | 0      | SKIPPED | 0.0                |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-fs-single     | QSPIF+LFS format                        | 1      | 0      | OK      | 1.42               |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-fs-threaded   | QSPIF+LFS 2 files, buff 4b/256b         | 0      | 0      | ERROR   | 0.0                |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-fs-threaded   | QSPIF+LFS 3 files, buff 256b/1kb/4kb    | 0      | 0      | SKIPPED | 0.0                |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-fs-threaded   | QSPIF+LFS format                        | 1      | 0      | OK      | 1.43               |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-net-single    | WiFi   128 buffer                       | 1      | 0      | OK      | 97.68              |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-net-single    | WiFi   256 buffer                       | 1      | 0      | OK      | 49.75              |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-net-single    | WiFi  1024 buffer                       | 1      | 0      | OK      | 16.9               |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-net-single    | WiFi  4096 buffer                       | 1      | 0      | OK      | 9.84               |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-net-single    | WiFi network setup                      | 1      | 0      | OK      | 4.49               |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-net-threaded  | WiFi 1 thread                           | 1      | 0      | OK      | 16.83              |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-net-threaded  | WiFi 2 threads                          | 1      | 0      | OK      | 33.41              |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-net-threaded  | WiFi network setup                      | 1      | 0      | OK      | 5.98               |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-stress-net-fs | QSPIF+LFS format                        | 1      | 0      | OK      | 1.44               |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-stress-net-fs | QSPIF+LFS+WiFi 1 thread, dl, file seq.  | 1      | 0      | OK      | 49.38              |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-stress-net-fs | QSPIF+LFS+WiFi 2 threads, dl, 1kb       | 0      | 0      | ERROR   | 0.0                |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-stress-net-fs | QSPIF+LFS+WiFi 3 threads, dl, 256b, 1kb | 0      | 0      | SKIPPED | 0.0                |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-stress-net-fs | Test memory allocation of 10 K bytes    | 1      | 0      | OK      | 0.06               |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-stress-net-fs | Test memory allocation of 20 K bytes    | 1      | 0      | OK      | 0.07               |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-stress-net-fs | Test memory allocation of 40 K bytes    | 1      | 0      | OK      | 0.14               |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-stress-net-fs | Test memory allocation of 60 K bytes    | 1      | 0      | OK      | 0.06               |
| CY8CKIT_062_WIFI_BT-ARMC6 | CY8CKIT_062_WIFI_BT | tests-integration-stress-net-fs | WiFi network setup                      | 1      | 0      | OK      | 5.97               |
mbedgt: test case results: 18 OK / 3 ERROR / 8 SKIPPED

Details:
Other toolchains pass all tests.

Workaround / Solution:

If the following lines are removed from the startup file, then all tests pass. I suspect that these tests are allocating memory, but there is a new regression introduced in Mbed OS 5.14 for the Cypress targets that is preventing the heap from being allocated properly for Mbed OS.

\targets\TARGET_Cypress\TARGET_PSOC6\TARGET_CY8CKIT_062_WIFI_BT\device\TOOLCHAIN_ARM\startup_psoc6_01_cm4.S

; Heap Configuration
; Heap Size (in Bytes) <0x0-0xFFFFFFFF:8>
;
IF :DEF:__HEAP_SIZE
Heap_Size EQU __HEAP_SIZE
ELSE
Heap_Size EQU 0x00000400
ENDIF
AREA HEAP, NOINIT, READWRITE, ALIGN=3
__heap_base
Heap_Mem SPACE Heap_Size
__heap_limit

cc @ARMmbed/team-cypress

Issue request type

[ ] Question
[ ] Enhancement
[ ] Bug

[ciarmcom]

Internal Jira reference: https://jira.arm.com/browse/MBOCUSTRIA-1736

[yarbcy]

@maclobdell @cy-opm
We tried comment:
\targets\TARGET_Cypress\TARGET_PSOC6\TARGET_CY8CKIT_062_WIFI_BT\device\TOOLCHAIN_ARM\startup_psoc6_01_cm4.S

; Heap Configuration
; Heap Size (in Bytes) <0x0-0xFFFFFFFF:8>
;
IF :DEF:__HEAP_SIZE
Heap_Size EQU __HEAP_SIZE
ELSE
Heap_Size EQU 0x00000400
ENDIF
AREA HEAP, NOINIT, READWRITE, ALIGN=3
__heap_base
Heap_Mem SPACE Heap_Size
__heap_limit

But still see the same issue that you described.

Could you please double check if this fix is work for you?

[yarbcy]

Please also give us branch that you use and commit.

[maclobdell]

Unfortunately, the workaround I proposed, does not seem to work anymore. It appears the failing tests are resulting in a hard fault.

[vmedcy]

Hi @maclobdell @hennadiykytsun @yarbcy

I executed mbed test -n tests-integration-fs-single* -DINTEGRATION_TESTS -t ARMC6 -m CY8CKIT_062_WIFI_BT -v on mbed-os-5.14 branch (b6e5a0a), the tests fail with assertion "could not mount block device" at the first test case (QSPIF+LFS format):

calling mbedhtrun: mbedhtrun -m CY8CKIT_062_WIFI_BT -p COM26:9600 -f "BUILD/tests/CY8CKIT_062_WIFI_BT/ARMC6/TESTS/integration/fs-single/fs-single.hex" -e "TESTS\host_tests" -d E: -c default -t 19001301f306191901f3061900000000000000002e127069 -r default -C 10 --sync 5 -P 60

...

[1569343210.00][CONN][RXD] >>> Running case #1: 'QSPIF+LFS format'...
[1569343210.05][CONN][INF] found KV pair in stream: {{__testcase_start;QSPIF+LFS format}}, queued...
[1569343210.09][CONN][RXD] lfs error:494: Corrupted dir pair at 0 1
[1569343210.14][CONN][RXD] lfs error:2216: Invalid superblock at 0 1
mbedgt: :70::FAIL: Expected 0 Was -84. could not mount block device

[1569343210.20][CONN][RXD] :70::FAIL: Expected 0 Was -84. could not mount block device
[1569343210.24][CONN][INF] found KV pair in stream: {{__testcase_finish;QSPIF+LFS format;0;1}}, queued...
[1569343210.32][CONN][RXD] >>> 'QSPIF+LFS format': 0 passed, 1 failed with reason 'Assertion Failed'
[1569343210.32][CONN][RXD]
[1569343210.39][CONN][RXD] >>> Test cases: 0 passed, 1 failed with reason 'Assertion Failed'
[1569343210.41][CONN][RXD] >>> TESTS FAILED!

...

mbedgt: test on hardware with target id: 19001301f306191901f3061900000000000000002e127069
mbedgt: test suite 'tests-integration-fs-single' ..................................................... FAIL in 33.10 sec
	test case: 'QSPIF+LFS 1 file, buff     1' .................................................... SKIPPED in 0.00 sec
	test case: 'QSPIF+LFS 1 file, buff     4' .................................................... SKIPPED in 0.00 sec
	test case: 'QSPIF+LFS 1 file, buff    16' .................................................... SKIPPED in 0.00 sec
	test case: 'QSPIF+LFS 1 file, buff    64' .................................................... SKIPPED in 0.00 sec
	test case: 'QSPIF+LFS 1 file, buff   256' .................................................... SKIPPED in 0.00 sec
	test case: 'QSPIF+LFS 1 file, buff  1024' .................................................... SKIPPED in 0.00 sec
	test case: 'QSPIF+LFS 1 file, buff  4096' .................................................... SKIPPED in 0.00 sec
	test case: 'QSPIF+LFS 1 file, buff 16384' .................................................... SKIPPED in 0.00 sec
	test case: 'QSPIF+LFS format' ................................................................ FAIL in 0.19 sec
mbedgt: test case summary: 0 passes, 1 failure

(full log attached: test_CY8CKIT_062_WIFI_BT_ARMC6.log)

I executed the same test with different toolchain (GCC_ARM 6), and with board from different vendor: DISCO_L475VG_IOT01A (also using QSPIFBlockDevice as the underlying block device for LittleFS). In all situations, test failed with the identical error.

I checked the QSPI transactions (set few breakpoints in cy_qspi_api.c), was not able to catch any failures.

The assertions from the LittleFS core functions: lfs error:494: Corrupted dir pair at 0 1, lfs error:2216: Invalid superblock at 0 1, FAIL: Expected 0 Was -84. could not mount block device - doesn't indicate where is the real root cause of the failure. Very likely, the issue is not target- or toolchain- specific.

In the problem description, I see different test case is failing - QSPIF+LFS 1 file, buff 4, while QSPIF+LFS format and QSPIF+LFS 1 file, buff 1 are padding. Is it possible that the test result depends on the order of integration tests? I have not tried executing everything in batch.

[maclobdell]

I will run these tests again and check with our team.

[maclobdell]

@jamesbeyond - can you confirm that these integration tests are working across other boards?

[shuopeng-deng]

Filed #12044
For fs integration test failures

[0xc0170]

Filed #12044

It was integrated. @maclobdell Is this issue still relevant? Or can be closed?