Skip to content

Conversation

@amq
Copy link
Contributor

@amq amq commented Nov 12, 2019

Description (required)

Adds ability to configure the many FAT ChaN options using mbed_lib.json.

Summary of change (What the change is for and why)

All defines in ffconf.h apart of Revision ID are made dynamically configurable using the standard Mbed OS mechanism.

All values remain the same, so this change should be fully transparent for everyone.


Pull request type (required)

[x] Patch update (Bug fix / Target update / Docs update / Test update / Refactor)
[] Feature update (New feature / Functionality change / New API)
[] Major update (Breaking change E.g. Return code change / API behaviour change)

Test results (required)

[] No Tests required for this change (E.g docs only update)
[x] Covered by existing mbed-os tests (Greentea or Unittest)
[] Tests / results supplied as part of this PR

@ciarmcom ciarmcom requested review from a team November 12, 2019 16:00
@ciarmcom
Copy link
Member

@amq, thank you for your changes.
@ARMmbed/mbed-os-storage @ARMmbed/mbed-os-maintainers please review.

@amq
Copy link
Contributor Author

amq commented Nov 12, 2019

Maybe it would make sense to move mbed_lib.json one level up, or rename the library from fat to fat_chan, in both cases I will be glad to do that.

@SeppoTakalo
Copy link
Contributor

How often do these needs to be changed?

This spawns a quite a lot of configurable parameters for such a rarely used component. We try to use LittleFS instead of FAT because it is more resilient for power losses.

But.. well, if the component name is changed to fat_chan, I don't have problems with it..

@amq
Copy link
Contributor Author

amq commented Nov 13, 2019

@SeppoTakalo changed the component name to fat_chan.

Copy link
Contributor

@0xc0170 0xc0170 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally a library would provide a way to override and our config would just use these,

@0xc0170
Copy link
Contributor

0xc0170 commented Nov 13, 2019

CI started

@mbed-ci
Copy link

mbed-ci commented Nov 13, 2019

Test run: FAILED

Summary: 1 of 4 test jobs failed
Build number : 1
Build artifacts

Failed test jobs:

  • jenkins-ci/mbed-os-ci_unittests

@SeppoTakalo
Copy link
Contributor

Ah.. true.. All unittests that include these headers now fail:

[2019-11-13T17:07:22.341Z] /builds/ws/mbed-os-ci_unittests@2/unitTest-4420/mbed-os/UNITTESTS/../features/storage/filesystem/fat/ChaN/ffconf.h:190:20: error: ‘MBED_CONF_FAT_CHAN_FF_MAX_SS’ was not declared in this scope

[2019-11-13T17:07:22.341Z]  #define FF_MAX_SS  MBED_CONF_FAT_CHAN_FF_MAX_SS

[2019-11-13T17:07:22.341Z]                     ^

[2019-11-13T17:07:22.341Z] /builds/ws/mbed-os-ci_unittests@2/unitTest-4420/mbed-os/UNITTESTS/../features/storage/filesystem/fat/ChaN/ff.h:130:11: note: in expansion of macro ‘FF_MAX_SS’

[2019-11-13T17:07:22.341Z]   BYTE win[FF_MAX_SS]; /* Disk access window for Directory, FAT (and file data at tiny cfg) */

[2019-11-13T17:07:22.341Z]            ^~~~~~~~~

[2019-11-13T17:07:22.341Z] [ 33%] Building CXX object CMakeFiles/empty_baseline.MbedOS.dir/builds/ws/mbed-os-ci_unittests@2/unitTest-4420/mbed-os/features/storage/blockdevice/ExhaustibleBlockDevice.cpp.o

[2019-11-13T17:07:22.341Z] /builds/ws/mbed-os-ci_unittests@2/unitTest-4420/mbed-os/UNITTESTS/../features/storage/filesystem/fat/ChaN/ffconf.h:190:20: note: suggested alternative: ‘MBED_CONF_LORA_FSB_MASK’

[2019-11-13T17:07:22.341Z]  #define FF_MAX_SS  MBED_CONF_FAT_CHAN_FF_MAX_SS

[2019-11-13T17:07:22.341Z]                     ^

[2019-11-13T17:07:22.341Z] /builds/ws/mbed-os-ci_unittests@2/unitTest-4420/mbed-os/UNITTESTS/../features/storage/filesystem/fat/ChaN/ff.h:130:11: note: in expansion of macro ‘FF_MAX_SS’

[2019-11-13T17:07:22.341Z]   BYTE win[FF_MAX_SS]; /* Disk access window for Directory, FAT (and file data at tiny cfg) */

[2019-11-13T17:07:22.341Z]            ^~~~~~~~~

[2019-11-13T17:07:22.341Z] /builds/ws/mbed-os-ci_unittests@2/unitTest-4420/mbed-os/UNITTESTS/../features/storage/filesystem/fat/ChaN/ffconf.h:190:20: error: ‘MBED_CONF_FAT_CHAN_FF_MAX_SS’ was not declared in this scope

[2019-11-13T17:07:22.341Z]  #define FF_MAX_SS  MBED_CONF_FAT_CHAN_FF_MAX_SS

[2019-11-13T17:07:22.341Z]                     ^

[2019-11-13T17:07:22.341Z] /builds/ws/mbed-os-ci_unittests@2/unitTest-4420/mbed-os/UNITTESTS/../features/storage/filesystem/fat/ChaN/ff.h:179:11: note: in expansion of macro ‘FF_MAX_SS’

[2019-11-13T17:07:22.341Z]   BYTE buf[FF_MAX_SS]; /* File private data read/write window */

[2019-11-13T17:07:22.341Z]            ^~~~~~~~~

[2019-11-13T17:07:22.341Z] /builds/ws/mbed-os-ci_unittests@2/unitTest-4420/mbed-os/UNITTESTS/../features/storage/filesystem/fat/ChaN/ffconf.h:190:20: note: suggested alternative: ‘MBED_CONF_LORA_FSB_MASK’

[2019-11-13T17:07:22.341Z]  #define FF_MAX_SS  MBED_CONF_FAT_CHAN_FF_MAX_SS

[2019-11-13T17:07:22.341Z]                     ^

One option would be that you provide the previous version of the header in the UNITTESTS/target_h/ or provide those configuration macros in the CMake file.

@amq
Copy link
Contributor Author

amq commented Nov 14, 2019

@SeppoTakalo will simply adding the original header file without any changes work, won't there be "redefined" errors? (if this options works, ignore the next question) How would the CMake change work, could you give an example based on one define?

@amq
Copy link
Contributor Author

amq commented Nov 17, 2019

@SeppoTakalo added the macros in the CMake file.

@amq amq requested a review from 0xc0170 November 21, 2019 09:17
@0xc0170
Copy link
Contributor

0xc0170 commented Nov 21, 2019

CI restarted

@mbed-ci
Copy link

mbed-ci commented Nov 21, 2019

Test run: FAILED

Summary: 1 of 4 test jobs failed
Build number : 2
Build artifacts

Failed test jobs:

  • jenkins-ci/mbed-os-ci_unittests

@0xc0170
Copy link
Contributor

0xc0170 commented Nov 21, 2019

I restarted the unittests, I could find this one in the logs:

10:43:32 /builds/ws/mbed-os-ci_unittests/unitTest-4477/mbed-os/UNITTESTS/events/equeue/test_equeue.cpp:680: Failure , that does not make sense for these changes.

@mbed-ci
Copy link

mbed-ci commented Nov 21, 2019

Test run: FAILED

Summary: 1 of 4 test jobs failed
Build number : 3
Build artifacts

Failed test jobs:

  • jenkins-ci/mbed-os-ci_build-ARM

@0xc0170
Copy link
Contributor

0xc0170 commented Nov 21, 2019

CI restarted

@mbed-ci
Copy link

mbed-ci commented Nov 21, 2019

Test run: FAILED

Summary: 1 of 11 test jobs failed
Build number : 4
Build artifacts

Failed test jobs:

  • jenkins-ci/mbed-os-ci_cloud-client-pytest

@0xc0170
Copy link
Contributor

0xc0170 commented Nov 22, 2019

Internal error, pytest restarted

@0xc0170 0xc0170 removed the needs: CI label Nov 22, 2019
@0xc0170 0xc0170 merged commit 04ed9c5 into ARMmbed:master Nov 22, 2019
@amq amq deleted the patch-2 branch November 22, 2019 15:33
@SeppoTakalo
Copy link
Contributor

Oh man.. This just build Bootloader build on all platform.

Compile [ 46.1%]: kv_config.cpp
[Error] ffconf.h@190,20: 'MBED_CONF_FAT_CHAN_FF_MAX_SS' was not declared in this scope
[Error] ffconf.h@190,20: 'MBED_CONF_FAT_CHAN_FF_MAX_SS' was not declared in this scope
[ERROR] In file included from ./mbed-os/features/storage/filesystem/fat/ChaN/ff.h:30,
                 from ./mbed-os/features/storage/filesystem/fat/FATFileSystem.h:34,
                 from ./mbed-os/features/storage/kvstore/conf/kv_config.cpp:24:
./mbed-os/features/storage/filesystem/fat/ChaN/ffconf.h:190:20: error: 'MBED_CONF_FAT_CHAN_FF_MAX_SS' was not declared in this scope
  190 | #define FF_MAX_SS  MBED_CONF_FAT_CHAN_FF_MAX_SS
      |                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
./mbed-os/features/storage/filesystem/fat/ChaN/ff.h:130:11: note: in expansion of macro 'FF_MAX_SS'
  130 |  BYTE win[FF_MAX_SS]; /* Disk access window for Directory, FAT (and file data at tiny cfg) */
      |           ^~~~~~~~~
./mbed-os/features/storage/filesystem/fat/ChaN/ffconf.h:190:20: error: 'MBED_CONF_FAT_CHAN_FF_MAX_SS' was not declared in this scope
  190 | #define FF_MAX_SS  MBED_CONF_FAT_CHAN_FF_MAX_SS
      |                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
./mbed-os/features/storage/filesystem/fat/ChaN/ff.h:179:11: note: in expansion of macro 'FF_MAX_SS'
  179 |  BYTE buf[FF_MAX_SS]; /* File private data read/write window */
      |           ^~~~~~~~~

Not sure if this also broke all Baremetal builds.

@0xc0170
Copy link
Contributor

0xc0170 commented Nov 25, 2019

Looks like the configuration is not there, a question is why? Neither our tests caught this one.

@SeppoTakalo
Copy link
Contributor

Apparently Baremetal builds now need to include chan-fat if they include FatFileSystem.h.

Which is logical.. but kv_config.cpp did not require this..

@amq
Copy link
Contributor Author

amq commented Nov 25, 2019

I am building a bootloader with CI, and I didn't have this issue when updating to 5.14.2

Could this be because in my case I'm not using KV, and the whole FAT ChaN is "self-confined" (only ff.cpp/h depends on ffconf.h)? Seems odd.

bootloader/mbed_app.json

{
    "requires": [
        "bare-metal",
        "rtos-api",
        "filesystem",
        "littlefs",
        "sd",
        "flashiap-block-device"
    ],
    "target_overrides": {
        "*": {
            "target.restrict_size": "0x20000",
            "target.macros_add": ["NDEBUG=1"],
            "platform.stdio-flush-at-exit": false
    }
}

bootloader/bootloader.cpp

#if !defined(BOOTLOADER_ADDR) && defined(POST_APPLICATION_ADDR)

#include "FlashIAPBlockDevice.h"
#include "LittleFileSystem.h"
#include "SDBlockDevice.h"
#include "FATFileSystem.h"
#include "mbed.h"

#define FS_MOUNT_PATH "fs"
#define FS_UPDATE_FILE_PATH "/" FS_MOUNT_PATH "/" MY_UPDATE_FILE_NAME

#define SD_MOUNT_PATH "sd"
#define SD_UPDATE_FILE_PATH "/" SD_MOUNT_PATH "/" MY_UPDATE_FILE_NAME

FlashIAPBlockDevice flashiap_bd(MY_FLASH_BD_START, MY_FLASH_BD_SIZE);
LittleFileSystem flashiap_fs(FS_MOUNT_PATH, &flashiap_bd);

SDBlockDevice sd_bd(SPI_MOSI, SPI_MISO, SPI_SCK, SPI_CS);
FATFileSystem sd_fs(SD_MOUNT_PATH, &sd_bd);

FlashIAP mcuflash;
DigitalOut led(LED_RED);

bool update(const char *path);
void apply_update(FILE *file, uint32_t address);

int main() {
    bool res = update(FS_UPDATE_FILE_PATH);

    if (!res) {
        update(SD_UPDATE_FILE_PATH);
    }

    mbed_start_application(POST_APPLICATION_ADDR);
}

bool update(const char *path) {
    bool res = false;

    FILE *file = fopen(path, "rb");
    if (file != NULL) {
        mcuflash.init();
        apply_update(file, POST_APPLICATION_ADDR);
        fclose(file);
        remove(path);
        mcuflash.deinit();
        res = true;
    }

    return res;
}

void apply_update(FILE *file, uint32_t address) {
    const uint32_t page_size = mcuflash.get_page_size();
    char *page_buffer = new char[page_size];

    uint32_t addr = address;
    uint32_t next_sector = addr + mcuflash.get_sector_size(addr);
    bool sector_erased = false;
    size_t pages_flashed = 0;

    while (true) {

        // Read data for this page
        memset(page_buffer, 0, sizeof(char) * page_size);
        int size_read = fread(page_buffer, 1, page_size, file);
        if (size_read <= 0) {
            break;
        }

        // Erase this page if it hasn't been erased
        if (!sector_erased) {
            mcuflash.erase(addr, mcuflash.get_sector_size(addr));
            sector_erased = true;
        }

        // Program page
        mcuflash.program(page_buffer, addr, page_size);

        addr += page_size;
        if (addr >= next_sector) {
            next_sector = addr + mcuflash.get_sector_size(addr);
            sector_erased = false;
        }

        if (++pages_flashed % 24 == 0) {
            led = !led;
        }
    }
    led = 0;

    delete[] page_buffer;
}

#endif
[mbed] Adding library "mbed-os" from "https://github.com/ARMmbed/mbed-os" at rev #cf4f12a123c0
mbed compile -t GCC_ARM -m ${TARGET} --source=./bootloader --source=./mbed-os --source=./targets -c
Building project bootloader (MY_TARGET, GCC_ARM)
Scan: bootloader
Scan: mbed-os
Scan: targets
Using ROM regions application, post_application in this build.
  Region application: size 0x20000, offset 0x0
  Region post_application: size 0xe0000, offset 0x20000
Compile [  0.5%]: mbed_tz_context.c
Compile [  1.0%]: fslittle_test.c
Compile [  1.5%]: FlashIAPBlockDevice.cpp
Compile [  2.1%]: bootloader.cpp
...
Compile [ 27.2%]: FileSystem.cpp
Compile [ 27.7%]: ffunicode.cpp
Compile [ 28.2%]: ff.cpp
Compile [ 28.7%]: FATFileSystem.cpp
...
Total Static RAM memory (data + bss): 5216(+5216) bytes
Total Flash memory (text + data): 83280(+83280) bytes

When I tried building with fat_chan in requires, the size stats were byte-equal.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants