Commit Graph

208 Commits

Author SHA1 Message Date
Shivani Tipnis
1f9f39fe46 nvs_util: Set previous page state to full before creating new page 2019-06-24 13:53:51 +00:00
Renz Christian Bagaporo
9eccd7c082 components: use new component registration api 2019-06-21 19:53:29 +08:00
suda-morris
3f7a571c90 fix errors when ci testing for esp32 2019-06-19 15:31:47 +08:00
Kirill Chalov
4faf2de035 Doc/review api ref storage 2019-06-17 14:23:52 +08:00
suda-morris
84b2f9f14d build and link hello-world for esp32s2beta 2019-06-11 13:07:37 +08:00
MartinValik
5268960235 Added support for NVS iterators
Closes https://github.com/espressif/esp-idf/issues/129
2019-06-10 16:42:39 +02:00
Angus Gratton
154fc74f76 Merge branch 'bugfix/nvs_typedef_naming_pr3239' into 'master'
nvs: Append _t to nvs_handle & nvs_open_mode types

Closes IDFGH-891

See merge request idf/esp-idf!4783
2019-05-29 08:14:00 +08:00
Angus Gratton
b2e54a9cc6 ci: Add IDF_DEPRECATED macro for things we only deprecate in our CI passes 2019-05-27 17:43:38 +10:00
GOPTIONS\pfrost
897bb55482 nvs: Append _t to nvs_handle & nvs_open_mode types
Merges https://github.com/espressif/esp-idf/pull/3239/
2019-05-22 02:53:28 +00:00
Roland Dobai
1ad2283641 Rename Kconfig options (components/bootloader) 2019-05-21 09:32:55 +02:00
Tim Nordell
c3fa2497b9 nvs: Check if an item is modified before writing out an identical copy
This prevents wear and tear on the flash, and it also is faster in some
cases since the read-out of flash is a cheaper operation than the erasure
of flash.  Some library modules (such as the esp_wifi) write out to NVS
upon every initialization without checking first that the existing value
is the same, and this speeds up initialization of modules that make
these design choices and moves it into a centralized place.

The comparison functions are based on the read-out functions of the same
name, and changes out the memcpy(...) operations for memcmp(...)
operations.

Signed-off-by: Tim Nordell <tim.nordell@nimbelink.com>
2019-04-30 11:39:58 -05:00
negativekelvin
9a319772e1 nvs_flash: Multi-page blob erased using nvs_erase_key should be cleaned properly
Earlier eraseItem function in Storage class would do lazy cleanup of
multi-page blobs if called using type "ANY" instead of "BLOB". It used to
just delete BLOB data and index would remain as is. Any subsequent read
would delete index entry as well. This however would return a valid
length without error if nvs_get_blob API was just used for finding
length and not reading the complete blob. This change fixes this issue.

Closes https://github.com/espressif/esp-idf/issues/3255
2019-04-10 11:45:29 +00:00
Shivani Tipnis
a88b40483d mfg_util: Add changes to mfg_util as per changes in nvs_util
revert changeson this file

Update README for mfg util

Update to correct coding style of script
2019-04-03 11:31:49 +05:30
Shivani Tipnis
66e38e9bac nvs_util: Add support to include comments in csv file
Closes https://github.com/espressif/esp-idf/issues/2965
2019-04-01 16:52:43 +05:30
Shivani Tipnis
cd8c42600c nvs_util: Add support for creation of unique encryption keys 2019-03-28 05:06:12 +00:00
Tim Nordell
eb5abca23d nvs_flash: Fix typo in README.rst
Merges https://github.com/espressif/esp-idf/pull/3216

This should read "except for the new key-value pair" instead of
"expect for the new key-value pair".

Signed-off-by: Tim Nordell <tim.nordell@nimbelink.com>
2019-03-24 19:56:10 +01:00
Ivan Grokhotkov
940a1f6e79 Merge branch 'feature/separate_rom_from_esp32' into 'master'
separate rom from esp32 component to esp_rom

Closes IDF-542

See merge request idf/esp-idf!4500
2019-03-21 22:10:45 +08:00
morris
c159984264 separate rom from esp32 component to esp_rom
1. separate rom include files and linkscript to esp_rom
2. modefiy "include rom/xxx.h" to "include esp32/rom/xxx.h"
3. Forward compatible
4. update mqtt
2019-03-21 18:51:45 +08:00
Shivani Tipnis
60b5cdde20 nvs_util: Fix to support write of multiple singlepage big blob data
Closes https://github.com/espressif/esp-idf/issues/3011
2019-03-20 08:15:36 +00:00
Ivan Grokhotkov
dae40f4ff1 nvs: add a blob fragmentation test case
Ref. TW12937
2019-02-26 01:58:20 +00:00
Ivan Grokhotkov
02a76abb8b nvs: do eager cleanup of HashListBlocks
Previously when HashList was removing items, HashListBlocks were
removed lazily. This resulted in empty HashListBlocks dangling around
in full pages, even when all items have been erased. These blocks
would only be deleted when NVS was re-initialized
(nvs_flash_deinit/nvs_flash_init).

This change does eager cleanup instead, based on the code from
@negativekelvin offered in
https://github.com/espressif/esp-idf/issues/1642#issuecomment-367227994.

Closes https://github.com/espressif/esp-idf/issues/1642.
2019-02-26 01:58:20 +00:00
Sagar Bijwe
d32128440d nvs_flash: Detect key partition as uninitialised even if encrypted by bootloader
Currently, only erase operation performed by the application leads
to detection of NVS key partition as uninitialised. This change
adds additional checks for detecting partition as uninitialised,
when device boots first time right after encryption by bootloader.
2019-02-01 13:28:13 +05:30
Roland Dobai
37126d3451 Correct Kconfigs according to the coding style 2019-01-29 13:37:01 +01:00
Gautier Seidel
b4c4f8514b nvs_flash: Improve nvs_flash_init_partition() speed
Signed-off-by: Sagar Bijwe <sagar@espressif.com>
2019-01-23 04:53:36 +00:00
Martin Válik
602d102403 Added support for NVS in console example 2019-01-11 09:51:50 +08:00
Sagar Bijwe
fc37dc1944 nvs_flash: Fix compilation of host testcases when g++-8.x is used.
When std::bind is used, it requires inclusion of <functional> header.
This was not mandatory with earlier versions of g++ (4.x), may be because they
had experimental support for c++11.
2018-12-07 09:40:45 +00:00
Shivani Tipnis
0c72b08872 nvs_util: update test case 2018-11-21 13:40:24 +05:30
Shivani Tipnis
582c547191 Update README 2018-11-21 12:55:47 +05:30
Shivani Tipnis
c44f15441f Closes https://github.com/espressif/esp-idf/issues/2472 2018-11-21 12:55:47 +05:30
Shivani Tipnis
9db1987892 nvs_util: Add changes to nvs part gen util
nvs_util: Add changes to nvs part gen util

nvs_host_test: Add test case for keygen and keyfile paratmeters
2018-11-21 12:55:47 +05:30
Ivan Grokhotkov
6091021e83 unity: separate common and IDF specific functionality
New unity component can be used for testing other applications.
Upstream version of Unity is included as a submodule.
Utilities specific to ESP-IDF unit tests (partitions, leak checking
setup/teardown functions, etc) are kept only in unit-test-app.
Kconfig options are added to allow disabling certain Unity features.
2018-11-19 12:36:31 +08:00
Renz Bagaporo
cc774111bf cmake: Add support for test build 2018-10-20 12:07:24 +08:00
Shivani Tipnis
7c08fcccaf util: Update docs as per review comments 2018-10-12 11:04:49 +05:30
Shivani Tipnis
5bee9faf49 tools: Add Python2 and Python3 compatibility support to nvs util tool and mfg tool 2018-10-12 11:04:49 +05:30
Shivani Tipnis
1d38d8a17f nvs_util: Add Python2 and Python3 compatible 2018-10-12 11:04:49 +05:30
Shivani Tipnis
0f6bcec2bc nvs_part_gen: Add changes to partition size input 2018-10-12 11:04:49 +05:30
Shivani Tipnis
60df21115e nvs_part_gen: Update to make python version compatible
(cherry picked from commit 3506b2d60792e5d4ed4f744b28f2da6733c6aae7)
2018-10-12 11:04:49 +05:30
Shivani Tipnis
1460f03065 nvs_util: Add Python2 and Python3 compatible 2018-10-12 11:04:49 +05:30
Shivani Tipnis
fb31788504 nvs_part_gen: Add changes to partition size input 2018-10-12 11:04:49 +05:30
Shivani Tipnis
4bc87a414f nvs_part_gen: Update to make python version compatible
(cherry picked from commit 3506b2d60792e5d4ed4f744b28f2da6733c6aae7)
2018-10-12 11:04:49 +05:30
Sagar Bijwe
454b69d2ea nvs_flash: Update documentation at different places to indicate NVS encryotion is supported 2018-10-05 14:05:21 +05:30
Angus Gratton
a4aadd1926 Merge branch 'feature/nvs_encryption' into 'master'
nvs_flash: Add support for nvs encryption

See merge request idf/esp-idf!2700
2018-09-25 08:15:55 +08:00
Shivani Tipnis
7c1360b10a nvs_util: Fix multipage blob data format generate 2018-09-24 15:44:31 +05:30
Sagar Bijwe
48fccbf5dd nvs_flash: Add support for nvs encryption 2018-09-24 11:25:21 +05:30
Angus Gratton
4d50427e87 Merge branch 'feature/nvs_util_backport' into 'master'
nvs_util: Add changes for utility to support old and new multipage blob

See merge request idf/esp-idf!3270
2018-09-24 11:53:29 +08:00
Shivani Tipnis
ca3acef82c nvs_util: Add support for old and new multipage blob 2018-09-19 22:00:08 +05:30
Roland Dobai
7183112cc4 nvs_flash: fix division for Python 2 & 3 2018-09-17 16:01:54 +02:00
Renz Christian Bagaporo
d9939cedd9 cmake: make main a component again 2018-09-11 09:44:12 +08:00
Angus Gratton
cff2ef695a Merge branch 'master' into feature/cmake 2018-09-06 20:23:04 +08:00
Sagar Bijwe
db7c257f33 nvs_flash: Minor changes to NVS documentation
1) Change API doc to reflect new limits on string and blob length.
2) Add "version" field to Page Header structure in README.

Fixes: https://github.com/espressif/esp-idf/issues/2321
2018-09-04 18:51:31 +05:30
Angus Gratton
a9c4ed7139 Merge branch 'master' into feature/cmake 2018-08-30 18:51:01 +08:00
Sagar Bijwe
911f96ad21 nvs_flash: Fix nvs_set_blob hang when partition is near to FULL
Marking a page full does not skip it from page selection process and the
same page might get returned if there is no other page with more unused
entries. Added a check for the same while storing blobs.

Fixes: https://github.com/espressif/esp-idf/issues/2313
2018-08-21 17:08:46 +05:30
Angus Gratton
3012b0f887 Merge branch 'feature/nvs_multi_page_util' into 'master'
Bugfix: multi page blob creation was failing if only one entry was left in page

See merge request idf/esp-idf!2971
2018-08-16 15:23:11 +08:00
Angus Gratton
ff2404a272 Merge branch 'master' into feature/cmake 2018-08-16 17:14:17 +10:00
Shivani Tipnis
dfde105f25 Add support for input custom partition size 2018-08-16 11:54:46 +05:30
Shivani Tipnis
5ce40fce0c Add nvs part gen utility multi page blob support update 2018-08-16 11:46:25 +05:30
Ivan Grokhotkov
5fc6397589 nvs: fix memory leak when writing a blob
Introduced in 5a27a63.
Clear usedPages list in all exit conditions, add test.
2018-08-09 10:35:19 +03:00
Sagar Bijwe
1df85e0039 nvs_flash: Version compatibility check for nvs storage
This change adds a check for compatibility between the nvs version
found on nvs flash and the one assumed by running code during nvs
initialization. Any mismatch is reported to the user using new error
code ESP_ERR_NVS_NEW_VERSION_FOUND.
2018-08-05 00:00:56 +00:00
Sagar Bijwe
8ef7a49cc3 nvs_flash: Bump up nvs version 2018-08-05 00:00:56 +00:00
Sagar Bijwe
5a27a63541 nvs-flash: Support for blobs larger than half of SPI Flash sector size
This change removes the earlier limitation of 1984 bytes for storing data-blobs.
Blobs larger than the sector size are split and stored on multiple sectors.
For this purpose, two new datatypes (multi-page index and multi-page data) are
added for entries stored in the sectors. The underlying read, write, erase and find
operations are modified to support these large blobs. The change is transparent
to users of the library and no special APIs need to be used to store these large
blobs.
2018-07-20 00:13:15 +05:30
Piyush Shah
e7ddd47716 nvs_partition_generator: Add support for base64 representation of Binary data 2018-07-02 15:40:21 +05:30
Piyush Shah
8369bd244b nvs_partition_generator: Modifications to enable using the utility as a Python library
And also use directly as an executable
2018-07-02 15:39:52 +05:30
Ivan Grokhotkov
d9f8311b9d docs: add error codes reference 2018-06-15 15:49:24 +08:00
Paul Reimer
c83c3b64cd Add declarations of private nvs functions (nvs_*) in nvs_flash component to fix C++ -Wmissing-declarations warning
Merges https://github.com/espressif/esp-idf/pull/1956
2018-06-04 16:36:44 +10:00
Roland Dobai
07acbb5664 ESP_ERROR_CHECK converts error code to text
If CONFIG_ESP_ERR_TO_NAME_LOOKUP is enabled, ESP_ERROR_CHECK
prints error message in addition to the error code.
2018-05-31 12:53:37 +02:00
Angus Gratton
6b9784cc45 Merge branch 'master' into feature/cmake_update 2018-05-31 14:46:23 +10:00
Ivan Grokhotkov
cb649e452f remove executable permission from source files 2018-05-29 20:07:45 +08:00
Angus Gratton
3d92c6cccd nvs README: Remove section about encryption capability 2018-05-15 18:02:28 +08:00
Ivan Grokhotkov
73ec743a0a Merge branch 'bugfix/nvs_page_compaction' into 'master'
nvs: Fix recovery from power-off while page is being freed

See merge request idf/esp-idf!2327
2018-05-07 09:08:23 +08:00
Sagar Bijwe
7ae1df1c5e nvs: Fix recovery from power-off while page is being freed
Currently when page is being freed, items are individually moved from
FREEING page to ACTIVE page and erased. If power-off happens during the
process, the remaining entries are moved to ACTIVE page during recovery.
The problem with this approach is there may not be enough space on
ACTIVE page for all items if an item was partially written before
power-off and erased during recovery. This change moves all the items
from FREEING to ACTIVE page and then erased the FREEING page, If
power-off happens during the process, then ACTIVE page is erased and the
process is restarted.
2018-05-02 12:18:02 +05:30
Ivan Grokhotkov
5f956125c1 Merge branch 'feature/mfg_bin_create_utility' into 'master'
nvs_flash: Add external binary creation support for NVS partition.

See merge request idf/esp-idf!2031
2018-05-02 11:58:38 +08:00
Angus Gratton
1cb5712463 cmake: Add component dependency support
Components should set the COMPONENT_REQUIRES & COMPONENT_PRIVATE_REQUIRES variables to define their
requirements.
2018-04-30 09:59:20 +10:00
Angus Gratton
4f1a856dbf cmake: Remove defaults for COMPONENT_SRCDIRS, COMPONENT_SRCS, COMPONENT_ADD_INCLUDEDIRS
* Philosophical: "explicit is better than implicit".
* Practical: Allows useful errors if invalid directories given in components as the defaults aren't
  always used. Also trims the -I path from a number of components that have no actual include
  directory.
* Simplifies knowing which components will be header-only and which won't
2018-04-30 09:59:20 +10:00
Angus Gratton
c671a0c3eb build system: Initial cmake support, work in progress 2018-04-30 09:59:20 +10:00
Amit Sheth
aa357a32bc nvs_flash: Add binary creation support for NVS partition.
Ideally suited for generating a binary externally, containing key-value pairs specific
to device manufacturers. Utility allows creation of a binary, compatible
with NVS structure, which can be separately flashed onto a new
partition. This helps device manufacturers set different values for
different devices, e.g. serial numbers, but using a single firmaware
image.
2018-04-23 00:27:29 +05:30
Ivan Grokhotkov
6f504deed0 nvs: run all test by default, exclude long tests by tag
Reduces the chance of new tests being skipped in CI
2018-04-19 15:08:08 +08:00
Ivan Grokhotkov
f8cb95d0b8 nvs: print progress in nvs API tests, reduce number of iterations 2018-04-19 15:08:08 +08:00
Ivan Grokhotkov
17ab60d642 Merge branch 'bugfix/nvs_failed_crc_assert' into 'master'
nvs: don’t expect items with bad CRC to be in cache

See merge request idf/esp-idf!2239
2018-04-18 11:01:21 +08:00
Sagar Bijwe
9a3c4b7134 nvs: Fix recovery after power-off during erase operation
Current code for recovery after power-off do not clean-up partially
erased items for FULL pages. If the erasure was part of modification
operation, this gets luckily cleaned-up because of duplicate detection
logic. For erase-only operation, the problem still exists. This patch
adds the recovery for FULL pages also.

Closes TW<20284>
2018-04-17 12:24:21 +05:30
Konstantin Kondrashov
6a4033b8a7 nvs: fix nvs_get_stats unit test
Changed check for the returned parameter from ESP_ERR_NVS_PART_NOT_FOUND to ESP_ERR_NVS_NOT_INITIALIZED.
2018-04-16 13:28:45 +00:00
Ivan Grokhotkov
2c3644a09f nvs: don’t expect items with bad CRC to be in cache
When erasing a variable length item with an incorrect CRC32, the span
value of the item can not be trusted, so the item will be erased with
span = 1. Subsequent entries represent the data of the variable
length item, and these will be treated as separate items. For each
entry CRC32 is checked, the check most likely fails (because the
entry contains arbitrary data, and not a proper NVS item), and the
entry is erased. Erase function assumed that every item should be
present in cache, but it is not the case for the entries which are
just parts of item’s payload. This change allows for the item to be
not found in the hashlist, if the CRC32 check fails.
2018-04-16 09:36:17 +00:00
Ivan Grokhotkov
20405fbbcc nvs: make nvs_flash_deinit useable from unit tests 2018-04-16 14:06:52 +08:00
Ivan Grokhotkov
fa3bd16d75 nvs: deinit NVS in nvs_stats unit test
This fixes the failure when this test runs after another test which
leaves NVS initialized.
2018-04-16 12:50:57 +08:00
Ivan Grokhotkov
d0d314d8f8 Merge branch 'bugfix/nvs_page_selection' into 'master'
nvs: Fix page selection algo to consider free entry counts as well

See merge request idf/esp-idf!2240
2018-04-16 10:32:01 +08:00
Sagar Bijwe
7e79471ece nvs: Fix page selection algo to consider free entry counts as well
Current page selection algorithm selects a page for compaction based on just erased counts
and gives up when it does not find any page with erased count greater than 0. This is
problematic since the current allocation procedure skips the active page if there is not
enough room for the item in that page leaving free chunks on the pages. This change modifies
the algorithm to consider both erased as well as free counts on the candidate pages.

Closes TW<20297>
2018-04-13 14:49:05 +05:30
konstantin
c93626db3f nvs:Add functions for calculating used/free entries
Users needs functions to count the number of free and used entries.

1. `nvs_get_stats()` This function return structure of statistic about the uspace NVS.
(Struct: used_entries, free_entries, total_entries and namespace_count)
2. `nvs_get_used_entry_count()` The second function return amount of entries in the namespace (by handler)
3. Added unit tests.

Closes TW<12282>
2018-04-13 10:34:13 +05:00
Roman Valls Guimera
519edc332d Fix compilation errors when using gcc-7.2.0 for the crosstool-ng toolchain
* Change snprintf for strlcat does not complain w/gcc7.2.0 and it is safer, thanks @projectgus
* Use proper quotes for character literals

Merges https://github.com/espressif/esp-idf/pull/1163
2017-12-04 10:18:40 +11:00
Ivan Grokhotkov
ab13db0e45 nvs: fix nvs_flash_init_partition initializing default partition
Fixes https://github.com/espressif/esp-idf/issues/1052.

Ref TW15698.
2017-10-13 07:19:16 +08:00
Ivan Grokhotkov
9325f2a7a4 nvs_flash: emulator: fix issues in load function, add save function 2017-09-19 10:39:44 +08:00
Ivan Grokhotkov
fe30789149 nvs_flash: delete Storage if init fails
Previously, nvs_flash_init_custom would not be called if Storage for a
particular partition was already created. This caused issues if the
first call to nvs_flash_init failed due to Storage init error. This issue
exhibited itself as random failures of NVS CI test.

With this change, storage object is deleted (and not added to storage
list) if initialization fails.
2017-09-19 10:39:44 +08:00
Ivan Grokhotkov
f32b25ebe2 nvs_flash: add functions to deinitialize storage 2017-09-19 10:39:44 +08:00
Amey Inamdar
c8945bb82a nvs: minor - removed unused header inclusion
Signed-off-by: Amey Inamdar <amey.inamdar@gmail.com>
2017-09-18 17:02:32 +05:30
baohongde
87031f9457 component/bt: fix nvs_get_str_or_blob length output error 2017-09-08 12:02:35 +08:00
baohongde
e1bd51af80 component/bt: fix nvs_get_str_or_blob length output error 2017-09-08 12:02:35 +08:00
baohongde
f3be0395d2 component/bt: fix nvs_get_str_or_blob length output error 2017-09-08 12:02:35 +08:00
Ivan Grokhotkov
c0269bc1cd nvs: check CRC of items on full pages
Previously NVS did check CRC values of key-value pairs on the active
page, but the check for full pages was missing. This adds the necessary
check and a test for it.
2017-09-04 14:37:33 +08:00
Amey Inamdar
a2dcf7faa8 nvs: Support for handling multiple NVS partitions
This commit adds support for multiple NVS partitions. This provides application a flexibility to have multiple NVS
partitions such as separate partition with read-only manufacturing data and read-write partition with configuration.
Application can also use this to separate out application's configuration storage from system configuration.

This feature does not change any of the basic property of NVS subsystem. The same-named namespaces across partitions are
considered to be different namespaces. The original NVS API available for the applications remains unchanged. The only
difference is that instead of first NVS partition in the partition table, it now operates on the partition with label
"nvs" (which is default in the IDF provided partition table files). Additional APIs are provided to open a handle and
erase NVS with partition name as a parameter.

A test case is added in the host tests and it is made sure that all the host tests pass. nvs_rw_value app is also tested
with multiple partitions.

Signed-off-by: Amey Inamdar <amey.inamdar@gmail.com>
2017-08-22 13:48:14 +05:30
Amey Inamdar
3e4e4dd07a nvs_flash: Reduced visibility of handle counter
Monotonically increasing handle counter need not be visible outside the HandleEntry class.

Signed-off-by: Amey Inamdar <amey.inamdar@gmail.com>
2017-08-22 12:58:56 +05:30
Ivan Grokhotkov
979fce0df5 bt: call nvs_flash_init in examples, show error if NVS is not initialized
NVS is used to store PHY calibration data, WiFi configuration, and BT
configuration. Previously BT examples did not call nvs_flash_init,
relying on the fact that it is called during PHY init. However PHY init
did not handle possible NVS initialization errors.

This change moves PHY init procedure into the application, and adds
diagnostic messages to BT config management routines if NVS is not
initialized.
2017-07-17 21:29:50 +08:00
Ivan Grokhotkov
f045123691 nvs: add test for ESP_ERR_NVS_INVALID_LENGTH 2017-06-27 11:28:37 +08:00