Non-volatile storage library

Introduction

Non-volatile storage (NVS) library is designed to store key-value pairs in flash. This sections introduces some concepts used by NVS.

Underlying storage

Currently NVS uses a portion of main flash memory through spi_flash_{read|write|erase} APIs. The range of flash sectors to be used by the library is provided to nvs_flash_init function.

Future versions of this library may add other storage backends to keep data in another flash chip (SPI or I2C), RTC, FRAM, etc.

Keys and values

NVS operates on key-value pairs. Keys are ASCII strings, maximum key length is currently 15 characters. Values can have one of the following types:

  • integer types: uint8_t, int8_t, uint16_t, int16_t, uint32_t, int32_t, uint64_t, int64_t
  • zero-terminated string
  • variable length binary data (blob)

Additional types, such as float and double may be added later.

Keys are required to be unique. Writing a value for a key which already exists behaves as follows:

  • if the new value is of the same type as old one, value is updated
  • if the new value has different data type, an error is returned

Data type check is also performed when reading a value. An error is returned if data type of read operation doesn’t match the data type of the value.

Namespaces

To mitigate potential conflicts in key names between different components, NVS assigns each key-value pair to one of namespaces. Namespace names follow the same rules as key names, i.e. 15 character maximum length. Namespace name is specified in the nvs_open call. This call returns an opaque handle, which is used in subsequent calls to nvs_read_*, nvs_write_*, and nvs_commit functions. This way, handle is associated with a namespace, and key names will not collide with same names in other namespaces.

Security, tampering, and robustness

NVS library doesn’t implement tamper prevention measures. It is possible for anyone with physical access to the flash chip to alter, erase, or add key-value pairs.

NVS is compatible with the ESP32 flash encryption system, and it can store key-value pairs in an encrypted form. Some metadata, like page state and write/erase flags of individual entries can not be encrypted as they are represented as bits of flash memory for efficient access and manipulation. Flash encryption can prevent some forms of modification:

  • replacing keys or values with arbitrary data
  • changing data types of values

The following forms of modification are still possible when flash encryption is used:

  • erasing a page completely, removing all key-value pairs which were stored in that page
  • corrupting data in a page, which will cause the page to be erased automatically when such condition is detected
  • rolling back the contents of flash memory to an earlier snapshot
  • merging two snapshots of flash memory, rolling back some key-value pairs to an earlier state (although this is possible to mitigate with the current design — TODO)

The library does try to recover from conditions when flash memory is in an inconsistent state. In particular, one should be able to power off the device at any point and time and then power it back on. This should not result in loss of data, expect for the new key-value pair if it was being written at the moment of power off. The library should also be able to initialize properly with any random data present in flash memory.

Internals

Log of key-value pairs

NVS stores key-value pairs sequentially, with new key-value pairs being added at the end. When a value of any given key has to be updated, new key-value pair is added at the end of the log and old key-value pair is marked as erased.

Pages and entries

NVS library uses two main entities in its operation: pages and entries. Page is a logical structure which stores a portion of the overall log. Logical page corresponds to one physical sector of flash memory. Pages which are in use have a sequence number associated with them. Sequence numbers impose an ordering on pages. Higher sequence numbers correspond to pages which were created later. Each page can be in one of the following states:

Empty/uninitialized
Flash storage for the page is empty (all bytes are 0xff). Page isn’t used to store any data at this point and doesn’t have a sequence number.
Active
Flash storage is initialized, page header has been written to flash, page has a valid sequence number. Page has some empty entries and data can be written there. At most one page can be in this state at any given moment.
Full
Flash storage is in a consistent state and is filled with key-value pairs. Writing new key-value pairs into this page is not possible. It is still possible to mark some key-value pairs as erased.
Erasing
Non-erased key-value pairs are being moved into another page so that the current page can be erased. This is a transient state, i.e. page should never stay in this state when any API call returns. In case of a sudden power off, move-and-erase process will be completed upon next power on.
Corrupted
Page header contains invalid data, and further parsing of page data was canceled. Any items previously written into this page will not be accessible. Corresponding flash sector will not be erased immediately, and will be kept along with sectors in uninitialized state for later use. This may be useful for debugging.

Mapping from flash sectors to logical pages doesn’t have any particular order. Library will inspect sequence numbers of pages found in each flash sector and organize pages in a list based on these numbers.

+--------+     +--------+     +--------+     +--------+
| Page 1 |     | Page 2 |     | Page 3 |     | Page 4 |
| Full   +---> | Full   +---> | Active |     | Empty  |   <- states
| #11    |     | #12    |     | #14    |     |        |   <- sequence numbers
+---+----+     +----+---+     +----+---+     +---+----+
    |               |              |             |
    |               |              |             |
    |               |              |             |
+---v------+  +-----v----+  +------v---+  +------v---+
| Sector 3 |  | Sector 0 |  | Sector 2 |  | Sector 1 |    <- physical sectors
+----------+  +----------+  +----------+  +----------+

Structure of a page

For now we assume that flash sector size is 4096 bytes and that ESP32 flash encryption hardware operates on 32-byte blocks. It is possible to introduce some settings configurable at compile-time (e.g. via menuconfig) to accommodate flash chips with different sector sizes (although it is not clear if other components in the system, e.g. SPI flash driver and SPI flash cache can support these other sizes).

Page consists of three parts: header, entry state bitmap, and entries themselves. To be compatible with ESP32 flash encryption, entry size is 32 bytes. For integer types, entry holds one key-value pair. For strings and blobs, an entry holds part of key-value pair (more on that in the entry structure description).

The following diagram illustrates page structure. Numbers in parentheses indicate size of each part in bytes.

+-----------+--------------+-------------+-----------+
| State (4) | Seq. no. (4) | Unused (20) | CRC32 (4) | Header (32)
+-----------+--------------+-------------+-----------+
|                Entry state bitmap (32)             |
+----------------------------------------------------+
|                       Entry 0 (32)                 |
+----------------------------------------------------+
|                       Entry 1 (32)                 |
+----------------------------------------------------+
/                                                    /
/                                                    /
+----------------------------------------------------+
|                       Entry 125 (32)               |
+----------------------------------------------------+

Page header and entry state bitmap are always written to flash unencrypted. Entries are encrypted if flash encryption feature of the ESP32 is used.

Page state values are defined in such a way that changing state is possible by writing 0 into some of the bits. Therefore it not necessary to erase the page to change page state, unless that is a change to erased state.

CRC32 value in header is calculated over the part which doesn’t include state value (bytes 4 to 28). Unused part is currently filled with 0xff bytes. Future versions of the library may store format version there.

The following sections describe structure of entry state bitmap and entry itself.

Entry and entry state bitmap

Each entry can be in one of the following three states. Each state is represented with two bits in the entry state bitmap. Final four bits in the bitmap (256 - 2 * 126) are unused.

Empty (2’b11)
Nothing is written into the specific entry yet. It is in an uninitialized state (all bytes 0xff).
Written (2’b10)
A key-value pair (or part of key-value pair which spans multiple entries) has been written into the entry.
Erased (2’b00)
A key-value pair in this entry has been discarded. Contents of this entry will not be parsed anymore.

Structure of entry

For values of primitive types (currently integers from 1 to 8 bytes long), entry holds one key-value pair. For string and blob types, entry holds part of the whole key-value pair. In case when a key-value pair spans multiple entries, all entries are stored in the same page.

+--------+----------+----------+---------+-----------+---------------+----------+
| NS (1) | Type (1) | Span (1) | Rsv (1) | CRC32 (4) |    Key (16)   | Data (8) |
+--------+----------+----------+---------+-----------+---------------+----------+

                                               +--------------------------------+
                         +->    Fixed length:  | Data (8)                       |
                         |                     +--------------------------------+
          Data format ---+
                         |                     +----------+---------+-----------+
                         +-> Variable length:  | Size (2) | Rsv (2) | CRC32 (4) |
                                               +----------+---------+-----------+

Individual fields in entry structure have the following meanings:

NS
Namespace index for this entry. See section on namespaces implementation for explanation of this value.
Type
One byte indicating data type of value. See ItemType enumeration in nvs_types.h for possible values.
Span
Number of entries used by this key-value pair. For integer types, this is equal to 1. For strings and blobs this depends on value length.
Rsv
Unused field, should be 0xff.
CRC32
Checksum calculated over all the bytes in this entry, except for the CRC32 field itself.
Key
Zero-terminated ASCII string containing key name. Maximum string length is 15 bytes, excluding zero terminator.
Data
For integer types, this field contains the value itself. If the value itself is shorter than 8 bytes it is padded to the right, with unused bytes filled with 0xff. For string and blob values, these 8 bytes hold additional data about the value, described next:
Size
(Only for strings and blobs.) Size, in bytes, of actual data. For strings, this includes zero terminator.
CRC32
(Only for strings and blobs.) Checksum calculated over all bytes of data.

Variable length values (strings and blobs) are written into subsequent entries, 32 bytes per entry. Span field of the first entry indicates how many entries are used.

Namespaces

As mentioned above, each key-value pair belongs to one of the namespaces. Namespaces identifiers (strings) are stored as keys of key-value pairs in namespace with index 0. Values corresponding to these keys are indexes of these namespaces.

+-------------------------------------------+
| NS=0 Type=uint8_t Key="wifi" Value=1      |   Entry describing namespace "wifi"
+-------------------------------------------+
| NS=1 Type=uint32_t Key="channel" Value=6  |   Key "channel" in namespace "wifi"
+-------------------------------------------+
| NS=0 Type=uint8_t Key="pwm" Value=2       |   Entry describing namespace "pwm"
+-------------------------------------------+
| NS=2 Type=uint16_t Key="channel" Value=20 |   Key "channel" in namespace "pwm"
+-------------------------------------------+

Item hash list

To reduce the number of reads performed from flash memory, each member of Page class maintains a list of pairs: (item index; item hash). This list makes searches much quicker. Instead of iterating over all entries, reading them from flash one at a time, Page::findItem first performs search for item hash in the hash list. This gives the item index within the page, if such an item exists. Due to a hash collision it is possible that a different item will be found. This is handled by falling back to iteration over items in flash.

Each node in hash list contains a 24-bit hash and 8-bit item index. Hash is calculated based on item namespace and key name. CRC32 is used for calculation, result is truncated to 24 bits. To reduce overhead of storing 32-bit entries in a linked list, list is implemented as a doubly-linked list of arrays. Each array holds 29 entries, for the total size of 128 bytes, together with linked list pointers and 32-bit count field. Minimal amount of extra RAM useage per page is therefore 128 bytes, maximum is 640 bytes.

Application Example

Two examples are provided in ESP-IDF examples directory:

07_nvs_rw_value

Demonstrates how to read and write a single integer value using NVS.

The value holds the number of ESP32 module restarts. Since it is written to NVS, the value is preserved between restarts.

Example also shows how to check if read / write operation was successful, or certain value is not initialized in NVS. Diagnostic is provided in plain text to help track program flow and capture any issues on the way.

08_nvs_rw_blob

Demonstrates how to read and write a single integer value and a blob (binary large object) using NVS to preserve them between ESP32 module restarts.

  • value - tracks number of ESP32 module soft and hard restarts.
  • blob - contains a table with module run times. The table is read from NVS to dynamically allocated RAM. New run time is added to the table on each manually triggered soft restart and written back to NVS. Triggering is done by pulling down GPIO0.

Example also shows how to implement diagnostics if read / write operation was successful.

API Reference

Macros

ESP_ERR_NVS_BASE 0x1100

Starting number of error codes

ESP_ERR_NVS_NOT_INITIALIZED (ESP_ERR_NVS_BASE + 0x01)

The storage driver is not initialized

ESP_ERR_NVS_NOT_FOUND (ESP_ERR_NVS_BASE + 0x02)

Id namespace doesn’t exist yet and mode is NVS_READONLY

ESP_ERR_NVS_TYPE_MISMATCH (ESP_ERR_NVS_BASE + 0x03)

TBA

ESP_ERR_NVS_READ_ONLY (ESP_ERR_NVS_BASE + 0x04)

Storage handle was opened as read only

ESP_ERR_NVS_NOT_ENOUGH_SPACE (ESP_ERR_NVS_BASE + 0x05)

There is not enough space in the underlying storage to save the value

ESP_ERR_NVS_INVALID_NAME (ESP_ERR_NVS_BASE + 0x06)

Namespace name doesn’t satisfy constraints

ESP_ERR_NVS_INVALID_HANDLE (ESP_ERR_NVS_BASE + 0x07)

Handle has been closed or is NULL

ESP_ERR_NVS_REMOVE_FAILED (ESP_ERR_NVS_BASE + 0x08)

The value wasn’t updated because flash write operation has failed. The value was written however, and update will be finished after re-initialization of nvs, provided that flash operation doesn’t fail again.

ESP_ERR_NVS_KEY_TOO_LONG (ESP_ERR_NVS_BASE + 0x09)

TBA

ESP_ERR_NVS_PAGE_FULL (ESP_ERR_NVS_BASE + 0x0a)

TBA

ESP_ERR_NVS_INVALID_STATE (ESP_ERR_NVS_BASE + 0x0b)

TBA

ESP_ERR_NVS_INVALID_LENGTH (ESP_ERR_NVS_BASE + 0x0c)

TBA

Type Definitions

typedef uint32_t nvs_handle

Opaque pointer type representing non-volatile storage handle

Enumerations

enum nvs_open_mode

Mode of opening the non-volatile storage.

Values:

NVS_READONLY

Read only

NVS_READWRITE

Read and write

Functions

esp_err_t nvs_open(const char *name, nvs_open_mode open_mode, nvs_handle *out_handle)

Open non-volatile storage with a given namespace.

Multiple internal ESP-IDF and third party application modules can store their key-value pairs in the NVS module. In order to reduce possible conflicts on key names, each module can use its own namespace.

Return
  • ESP_OK if storage handle was opened successfully
  • ESP_ERR_NVS_NOT_INITIALIZED if the storage driver is not initialized
  • ESP_ERR_NVS_NOT_FOUND id namespace doesn’t exist yet and mode is NVS_READONLY
  • ESP_ERR_NVS_INVALID_NAME if namespace name doesn’t satisfy constraints
  • other error codes from the underlying storage driver
Parameters
  • name: Namespace name. Maximal length is determined by the underlying implementation, but is guaranteed to be at least 16 characters. Shouldn’t be empty.
  • open_mode: NVS_READWRITE or NVS_READONLY. If NVS_READONLY, will open a handle for reading only. All write requests will be rejected for this handle.
  • out_handle: If successful (return code is zero), handle will be returned in this argument.

esp_err_t nvs_set_i8(nvs_handle handle, const char *key, int8_t value)

set value for given key

This family of functions set value for the key, given its name. Note that actual storage will not be updated until nvs_commit function is called.

Return
  • ESP_OK if value was set successfully
  • ESP_ERR_NVS_INVALID_HANDLE if handle has been closed or is NULL
  • ESP_ERR_NVS_READ_ONLY if storage handle was opened as read only
  • ESP_ERR_NVS_INVALID_NAME if key name doesn’t satisfy constraints
  • ESP_ERR_NVS_NOT_ENOUGH_SPACE if there is not enough space in the underlying storage to save the value
  • ESP_ERR_NVS_REMOVE_FAILED if the value wasn’t updated because flash write operation has failed. The value was written however, and update will be finished after re-initialization of nvs, provided that flash operation doesn’t fail again.
Parameters
  • handle: Handle obtained from nvs_open function. Handles that were opened read only cannot be used.
  • key: Key name. Maximal length is determined by the underlying implementation, but is guaranteed to be at least 16 characters. Shouldn’t be empty.
  • value: The value to set.

esp_err_t nvs_set_u8(nvs_handle handle, const char *key, uint8_t value)
esp_err_t nvs_set_i16(nvs_handle handle, const char *key, int16_t value)
esp_err_t nvs_set_u16(nvs_handle handle, const char *key, uint16_t value)
esp_err_t nvs_set_i32(nvs_handle handle, const char *key, int32_t value)
esp_err_t nvs_set_u32(nvs_handle handle, const char *key, uint32_t value)
esp_err_t nvs_set_i64(nvs_handle handle, const char *key, int64_t value)
esp_err_t nvs_set_u64(nvs_handle handle, const char *key, uint64_t value)
esp_err_t nvs_set_str(nvs_handle handle, const char *key, const char *value)
esp_err_t nvs_set_blob(nvs_handle handle, const char *key, const void *value, size_t length)

set variable length binary value for given key

This family of functions set value for the key, given its name. Note that actual storage will not be updated until nvs_commit function is called.

Return
  • ESP_OK if value was set successfully
  • ESP_ERR_NVS_INVALID_HANDLE if handle has been closed or is NULL
  • ESP_ERR_NVS_READ_ONLY if storage handle was opened as read only
  • ESP_ERR_NVS_INVALID_NAME if key name doesn’t satisfy constraints
  • ESP_ERR_NVS_NOT_ENOUGH_SPACE if there is not enough space in the underlying storage to save the value
  • ESP_ERR_NVS_REMOVE_FAILED if the value wasn’t updated because flash write operation has failed. The value was written however, and update will be finished after re-initialization of nvs, provided that flash operation doesn’t fail again.
Parameters
  • handle: Handle obtained from nvs_open function. Handles that were opened read only cannot be used.
  • key: Key name. Maximal length is determined by the underlying implementation, but is guaranteed to be at least 16 characters. Shouldn’t be empty.
  • value: The value to set.
  • length: length of binary value to set, in bytes.

esp_err_t nvs_get_i8(nvs_handle handle, const char *key, int8_t *out_value)

get value for given key

These functions retrieve value for the key, given its name. If key does not exist, or the requested variable type doesn’t match the type which was used when setting a value, an error is returned.

In case of any error, out_value is not modified.

All functions expect out_value to be a pointer to an already allocated variable of the given type.

// Example of using nvs_get_i32:
int32_t max_buffer_size = 4096; // default value
esp_err_t err = nvs_get_i32(my_handle, "max_buffer_size", &max_buffer_size);
assert(err == ESP_OK || err == ESP_ERR_NVS_NOT_FOUND);
// if ESP_ERR_NVS_NOT_FOUND was returned, max_buffer_size will still
// have its default value.

Return
  • ESP_OK if the value was retrieved successfully
  • ESP_ERR_NVS_NOT_FOUND if the requested key doesn’t exist
  • ESP_ERR_NVS_INVALID_HANDLE if handle has been closed or is NULL
  • ESP_ERR_NVS_INVALID_NAME if key name doesn’t satisfy constraints
  • ESP_ERR_NVS_INVALID_LENGTH if length is not sufficient to store data
Parameters
  • handle: Handle obtained from nvs_open function.
  • key: Key name. Maximal length is determined by the underlying implementation, but is guaranteed to be at least 16 characters. Shouldn’t be empty.
  • out_value: Pointer to the output value. May be NULL for nvs_get_str and nvs_get_blob, in this case required length will be returned in length argument.

esp_err_t nvs_get_u8(nvs_handle handle, const char *key, uint8_t *out_value)
esp_err_t nvs_get_i16(nvs_handle handle, const char *key, int16_t *out_value)
esp_err_t nvs_get_u16(nvs_handle handle, const char *key, uint16_t *out_value)
esp_err_t nvs_get_i32(nvs_handle handle, const char *key, int32_t *out_value)
esp_err_t nvs_get_u32(nvs_handle handle, const char *key, uint32_t *out_value)
esp_err_t nvs_get_i64(nvs_handle handle, const char *key, int64_t *out_value)
esp_err_t nvs_get_u64(nvs_handle handle, const char *key, uint64_t *out_value)
esp_err_t nvs_get_str(nvs_handle handle, const char *key, char *out_value, size_t *length)

get value for given key

These functions retrieve value for the key, given its name. If key does not exist, or the requested variable type doesn’t match the type which was used when setting a value, an error is returned.

In case of any error, out_value is not modified.

All functions expect out_value to be a pointer to an already allocated variable of the given type.

nvs_get_str and nvs_get_blob functions support WinAPI-style length queries. To get the size necessary to store the value, call nvs_get_str or nvs_get_blob with zero out_value and non-zero pointer to length. Variable pointed to by length argument will be set to the required length. For nvs_get_str, this length includes the zero terminator. When calling nvs_get_str and nvs_get_blob with non-zero out_value, length has to be non-zero and has to point to the length available in out_value. It is suggested that nvs_get/set_str is used for zero-terminated C strings, and nvs_get/set_blob used for arbitrary data structures.

// Example (without error checking) of using nvs_get_str to get a string into dynamic array:
size_t required_size;
nvs_get_str(my_handle, "server_name", NULL, &required_size);
char* server_name = malloc(required_size);
nvs_get_str(my_handle, "server_name", server_name, &required_size);

// Example (without error checking) of using nvs_get_blob to get a binary data
into a static array:
uint8_t mac_addr[6];
size_t size = sizeof(mac_addr);
nvs_get_blob(my_handle, "dst_mac_addr", mac_addr, &size);

Return
  • ESP_OK if the value was retrieved successfully
  • ESP_ERR_NVS_NOT_FOUND if the requested key doesn’t exist
  • ESP_ERR_NVS_INVALID_HANDLE if handle has been closed or is NULL
  • ESP_ERR_NVS_INVALID_NAME if key name doesn’t satisfy constraints
  • ESP_ERR_NVS_INVALID_LENGTH if length is not sufficient to store data
Parameters
  • handle: Handle obtained from nvs_open function.
  • key: Key name. Maximal length is determined by the underlying implementation, but is guaranteed to be at least 16 characters. Shouldn’t be empty.
  • out_value: Pointer to the output value. May be NULL for nvs_get_str and nvs_get_blob, in this case required length will be returned in length argument.
  • length: A non-zero pointer to the variable holding the length of out_value. In case out_value a zero, will be set to the length required to hold the value. In case out_value is not zero, will be set to the actual length of the value written. For nvs_get_str this includes zero terminator.

esp_err_t nvs_get_blob(nvs_handle handle, const char *key, void *out_value, size_t *length)
esp_err_t nvs_erase_key(nvs_handle handle, const char *key)

Erase key-value pair with given key name.

Note that actual storage may not be updated until nvs_commit function is called.

Return
  • ESP_OK if erase operation was successful
  • ESP_ERR_NVS_INVALID_HANDLE if handle has been closed or is NULL
  • ESP_ERR_NVS_READ_ONLY if handle was opened as read only
  • ESP_ERR_NVS_NOT_FOUND if the requested key doesn’t exist
  • other error codes from the underlying storage driver
Parameters
  • handle: Storage handle obtained with nvs_open. Handles that were opened read only cannot be used.
  • key: Key name. Maximal length is determined by the underlying implementation, but is guaranteed to be at least 16 characters. Shouldn’t be empty.

esp_err_t nvs_erase_all(nvs_handle handle)

Erase all key-value pairs in a namespace.

Note that actual storage may not be updated until nvs_commit function is called.

Return
  • ESP_OK if erase operation was successful
  • ESP_ERR_NVS_INVALID_HANDLE if handle has been closed or is NULL
  • ESP_ERR_NVS_READ_ONLY if handle was opened as read only
  • other error codes from the underlying storage driver
Parameters
  • handle: Storage handle obtained with nvs_open. Handles that were opened read only cannot be used.

esp_err_t nvs_commit(nvs_handle handle)

Write any pending changes to non-volatile storage.

After setting any values, nvs_commit() must be called to ensure changes are written to non-volatile storage. Individual implementations may write to storage at other times, but this is not guaranteed.

Return
  • ESP_OK if the changes have been written successfully
  • ESP_ERR_NVS_INVALID_HANDLE if handle has been closed or is NULL
  • other error codes from the underlying storage driver
Parameters
  • handle: Storage handle obtained with nvs_open. Handles that were opened read only cannot be used.

void nvs_close(nvs_handle handle)

Close the storage handle and free any allocated resources.

This function should be called for each handle opened with nvs_open once the handle is not in use any more. Closing the handle may not automatically write the changes to nonvolatile storage. This has to be done explicitly using nvs_commit function. Once this function is called on a handle, the handle should no longer be used.

Parameters
  • handle: Storage handle to close

esp_err_t nvs_flash_init(void)

Initialize NVS flash storage with layout given in the partition table.

Return
ESP_OK if storage was successfully initialized.

Warning

doxygenfunction: Cannot find function “nvs_flash_init_custom” in doxygen xml output for project “esp32-idf” from directory: xml/