# Nanopb: Basic concepts

The things outlined here are the underlying concepts of the nanopb
design.

## Proto files

All Protocol Buffers implementations use .proto files to describe the
message format. The point of these files is to be a portable interface
description language.

### Compiling .proto files for nanopb

Nanopb comes with a Python script to generate `.pb.c` and
`.pb.h` files from the `.proto` definition:

    user@host:~$ nanopb/generator/nanopb_generator.py message.proto
    Writing to message.pb.h and message.pb.c

Internally this script uses Google `protoc` to parse the
input file. If you do not have it available, you may receive an error
message. You can install either `grpcio-tools` Python
package using `pip`, or the `protoc` compiler
itself from `protobuf-compiler` distribution package.
Generally the Python package is recommended, because nanopb requires
protoc version 3.6 or newer to support all features, and some distributions come with an older
version.

### Modifying generator behaviour

Using generator options, you can set maximum sizes for fields in order
to allocate them statically. The preferred way to do this is to create
an .options file with the same name as your .proto file:

    # Foo.proto
    message Foo {
       required string name = 1;
    }

    # Foo.options
    Foo.name max_size:16

For more information on this, see the [Proto file
options](reference.html#proto-file-options) section in the reference
manual.

## Streams

Nanopb uses streams for accessing the data in encoded format. The stream
abstraction is very lightweight, and consists of a structure
(`pb_ostream_t` or `pb_istream_t`) which contains a pointer to a
callback function.

There are a few generic rules for callback functions:

1)  Return false on IO errors. The encoding or decoding process will
    abort immediately.
2)  Use state to store your own data, such as a file descriptor.
3)  `bytes_written` and `bytes_left` are updated by pb_write and
    pb_read.
4)  Your callback may be used with substreams. In this case
    `bytes_left`, `bytes_written` and `max_size` have smaller values
    than the original stream. Don't use these values to calculate
    pointers.
5)  Always read or write the full requested length of data. For example,
    POSIX `recv()` needs the `MSG_WAITALL` parameter to accomplish
    this.

### Output streams

    struct _pb_ostream_t
    {
       bool (*callback)(pb_ostream_t *stream, const uint8_t *buf, size_t count);
       void *state;
       size_t max_size;
       size_t bytes_written;
    };

The `callback` for output stream may be NULL, in which case the stream
simply counts the number of bytes written. In this case, `max_size` is
ignored.

Otherwise, if `bytes_written` + bytes_to_be_written is larger than
`max_size`, pb_write returns false before doing anything else. If you
don\'t want to limit the size of the stream, pass SIZE_MAX.

**Example 1:**

This is the way to get the size of the message without storing it
anywhere:

    Person myperson = ...;
    pb_ostream_t sizestream = {0};
    pb_encode(&sizestream, Person_fields, &myperson);
    printf("Encoded size is %d\n", sizestream.bytes_written);

**Example 2:**

Writing to stdout:

    bool callback(pb_ostream_t `stream, const uint8_t `buf, size_t count)
    {
       FILE *file = (FILE*) stream->state;
       return fwrite(buf, 1, count, file) == count;
    }

    pb_ostream_t stdoutstream = {&callback, stdout, SIZE_MAX, 0};

### Input streams

For input streams, there is one extra rule:

6)  You don't need to know the length of the message in advance. After
    getting EOF error when reading, set `bytes_left` to 0 and return
    `false`. `pb_decode()` will detect this and if the EOF was in a proper
    position, it will return true.

Here is the structure:

    struct _pb_istream_t
    {
       bool (*callback)(pb_istream_t *stream, uint8_t *buf, size_t count);
       void *state;
       size_t bytes_left;
    };

The `callback` must always be a function pointer. `Bytes_left` is an
upper limit on the number of bytes that will be read. You can use
SIZE_MAX if your callback handles EOF as described above.

**Example:**

This function binds an input stream to stdin:

    bool callback(pb_istream_t *stream, uint8_t *buf, size_t count)
    {
       FILE *file = (FILE*)stream->state;
       bool status;

       if (buf == NULL)
       {
           while (count-- && fgetc(file) != EOF);
           return count == 0;
       }

       status = (fread(buf, 1, count, file) == count);

       if (feof(file))
           stream->bytes_left = 0;

       return status;
    }

    pb_istream_t stdinstream = {&callback, stdin, SIZE_MAX};

## Data types

Most Protocol Buffers datatypes have directly corresponding C datatypes,
such as `int32` is `int32_t`, `float` is `float` and `bool` is `bool`. However, the
variable-length datatypes are more complex:

1)  Strings, bytes and repeated fields of any type map to callback
    functions by default.
2)  If there is a special option `(nanopb).max_size` specified in the
    .proto file, string maps to null-terminated char array and bytes map
    to a structure containing a char array and a size field.
3)  If `(nanopb).fixed_length` is set to `true` and
    `(nanopb).max_size` is also set, then bytes map to an inline byte
    array of fixed size.
4)  If there is a special option `(nanopb).max_count` specified on a
    repeated field, it maps to an array of whatever type is being
    repeated. Another field will be created for the actual number of
    entries stored.
5)  If `(nanopb).fixed_count` is set to `true` and
    `(nanopb).max_count` is also set, the field for the actual number
    of entries will not by created as the count is always assumed to be
    max count.

### Examples of .proto specifications vs. generated structure

**Simple integer field:**\
.proto: `int32 age = 1;`\
.pb.h: `int32_t age;`

**String with unknown length:**\
.proto: `string name = 1;`\
.pb.h: `pb_callback_t name;`

**String with known maximum length:**\
.proto: `string name = 1 [(nanopb).max_length = 40];`\
.pb.h: `char name[41];`

**Repeated string with unknown count:**\
.proto: `repeated string names = 1;`\
.pb.h: `pb_callback_t names;`

**Repeated string with known maximum count and size:**\
.proto: `repeated string names = 1 [(nanopb).max_length = 40, (nanopb).max_count = 5];`\
.pb.h: `size_t names_count;` `char names[5][41];`

**Bytes field with known maximum size:**\
.proto: `bytes data = 1 [(nanopb).max_size = 16];`\
.pb.h: `PB_BYTES_ARRAY_T(16) data;`, where the struct contains `{pb_size_t size; pb_byte_t bytes[n];}`

**Bytes field with fixed length:**\
.proto: `bytes data = 1 [(nanopb).max_size = 16, (nanopb).fixed_length = true];`\
.pb.h: `pb_byte_t data[16];`

**Repeated integer array with known maximum size:**\
.proto: `repeated int32 numbers = 1 [(nanopb).max_count = 5];`\
.pb.h: `pb_size_t numbers_count;` `int32_t numbers[5];`

**Repeated integer array with fixed count:**\
.proto: `repeated int32 numbers = 1 [(nanopb).max_count = 5, (nanopb).fixed_count = true];`\
.pb.h: `int32_t numbers[5];`

The maximum lengths are checked in runtime. If string/bytes/array
exceeds the allocated length, `pb_decode()` will return false.

> **Note:** For the `bytes` datatype, the field length checking may not be
exact. The compiler may add some padding to the `pb_bytes_t`
structure, and the nanopb runtime doesn't know how much of the
structure size is padding. Therefore it uses the whole length of the
structure for storing data, which is not very smart but shouldn't cause
problems. In practise, this means that if you specify
`(nanopb).max_size=5` on a `bytes` field, you may be able to store 6
bytes there. For the `string` field type, the length limit is exact.

> **Note:** The decoder only keeps track of one `fixed_count` repeated field at a time. Usually this it not an issue because all elements of a repeated field occur end-to-end. Interleaved array elements of several `fixed_count` repeated fields would be a valid protobuf message, but would get rejected by nanopb decoder with error `"wrong size for fixed count field"`.

## Field callbacks

When a field has dynamic length, nanopb cannot statically allocate
storage for it. Instead, it allows you to handle the field in whatever
way you want, using a callback function.

The [pb_callback_t](reference.html#pb-callback-t) structure contains a
function pointer and a `void` pointer called `arg` you can use for
passing data to the callback. If the function pointer is NULL, the field
will be skipped. A pointer to the `arg` is passed to the function, so
that it can modify it and retrieve the value.

The actual behavior of the callback function is different in encoding
and decoding modes. In encoding mode, the callback is called once and
should write out everything, including field tags. In decoding mode, the
callback is called repeatedly for every data item.

To write more complex field callbacks, it is recommended to read the
[Google Protobuf Encoding Specification](https://developers.google.com/protocol-buffers/docs/encoding).

### Encoding callbacks

    bool (*encode)(pb_ostream_t *stream, const pb_field_iter_t *field, void * const *arg);

|            |                                                                    |
| ---------- | ------------------------------------------------------------------ |
| `stream`   | Output stream to write to                                          |
| `field`    | Iterator for the field currently being encoded or decoded.         |
| `arg`      | Pointer to the `arg` field in the `pb_callback_t` structure.       |

When encoding, the callback should write out complete fields, including
the wire type and field number tag. It can write as many or as few
fields as it likes. For example, if you want to write out an array as
`repeated` field, you should do it all in a single call.

Usually you can use [pb_encode_tag_for_field](reference.html#pb-encode-tag-for-field) to
encode the wire type and tag number of the field. However, if you want
to encode a repeated field as a packed array, you must call
[pb_encode_tag](reference.html#pb-encode-tag) instead to specify a
wire type of `PB_WT_STRING`.

If the callback is used in a submessage, it will be called multiple
times during a single call to [pb_encode](reference.html#pb-encode). In
this case, it must produce the same amount of data every time. If the
callback is directly in the main message, it is called only once.

This callback writes out a dynamically sized string:

    bool write_string(pb_ostream_t *stream, const pb_field_iter_t *field, void * const *arg)
    {
        char *str = get_string_from_somewhere();
        if (!pb_encode_tag_for_field(stream, field))
            return false;

        return pb_encode_string(stream, (uint8_t*)str, strlen(str));
    }

### Decoding callbacks

    bool (*decode)(pb_istream_t *stream, const pb_field_iter_t *field, void **arg);

|            |                                                                    |
| ---------- | ------------------------------------------------------------------ |
| `stream`   | Input stream to read from                                          |
| `field`    | Iterator for the field currently being encoded or decoded.         |
| `arg`      | Pointer to the `arg` field in the `pb_callback_t` structure.       |

When decoding, the callback receives a length-limited substring that
reads the contents of a single field. The field tag has already been
read. For `string` and `bytes`, the length value has already been
parsed, and is available at `stream->bytes_left`.

The callback will be called multiple times for repeated fields. For
packed fields, you can either read multiple values until the stream
ends, or leave it to [pb_decode](reference.html#pb-decode) to call your
function over and over until all values have been read.

This callback reads multiple integers and prints them:

    bool read_ints(pb_istream_t *stream, const pb_field_iter_t *field, void **arg)
    {
        while (stream->bytes_left)
        {
            uint64_t value;
            if (!pb_decode_varint(stream, &value))
                return false;
            printf("%lld\n", value);
        }
        return true;
    }

### Function name bound callbacks

    bool MyMessage_callback(pb_istream_t *istream, pb_ostream_t *ostream, const pb_field_iter_t *field);

|            |                                                                    |
| ---------- | ------------------------------------------------------------------ |
| `istream`  | Input stream to read from, or NULL if called in encoding context.  |
| `ostream`  | Output stream to write to, or NULL if called in decoding context.  |
| `field`    | Iterator for the field currently being encoded or decoded.         |

Storing function pointer in `pb_callback_t` fields inside
the message requires extra storage space and is often cumbersome. As an
alternative, the generator options `callback_function` and
`callback_datatype` can be used to bind a callback function
based on its name.

Typically this feature is used by setting
`callback_datatype` to e.g. `void\*` or other
data type used for callback state. Then the generator will automatically
set `callback_function` to
`MessageName_callback` and produce a prototype for it in
generated `.pb.h`. By implementing this function in your own
code, you will receive callbacks for fields without having to separately
set function pointers.

If you want to use function name bound callbacks for some fields and
`pb_callback_t` for other fields, you can call
`pb_default_field_callback` from the message-level
callback. It will then read a function pointer from
`pb_callback_t` and call it.

## Message descriptor

For using the `pb_encode()` and `pb_decode()` functions, you need a
description of all the fields contained in a message. This description
is usually autogenerated from .proto file.

For example this submessage in the Person.proto file:

~~~~ protobuf
message Person {
    message PhoneNumber {
        required string number = 1 [(nanopb).max_size = 40];
        optional PhoneType type = 2 [default = HOME];
    }
}
~~~~

This in turn generates a macro list in the `.pb.h` file:

    #define Person_PhoneNumber_FIELDLIST(X, a) \
    X(a, STATIC,   REQUIRED, STRING,   number,            1) \
    X(a, STATIC,   OPTIONAL, UENUM,    type,              2)

Inside the `.pb.c` file there is a macro call to
`PB_BIND`:

    PB_BIND(Person_PhoneNumber, Person_PhoneNumber, AUTO)

These macros will in combination generate `pb_msgdesc_t`
structure and associated lists:

    const uint32_t Person_PhoneNumber_field_info[] = { ... };
    const pb_msgdesc_t * const Person_PhoneNumber_submsg_info[] = { ... };
    const pb_msgdesc_t Person_PhoneNumber_msg = {
      2,
      Person_PhoneNumber_field_info,
      Person_PhoneNumber_submsg_info,
      Person_PhoneNumber_DEFAULT,
      NULL,
    };

The encoding and decoding functions take a pointer to this structure and
use it to process each field in the message.

## Oneof

Protocol Buffers supports
[oneof](https://developers.google.com/protocol-buffers/docs/reference/proto2-spec#oneof_and_oneof_field)
sections, where only one of the fields contained within can be present. Here is an example of `oneof` usage:

~~~~ protobuf
message MsgType1 {
    required int32 value = 1;
}

message MsgType2 {
    required bool value = 1;
}

message MsgType3 {
    required int32 value1 = 1;
    required int32 value2 = 2;
}

message MyMessage {
    required uint32 uid = 1;
    required uint32 pid = 2;
    required uint32 utime = 3;

    oneof payload {
        MsgType1 msg1 = 4;
        MsgType2 msg2 = 5;
        MsgType3 msg3 = 6;
    }
}
~~~~

Nanopb will generate `payload` as a C union and add an additional field
`which_payload`:

    typedef struct _MyMessage {
      uint32_t uid;
      uint32_t pid;
      uint32_t utime;
      pb_size_t which_payload;
      union {
          MsgType1 msg1;
          MsgType2 msg2;
          MsgType3 msg3;
      } payload;
    } MyMessage;

`which_payload` indicates which of the `oneof` fields is actually set.
The user is expected to set the field manually using the correct field
tag:

    MyMessage msg = MyMessage_init_zero;
    msg.payload.msg2.value = true;
    msg.which_payload = MyMessage_msg2_tag;

Notice that neither `which_payload` field nor the unused fields in
`payload` will consume any space in the resulting encoded message.

When a field inside `oneof` contains `pb_callback_t`
fields, the callback values cannot be set before decoding. This is
because the different fields share the same storage space in C
`union`. Instead either function name bound callbacks or a
separate message level callback can be used. See
[tests/oneof_callback](https://github.com/nanopb/nanopb/tree/master/tests/oneof_callback)
for an example on this.

## Extension fields

Protocol Buffers supports a concept of [extension
fields](https://developers.google.com/protocol-buffers/docs/proto#extensions),
which are additional fields to a message, but defined outside the actual
message. The definition can even be in a completely separate .proto
file.

The base message is declared as extensible by keyword `extensions` in
the .proto file:

~~~~ protobuf
message MyMessage {
    .. fields ..
    extensions 100 to 199;
}
~~~~

For each extensible message, `nanopb_generator.py` declares an
additional callback field called `extensions`. The field and associated
datatype `pb_extension_t` forms a linked list of handlers. When an
unknown field is encountered, the decoder calls each handler in turn
until either one of them handles the field, or the list is exhausted.

The actual extensions are declared using the `extend` keyword in the
.proto, and are in the global namespace:

~~~~ protobuf
extend MyMessage {
    optional int32 myextension = 100;
}
~~~~

For each extension, `nanopb_generator.py` creates a constant of type
`pb_extension_type_t`. To link together the base message and the
extension, you have to:

1.  Allocate storage for your field, matching the datatype in the
    .proto. For example, for a `int32` field, you need a `int32_t`
    variable to store the value.
2.  Create a `pb_extension_t` constant, with pointers to your variable
    and to the generated `pb_extension_type_t`.
3.  Set the `message.extensions` pointer to point to the
    `pb_extension_t`.

An example of this is available in `tests/test_encode_extensions.c`
and `tests/test_decode_extensions.c`.

## Default values

Protobuf has two syntax variants, proto2 and proto3. Of these proto2 has
user definable default values that can be given in .proto file:

~~~~ protobuf
message MyMessage {
    optional bytes foo = 1 [default = "ABC\x01\x02\x03"];
    optional string bar = 2 [default = "åäö"];
}
~~~~

Nanopb will generate both static and runtime initialization for the
default values. In `myproto.pb.h` there will be a
`#define MyMessage_init_default {...}` that can be used to initialize
whole message into default values:

    MyMessage msg = MyMessage_init_default;

In addition to this, `pb_decode()` will initialize message
fields to defaults at runtime. If this is not desired,
`pb_decode_ex()` can be used instead.

## Message framing

Protocol Buffers does not specify a method of framing the messages for
transmission. This is something that must be provided by the library
user, as there is no one-size-fits-all solution. Typical needs for a
framing format are to:

1.  Encode the message length.
2.  Encode the message type.
3.  Perform any synchronization and error checking that may be needed
    depending on application.

For example UDP packets already fulfill all the requirements, and TCP
streams typically only need a way to identify the message length and
type. Lower level interfaces such as serial ports may need a more robust
frame format, such as HDLC (high-level data link control).

Nanopb provides a few helpers to facilitate implementing framing
formats:

1.  Functions `pb_encode_ex` and `pb_decode_ex` prefix the message
    data with a varint-encoded length.
2.  Union messages and oneofs are supported in order to implement
    top-level container messages.
3.  Message IDs can be specified using the `(nanopb_msgopt).msgid`
    option and can then be accessed from the header.

## Return values and error handling

Most functions in nanopb return bool: `true` means success, `false`
means failure. There is also support for error messages for
debugging purposes: the error messages go in `stream->errmsg`.

The error messages help in guessing what is the underlying cause of the
error. The most common error conditions are:

1)  Invalid protocol buffers binary message.
2)  Mismatch between binary message and .proto message type.
3)  Unterminated message (incorrect message length).
4) Exceeding the max_size or bytes_left of a stream.
5) Exceeding the max_size/max_count of a string or array field
6) IO errors in your own stream callbacks.
7) Errors that happen in your callback functions.
8) Running out of memory, i.e. stack overflow.
9) Invalid field descriptors (would usually mean a bug in the generator).

## Static assertions

Nanopb code uses static assertions to check size of structures at the compile
time. The `PB_STATIC_ASSERT` macro is defined in `pb.h`. If ISO C11 standard
is available, the C standard `_Static_assert` keyword is used, otherwise a
negative sized array definition trick is used.

Common reasons for static assertion errors are:

1. `FIELDINFO_DOES_NOT_FIT_width2` with `width1` or `width2`:
    Message that is larger than 256 bytes, but nanopb generator does not detect
    it for some reason. Often resolved by giving all `.proto` files as argument
    to `nanopb_generator.py` at the same time, to ensure submessage definitions
    are found. Alternatively `(nanopb).descriptorsize = DS_4` option can be
    given manually.

2. `FIELDINFO_DOES_NOT_FIT_width4` with `width4`:
    Message that is larger than 64 kilobytes. There will be a better error
    message for this in a future nanopb version, but currently it asserts here.
    The compile time option `PB_FIELD_32BIT` should be specified either on
    C compiler command line or by editing `pb.h`. This will increase the sizes
    of integer types used internally in nanopb code.

3. `DOUBLE_MUST_BE_8_BYTES`:
    Some platforms, most notably AVR, do not support the 64-bit `double` type,
    only 32-bit `float`. The compile time option `PB_CONVERT_DOUBLE_FLOAT` can
    be defined to convert between the types automatically. The conversion
    results in small rounding errors and takes unnecessary space in transmission,
    so changing the `.proto` to use `float` type is often better.

4.  `INT64_T_WRONG_SIZE`:
    The `stdint.h` system header is incorrect for the C compiler being used.
    This can result from erroneous compiler include path.
    If the compiler actually does not support 64-bit types, the compile time
    option `PB_WITHOUT_64BIT` can be used.

5.  `variably modified array size`:
    The compiler used has problems resolving the array-based static assert at
    compile time. Try setting the compiler to C11 standard mode if possible.
    If static assertions cannot be made to work on the compiler used, the
    compile-time option `PB_NO_STATIC_ASSERT` can be specified to turn them off.


# Nanopb: Overview

Nanopb is an ANSI-C library for encoding and decoding messages in
Google's [Protocol Buffers](https://developers.google.com/protocol-buffers/docs/reference/overview)
format with minimal requirements for RAM and code space. It is primarily
suitable for 32-bit microcontrollers.

Documentation version
---------------------

This documentation applies for nanopb 0.4.0 and later versions. For
documentation of older releases,
[see here](https://github.com/nanopb/nanopb/blob/maintenance_0.3/docs/index.rst).

Overall structure
-----------------

For the runtime program, you always need `pb.h` for type declarations
and `pb_common.h/c` for base functions. Depending on whether you want
to encode, decode, or both, you also need `pb_encode.h/c` or
`pb_decode.h/c`.

The high-level encoding and decoding functions take a pointer to
`pb_msgdesc_t` structure, which describes the fields of a message
structure. Usually you want these autogenerated from a `.proto` file.
The tool script `nanopb_generator.py` accomplishes this.

![Image: Nanopb generator flow](generator_flow.svg)

So a typical project might include these files:

1. Nanopb runtime library:
    -   pb.h
    -   pb_common.h and pb_common.c (always needed)
    -   pb_decode.h and pb_decode.c (needed for decoding messages)
    -   pb_encode.h and pb_encode.c (needed for encoding messages)

2. Protocol description (you can have many):
    -   person.proto (just an example)
    -   person.pb.c (autogenerated, contains message descriptors)
    -   person.pb.h (autogenerated, contains type declarations and macros)

Features and limitations
------------------------

**Features**

1)  Pure C runtime
2)  Small code size (5--10 kB depending on processor and compilation options, plus any message definitions)
3)  Small ram usage (typically \~300 bytes stack, plus any message structs)
4)  Allows specifying maximum size for strings and arrays, so that they can be allocated statically.
5)  No malloc needed: everything can be allocated statically or on the stack. Optional malloc support available.
6)  You can use either encoder or decoder alone to cut the code size in half.
7)  Support for most protobuf features, including: all data types,
    nested submessages, default values, repeated and optional fields,
    oneofs, packed arrays, extension fields.
8)  Callback mechanism for handling messages larger than can fit in available RAM.
9)  Extensive set of tests.

**Limitations**

1)  Some speed has been sacrificed for code size.
2)  Encoding is focused on writing to streams. For memory buffers only it could be made more efficient.
3)  The deprecated Protocol Buffers feature called "groups" is not supported.
4)  Fields in the generated structs are ordered by the tag number, instead of the natural ordering in .proto file.
5)  Unknown fields are not preserved when decoding and re-encoding a message.
6)  Reflection (runtime introspection) is not supported. E.g. you can't request a field by giving its name in a string.
7)  Numeric arrays are always encoded as packed, even if not marked as packed in .proto.
8)  Cyclic references between messages are supported only in callback and malloc mode.
9)  Nanopb doesn't have a stable ABI (application binary interface)
    between versions, so using it as a shared library (.so / .dll)
    requires extra care.

Getting started
---------------

For starters, consider this simple message:

~~~~ protobuf
message Example {
    required int32 value = 1;
}
~~~~

Save this in `message.proto` and compile it:

    user@host:~$ python nanopb/generator/nanopb_generator.py message.proto

You should now have in `message.pb.h`:

    typedef struct {
       int32_t value;
    } Example;

    extern const pb_msgdesc_t Example_msg;
    #define Example_fields &Example_msg

Then you have to include the nanopb headers and the generated header:

    #include <pb_encode.h>
    #include "message.pb.h"

Now in your main program do this to encode a message:

    Example mymessage = {42};
    uint8_t buffer[10];
    pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
    pb_encode(&stream, Example_fields, &mymessage);

After that, buffer will contain the encoded message. The number of bytes
in the message is stored in `stream.bytes_written`. You can feed the
message to `protoc --decode=Example message.proto` to verify its
validity.

For a complete example of the simple case, see [examples/simple/simple.c](https://github.com/nanopb/nanopb/blob/master/examples/simple/simple.c).
For a more complex example with network interface, see the [examples/network_server](https://github.com/nanopb/nanopb/tree/master/examples/network_server) subdirectory.

Compiler requirements
---------------------

Nanopb should compile with most ansi-C compatible compilers. It however
requires a few header files to be available:

1)  `string.h`, with these functions: `strlen`, `memcpy`, `memset`
2)  `stdint.h`, for definitions of `int32_t` etc.
3)  `stddef.h`, for definition of `size_t`
4)  `stdbool.h`, for definition of `bool`
5)  `limits.h`, for definition of `CHAR_BIT`

If these header files do not come with your compiler, you can use the
file `extra/pb_syshdr.h` instead. It contains an example of how to
provide the dependencies. You may have to edit it a bit to suit your
custom platform.

To use the pb_syshdr.h, define `PB_SYSTEM_HEADER` as
`"pb_syshdr.h"` (including the quotes). Similarly, you can provide a
custom include file, which should provide all the dependencies listed
above.

Running the test cases
----------------------

Extensive unittests and test cases are included under the `tests`
folder.

To build the tests, you will need the [scons](http://www.scons.org/)
build system. The tests should be runnable on most platforms. Windows
and Linux builds are regularly tested. The tests also support embedded
targets: STM32 (ARM Cortex-M) and AVR builds are regularly tested.

In addition to the build system, you will also need a working Google
Protocol Buffers `protoc` compiler, and the Python bindings for Protocol
Buffers.

Easiest way to install dependencies is to use the Python package manager
[pip](https://pypi.org/project/pip/), which works on all platforms supported by Python:

    pip3 install scons protobuf grpcio-tools

This is the latest version of the internal repository from Pebble Technology providing the software to run on Pebble watches. Proprietary source code has been removed from this repository and it will not compile as-is. This is for information only.


# Nanopb: Migration from older versions

This document details all the breaking changes that have been made to
nanopb since its initial release. For each change, the rationale and
required modifications of user applications are explained. Also any
error indications are included, in order to make it easier to find this
document.

Nanopb-0.4.6 (2022-05-30)
-------------------------

### NANOPB_VERSION define is now a string

**Changes:** To ease `NANOPB_VERSION` macro usage, the value is directly a string.

**Required actions:** Most nanopb users probably never used that macro. If so,
you certainly use the `#` preprocessor to convert it as string. You, now,
only have to call it directly, like this for example:
`strcpy(myvar, NANOPB_VERSION);`

### FindNanopb.cmake now requires protoc 3.6.0 or newer by default

**Changes:** The default options passing method now uses `--plugin-opt` which
is supported by protoc 3.6.0 and newer (released in 2018).

**Required actions:** Update `protoc` if needed, or alternatively install
`grpcio-tools` package from `pip`. If neither is possible, the
`NANOPB_PROTOC_OLDER_THAN_3_6_0` cmake option can be used to restore the old
style option passing. Note that it has problems with special characters such
as `:`.

**Error indications:** "`protoc: Unknown flag: --nanopb_opt`"

### pb.h uses C11 _Static_assert keyword by default

**Rationale:** The nanopb generated headers use static assertions to catch
errors at compile time. There are several mechanisms to implement this.
The most widely supported is C11 `_Static_assert` keyword.
Previously the code used negative size array definition trick, which is
supported already in C99 but does not work with every compiler and can
produce confusing error messages.

**Changes:** Now `_Static_assert` is used by default.

**Required actions:** If the keyword is not recognized, set the compiler to
C11 standard mode if available. If it is not available, define either `PB_C99_STATIC_ASSERT`
or `PB_NO_STATIC_ASSERT` in `pb.h` or on compiler command line.

**Error indications:** `Undefined identifier _Static_assert`

Nanopb-0.4.4 (2020-11-25)
-------------------------

### Remove outdated generator/nanopb/options.proto

**Changes:** Back in 2018, it was considered in pull request #241 to
move nanopb generator options to a separate namespace. For this reason,
a transitional file was added. It was later abandoned and is now removed
to avoid confusion.

**Required actions:** Most nanopb users probably never used that transitional
file at all. If your `.proto` files import it, change to using `generator/proto/nanopb.proto`.

**Error indications:** Errors about missing file `options.proto` when running
the generator.

Nanopb-0.4.3 (2020-09-21)
-------------------------

### pb_msgdesc_t struct has new fields

**Changes:** New fields `required_field_count` and
`largest_tag` were added to `pb_msgdesc_t`
and existing fields were reordered.

**Required actions:** All `.pb.c` files must be recompiled.
Regeneration is not needed.

**Error indications:** Messages may fail to encode or decode, or the
code can crash inside `load_descriptor_values()` in
`pb_common.c`.

Nanopb-0.4.2 (2020-06-23)
-------------------------

### Generator now uses Python 3 by default

**Rationale:** Previously `nanopb-generator.py` had hashbang
of `#!/usr/bin/env python`, which would execute with Python
2 on most systems. Python 2 is now deprecated and many libraries are
dropping support for it, which makes installing dependencies difficult.
While `nanopb_generator.py` has worked with Python 3 for
years now, and overriding the python version was possible with
virtualenv, that was an extra complication.

**Changes:** Hashbang now uses `#!/usr/bin/env python3`.
New file `nanopb_generator.py2` can be used to run with
Python 2, if necessary.

**Required actions:** If possible, just verify Python 3 is installed and
necessary dependencies are installed for it. For example `pip3 install protobuf grpcio-tools`
should take care of it. If this is not possible, call `nanopb_generator.py2` from your build
scripts instead.

**Error indications:** `python3: command not found` if
Python 3 is not installed.
`Could not import the Google protobuf Python libraries` if dependencies are only installed for Python 2.

Nanopb-0.4.0 (2019-12-20)
-------------------------

### New field descriptor format

**Rationale:** Previously information about struct fields was stored as
an array of `pb_field_t` structures. This was a
straightforward method, but required allocating space for e.g.
submessage type and array size for all fields, even though most fields
are not submessages nor arrays.

**Changes:** Now field information is encoded more efficiently in
`uint32_t` array in a variable-length format. Old
`pb_field_t` structure has been removed and it is now a
typedef for `pb_field_iter_t`. This retains compatibility
with most old callback definitions. The field definitions in
`.pb.h` files are now of type `pb_msgdesc_t`.

**Required actions:** If your own code accesses the low-level field
information in `pb_field_t`, it must be modified to do so
only through the functions declared in `pb_common.h`.

**Error indications:** `incompatible pointer type` errors
relating to `pb_field_t`

### Changes to generator default options

**Rationale:** Previously nanopb_generator added a timestamp header to
generated files and used only basename of files in
`#include` directives. This is different than what the
`protoc` C++ backend does.

**Changes:** Now default options are `--no-timestamp` and
`--no-strip-path`.

**Required actions:** If old behaviour is desired, add
`--timestamp` and `--strip-path` options to
`nanopb_generator.py` or on `protoc` command
line as `--nanopb_out=--timestamp,--strip-path:outdir`.

**Error indications:** Compiler error: cannot find include file
`mymessage.pb.h` when compiling
`mymessage.pb.c`.

### Removal of bundled plugin.proto

**Rationale:** Google's Python protobuf library, which is used in
nanopb generator, has included `plugin_pb2` with it since
version 3.1.0. It is not necessary to bundle it with nanopb anymore.

**Required actions:** Update `python-protobuf` to version
3.1.0 or newer.

**Error indications:** `ImportError: No module named compiler.plugin_pb2`

### .options file is now always case-sensitive

**Rationale:** Previously field names in `.options` file
were case-sensitive on Linux and case-insensitive on Windows. This was
by accident. Because `.proto` files are case-sensitive,
`.options` files should be too.

**Changes:** Now field names in `.options` are always
case-sensitive, and matched by `fnmatchcase()` instead of
`fnmatch()`.

**Required actions:** If field names in `.options` are not
capitalized the same as in `.proto`, they must be updated.

### `CHAR_BIT` define is now needed

**Rationale:** To check whether the platform has 8-bit or larger chars,
the C standard `CHAR_BIT` macro is needed.

**Changes:** `pb.h` now includes `limits.h` for this macro.

**Required actions:** If your platform doesn't have `limits.h`
available, you can define the macro in `pb_syshdr.h`. There is an
example in `extra` directory.

**Error indications:** `"Cannot find include file <limits.h>."` or
`"Undefined identifier: CHAR_BIT."`

### Strings must now always be null-terminated

**Rationale:** Previously `pb_encode()` would accept non-terminated
strings and assume that they are the full length of the defined array.
However, `pb_decode()` would reject such messages because null
terminator wouldn't fit in the array.

**Changes:** `pb_encode()` will now return an error if null terminator
is missing. Maximum encoded message size calculation is changed
accordingly so that at most `max_size-1` strings are assumed. New field
option `max_length` can be used to define the maximum string length,
instead of the array size.

**Required actions:** If your strings were previously filling the whole
allocated array, increase the size of the field by 1.

**Error indications:** `pb_encode()` returns error `unterminated string`.

### Removal of per-field default value constants

**Rationale:** Previously nanopb declared a
`fieldname_default` constant variable for each field with a
default value, and used these internally to initialize messages. This
however used unnecessarily large amount of storage for the values. The
variables were mostly for internal usage, but were available in the
header file.

**Changes:** Default values are now stored as an encoded protobuf
message.

**Required actions:** If your code previously used default constants, it
will have to be adapted to take the default value in some other way,
such as by defining
`static const MyMessage msg_default = MyMessage_init_default;` and accessing
`msg_default.fieldname`.

**Error indications:** Compiler error about `fieldname_default` being undeclared.

### Zero tag in message now raises error by default

**Rationale:** Previously nanopb has allowed messages to be terminated
by a null byte, which is read as zero tag value. Most other protobuf
implementations don't support this, so it is not very useful feature.
It has also been noted that this can complicate debugging issues with
corrupted messages.

**Changes:** `pb_decode()` now gives error when it
encounters zero tag value. A new function `pb_decode_ex()`
supports flag `PB_DECODE_NULLTERMINATED` that supports
decoding null terminated messages.

**Required actions:** If application uses null termination for messages,
switch it to use `pb_decode_ex()` and
`pb_encode_ex()`. If compatibility with 0.3.9.x is needed,
there are also `pb_decode_nullterminated()` and
`pb_encode_nullterminated()` macros, which work both in
0.4.0 and 0.3.9.

**Error indications:** Error message from `pb_decode()`: `zero_tag`.

### Submessages now have has_field in proto3 mode

**Rationale:** Previously nanopb considered proto3 submessages as
present only when their contents was non-zero. Most other protobuf
libraries allow explicit null state for submessages.

**Changes:** Submessages now have separate `has_field` in
proto3 mode also.

**Required actions:** When using submessages in proto3 mode, user code
must now set `mymsg.has_submsg = true` for each submessage
that is present. Alternatively, the field option
`proto3_singular_msgs` can be used to restore the old
behavior.

**Error indications:** Submessages do not get encoded.

### PB_OLD_CALLBACK_STYLE option has been removed

**Rationale:** Back in 2013, function signature for callbacks was
changed. The `PB_OLD_CALLBACK_STYLE` option allowed
compatibility with old code, but complicated code and testing because of
the different options.

**Changes:** `PB_OLD_CALLBACK_STYLE` option no-longer has
any effect.

**Required actions:** If `PB_OLD_CALLBACK_STYLE` option
was in use previously, function signatures must be updated to use double
pointers (`void**` and `void * const *`).

**Error indications:** Assignment from incompatible pointer type.

### protoc insertion points are no longer included by default

**Rationale:** Protoc allows including comments in form
`@@protoc_insertion_point` to identify locations for
other plugins to insert their own extra content. Previously these were
included by default, but they clutter the generated files and are rarely
used.

**Changes:** Insertion points are now included only when
`--protoc-insertion-points` option is passed to the
generator.

Nanopb-0.3.9.4, 0.4.0 (2019-10-13)
----------------------------------

### Fix generation of min/max defines for enum types

**Rationale:** Nanopb generator makes \#defines for enum minimum and
maximum value. Previously these defines incorrectly had the first and
last enum value, instead of the actual minimum and maximum. (issue
#405)

**Changes:** Minimum define now always has the smallest value, and
maximum define always has the largest value.

**Required actions:** If these defines are used and enum values in
.proto file are not defined in ascending order, user code behaviour may
change. Check that user code doesn\'t expect the old, incorrect
first/last behaviour.

### Fix undefined behavior related to bool fields

**Rationale:** In C99, `bool` variables are not allowed to
have other values than `true` and `false`.
Compilers use this fact in optimization, and constructs like
`int foo = msg.has_field ? 100 : 0;` will give unexpected results
otherwise. Previously nanopb didn\'t enforce that decoded bool fields
had valid values.

**Changes:** Bool fields are now handled separately as
`PB_LTYPE_BOOL`. The `LTYPE` descriptor
numbers for other field types were renumbered.

**Required actions:** Source code files must be recompiled, but
regenerating `.pb.h`/`.pb.c` files from
`.proto` is not required. If user code directly uses the
nanopb internal field representation (search for
`PB_LTYPE_VARINT` in source), it may need updating.

Nanopb-0.3.9.1, 0.4.0 (2018-04-14)
----------------------------------

### Fix handling of string and bytes default values

**Rationale:** Previously nanopb didn't properly decode special
character escapes like `\200` emitted by protoc. This caused these
escapes to end up verbatim in the default values in .pb.c file.

**Changes:** Escapes are now decoded, and e.g. `\200` or `\x80`
results in {0x80} for bytes field and `"\x80"` for string field.

**Required actions:** If code has previously relied on `\` in default
value being passed through verbatim, it must now be changed to `\\`.

Nanopb-0.3.8 (2017-03-05)
-------------------------

### Fully drain substreams before closing

**Rationale:** If the substream functions were called directly and the
caller did not completely empty the substring before closing it, the
parent stream would be put into an incorrect state.

**Changes:** `pb_close_string_substream` can now error and returns a
boolean.

**Required actions:** Add error checking onto any call to
`pb_close_string_substream`.

### Change oneof format in .pb.c files

**Rationale:** Previously two oneofs in a single message would be
erroneously handled as part of the same union.

**Changes:** Oneofs fields now use special `PB_DATAOFFSET_UNION`
offset type in generated .pb.c files to distinguish whether they are the
first or following field inside an union.

**Required actions:** Regenerate `.pb.c/.pb.h` files with new nanopb
version if oneofs are used.

Nanopb-0.3.5 (2016-02-13)
-------------------------

### Add support for platforms without uint8_t

**Rationale:** Some platforms cannot access 8-bit sized values directly,
and do not define `uint8_t`. Nanopb previously didn\'t support these
platforms.

**Changes:** References to `uint8_t` were replaced with several
alternatives, one of them being a new `pb_byte_t` typedef. This in
turn uses `uint_least8_t` which means the smallest available type.

**Required actions:** If your platform does not have a
standards-compliant `stdint.h`, it may lack the definition for
`[u]int_least8_t`. This must be added manually, example can be found
in `extra/pb_syshdr.h`.

**Error indications:** Compiler error: `"unknown type name 'uint_least8_t'"`.

Nanopb-0.3.2 (2015-01-24)
-------------------------

### Add support for OneOfs

**Rationale:** Previously nanopb did not support the `oneof` construct
in `.proto` files. Those fields were generated as regular `optional`
fields.

**Changes:** OneOfs are now generated as C unions. Callback fields are
not supported inside oneof and generator gives an error.

**Required actions:** The generator option `no_unions` can be used to
restore old behaviour and to allow callbacks to be used. To use unions,
one change is needed: use `which_xxxx` field to detect which field is
present, instead of `has_xxxx`. Compare the value against
`MyStruct_myfield_tag`.

**Error indications:** Generator error: `"Callback fields inside of
oneof are not supported"`. Compiler error: `"Message"` has no member
named `"has_xxxx"`.

Nanopb-0.3.0 (2014-08-26)
-------------------------

### Separate field iterator logic to pb_common.c

**Rationale:** Originally, the field iteration logic was simple enough
to be duplicated in `pb_decode.c` and `pb_encode.c`. New field types
have made the logic more complex, which required the creation of a new
file to contain the common functionality.

**Changes:** There is a new file, `pb_common.c`, which must be included
in builds.

**Required actions:** Add `pb_common.c` to build rules. This file is
always required. Either `pb_decode.c` or `pb_encode.c` can still be
left out if some functionality is not needed.

**Error indications:** Linker error: undefined reference to
`pb_field_iter_begin`, `pb_field_iter_next` or similar.

### Change data type of field counts to pb_size_t

**Rationale:** Often nanopb is used with small arrays, such as 255 items
or less. Using a full `size_t` field to store the array count wastes
memory if there are many arrays. There already exists parameters
`PB_FIELD_16BIT` and `PB_FIELD_32BIT` which tell nanopb what is the
maximum size of arrays in use.

**Changes:** Generator will now use `pb_size_t` for the array
`_count` fields. The size of the type will be controlled by the
`PB_FIELD_16BIT` and `PB_FIELD_32BIT` compilation time options.

**Required actions:** Regenerate all `.pb.h` files. In some cases casts
to the `pb_size_t` type may need to be added in the user code when
accessing the `_count` fields.

**Error indications:** Incorrect data at runtime, crashes. But note that
other changes in the same version already require regenerating the files
and have better indications of errors, so this is only an issue for
development versions.

### Renamed some macros and identifiers

**Rationale:** Some names in nanopb core were badly chosen and
conflicted with ISO C99 reserved names or lacked a prefix. While they
haven\'t caused trouble so far, it is reasonable to switch to
non-conflicting names as these are rarely used from user code.

**Changes:** The following identifier names have changed:

 -   Macros:
     -   STATIC_ASSERT(x) -> PB_STATIC_ASSERT(x)
     -   UNUSED(x) -> PB_UNUSED(x)
 -   Include guards:
     -   PB_filename -> PB_filename_INCLUDED
 -   Structure forward declaration tags:
     -   _pb_field_t -> pb_field_s
     -   _pb_bytes_array_t -> pb_bytes_array_s
     -   _pb_callback_t -> pb_callback_s
     -   _pb_extension_type_t -> pb_extension_type_s
     -   _pb_extension_t -> pb_extension_s
     -   _pb_istream_t -> pb_istream_s
     -   _pb_ostream_t -> pb_ostream_s

**Required actions:** Regenerate all `.pb.c` files. If you use any of
the above identifiers in your application code, perform search-replace
to the new name.

**Error indications:** Compiler errors on lines with the macro/type
names.

Nanopb-0.2.9 (2014-08-09)
-------------------------

### Change semantics of generator -e option

**Rationale:** Some compilers do not accept filenames with two dots
(like in default extension .pb.c). The `-e` option to the generator
allowed changing the extension, but not skipping the extra dot.

**Changes:** The `-e` option in generator will no longer add the
prepending dot. The default value has been adjusted accordingly to
`.pb.c` to keep the default behaviour the same as before.

**Required actions:** Only if using the generator -e option. Add dot
before the parameter value on the command line.

**Error indications:** File not found when trying to compile generated
files.

Nanopb-0.2.7 (2014-04-07)
-------------------------

### Changed pointer-type bytes field datatype

**Rationale:** In the initial pointer encoding support since
nanopb-0.2.5, the bytes type used a separate `pb_bytes_ptr_t` type to
represent `bytes` fields. This made it easy to encode data from a
separate, user-allocated buffer. However, it made the internal logic
more complex and was inconsistent with the other types.

**Changes:** Dynamically allocated bytes fields now have the
`pb_bytes_array_t` type, just like statically allocated ones.

**Required actions:** Only if using pointer-type fields with the bytes
datatype. Change any access to `msg->field.size` to
`msg->field->size`. Change any allocation to reserve space of amount
`PB_BYTES_ARRAY_T_ALLOCSIZE(n)`. If the data pointer was begin
assigned from external source, implement the field using a callback
function instead.

**Error indications:** Compiler error: unknown type name
`pb_bytes_ptr_t`.

Nanopb-0.2.4 (2013-11-07)
-------------------------

### Remove the NANOPB_INTERNALS compilation option

**Rationale:** Having the option in the headers required the functions
to be non-static, even if the option is not used. This caused errors on
some static analysis tools.

**Changes:** The `\#ifdef` and associated functions were removed from
the header.

**Required actions:** Only if the `NANOPB_INTERNALS` option was
previously used. Actions are as listed under nanopb-0.1.3 and
nanopb-0.1.6.

**Error indications:** Compiler warning: implicit declaration of
function `pb_dec_string`, `pb_enc_string`, or similar.

Nanopb-0.2.1 (2013-04-14)
-------------------------

### Callback function signature

**Rationale:** Previously the auxiliary data to field callbacks was
passed as `void*`. This allowed passing of any data, but made it
unnecessarily complex to return a pointer from callback.

**Changes:** The callback function parameter was changed to `void**`.

**Required actions:** You can continue using the old callback style by
defining `PB_OLD_CALLBACK_STYLE`. Recommended action is to:

-   Change the callback signatures to contain `void**` for decoders and `void * const *` for encoders.
-   Change the callback function body to use **arg` instead of `arg`.

**Error indications:** Compiler warning: assignment from incompatible
pointer type, when initializing `funcs.encode` or `funcs.decode`.

Nanopb-0.2.0 (2013-03-02)
-------------------------

### Reformatted generated .pb.c file using macros

**Rationale:** Previously the generator made a list of C `pb_field_t`
initializers in the .pb.c file. This led to a need to regenerate all
.pb.c files after even small changes to the `pb_field_t` definition.

**Changes:** Macros were added to pb.h which allow for cleaner
definition of the .pb.c contents. By changing the macro definitions,
changes to the field structure are possible without breaking
compatibility with old .pb.c files.

**Required actions:** Regenerate all .pb.c files from the .proto
sources.

**Error indications:** Compiler warning: implicit declaration of
function `pb_delta_end`.

### Changed pb_type_t definitions

**Rationale:** The `pb_type_t` was previously an enumeration type.
This caused warnings on some compilers when using bitwise operations to
set flags inside the values.

**Changes:** The `pb_type_t` was changed to *typedef uint8_t*. The
values were changed to `#define`. Some value names were changed for
consistency.

**Required actions:** Only if you directly access the
`pb_field_t` contents in your own code, something which is
not usually done. Needed changes:

-   Change `PB_HTYPE_ARRAY` to `PB_HTYPE_REPEATED`.
-   Change `PB_HTYPE_CALLBACK` to `PB_ATYPE()` and `PB_ATYPE_CALLBACK`.

**Error indications:** Compiler error: `PB_HTYPE_ARRAY` or
`PB_HTYPE_CALLBACK` undeclared.

Nanopb-0.1.6 (2012-09-02)
-------------------------

### Refactored field decoder interface

**Rationale:** Similarly to field encoders in nanopb-0.1.3.

**Changes:** New functions with names `pb_decode_*` were added.

**Required actions:** By defining NANOPB_INTERNALS, you can still keep
using the old functions. Recommended action is to replace any calls with
the newer `pb_decode_*` equivalents.

**Error indications:** Compiler warning: implicit declaration of
function `pb_dec_string`, `pb_dec_varint`, `pb_dec_submessage` or
similar.

Nanopb-0.1.3 (2012-06-12)
-------------------------

### Refactored field encoder interface

**Rationale:** The old `pb_enc_*` functions were designed mostly for
the internal use by the core. Because they are internally accessed
through function pointers, their signatures had to be common. This led
to a confusing interface for external users.

**Changes:** New functions with names `pb_encode_*` were added. These
have easier to use interfaces. The old functions are now only thin
wrappers for the new interface.

**Required actions:** By defining NANOPB_INTERNALS, you can still keep
using the old functions. Recommended action is to replace any calls with
the newer `pb_encode_*` equivalents.

**Error indications:** Compiler warning: implicit declaration of
function `pb_enc_string`, *pb_enc_varint,`pb_enc_submessage\` or
similar.