kotlinx.serialization: (de)serializing JSON’s nullable, optional properties

The APIs of this world can be a real mess. For any data that has newly arrived in your nicely structured, type-safe Kotlin environment, cleaning it up a bit is probably a good idea, if not absolutely necessary for your own sanity. Even for well-behaved APIs, the format can be unwieldy or a bit annoying to work with.

October 2020 edit: Updated to match kotlinx.serialization 1.0.0 interface.

The Problem

In Kotlin, one of those slightly awkward cases is handling nullable, optional properties, which the OpenAPI spec specifically allows for.

For a concrete example of when this could be useful, consider an API that supports partial updates of objects. Using this API, a JSON object would be used to communicate a patch for some long-lived object. Any included property specifies that the corresponding value of the object should be updated, while the values for any omitted properties should remain unchanged. If any of the object’s properties are nullable, then a value of null being sent for a property is fundamentally different than a property that is missing, so these cases must be distinguished.

Individually, nullable and optional properties can be handled trivially by Kotlin’s own serialization library, kotlinx.serialization. However, handling both cases simultaneously takes a little bit more work.

Nullable

The solution for a required, nullable property is as straightforward as it gets, thanks to Kotlin’s null safety:

Thus, { "value": null } maps to ResponseJson(null) , and { "value": "string" } maps to ResponseJson("string").

Optional

If a property is given a default value, then kotlinx.serialization automatically considers it to be optional. Thus, if our optional property isn’t nullable we can use null to represent the case where the key and value is missing from the JSON object entirely:

The empty JSON object { } then deserializes to ResponseJson(null) , and
{ "value": "string" } maps to ResponseJson("string"). If encodeDefaults is false, then ResponseJson(null) will also serialize to the empty JSON object { }.

Optional + Nullable

Take another look at those two response objects. Once created, they look exactly the same! If we tried to combine the approaches for an optional and nullable field, we wouldn’t be able to distinguish between the case of a null value or a missing property. To do so, we are going to have to get a bit more creative.

The Solution

There are two main goals for a solution:

• The underlying structure of the JSON must remain exactly the same during serialization or deserialization
• It should be generic, so that any type can be represented as an optional property

To support any nullable type as the value, we can’t use null as the token to mark a property that was missing. Instead, we can set up the following sealed class to encode all of the information we need:

To avoid changing the JSON structure, we also need a custom, generic serializer for OptionalProperty:

This KSerializer constructs and deconstructs the Present object, serializing and deserializing the underlying value object of type T using the generic valueSerializer passed in as a constructor parameter.

It might seem strange that this serializer blows up when confronted with a NotPresent value. However, we don’t have to serialize or deserialize a value that will never be represented in JSON! NotPresent will always be used as the default value to make a property optional, and we should never attempt to serialize NotPresent since it represents a value that isn’t present in the JSON.

To ensure that the last part is true for JSON objects created by us via encodeToString, the last step in our solution requires that encodeDefaults is set to false for our Json instance¹:

Putting it all together, we can now represent our nullable, optional property in a distinguishable way²:

The empty JSON object { } will serialize to ResponseJson(OptionalProperty.NotPresent), the object { "value": null } will map to ResponseJson(OptionalProperty.Present(null)), and the object
{ "value": "string" } will map to ResponseJson(OptionalProperty.Present("string")).

Even if you only have to deal with non-nullable, optional properties in the models you work with, you might still want to consider using OptionalProperty. As noted above, the easiest way to handle nullable properties and optional properties independently result in objects that look identical when created. Thus, the only way to keep track of whether a property is nullable or optional is by documentation. Enforcing this difference through the structure of the code itself is far less error-prone than having to rely on documentation.

Alex works at Livefront, where packing and unpacking data is fun!

¹: ^ If you were depending on the previous behavior of encodeDefaults with true elsewhere, you can preserve that behavior by adding another data model that exists in parallel with the serializable model. Converting between “network” models and “datalayer” models can be a super useful pattern, to do some common logic, lenient enum parsing, or creating sealed class hierarchies that aren’t directed represented as such in the JSON before doing business logic.

²: ^ If you are using the kotlin-kapt plugin (used by data binding, Dagger, etc.), then directly using @Serializable(with = OptionalPropertySerializer::class) can currently lead to a compiler error. As a workaround, create a subclass of OptionalPropertySerializer with the generic type explicitly defined: