Jackson is a mature and feature-rich open source project that we use, support, and contribute to here at Indeed. In my previous post, I introduced Jackson’s core competency as a JSON library for Java. I went on to describe the additional data formats, data types, and JVM languages Jackson supports. In this post, I will […]
Jackson is a mature and feature-rich open source project that we use, support, and contribute to here at Indeed. In my previous post, I introduced Jackson’s core competency as a JSON library for Java. I went on to describe the additional data formats, data types, and JVM languages Jackson supports. In this post, I will present challenges resulting from Jackson’s expansion, in my view, as Jackson’s creator and primary maintainer. I’ll also share our plans to address these challenges as a community.
“The other side of Cape Flattery” by Tatu Saloranta
Over its 13 years, the Jackson project added a lot of new functionality with the help of a large community of contributors. Users report issues and developers contribute fixes, new features, and even new modules. For example, the Java 8 date/time datatype module—which supports Java 8 date/time types, java.time.*, specified in JSR-310—was developed concurrently with the specification itself by a member of the community.
Because of its ever-expanding functionality and user base, Jackson is experiencing growing pains and could use help with:
The availability, accessibility, and freshness of documentation is always a challenge for widely used libraries and frameworks. Jackson is no exception. It may even suffer from more documentation debt than other libraries with a comparable user base.
The problem is not so much a lack of documentation per se. A lot of content exists already:
|Jackson project repos|
Jackson also has a Javadoc reference that documents most classes of interest. However, Jackson is a vast library. The Javadoc reference can seem overwhelming, especially to new users. And the Javadoc reference does not include usage examples.
Our first priority is creating How-To guides that walk library consumers through the most typical use cases, such as:
f.exJSON structure to match Java objects.
We also want to improve the Javadoc reference. Some classes and methods have no descriptions, while descriptions for others are incomplete. In addition to the auto-generated Javadoc reference, we publish manually created wiki pages that detail the most used features and objects. We’d like to find ways to auto-update these wiki pages.
To implement these modifications, we need the following documentation structure, tooling, and process changes:
Even though the project is 13 years old and widely used, the main project home page still has the stock GitHub project look. This repo contains a lot of helpful information about Jackson, but it doesn’t have a distinct style, brand, or logo.
For Hacktoberfest 2020, we created an issue for a Jackson logo design. Examples we like:
We appreciate the healthy stream of code contributions—mostly to fix reported issues, but for new features as well.
The most valuable type of contribution for the Jackson project has historically been user bug reports filed as GitHub issues. We get a dozen issues each week, leading to fixes and quality improvements. Without these reports, Jackson would not be half as good as it is. The new feature requests we receive regularly are another valuable source of improvements. New feature requests also help us prioritize new development work.
Better management of larger changes and features is an area for improvement. These efforts take a long time to execute and benefit from careful planning and collaboration. While it’s possible to handle bigger changes under a single issue, this can get unwieldy quickly.
The Jackson project could use help with setting the priority of new feature requests.
To address the need for more community feedback, I introduced the idea of “mini-specifications” called Jackson STrategic Enhancement Plan: JSTEP. Similar to KIPs in Kafka, JSTEPs are intended to foster collaboration. So far this has had limited success, partly due to the limitations of the tool used: GitHub wiki pages do not have the same feature set as GitHub issues or Google documents.
I also started keeping a personal but public todo or work-in-progress (WIP) list—Jackson WIP—as a GitHub wiki page. I wanted a low-maintenance mechanism for me to track and share my own near-term plans.
As mentioned above, bug reports have historically been the most useful type of feedback. But while useful, managing reported issues and new feature requests takes a lot of effort. This has become especially challenging now that the data formats, data types, JVM languages, and user base have grown.
Basic issue triage has become time consuming and includes the following steps:
The triage time takes away from the limited time we have to work on functionality and documentation. It can also slow down responses to issue reports, which in turn leads to a poor reporter experience.
We would like to find ways to train and include volunteer issue triagers. They can help with refining issues in a timely manner and improving user engagement by:
Starting with jackson-databind issues, we improved the workflow and refined GitHub issues by creating:
If these changes are successful, we’ll propagate them to other project repos. Similarly, we can make other incremental changes along these lines.
The unit test coverage in the Jackson project’s code is reasonable, and coverage for core components is quite good. There’s even a nascent effort to write more inter-component functional tests in the jackson-integration-tests repo. However, there is currently a sizable gap in testing the compatibility between the minor Jackson version that is under development and downstream dependencies, such as frameworks like Spring Boot.
One challenge is the difference between the development lifecycles of Jackson and the libraries and frameworks that use Jackson. Existing libraries and frameworks depend on the stable Jackson minor version and test everything only against that version. Version 2.11 is the current stable version. They report any issues they find. We may fix the issues in a patch release, like 2.11.1.
Simultaneously, the Jackson project is developing a new minor version, 2.12, which will be released and published as version 2.12.0. Working on versions 2.11.1 and 2.12 concurrently should not be an issue. We use the standard semantic versioning scheme for the Jackson public API, which should guard against creating new issues.
In practice, standard semantic versioning cannot protect against all actual issues. Semantic versioning is used for Jackson’s public API, but internal APIs across core components do not have the same level of backwards compatibility. While Jackson maintainers consider the compatibility between the published and documented API, users tend to rely on the API’s actual observed behavior rather than its intended behavior. Sometimes what maintainers consider bugs or unexpected behaviors, users consider the behavior an actual feature or working as expected.
That said, we can catch and address such issues during the development cycle of the new version if we test downstream systems using the under-development SNAPSHOT version of Jackson. At the time of writing this article, the current version of Spring Boot ships with a dependency on Jackson 2.11.2, the latest patch version. We can create an alternative set of Spring Boot unit and integration tests that instead rely on Jackson 2.12.0-SNAPSHOT. The snapshot version is published regularly, and the Jackson project provides automatic snapshot builds.
Such cross-project integration tests would benefit all involved projects. This prevents the release of some (or perhaps most) regression issues, reducing the need for patch versions. Over time it should also significantly improve compatibility across Jackson versions, closing the gap between expected and actual behavior of the API.
Jackson’s growing user base and features have presented us with documentation, branding, issue triaging, issue prioritization, and code testing challenges. I’ve shared some of the steps we’ve taken to address them. You can help us by implementing some of these solutions or suggesting alternates. Together we can lay a foundation for Jackson’s future growth. For contribution opportunities, visit Jackson Hacktoberfest 2020. To find out more, join us on Gitter chat or sign up for the jackson-dev mailing list.
Tatu Saloranta is a staff software engineer at Indeed, leading the team that is integrating the next-generation continuous deployment system. Outside of Indeed, he is best known for his open source activities. Under the handle @cowtowncoder, he has authored many popular open source Java libraries, such as Jackson, Woodstox, lzf compression codec, and Java ClassMate. For a complete list see FasterXML Org.