To boost the performance of a mission-critical instance of Cassandra, Instagram engineers replaced the storage engine of this Java-based distributed open source database with a faster C++-based one from another database, RocksDB.
With the resulting high-performance database, dubbed “Rocksandra,” engineers from the company are hoping the move will help propel the Apache Cassandra development team to adopt this plug-in storage engine architecture that they developed, which could pave the way for more flexible use of Cassandra. The company has released as open source the Rocksandra code base and benchmark framework as open source.
Rocksandra is a fork, said Francois Deliege, Instagram engineering manager, but not one designed to compete with Cassandra itself. “We are not trying to sell the community on the particular RocksDB implementation, but it’s just a showcase of how it could help. If we had any other database or storage engine supporting this, it would be a great win for the community,” he said.
Instagram uses Cassandra as a general key-value storage service, to support the user photo feed, direct messages, as well as for fraud detection. While exceptionally fast at writing data to disk, Instagram was finding that read times were starting to increase, resulting in slow performance for users. The performance lag came from Java’s garbage collection routine, which periodically stops all client requests to clean up unused memory. “Cassandra is really good at writing queries, but on the read side, it’s a little bit of a pain,” Deliege said.
“Cassandra is really good at communication across all the different nodes, but Java is not the most efficient language to interact with the kernel of the OS,” explained Dikang Gu, a Facebook staff software engineer who is working on the project (Facebook owns Instagram).
A storage engine can be seen as the heart of the database. It is responsible for formatting the data so it can be placed on disk, as well as reading the data from disk later. The leaner RocksDB storage engine, written in C++, did not have this issue. This storage engine, developed at Facebook, was optimized for performance, especially on fast storage like SSD. Others have plugged it into MySQL, MongoDB, and other popular databases, to speed performance.
The initial challenge to this task was that Cassandra did not have a pluggable storage engine architecture into which the RocksDB engine could be easily embedded. So the engineering team went ahead defined a new storage engine API that cleanly separated the distribution layers from the storage engine, one that incorporated the most common read/write and streaming interfaces.
“This way we could implement the new storage engine behind the API and inject it into the related code paths inside Cassandra,” the developers wrote in a blog post explaining the work.
They faced other challenges in the transplant as well. They had to match Cassandra’ support of rich data types against RocksDB simpler key-value interfaces. This entailed work in defining encoding/decoding algorithms for these data models.
Lastly, the company needed to accommodate its streaming data architecture. Whenever a node is entered into a cluster or removed from one, the stream of incoming data flows across all the nodes. This model had to be changed to accommodate the RocksDB APIs.
Results
The team had worked on the project for a better part of a year and was able to test it in production, where they found that the red latency with the new architecture from 60ms to 20ms. Moreover, the temporary operational stalls owing from garbage collection dipped from 2.5 percent to 0.3 percent, a 10X reduction.
The Instagram engineers are looking for ways to add more C/C++ features into the codebase, to cover other features such as secondary indexes, repair, and so on.
Photo by RKTKN on Unsplash.
