Introducing Shotover for Database Ops and Interoperability – InApps Technology 2022

We’re proud to introduce Shotover proxy, a fully open source software project designed to simplify database operations and interoperability. From its initial prototyping and throughout its various iterations, the technology has been under internal development for two years now and has already proven its value in building powerful new data-layer capabilities. We’re excited to bring it to the open source community.

The idea for the Shotover proxy occurred to us after hearing myriad stories of organizations discovering that a chosen database didn’t fulfill their application needs, leading to painstaking migrations. In practice, this reality has likely cost us as many customers as it has gained us, with shifts to and from the databases within our realm of expertise happening in both directions. As ardent supporters of open source data infrastructure technologies in their 100% open source versions, we were eager to open source our Shotover solution once it became ready. We want to help organizations solve this all-too-common hurdle, and enable the project itself to benefit from the community’s greater breadth of perspectives.

What Is Shotover?

Shotover is a high-performance Layer 7 database proxy that enables users to control, manage and modify the flow of in-flight database requests. For organizations using multiple databases, it’s common to identify operational shortcomings that arise when data models, queries and database behaviors aren’t in congruence with the needs of particular workloads.

Shotover lets organizations implement new database behavior without changing their applications.

As organizations gain operational experience with applications and clarify their true requirements, it’s not unusual for teams to regret their initial database choice. A database originally selected to persist the state of applications may turn out to offer performance that doesn’t live up to its benchmarking. A data model that’s a poor fit will lead to slow and inefficient queries, oversized and inefficient tables, queries occurring too often because of hot partitions, slowdowns and other issues. A database might also prove a bad match due to an organization’s needs around pricing, service-level guarantees or simply user behavior that veers away from expectations. Ultimately, these unforeseen issues become obstacles to developer velocity.

Unfortunately, these database issues often emerge in production and only after the organization is deeply committed, so they must perform challenging and expensive application and schema alterations to change course. Organizations may have to completely rewrite applications or adopt new services to use a new database of choice. Data models, data storage, query patterns and configurations may also need to undergo extensive revisions.

Same Old Database and App, Brand New Database Behavior

The open source Shotover project provides welcome relief for organizations in this situation, by offering predefined transforms that empower teams to modify and route queries from any source to any upstream database of their choice. In this way, organizations using Shotover can effectively decouple applications from particular database performance, storage and data locality limitations. By assembling chains of these predefined query transforms, teams can immediately specify new query and data storage options for use with their existing applications. There are no code alterations or redeployments required.

Put simply, Shotover lets organizations implement new database behavior without changing their applications. Suddenly, teams can leverage the optimal capabilities from the multiple databases available to them to address their entire range of needs. Inefficient queries, queries that are slow for certain keys, tables that require a more appropriate data model, queries with poor observability and more can be addressed easily and with core business logic left intact.

With Shotover, organizations can pick and choose from the best of all database worlds. It becomes no big deal to cache data stored in a Cassandra database using Redis, or to index data in Redis using Elasticsearch. Databases without native cross-region support or saddled with “open core” proprietary limits can still achieve cross-region data routing and replication. Shotover similarly solves compliance issues familiar to organizations using multiple databases. Shotover provides field-level data encryption and common key management across an organization’s databases. The project also offers a common audit and AuthZ/AuthN point to support SOX, PCI-DSS, HIPAA and further compliance requirements.

Open Sourcing Shotover: A More Powerful Solution

Beyond our own strong inclination toward an open source approach to software, the flexible and inclusive nature of Shotover naturally demands to be free and available to the global community. The universal opportunity to leverage the ideal database for performing any given data-layer function is too powerful to be limited by the finite support any single company could offer.

As Shotover now leaves the nest and grows in the world, it’s our hope that it will flourish in breaking down barriers in many more database use cases,  and for applications enabled by optimized data layers to become far more commonplace.

Photo by Lucas DC from Pexels.