Why Our Agile Journey Led Us to Ditch the Relational Database – InApps 2022

For almost 170 years, Travelers Insurance has been one of the top writers of commercial, property and casualty insurance in the United States. While the insurance industry traditionally has about 20 different use cases, over the past decade-plus, we’ve been witnessing an evolution of the demands from our data.

As a senior architect, I saw demands to build features that required us to use data at rest, then data in motion and finally, data in consumption. Around the same time, our board of directors mandated that Travelers leverage technology in new ways to redesign the way we manufacture and sell our products in order to improve our productivity and efficiency as a company.

It stood to reason that software would reduce the bottlenecks, which meant we needed to become better at building and delivering software consistently. While we considered ourselves agile zealots and wanted our teams to be as self-sufficient and independent as possible using microservices, we did not wake up one day and suddenly change. It took several years of iterating before we finally replaced the underlying relational database with a document database that would enable us to capture the value of using microservices and increase our developer productivity and velocity.

Becoming More Agile with an Operational Data Store

Initially, our goal was to build a single-view application for our brokers, who at the time were logging into 12 different services required for a single use case. What continued to hold us back was the relational data model. Let me explain.

Within modern software development practices, you are expected to build software from a two- or three-sentence business feature. The name of the game is to not go too deep and to constantly iterate. This is far different from the traditional waterfall method, where you would spend six months figuring out the requirements analysis before you wrote a single line of code. With a waterfall approach, this is fine because you will know the end state in order to create your database objects. However, you simply can’t do this if you’re following agile methods because there is no way to build a data model from a three-sentence business requirement, and you’re constantly having to rework your database.

We stood up the single-view app in 2014. At this time it was still ETL-dependent and had a monolithic code base as well as consistent integration challenges. But we were now deploying software five times annually — major for us — and had built buzz internally from people seeing the business impact from using our app. We realized that for our engineering teams to be able to deliver the software necessary for the business goals, we ultimately needed to move away from the relational database.

RDBMS Growth at Travelers

Goodbye Tables, Hello JSON!

We made the decision to use MongoDB’s document database to fully commit to our internal transformation. However, to be successful after this massive change, we knew we had to do more than just learn how to program against a different database. This was a massive change we were introducing to a company that had never used anything other than a relational database. As much of a technology change it was, to succeed, Travelers needed to also undergo a culture change.

While our developers are building the software, there are many other groups of people we took the time to build relationships with and bring them along on our journey. We did this to make sure we could use their expertise and to quiet the noise. We taught every team how to data model in JSON, as opposed to the rigid tables and rows of a relational database. This was an eye-opening experience for many people, who now understood how this affected the speed at which our teams could deliver software into production.

Quickly, backlogs for business product owners began declining as our dev teams were able to deliver features into production faster. This validated our technology choice and began a flywheel of momentum. As the word got out internally of what our business unit was doing, we began to see a massive uptick of interest from other teams who were curious about our results.

OK, Now Come the Microservices

Despite our developers having zero prior experience with MongoDB prior to our first release, they still were able to ship to production in eight weeks while eliminating more than 600 lines of code, coming in under time and budget. Pretty good, right?

Additionally, the feedback provided was that the document data model helped eliminate the tedious work of data mapping and modeling they were used to from a relational database. This amounted to more time that our developers could allocate on high-priority projects.

When we first began using MongoDB in summer 2017, we had two collections into production. A year later, that had grown into 120 collections deployed into production, writing 10 million documents daily. Now, each team was able to own its own dependency, have its own dedicated microservice and database leading to a single pipeline for application and database changes. These changes, along with the hours saved not spent refactoring our data model, allowed us to cut our deployment time to minutes, down from hours or even days.

Setting the Pace for Future Innovation

At the end of the day, if you’re doing it right in relational, you have a lot of tables, and if you’re modeling your data properly, you will have a lot of objects even for the simplest use cases. Once we determined that our database was slowing us down, we knew it was time for a change.

Our initial decision to veer away from the relational database for an experimental application ended up having a profound effect on the company, with MongoDB becoming the de-facto standard for software development. We found that we could really embrace a lean product mindset and set our development teams up for success to achieve our original goal of significantly reducing time from business ask to production delivery.

Interested in data modeling in JSON, but don’t know where to start? Check out our white paper or attend our weekly webinar series.

Photo by Matthis Volquardsen from Pexels.