Update The Slow Climb of Postgres and the Value of Persistence

Those intrepid individuals trying to bring a new innovation into the world, or a new start-up, should keep in mind the power of perseverance. Even if a technology is a hit, the creator must still face any number of hurdles to maintain and grow its success.

Attendees at the PostgresVision conference in Boston this week certainly heard a great testimonial for this idea, as database pioneer Dr. Michael Stonebraker offered an account of how he helped bring PostgreSQL database into the world, a 15 year journey of amazing highs and lows.

Stonebraker had already helped birth the first world’s first relational database, Ingres, a decade before, but he was back in the laboratory in early 1980s to create Postgres (later amended to “PostgreSQL” to reflect its SQL powers). Ingres was such a success. In fact, that users wanted to add their more arbitrary data types and operators to its schema, for use cases such as geolocation, he explained. The initial workaround to this problem with Ingres to encapsulate that logic (often very slow) application logic.

Click to embiggen.

So the idea with Postgres, which debuted in 1984, would be to offer a relational database system with support for “advanced data types” (ADTs). ADTs set the stage for much wider use of relational databases, the new capability — built on object-relational mapping — saving untold hours in developer effort. Postgres was also an early database system not to overwrite data when an update was needed, and instead, just add a new entry and delete the old one later, which opened the door to capturing a verifiable transactional history.

But Postgres was anything but an immediate success, no matter how obvious its value. First Stonebraker and his team had the seemingly good idea to write this new database in the best possible language of the day, LISP. LISP, however, turned out to be a terrible choice, because of its slow performance — not a characteristic you’d want in a database. So a rewrite into the much more performant C was necessary.

They also ran into a potentially-crippling catch 22. Early users wanted not just the crude ADTs created by the Postgres developers. They wanted sophisticated ADTs to work with the commercial applications they used, such as geo-mapping software from Arcinfo. But when they approached companies such as Arcinfo and asked for their help, they were told the Postgres user base wasn’t then large enough to warrant such an effort. The lack of commercial ADTs would keep Postgres on the sideline, yet software vendors weren’t interested in creating ADTs for their own products no matter how useful they would be to Postgres end-users.

These days, the core of Postgres is maintained by, among others, Bruce Momjian, among many others.

The database designers were seemingly stuck, at least until another use-case came into view, supporting the then-nascent but rapidly-growing Internet, and gave Postgres a bountiful user-base. In 1996, Stonebraker sold the company he formed, Illustra, to commercialize the database system, and went on to create other databases for other use-cases, such as analytics oriented Vertica.

These days, Postgres is the currently fourth most widely-used database system, according to the DB-Engines Ranking, just ahead of NoSQL MongoDB. EnterpriseDB, which offers commercial support for the open source database (and is the organizing sponsor for Postgres Vision),  sponsored an independently-executed benchmark test to compare Postgres and MongoDB, which was just enhanced with multi-transactional capabilities.

Despite MongoDB’s fast performance, the benchmarking company, Ongres, found that MongoDB had difficulty matching the transactional performance of Postgres. And, with more than 16 threads, Postgres left MongoDB in the dust entirely.

“PostgreSQL 11 was found to be faster than MongoDB 4.0 in almost every benchmark,” the report concluded. “Throughput was higher, ranging from dozens of percentage points up to one and even two orders of magnitude on some benchmarks. Latency, when measured by the benchmark, was also lower on PostgreSQL.”

Work still needs to be done to give Postgres multi-master capability, which would make it truly scalable, across multiple servers. Still, its presence in today’s business stack is not too shabby for software that is 35 years old.

Good ideas never come from “communing with nature on a mountaintop,” Stonebraker said. They come from talking with users, many users. Serendipity always helps (and anti-serendipity always hurts), but that is beyond your control.

And most importantly, Stonebraker reminds us that success comes from persistence, weathering the inevitable lows as well as the highs.