Why Bloomberg Bet Its Data On Kubernetes
Also available on Apple Podcasts, Google Podcasts, Overcast, PlayerFM, Pocket Casts, Spotify, Stitcher, TuneIn
Bloomberg owes its success in maintaining its status as a financial information and data giant, over the past few decades, thanks largely to technology. After its creation in 1981 when Bloomberg shortly thereafter became synonymous with the large computer terminals on the desks of Wall Street and financial trading floors around the U.S., the company now describes itself as an information and technology company. With 19,000 employees worldwide, it offers software, financial, media and data services — with data heavily embedded in its DNA. Recently added to its immense data-management structure is a Kubernetes layer, which Bloomberg adopted about three years ago.
For a podcast recorded at KubeCon + CloudNativeCon North America 2018, Steve Bower, data and analytics infrastructure lead at Bloomberg, discussed why and how Bloomberg transitioned its data and infrastructure operations to Kubernetes and a largely cloud native platform and what he learned from making the move.
Among the services supported by Bower’s data and infrastructure group are databases, search engines, and Hadoop infrastructure, culminating in thousands of instances of databases and a large range of analytic platforms that rely on the different data services, Bower said.
The magnitude of the range and quantity of data Bloomberg manages is obviously vast. Data, and data management, are what power, for example, Bloomberg’s recruiting site and bond screening on Bloomberg terminals, its Hadoop infrastructure, financial data and straight news for its media feeds, Bower said.
About three years ago, Bloomberg had built what Bower described as “kind of our own hack version of what is Kubernetes in order to manage and run [our data services].” The problem: getting the infrastructure to scale in order to keep up with the data services’ growing demands.
“About three years ago, we had built a system for ourselves and recognized that it wasn’t going to scale to where we want it to be, so we started looking at things out in the ecosystem. We looked at different stuff and kind of came to Kubernetes really because it actually has this very similar model to what we built, which is a very declarative model where you declare something should exist, and then something goes and makes it happen,” Bower said. “And that’s what we had built and it kind of fit our pattern.”
For microservices, Bower said: “it’s not necessarily the act of breaking it down, it’s about changing the actual model by which people build and deploy and manage their software. “So just breaking things down for the sake of breaking things out. I feel like you hear that a lot like, ‘Oh, I want to break everything to microservices’ and I’m like, “okay, do you have a good reason for that?’” Bower said. “And for me, actually, the reason is testability. Like we have some services that we use, for example, for managing how we change configurations in Solr and move data from deployed schemas and configurations into our Solr clusters. We have a service that we built that does nothing but that, and we haven’t re-deployed it for like three years.”
Bower’s group, for example, saw the real value in centralizing its data management infrastructure into a consistent platform. “As I see it, the business value for us, is that [team members] can now focus on the actual tools that they’re working on. We are a large contributor, for example, of the Solr ecosystem with a bunch of committers at Bloomberg,” Bower said. “I would rather them spend all of their time doing that than figuring out how to collect metrics and orchestrate their software across a bunch of servers. That’s where the business value is for us.”
An outlier of sorts, Bloomberg’s data infrastructure group does not necessarily have to have a stated business objective per se when it develops new infrastructures and technologies. “Bloomberg is an interesting company in that way,” Bower said. “ Like, a lot of this actually, is from the bottom up. It [is] us, as engineers looking at how we build and manage.”
Alex Williams, founder and editor-in-chief, of InApps Technology, hosted the podcast, along with Joab Jackson, TNS managing editor.
KubeCon + CloudNativeCon is a sponsor of InApps Technology.
