KubeCon + CloudNativeCon sponsored this post.
What is DataOps? Why is a real-time data platform essential to the use cases driving it? How can you build data pipelines with open source complexity?
In this episode of InApps Makers podcast — recorded live for KubeCon+CloudNativeCon North America, we talk to Andrew Stevenson, chief technical officer and co-founder of Lenses.io, about how Apache Kafka and Kubernetes can together dramatically increase the agility, efficiency and security of building real-time data applications.
“Data is a constant, and it’s moving around, no matter what technology we’re using,” Stevenson said.
Why Kubernetes and Kafka are the Combo for DataOps Success
Also available on Apple Podcasts, Google Podcasts, Overcast, PlayerFM, Pocket Casts, Spotify, Stitcher, TuneIn
So how do organizations make the most of that bigger-than-big data? Well, the majority of them have adopted open source tools Kafka stream processing platform or Kubernetes container orchestration system — or most likely both. But the complexity of these huge systems can be overwhelming to the point of ineffective. And with open source tooling, as Stevenson says, can come uncertainty.
He explained, “It’s a real struggle making sure the open source technology has the governance around it. It’s not that data is something new to people in organizations — they have data. But it’s the security aspect around it. And if you couple that with, quite often, the lack of visibility on open source technology — that is where it starts to fail.”
That’s where DataOps comes in, playing the role of management and self-service. It’s about creating a self-service pipeline and platform, like Lenses.io, which sits atop this open source technology, enabling all the data management including policies, governance, and internal knowledge and data stewardship, coupled with operations. Then, with all that in place, you can integrate with partners that can help you scale, all while compliant for data regulations like Europe’s GDPR. And, you can finally take your data management into production.
Everything is done in a controlled manner, within the context an organization’s data, so that the complexity of data no longer has to come at a cost.
Next Lenses.io sees its roadmap going towards answer key enterprise questions like:
- How can I discover data?
- How can I access it in a secure way?
- How can I make it actionable?
And this can all occur within the context, governance and policies of an organization.
