Posts

This article was re-purposed into an article form the Keynote speech by James Watters Senior Vice President, Strategy, Pivotal from Kafka Summit NYC 2019 in an effort to expand its reach.

Official Conference website: https://kafka-summit.org/

Below is the transcript of the keynote with minor changes:

James waters: Great to be here  in New York, I came here five years ago for the first time in March of 2014 to talk to the New York Technology audience about micro services and platforms. So it’s really great to be here again with the Kafka community today, kind of taking a health check on where we’ve gone as a community and I think there’s an exciting next gear that we’re all finding together what we’re doing with these applications.

Slide 1:

1 . My talk today is basically on the two fundamental things that I think you need in your organization as technology primitives to help you. And the two that I’ve come up with or everyone’s gonna need are spring boot and kafka, you might need other things but those are the two things I see everywhere.

 

When teams come to San Francisco to talk to me about where they’re going as organizations, they kind of come in and they give me the quick introduction and they’re like well we’ve chosen spring boot and we’ve chosen Kafka and we’d like your advice from there.

Slide 2:

2. So the talk today is a little bit about how we’re making a big bet in that ecosystem and also the evolution of how we got there, how our thinking has evolved and how I think it fits into that transition we’ve already been making to continuous delivery and microservices.

 

The thing that our clients asked us to do are two primary things- They want to learn how as an enterprise to start competing with those new up-and-comers that are attacking them in their traditional industry often or to accelerate their attack on incumbents (and that centers around continuous delivery).

 

But the other thing is that these enterprises can’t really take on all these new micro services, all these demanding new projects without thinking about a platform. They have to think of these platform primitives-what they can get good at as an organization in a standardized way. So these are two big things that our organization asks us for and why I really care about the right emerging technologies.

Slide 3:

3. I got the whole way through all of my slides without saying the word paradigm shift, it was really an act of discipline for me it’s one of my favorite terms and anytime you’re a technologist you’re really excited to participate in a paradigm shift but that’s sort of something that you let others say you’ve accomplished a paradigm shift but when I look at what’s happened in these last five years, if you look at how people used to do IT projects- They would start a project and then 18 months later they get feedback from their customers or 24 months later.

 

I was talking to a bank in New York once and they revealed that  it took us 18 months to change a button on an ATM but that was how the organization had been built, that was sort of the model You’d have a big bang project everyone would fill their role in the swimlane and then you’d go execute sequentially that was sort of how that model ran and the architecture of the organization also reflected that, you had a monolith for all of your application infrastructure lived,  you had a monolith for all of your data lived.

 

What’s happening is that these bonds are being changed by continuous delivery and so from left to right continuous delivery is radically reimagining- “what’s going on” and so you know instead of this old ITIL process ICD took the precedence. We used to think of just one virtual machine or several virtual machines for your project and then good luck and now developers are demanding APIs to be productive immediately, that’s the core that we were talking about.

 

And fundamentally the lower level infrastructure in the application has gotten changed as well ,we used to have heavyweight Java EEE and now spring boot is everywhere.

 

Asks a question to the audience: People heard of spring boot raise your hand.

pretty good, almost  90 percent people raised the hand.

 

He continues: and frankly the other thing where we talk a little bit about but you’ll hear all about at this conference is the expectations of what was a batch process that might have result in sending a letter in the mail and some of these applications and now needs to be a real-time application that connects with you on your mobile device so this I do believe is a paradigm change and we’re helping organizations get through it.

Slide 4:

4. The gravity of this situation is so real that even the business intuitions suggest that businesses need a platform to enable their teams, as they just can’t ask their teams to go through all of that transition that is shown in below slide without some sort of enabling platform. So even the folks at McKinsey now said hey you’re gonna need a platform and largely that’s our business “Pivotal” has been providing and we made some progress.

 

I’m going to talk about our progress a little bit and then I’ll talk about some of the you know the obstacles we encountered and how I think this new streaming ecosystem is going to be an exciting new addition to that.

Slide 5:

5. With spring we’ve had to make a couple of bets on what the right distributed programming model for micro services are and our biggest bet that we made years ago was betting on Netflix OSS and we’ve made some progress there, where even the Netflix team is then adopted spring boot as their core framework. So we’ve got this great big web scale company that’s now gone all-in and what we do is spring boot that’s how we’ve done it and I would say we’ve largely solved the micro services with Java core framework problem in partnership with Netflix.

And I see the relationship we have with a Kafka ecosystem as another double down on what we’re doing with this microservices ecosystem to make all of these new primitives easily accessible to developers.

Slide 6:

6. We also gave enterprises a platform. They were facing the need to go compete. If you look at a company that’s a disruptor in their space – t-mobile is the only mobile carrier right now taking share and they’ve taken this aggressive approach, they’re using the pivotal platform. You can just get an idea of some of the results that you can get from this platform thinking they’ve gone from six months per release to 10x two weeks.  

 

When they ran in a project and wanted to scale one function in one of their applications, it was used to take seven months. There was this one function of an application they needed to scale up and scale down and they’d have to use hardware and stack it out. Now with our platform they’re getting much greater efficiencies and these things come back to me in moments with them like over dinner where they said hey it used to take us 500 people to keep that application alive it now takes four. That to me is worthy of the term paradigm shift. So we’ve given people this platform and micro services and their process of doing is transformed. .

Slide 7:

7. If you haven’t reimagined the fundamental data infrastructure,if you haven’t re-conceived of how the teams are working together, you can get the quick early wins and then hit the wall of -”yeah but the data team owns that, “yeah but that’s a pretty thick monolith for us to go to unpack”. So you get these wins and oftentimes this is a huge win for them, they save tens of millions of dollars, they deliver for the customer but they get stuck here. My passion on Kafka and this ecosystem is largely about what we do after this. It is familiar to folks like you might get some micro-services wins and then there’s still work to be done.

Slide 8:

8. So we really needed to help enterprises do a further fundamental rethink and there’s this term “convergent evolution”, convergent evolution is why a shark and a dolphin look a little bit alike because they’re both competing and adapting to an environment that’s similar. And I think what was happening is that some of the fundamental work being done by the team at Confluent and the Kafka ecosystem was a convergent evolution to this event based microservices world that we’ve been in at Pivotal. So there’s this fundamental ubiquitous language around events that we discovered to help people think of how they do micro services.

Slide 9:

9. One of my favorite quotes to come out of Neha Narkhede’s book Shown in image) and spoke at spring one, she basically hinted that not only when you adopt a core infrastructure technology do you get a tool, you also get a way of thinking. And the biggest challenge for us is that the way some of those back-end systems think today, when they think in terms of oh they’ll be a batch process that in 72 hours will spit out a result. That’s a different way of thinking, the way monolithic technologies were built were also a way of thinking and I’m very excited about this new way of thinking we have with microservices. So what did we do. Confronted with having gotten only part of the way through a transformation of these organizations, we turn to a tool called Events storming and we’ve done hundreds of these app transformation engagements with a born on the web company named StubHub. And StubHub came to us and said – “hey we’ve got this good web application but the way we’ve architected it, we really want a lot more velocity. We’ve gotten to cloud but we need to get to continuous delivery and I would say that’s the trend It’s we’re not just trying to change the infrastructure, we’re trying to get to continuous delivery and so we help them unpack their monolith through the approach called event storming.

Slide 10: Kafka summit 2019

And we learned something to hundreds of these engagement with enterprises.We learned is that the ubiquitous language that we can approach both a business person who’s doing product management as well as a developer is thinking of these fundamental events. We also learned that the closer we stick to these events and to these bounded contexts, the better we keep the incidental coupling between services apart. Like as you start to do these transformations, as you start to maybe decouple deployment but topologically you still have incidental coupling and making a change without affecting everybody else as a distributed monolith. The the number one phrase that the team told me to repeat to everyone here in the audience today is “tell don’t ask” and that helps you limit the incidental coupling .

Slide 12:

12. So we see this as the next evolution of where we want to go with micro services. We want to make the contracts between these teams more event-based and more explicit and we also want to enable that next exciting use case and I think as we talked about that convergence of the data centric use cases as well as the next business application use case and these platforms is what’s so exciting I’m going to talk a little bit about those next.

Slide 13:

13. So we believe this is the new platform based on a new set of principles, it’s built for the world where you want to do continuous delivery with autonomous teams and you want to have data enriched applications that are really gonna allow you to go compete. They achieve arbitrary scaling a scope across enterprises. One of the coolest things about building a micro service platform for me was watching how fast you could get onboard it. Like we had customers where they said we went from zero to forty thousand containers in nine months, we’ve onboarded 900 developers in six months and when you have this scalable primitive at the heart of your enterprise that anyone can come in grab self-service. Go be productive, Go show their boss, that’s just an incredibly powerful thing and that’s our platform.

 

So you might think of this as oh hey we’ve got these new autonomous teams, they’re working across the stream platform, they’re using spring boot etc. I want to show you the most important line on this diagram though it’s between the people. No matter what we do with the fundamental application infrastructure, the number one critical factor to success is these teams have to talk, they have to discuss how they’re actually going to evolve these contracts between these micro services and the work that we do is really here just to enable that. If you can take one thing away is that Pivtol not just a technology solution, the technology is there to reduce the toil allow you to be productive quickly but if unless you’re having conversations about how you’re evolving the product, how the API is between those teams need to work and keeping awareness about the incidental coupling. We’re here as a platform to help enable you to have those conversations don’t forget that part .

 

Probably the number one key to success I’ve seen is you have everyone betting on a few common primitives and then everyone in a slack Channel talking about it because imagine one team, one person in organization trying to go tell everybody- here’s how to do this new pattern versus suddenly everyone’s using similar patterns and educating each other that’s the number one key to success we’ve seen.

 

So this approach has had immediate impacts , I’m going to go through them and quickly give you some highlights. So this is during one of the days in San Francisco and a really big important Bank comes in and they say the classic thing – we’re using spring boot and Kafka and we start diagramming their architecture and what they want to do and they said a really important thing and this is the back to that critical success factor. They said we know we need platforms to allow us to focus on the evolution of our micro-services, that’s what we want to focus on. So now I got really interested I was like- “these are real players”.

Slide 14:

14. The second thing they did is they started talking about Kafka in the same way that Martin Kleppman talks about Kafka and they showed me their diagram where everything coming in externally was written in Kafka first and then this polyglot approach after that and they started saying that we really count on Kafka for consistency, strict ordering, we can replay everything there’s durability, there’s audibility and these are these exciting paradigm shifted moments where the technology has allowed them to reimagine how they think of data.

 

They got cross called cloud replication out-of-the-box as well. This is happening all over banking like I really think in the banking ecosystem this approach of event based micro services is going to become the norm over the next five years as people take on new payment api’s even new core banks very exciting use cases there.

Slide 15:

15. There’s also you know as Jay mentioned real-time inventory we do work with shipping and logistics and client came to us and said well right now every service has its own dedicated team and trucks and they do one service. But we want to overlay other businesses on top of this what if we want to build that next service on that logistics infrastructure but today we don’t have the data to allow us to do it. So what we did is we worked with them to put RFID scanners into all their shipment centers (300 of them) and then pull that back to a core platform and we can now compute in real time how much room there is left on every truck every day and were there opportunities for.

The other thing we did though is we really you know we prepared them for the unexpected or unanticipated use cases and now they’ve added additional business services to that logistics shipping. The thing I’m going to point out in the next two slides is tell me where the data processing spring boot apps begin and end and where this more traditional business API is beginning at, you can’t tell on these architectures. Data processing to real-time applications are in this beautiful spectrum across the spring but–and kafka platform.

Slide 16:

16. This one is a mind-blowing project we’re working on right now and we have permission to talk just about enough of it how would you build a service to monitor a whole country?  Like how would you do that in the old crud app my sequel world, could you even do it? I don’t believe you could and this actually has far-reaching consequences you know beyond technology that I won’t touch but you know people came to us they said we want to monitor entire European country to keep it safe. Again the approach and the architecture here without going too deep into it is that you know between data ingestion, stream classification and then ultimately building event centric micro services to take action. At the end of the day what this application does is -it watches everything going on in a country it ingests that data and it helps analysts curate and do connections between everything that’s happening. So it says- oh if someone was just arrested here and there was previously you know an issue here and then we saw this on a video here oh hey take a look.  I think that’s incredible. But you couldn’t do this in the old project way either just thinking it from your organizational structure, if you tried to plan this out three years in advance ,you wouldn’t be able to do it. So they’re constantly changing the models and the applications to generate these alerts. And that’s where continuous delivery in this new architecture are really shining, you could never built this application before it’s pretty exciting.

 

Finally the power of working with legacy and enterprises is paramount. Things don’t really matter in an enterprise to some extent until you can show them how it helps them on their legacy. Because there’s such an organizational investment in that, they’ve got so much going on with it. I’m really excited to see you know this is expressed by many who’s spoken at Kafka some before as one example but many others using this pattern. Where you use change data capture to allow that legacy to enrich the event microservices stream.

Slide 17:

17.We’ve worked with organizations as part of app transformation to do these CDC like use cases, where we say- we’re going to enable these microservices teams to use to access legacy data without going through the legacy interfaces. And I think this is a huge promising area for the streaming platform ecosystem to really get inevitability because when you can do these new greenfield application, when you can build applications that could never been built before and you’re applicable to legacy and you have these scalable operational abilities,  that’s really profound.

 

So what does that mean for Pivotal? What’s our approach?

18. Well we’re all in, we’ve been doubling down on our investments in spring Kafka there’s some great demos that you’ll see here today and you know the idea is that every spring boot developer and there’s four or five million of those in the world 60 million downloads a month of spring boot we want to make it as simple as possible to add the streaming paradigm to their tool belt. So if you want to add case streams right within spring streams, you can do that and you don’t have to know too much about configuring and managing on the interfaces to kafka you can just be a spring boot and developer, I do that I think this is really powerful.

 

Building around that, of course  we’ve got a caching product called PCC as part of that CDC style use case cache and validation is a big one we’re gonna plum that right into our cache system.

We’ve got a great new thing called Pivotal function service which brings this event centric approach to infrastructure management.  

We have service brokers that allow you to bind from your applications into into Kafka to do real-time provisioning.

We have Pivotal kubernetes service and very excited about the engineering investments we’re making here to make confluence platform run on pivotal kubernetes service. I think it’s gonna be one of the number one workloads that we have, super excited about the idea that you have programmable infrastructure for these two you know distributed data systems.

Finally we’re doing a lot of app transformation if there’s one thing that I think come talk to us about is how do you transform your legacy, how do you think about decoupling in that legacy and we’re using Kafka patterns more and more.  I was reading through last night the last thing I read was sixty pages of notes about everything we’ve done with Kafka and app transformation.

So come talk to us about that during networking sessions, so that’s what I have today I think it’s a really exciting ecosystem to take notes in to learn about this paradigm shift we’re all in on it and I think it’s the next evolution of what I started that five years ago when I came to New York for the first time to talk about micro services and so really excited about the future and I hope you have a great summit go see Josh long speak, he is the energy of the entire spring ecosystem in one human and I can’t wait for his talk thank you very much.