The Shared Responsibilities of DevSecOps

Watch

Listen

Show Notes

In this episode, we sit down with Mackenzie Jackson from GitGuardian to discuss #devsecops and the way engineering teams can (and should) implement security into their development lifecycle. Spoiler alert: it is a shared responsibility.

We had a great conversation and touched on a ton of topics, including:

0:00  Intro
3:34  How common is it to leak secrets?
5:05  How does GitGuardian fit into a DevSecOps lifecycle?
8:04  Is DevSecOps just a silly acronym?
10:43  Creating a shared responsibility model for dev and security teams
13:48  What does it mean to incorporate security into the development lifecycle?
16:40  What are ways to bridge the gap between application, security and #devops teams?
20:57  Bringing security into the dev cycle creates proactive postures vs reactive postures
23:07  Tips on improving communication between teams.

Transcript

It’s definitely really challenging to be able to harmoniously bring all that together. You have these different teams that all create friction with one another. You have this security team, which is known as a team of no developers. Don’t actually get to kind of be part of the solution or understand why this is a problem.

That’s operations teams. Oh my God. Why are the developers doing this? The solution. Uh, I’m not harmonious. And so this is kind of like where I feel very passionate about it is that we need to have these conversations from day one. Welcome. My name is Sean Odell and welcome to my first episode. And I believe the second episode of, uh, the mission control podcast, , by space lift.

Really excited to get in today’s episode. And I am glad to welcome, uh, McKinsey, McKinsey, uh, you know, give us a little bit of about yourself and, uh, we’ll get this thing going right.

Yeah, sure. Well, like great to great to be here, Sean. It’s awesome to be on the podcast. I have to give a shout out to the name. It’s a, I think this is the best podcast name, a vegetable, but, uh, yeah, as I said, I’m McKenzie. So, uh, I’m actually, uh, in dev REL as well. Working in developer relations for a company called gifts guardian.

So, I mean, you know, I first started off as a founder. I was a CTO over health tech startup, which is still, still around and kicking today Pago. And when I left that, I was kind of, uh, it really taught me a lot about security being in that health space, uh, building robust apps. I really loved that. And once.

Focus on that for, for the rest of my career. So I started working for good guardian. We’re a security company that specializes in securing source code and more specifically making sure that we can keep secrets like API keys and passwords out of our source code out of our kit repositories. Um, so I’ve been with guardian for just over there for, for two, two years now and just kind of a.

Being an advocates and teaching tooting the horn for DevSecOps and getting developers interested in security and providing the right kind of tools to make that possible. Very cool. I know. I’m glad to have you in and, uh, that does not sound like a Texas accent.

So where are you based McKinsey? Where are you from? Uh, it’s a confusing. Okay. So I’m from, uh, I’m from New Zealand. Um, but I, uh, I live mostly between Paris and France and. in the Netherlands. So I kind of do half and half between those two, but I don’t know where my accent is from now. Like pretty much every country, every person in the world.

I have an accent too. And if you think that I have an exit, now you should hear me trying to speak French. Yeah. That then you’ll hear an accent. Ah, very fair. Well, no, I mean it it’s it’s it’s good. It’s good to, uh, to jump on with you. I think that’s one of the fun parts of that being a developer relations or advocacy in general is just really working with folks like yourself all over the world.

We come from different backgrounds, different, you know, aspects and walks of life and what you probably didn’t know. And I probably should have told you before you actually got on a podcast. Yes. I have uploaded one of my coworkers, AWS keys to get hub. Uh, I didn’t know that, that it was varied in the Jenkins code.

Yes. It was a public repository. So, sorry. Uh, yeah, this happens way more than you would even think. I can tell you right now that you are not alone in this, uh, every year we, we publish a report. How many credentials we find in public get repositories. So keep guardian scans every single. For the year. So last year we scanned about a little over a billion commits and we found 6 million credentials.

So you’re definitely, definitely not alone in that. Uh, very, very fair. So, so you, you mentioned get guardian right in, in my case, I tend to come from the, or I guess originally I came from the infrastructure side of the house, uh, and then grew fond of. Application and the application development side, uh, of the house.

Um, and so, you know, I, Lynn, I tend to come from the infrastructure to applications. It sounds like in your case, you probably coming from a little bit of the, uh, of the application side, but you really need infrastructure. So tell me a little bit about get guardian and, um, and, and kind of just the landscape and where you fit into.

Yeah. So, I mean, get guardian is kind of a solution that, that fits into a lot of different areas. So were hyper-focused on making sure of detecting credentials. It’s essentially a lot harder than you think, because credentials are probabilistic and they may be high into the strings, but most high end could be straight.

I’m not secrets. So you have to really analyze all of the code and different points to be able to detect them correctly. And we come at it from a couple of areas. So, you know, for your, your security teams, We have, you know, dashboards and products for them to be able to monitor their source code and kind of get visibility of aware their secrets out where the API keys are.

But then on my side, you know, we’re really focused in, on that developer bottom up, uh, area as well, where we kind of create these open source tools, uh, that fit into your CLI that can. Implemented get hooks or other areas so that your developers can prevent leaking secrets in as well. And then you have that, you know, uh, those operational teams, you know, because I think people forget about that.

There’s a lot of secrets that ops ops guys and the infrastructure guys have to handle, um, uh, you know, Can be really, really challenging and getting that over there. All right. We find a lot of secrets in GMO files and, and everything. So w you know, this is a problem that kind of needs to fit across the entire software development life cycle, and as a product that is part of a security.

Uh, is part of, you know, like a security program. I, I hate the vendors that kind of pretend to be this magic bullet solution for everything. You know, we are very good at doing one thing, uh, that fits across many different teams, you know, but it’s gotta be part of, maybe you have some dynamic application security testing.

Uh, maybe you have some open source vulnerability detection, you know, different, different tools out there. And this will be one of them that fits together. If that makes. Yeah, no, it makes perfect sense that I’m glad you mentioned that there is no one tool that fits them all. Um, I also, you know, I guess my previous two or three years, I’ve focused in the security space around infrastructure as code, uh, at a couple of different iterations.

Obviously I’m concerned about that now in my role at, uh, it’s facelift. But the one thing that I’ve always come back to is. We are providing solutions that meet developers, where they are. Right. And you mentioned it utilizing good hooks, utilizing. Basically the interfaces that developers are accustomed to, right?

Whether it’s vs code, uh, in their SCM, right. Rather than, Hey, here’s my cool new little screen. You need to use it. And we’re not going to integrate in any form or fashion into what you’re doing as your day in and day out development. Right. So I have a huge pattern for that. It sounds like you have a huge passion for that.

Um, and that leads me to my first question. And I think this is a fun question, right? Cause we’re, we’re wanting to meet developers where they are. Um, but I often hear like we have way too many acronyms and why do we need DevSecOps? Or why don’t we just say dev ops and encompass that with security. So real quick, what are your thoughts on dev sec ops?

They use of sec or security inside of ops or dev ops, and we’ll just start this debate, fight or agreement, whatever this turns into. I thought we agreed. We were going to go and he polarizing political, uh, conflicts. No, I look, I th I think it’s a problem. I get that everyone kind of gets tired of these new acronyms coming out where it feels like, and then someone describes it to you and it’s like, yeah, like, and.

What we’ve been doing for awhile, but, but I think it’s important because it kind of describes how these teams fit together. You know, security hasn’t really kept up with just the agile way of development that we’re kind of used to, and it’s been lingering a bit and kind of like that DevSecOps thing to me, it encompasses what really it should be.

And that’s, that’s this one, this is one function. Security. Within our devs securities in our operations teams doesn’t necessarily mean that it looks the same for everyone, but it means that there’s a shared responsibility of security of operations with everyone and making sure that we have a secure, smoothly running application from the design stage all the way through and every iteration there for after.

Fair. It sounds like to me, there’s a shared responsibility model. Now that that’s a, that’s a term. I think we overuse in a lot of ways. Um, but I go back to a conversation. I was, I was actually in London. This would have been four, four and a half years ago, meeting with a. Fortune 10 financial institution.

This is the CTO and CSO. And he’s like, yeah, I just had a meeting and an application, you know, team came in and they highlighted their new application. They were ready to go, you know, to launch it and put it on the public facing internet. And we looked at him and said, have you passed any sort of security?

Uh, any of the security checks our guardrails and they looked at him and were like, I don’t know what you’re talking about. Right. So, so I mean, that was a couple of years ago now, but that example still happens today, where there is a disconnect between app dev operations and security. Right. So kind of detail that a little bit of what you have seen and ways to be successful in bringing this all together.

Yeah. It’s, it’s definitely really challenging. Harmoniously bring all that together. Uh, so I mean, what do you kind of see in the industry is that you have these different teams that all create friction with one another. If you haven’t implemented kind of a shared responsibility model, you have this security team, which is known as a team of know, you know, but may not provide answers as to why.

The developers feel like they’re just making their life more difficult, you know? And they don’t actually, they end developers. Don’t actually get to kind of be part of the solution or understand why this is a problem. It’s just kind of like, no, you know, and we find the same thing with, of those operations teams that kind of know, oh my God, why are the developers doing this?

This is critic so much more load or, oh, you know, and the solutions. Uh, I’m not harmonious and they don’t have conversations at the beginning stages. And so this is kind of like where I feel very passionate about is that we need to have these conversations from day one. We need to get developers and security teams in the room, and we need to talk about topics like fundability that developers can take ownership over and we also need to provide.

The tooling for the ops teams and developers teams, so that they can take on some responsibility for security because, uh, oh, you touched on it before we have to fit this into their workflow, right? No, one’s sitting here saying that developers or ops guys or security guys don’t have enough to do. Right.

And now you have to add this new thing and it’s, it’s not about creating more work because ultimately it speeds up the, the process it’s about. Having discussions talking about it, why this is a problem. And then when a developer faces a block, then if they can’t use this package, because it has a vulnerability, if they can’t, uh, if their pull request was blocked because they’re containing secrets, right.

They understand why they understand why this is an issue. And they have tooling on their side, which allows them to fix it without having to talk to the scary, scary security guys or, you know, like, so it’s about bringing everyone to. And reducing this, this friction where everyone feels like everyone’s just against each other.

Oh, yeah, the, uh, it’s funny. It we’ve been doing it this in the industry for years. I’m going to, even, even when I was on the customer side many, many years ago, I’m like we would, we would go to, you know, I’m an infrastructure, I’d go with my app dev representative. We’d go to the, at that time, you know, in that case, it was the network team and the network team.

Can’t do it. And we’re like, wait a minute. You’re not even going to listen to the conversation. You know, you’re going to understand, you’re just going to flat out say no. Right. Um, and so having those teams and having that friction is nothing new. Right. But now that it seems like in some cases or what we’ve made some progress in some areas now, now we kinda like to, into, you know, in ways pick on security at times, or maybe they pick on us one of the two.

How, how does incorporating, um, dev ops or dev sec ops into the software development life cycle really come to light. If we do this correctly, if we do this in a way that makes sense to the organization, what does that look like for you? So, I mean, it, I think what it really means, it’s just, I like, I complete step away from the waterfall approach in every.

That we’re doing so not just from development, but also kind of through that whole process. It’s about everyone having conversations at each point and the never being a kind of a handoff, like, we’re not going to hand off this application, like, okay, we’ve built application now it’s ready to go to operations and oh, and now operations, it’s ready to go to security.

Like it’s, it’s a continuous. Project right. That continuous integration, continuous deployment. We’re going through the whole cycle. And at each point everyone has a say, and everyone has responsibilities over what they can do to make the process faster. Because the minute that the minute that something has to go to a team and then come back, like not only have you annoyed everyone, you’ve wasted so much more time and it’s infinitely more complicated to kind of retrofit like retro.

Kind of, I forgotten how to speak English from it, retroactively fix issues, uh, you know, once they’ve been built into that design into it. So I think if you’re going to have successful DevSecOps, then we have, you know, we have the three teams working together. They have a say at each point, uh, of the design, you know, and that when it comes through our final.

Everything’s already been worked on together. Everyone knows everything. And if there is still an issue, then we know immediately how to get back to it. And so I think that’s like what it looks like in a successful organization. It’s smoother, it’s faster. Everyone’s friendly. Once the application is deployed, you can all go off and have beers and everyone’s still friends.

I probably should have started this off or started this, you know, this, this thought off, uh, at the beginning. But I actually think in some ways, because of the friction that has happened, you know, whether it’s organic or man-made, or, you know, made up right or perception versus reality, I actually think security plays a significant part in this.

Um, they have a skillset that I may not have. Uh, and, and it’s, you know, in some cases I do, right. I, I, you know, maybe I do, maybe I don’t, I’m a man, I’m a, I’m an application developer and all I care about is go Lang and that is it. Right. I have a few libraries that work. But I know nothing about the overall security posture, right?

So I believe security has a skill set that we should bring to the table. We should incorporate, um, that being said, you know, what are ways that we can try and maybe bridge the gap or, or welcome security or give them an equal spot at the table? Yeah, definitely. I think everyone has something to bring to the table.

I think one of the, one of the processes and making sure that security is heard, uh, you know, is to, you know, I guess we need to have different types of security at different areas. And I, what I, what I, what I know that developers really react well to is kind of. Learning about security in a way that really affects them, you know, showing them how they can, you can do a malicious take over from some insecure code, showing them in real time, you know, how cross site scripting works or how you can inject something in there because then everyone becomes invested in this.

And it’s interesting. Um, but if security kind of can continue to talk in their own language, and I know, you know, like we, you know, I’m, I’m very passionate about security and I guilty of doing this myself. Talking off in jargon terms that no one understands it makes us feel really important. And no one can, no one can decipher what we’re saying.

So, you know, we feel really smart. Oh yeah. But at the end of the day, it’s not really helping anyone. Um, so, you know, so I think that’s one step when security can actually come in and show. What they’re doing and how it affects the developers. I think it all needs to relate to everything because we’re not going to take away security’s role or operations role, but we need those people need to be invested in seeing how it affects them.

Right. We’re not going to teach the developer how to be a security engineer, but we can’t take the portion of security that really affects them and shine a light on that. And then the other area is that a security has a whole bunch of blocking tools. You know, w that, that caused all kinds of, that I have, that can disrupt kind of SMO, uh, disrupt the development workflow.

So we need to give developers their versions of the security tools, and they need to kind of, you know, be on board with how they work. And I, my experience from. Endless conversations like this and implementing this in actuality is that developers are interested. No one wants to kind of be blocked and it’s embarrassing if it is a real vulnerability.

So give them tools that fit into the CLI that fit into vs. Code that fit into whatever they use it. That they can use to be able to kind of, you know, make progress on their own. And one example is that we did a rollout with, uh, a company, uh, ran about a thousand devs and, uh, they reduced, we implemented. Uh, tools for the developers.

And we reduce the amount of kind of incidents that the AppSec team had to deal with by 85%, because a lot of it was not fundamentally terrible stuff. It’s just little mistakes or little things that can create big problems. So we need to kind of sort that out. Yeah, I’m a human, uh, I think we all are, that are working in this space and we are going to make mistakes.

Um, we can do everything we can to put. You know, guardrails, uh, in place, but a guard rail is very different than a roadblock. Um, I go back to, uh, I, I was blessed or cursed depending on how you look at this to work in the old school change board. ITSMs. Right where we wanted to propose a change and we had to wait two weeks for the next change control meeting and we’d get in the room and everybody would say, what are you doing?

And you’d give them the rundown. And there’d be one person in the room. And really not, I don’t, I’m not really ready for that yet. And so it would be another two weeks before we could get back in. Right. And so in, in the case of, of dev sec ops, and this is something I continue to, to, to, to want to just almost continue.

Speak as loud as possible is. There’s a difference between a reactive tool and a proactive tool. And security can be very proactive in a lot of ways. Or if I have, you know, it, you know, whether it’s infrastructure as code or new libraries that I’m implementing or creation of new containers, um, that are a part of my application.

I don’t necessarily have to wait for another human interact and tell me that this code is insecure. This container’s insecure, or this, uh, this infrastructure that you’re building is empty is insecure. Right? You can create proactive steps and measures ahead toward if I just go do it, just a simple merge.

I don’t even have to wait for somebody to come around and say, no, you can’t do that. You can actually give me the results inside of, you know, whatever my SCM tool is. And then. I I’ve never, my productivity never stops as a developer. Right. And I think that’s one of the keys to re uh, being proactive versus reactive and not being a roadblock, but actually providing some stable guardrails, which are okay.

Um, and so for me, I think it’s, it’s great that you’re kind of bringing it. At least the conversation around DevSecOps puts an emphasis on security and gives us something to talk about rather than something. Full-stop Nope, we’re going to you do your thing and then bring it to us. And that just creates a lot of friction and a lot of unnecessary headache for all of us.

Yeah, no, I totally completely agree. And you know, and that’s, and that’s it. And that’s about, uh, kind of getting everyone in the same room, getting everyone to kind of communicate and become friends. And, uh, and I think that that, that process is makes so much sense and it’s so much smoother when it’s implemented.

Um, it’s, it’s, it’s not a, it’s not a, it’s not a, it’s not an. Thing. It’s not a difficult thing either. I think we’ve put a lot more. I think we create a lot more friction than, uh, the necessary, uh, man McKinsey. I’ve, I’ve appreciated the conversation. I know it’s a, you know, we try to keep these short quick to the point.

Um, and I think we, we’ve highlighted a very important, valuable topic today. Um, I, you know, what are some things, you know, Two or three tips that you could give to somebody, maybe that’s on the dev op side and is struggling to get their security team involved or somebody that’s on the security side that struggling to get a seat at the table for the dev ops side.

And this will close this out. So it’s all on you. Well, I think, you know, like when you’re in the operations and trying to communicate to, to security, I think, or even vice versa. I think everyone, when we were in, we were kind of talking, if you want to propose new changes that, that you think then to talk about it in a language that affects the other person’s job.

So we can reduce the amount of vulnerabilities being sent to you. Uh, we can speed up the operational times here. Uh, we can remove this human element. Um, and when you’re talking to each other in ways like that, then you’re, you know, you’re doing a couple of things, right? You’re acknowledging that their time time’s valuable.

We acknowledging that what they’re doing is important and then you can actually make, to make that step to move forward. And, uh, you know, and if you’re in the ops team and you’re getting frustrated by security, always being the, the, the, the team of nos. Then, uh, take, you know, be proactive and have a look at what security measures that you can implement to, to kind of get rid of some of their work.

And some of that interaction, you know, can you do put something in the CIC pipeline to check your containers, make sure that you don’t have secrets in your Docker images or make sure, you know, other areas like that, um, and, and re and remove that. It’d be part of that conversation. And. Extend the olive branch, because I know that a lot of y’all and friends at the moment, but I think it’s time that we move on to stat.

No, that that makes perfect sense. Right? It’s, it’s a mutual, um, mutual respect, mutual appreciation. Uh, and if we meet each other where we are, um, it just, it just helps alleviate a lot of the struggle and, um, you know, it makes, it makes the, it makes the entire process, uh, even better than when it started. So McKinsey, man, I appreciate you coming on today.

Uh, we may end up jumping on at a later time and do like an office hours, a demo or some crazy stuff at the command line and actually get into the weeds. Um, and I look forward to that if it’s not you, maybe somebody else from your team, but really do appreciate you coming on today. Excellent. It was awesome compensation.

Thanks, Sean. I appreciate being invited. Absolutely. Thanks everybody for joining I’m mission control, uh, podcast. I know I’m excited about what we’re doing here and really look forward to the next episode. So stay tuned and we’ll see you then.