DevSecOps Guide to Bulletproof Your Development Workflow: part 2

Ricardo Miguel Magalhães

Jan 15, 2025 • 11 min read

In part 1 of this article, we've learned about what you need to protect your software, and the tools you can use for that purpose.

Here's what we'll cover in part 2:

Let's bulletproof our development workflow
Embrace the DevOps culture and adapt
The challenges you might face
When to break the build

Now we're ready to start!

Let's bulletproof our development workflow

This is where things start to vary from project to project and the way you set up your workflow. But this article aims to address things in an objective way, so let's work with a scenario where we have a pipeline for Continuous Integration (CI) and a different one for Continuous Deployment (CD).

For the remainder of this article let's assume our CI pipeline executes on every Pull Request (PR) open to main, and the CD pipeline every time we commit to main.

Software Composition Analysis (SCA)

Starting with Software Composition Analysis (SCA). These are quick scans that you can get up and running in a couple of minutes. So, your CI pipeline is the perfect place for this scan to live in. When you add a step to your pipeline, it won't take long to execute and you'll have immediate feedback on what vulnerabilities the third-party tools in your code have.

Static Application Security Testing (SAST)

Then for Static Application Security Testing (SAST) scans, the CI is also the place where you'll want to run these. Since we're running these on every PR we have the chance to check vulnerabilities before we actually merge them on our main branch. Also, depending on how many branches and how you promote your code through them you might decide if it's worth it or not to have it in the CD pipeline or if you're just having repeated unnecessary scans. Since SAST scans usually execute quickly the CI is perfect for them.

Dynamic Application Security Testing (DAST)

For Dynamic Application Security Testing (DAST) scans we need an actual app running so if you want to set them up in your pipelines you'll want a CD pipeline deploying your application somewhere and then chose a tool that offers a CLI to run the scan once the deploy is done. You can always chose to set this up according to your own needs and liking. Let's say that at the end of every week (or at the start) you want to make sure everything is up and running, in this scenario you want to schedule your DAST scan to run every Friday at 7pm, for example.

Infrastructure Scanning

Infra scanning is a perfect integration for CI as well. They are quick to execute and offer some flexibility. If you have your infra files isolated in a dedicated repository you can trigger a scan on every Pull Request for example. On the other hand, if you have your infra files together with some other files (whatever they are) you can filter these scans to be triggered based on folders updated, so you can easily set this up to run every time your infrastructure folder has been changed.

Kubernetes Platform Scanning

Lastly for Kubernetes Platform Scanning we can take a look into a tool like Kubescape for example, and in here you have a couple of possibilities:

If you have a development or some sort of testing cluster where you deploy your services you can choose to perform the scan directly on the running resources on the cluster itself. Based on the response, fix those problems before you progress updates between environments. With this approach you will integrate this scan after you run your CD pipeline.
Or if you want to capture the findings pre-deployment you can run the tool directly on your Kubernetes manifests and choose to block deployments all-together based on the scan. With this approach you will integrate this scan in a CD pipeline as well, but as a step pre-deployment.

This was just an objective way to approach the integration of these scans in your workflow. Now, you can take these and tweak for your own use case and for your own liking.

Embrace the DevOps culture and adapt

Improvise. Adapt. Overcome. — Embrace the DevOps culture

Let's say one of these scans is a CI enemy, meaning it takes so much time that CI just becomes absolutely useless, breaking its purpose. Say a scan takes like 10 or 12 minutes consistently. Might be time to cut your losses, have that scan run on a schedule instead of running on the CI. Make it run everyday at a certain time. Improvise, Adapt, Overcome.

Can you afford to take those 10 minutes in your CD pipeline? Then by any means, have it run there.

Don't want your DAST scan to run in your CD pipeline? Run it everyday at midnight or after scheduled weekly deployments and then setup a process to deal with the results. The same can be done with Kubernetes resources scanning.

Embrace the DevOps culture and adapt, adapt, adapt

Challenges come with everything and with DevSecOps a big part of these can be quantified based on the maturity of the project that will start implementing this concept.

Implementing on something that is just starting is a lot easier than doing the same on an already matured project, simply because there are no other processes in place and everything can be defined from the ground up with no bad habits to break.

The challenges you might face.

Let's look into a couple of the challenges you might face.

Trying to integrate all at once? Don't.

You've browsed the web and found all of these cool scans you can integrate in your project, I know it's really tempting to just start implementing them as you go. But beware! These tools come with a learning curve, so don't fall into this trap.

In a new project, this problem kind of solves itself. You maybe haven't started with infra-as-code yet in the project, you probably don't have a tool like Kubernetes and maybe not even a Dockerfile for a while. So you don't have to worry about scanning these components until you actually have them in place!

So when starting something new you can go slow and then build the process as you proceed. Start with the basics, SCA and SAST scanning. Later as you keep adding more pieces you can take your time and integrate the other scans.

Adding a Dockerfile in this sprints planning? Also plan to implement a scan for container images.

Then the project gets bigger and we now need Kubernetes to orchestrate the containers for us, perfect, scan those Kubernetes manifests. The same for infra scanning.

On a more mature project all of these pieces (or at least some) might be in place already but the logic remains the same. Start going one by one and never all at once. Harden your system little by little, that's how you win consistently.

Building a DevSevOps culture.

Now this one is a doozy.

Make sure you bring people in! Not only developers but from the products' side as well. Everyone has to understand what it means to adopt DevSecOps into a project. This will take:

Time to ingrain;
Commitment from the team;
Your sanity (Just joking...maybe);

Once again it will be a lot easier to build and establish a culture on a new project or a new team versus adapting it in a mature project. If you're starting fresh, people are more open to adopt new processes but when they are used to work in a certain way for a long time, there will be resistance to change.

Building culture is mainly about changing how a group of people think, whether that is a team of 5 engineers or a company of 5000 people. And when starting to think about implementing a new culture you have to objectively think about what that group of people will gain with it. Let's say it's a company, how will my company benefit from implementing a DevSecOps culture? What examples of previous failures in adopting these practices exists that have lead to big losses of money and trust to companies?

Once the benefits and what the company might lose by not implementing DevSecOps are understood, it's important to start shaping up a plan for this implementation. Not necessarily from the technical perspective, but about how to start spreading this culture in a natural and organic way through the organisation.

Starting from the top down is a great strategy, this means having team leads understand the benefits of implementing a DevSecOps culture and if they are great leaders people will listen and understand just like they did.

Do you have DevSecOps afficionados in your company? Let them prepare a workshop for the other people in the company. Hearing someone passionate about a topic speak about it, is very different than hearing someone that has been "mandated" to do so. The passion of someone speaking of something they like spreads like wildfire.

It might be hard, but it can be done. And you have to be sure you enforce the culture every time you can do so. This might mean little things like not allowing a team member to bypass a failing SAST scanning...

Learn to say no, it's also how you build culture.

Fighting accumulated findings

One thing is guaranteed...there will always be security debt, always.

In an ideal world you would mitigate every single finding that shows up. But let's say you're introducing a new scan to the project you've been working on for a year or so. Imagine the amount of accumulated security debt in the form of vulnerabilities that you're going to find...yep.

Will you be able to allocate immediate time to deal with those? Probably not, and over the years I've learned that knowing how to manage these problems is truly a fundamental skill.

So how do you deal with this situation?

You need to have a process in place. You won't be able to allocate time to fixing these vulnerabilities, so you need a place to start. So let's discuss the concept of baselines.

A baseline is like a place to start, it's a deal you make with everyone and yourself where you recognise how many vulnerabilities are in your service and you make an agreement around it. That agreement comes in the form of numbers.

Let's say you cannot allow Critical vulnerabilities to enter your main branch, so those ones need to be immediately dealt with, no questions asked. Then you have 5 high vulnerabilities, 8 medium ones and 14 low ones, for example.

So your baseline is:

No Critical vulnerabilities are allowed;
5 Highs;
8 Mediums;
14 Lows;

This is your baseline, which is just a starting point. And from this point on you agree that you are working over these numbers and acting upon them. This allows you to not completely stop the work you have to do in order to mitigate vulnerabilities, and at the same time you still make a commitment with security as you acknowledge what issues are in your code and those numbers have to be kept compliant with the baseline.

And this last point takes me to...

How to deal with security debt.

Just because you won't immediately deal with all of those vulnerabilities doesn't mean you can cover your eyes and pretend they are not there. Absolutely not!

You still need to tackle them and that requires building a well defined process around it.

First, you need to create mitigation timelines. Let's say that:

Critical vulnerabilities have to be dealt immediately, so stop trying to align that div to the left and go fix all your criticals;
If a new High vulnerability is found you have to deal with it in 5 working days - Allocate time in this sprint to do so;
If a new Medium vulnerability is found you have to deal with it in 15 working days - Do we have time to tackle it this sprint? If not, maybe we have some leeway to do it during the next sprint;
If a new Low vulnerability is found you have to deal with it in a month - More time flexibility to fix due to lower severity;

This is just an example.

Once you have this in place you can start looking into that baseline. You now know how to act when new vulnerabilities appear over the ones you have acknowledged in your baseline, now we have to start lowering those baseline numbers.

This comes with commitment, so make sure when you are planning your sprint, you allocate time for those vulnerabilities on your baseline.

This can come in the form of:

We are tackling 2 High's of the baseline in this sprint (depending on the estimation of each High vulnerability you can fully focus on dealing with High's first, as you should);
We are tackling 5 Medium's in this sprint;
Maybe we can leave Low's for last;

And then when you start acting on these, start lowering those baseline numbers!

When to break the build

Another important choice to make is when to break the build, but let's take the chance to build upon a concept that we just talked in the previous "challenge", building baselines.

Once we have a baseline in place we always have clear numbers to work with and easily define when we can break the build. We set up that Critical vulnerabilities have to be dealt with immediately, so if you find even just one, break the build!

We have currently 5 high findings in one of our services and we defined that number as the baseline, once a sixth high finding is found, break the build!

And once you start dealing with security debt and lowering those baselines you also can adapt when to break the pipeline in a consistent way.

Embrace breaking the build

Everyone, DevOps engineer or not, loves a successful pipeline, to look at GitHub Actions or Jenkins little steps and seeing them all in beautiful green colouring. There's a hidden pleasure there, right? It can't just be me...

But we have to mention this here, because at the end of the day it's just as important: We have to embrace breaking the build.

We need to embrace it, accept it just as a natural step, because it is. Of course it means more work is about to come, but it also means our doings are working, we're enforcing security, we're creating culture and accepting it just as another step on the software development process.

Then by the end of it we can rejoice as all the pipeline steps have been successful!

Conclusion

To finish it up and even though I tried to approach things in an objective way, this guide is not a one size fit all kind of thing. There is not a definitive answer as use cases are very different from one another and there is not a one true path to integrate DevSecOps. My goal was to show you the main concepts, the tools and an objective way to think about the implementation.

Nowadays security can't be pushed aside, so embracing DevSecOps in your development workflow is the best way to make sure you deliver the best, more secure software and do it in a fast way.

Once you realize what you need to secure in your product it's a matter of starting integrating slowly each different scan. Here you will need to see where each one fits the best in your workflow but no matter how complicated you have things setup there's always a way to do it!

The day to day of a DevOps Engineer (or whatever you want to call our species) is based on adapting and learning on the go, so grab that bull by the horns and get it done.

Liked this article? Here's one that might interest you: