In my last blog post, I wrote about what an application security program is and why it matters. In this post, I’ll cover what it takes to build and scale an effective application security program.
I’ve seen many different ways that a well-intentioned program can fail to meet its objectives. While there may be many ways to fail, there are just a few key characteristics that lead to success.
The program must be:
1. Tied to the business.
2. Able to measure itself.
3. Able to improve over time.
That’s it! All the rest of a successful program lies in a set of details that you can figure out and optimize as long as you have this framework in place. And if you are wondering about these details, rest assured, I’ll cover that in future posts. For now, let’s focus on these critical strategic pieces, because if you don’t get the strategy right, nothing else matters.
The Program Must be Tied to the Business
The security program must justify the money spent. The only way for this justification to work is if the program can be tied directly to a set of business goals that support it. These goals vary from business to business, but they generally take the form of security policies written at the C-Level and agreed to by the board of directors. By their nature, these business goals will be non-technical, abstract enough that they require some level of interpretation by engineering teams, and they will be generally non-negotiable. They will nearly always focus on legislative compliance, globally acknowledged standards, customer requirements, or all the above.
If you cannot tightly tie your application security program to a set of corporate business objectives, your program will fail. This is one of the first questions I ask when talking to a new customer, and when they don’t meet this test, I think of it as a zombie program. There are still things I can do to help, but until the dynamic changes, the application security program is going to be like the walking dead. Acknowledging that fact can be a hard, but necessary first step toward improvement.
The Program Must be Able to Measure Itself
A successful security program must be able to measure itself and demonstrate value through a set of KPIs or metrics that are agreed upon, can be accurately measured, and are improvable over time. Security KPIs can be hard to come up with, especially for application security, but that doesn’t make them any less critical.
The first step is to have a discussion with your stakeholders in terms of what KPIs matter to them. For your executive management, it’s probably going to be some form of measurement against the corporate security policies. In other words, are you meeting the business objectives that are driving this program or not? If you aren’t, can you demonstrate progress toward that goal? For your engineering team, it’s probably going to be a more technical measure, such as the number of vulnerabilities discovered and fixed over time, the time it takes to fix a newly discovered vulnerability, the amount of time and money spent on security by members of the team, the number of externally reported vulnerabilities, security engineering activities achieved, penetration testing and code review coverage, and more.
Once you’ve established a set of KPIs, you’ll need to determine which you can reasonably hope to measure now vs. those that need to wait for the future vs. those that may never be achievable. It’s ok to have identified a metric that you cannot use because it is too expensive or impossible to measure. There is value in knowing it matters, even if you cannot formally measure it. But for those that you can measure, start doing so as soon as possible. The more data you have, and the longer you have to identify trends, the better. Every day you miss is data that you could have been using to improve your program and that is now lost forever. A word of caution is warranted here. Every metric you choose to include in your KPIs will take on a life of its own. Once you have a metric, the team will be motivated to drive its improvement. Think about side-effects and unintended consequences. It can be tough to remove a metric once it has been established, so take the addition of each one seriously.
The Program Must be Able to Improve Over Time
Once you have established and started tracking your KPIs, improvement may seem so obvious that it needs no mention, and to a certain extent, that’s true. Engineering teams like to optimize, and once you give them numbers to drive, they will almost always get to work doing so. There are some exceptions, however, that are worth calling out. Each of these exceptions represents a hurdle you’ll need to get over before your program can be successful.
I mentioned in the previous section that you should get agreement from the stakeholders, but I think it’s worth digging into this a little further. I implied that you need agreement from your executives on the executive level KPIs and from your engineering team on the engineering KPIs, but that’s not the case. You need to get agreement from the engineering team on both the executive KPIs and the engineering KPIs. Since the executive KPIs are tied to business objectives that are probably non-negotiable, that may seem difficult. A common mistake is that executive KPIs are not discussed with the engineering team other than to roll them out from on-high as an edict (Thou Shalt). The faulty assumption that some people make, is that since the business objectives are non-negotiable, the related KPIs don’t deserve discussion. That couldn’t be further from the truth. An engineering team will not fully commit to a goal that it doesn’t understand. If the business objective is real and the KPIs are tied to that, then it is a discussion worth having. Once the engineering team understands these KPIs, they will agree to them. Once they have agreed to them, they will commit themselves to optimize them. On the other hand, if they do not understand and they do not agree, you may get nods of the head, but you will not get a full commitment. Instead, you will see ‘check the box’ behavior where a minimum of effort is put in. When a customer comes to me and they list engineering team behavior as a problem, I usually uncover a buy-in problem. Getting buy-in from the engineering team doesn’t have to be hard, but it does have to be done.
The other common mistake is to get agreement to drive the KPIs but fail to equip the team to succeed. It is not enough to load an already fully loaded team with more work and expect them to make it happen. If you are driving new KPIs in order to achieve a successful application security program, you need to fully fund it. Fully funding it means you need to provide enough work cycles to get it done, you need to purchase the tooling necessary to improve efficiency, and you need to provide training to ensure the team knows how to get the job done.
Notice that this is the first time I’ve mentioned tooling. Don’t make the mistake of buying tools before your application security program is defined. Tools add efficiency to a program that is already in place. They cannot, and will not, provide program definition for you. Anyone who tells you otherwise is trying to sell you something.
—
Once you’ve taken these important steps, you are on your way to a successful program. You’ve defined a good set of requirements, you’ve agreed upon how to measure success, and you’ve set the team up so that they can iterate and improve over time. Once you’ve put this virtuous cycle into motion, you are all but guaranteed a successful outcome. You will start with an immature program, but over time you will achieve full maturity and become an example of security best practices that others can look up to and will want to emulate. Now get to work!