In the late 1990s I worked on the security team for Internet Explorer. In fact, I was the first hire that Microsoft made in response to an influx of browser-based security vulnerabilities. I got to see what it looks like when a development team is bombarded by security problems that are serious enough to require a response and yet there’s no process to handle it. In the early days we would get at least one new vulnerability each week. The cost to respond was over one million dollars - per vulnerability! Every time it happened, we had to stop development, understand the problem, understand the fix, test the fix, and then release to customers. A team of over four hundred developers and testers was stopped in their tracks on a regular basis. You can only do that for so long before you realize that something has to change. On the Internet Explorer team we developed a new set of processes, skills, and tooling that allowed us to rise to the challenge. We did what we had to do to solve the problem while under a constant barrage of enemy fire. In the end, we built and scaled an application security program that worked not only for Internet Explorer but for the rest of Microsoft product development as well. Today that process is called the Microsoft SDL and Trustworthy Computing.
Everybody has the right to use software that they trust to keep their sensitive data safe and to keep them personally from harm. I’ve spoken with many hundreds of companies over the years, and I’ve never talked to anyone who wanted to cause harm to their users. And yet, users continue to be hurt by the technology they depend upon. If development teams consistently have the best of intentions, why does this harm continue to occur?
I used to think that we could treat security like performance, or reliability, or any of the other quality measures of good software. I realize now that I was wrong. Security is fundamentally different because it is the only aspect of quality in which there is an adversary on the other side with whom you are competing. If you win, your software runs as intended. If the adversary wins, your business pays the price.
An application security program is your best chance at fighting back. It is a disciplined software engineering process in which you use every tool at your disposal to multiply your odds of being able to get trustworthy software into your users hands. There are more hackers out there than you can hope to match in numbers and they have seemingly infinite time to look for defects. You cannot hire enough developers and security professionals to compete in quantity, so you have to make it up elsewhere. What you have that they don’t is discipline, process, tooling, and knowledge. An application security program builds upon each of these areas and maximizes your effectiveness while minimizing your cost.
The first hurdle to get over is justifying the cost. You may know that investing in trustworthy software is worthwhile, but you have to convince the people who own the budget that this is money worth spending. If you look at it through their eyes, it is a tough sell. Every dollar that you spend on building security into your product is a dollar that you don’t get to spend on new features, testing, product management, marketing, sales, and a multitude of other areas that can be tied directly to generating more revenue for the company. Not only that, but security can sometimes run counter to user experience. There are some features that cannot be done, or at least not in the way intended, without compromising the security of what you are developing. So not only do these dollars get pulled from other worthy areas, sometimes you are spending more to do less. How do you sell that internally?
It’s easiest is if your company has already fallen victim to a major breach and has seen the cost of being vulnerable. Once bitten, twice shy as they say. Failing that, you can point to a competitor that has been hit and draw the obvious parallels. Better to learn from other’s misfortune before it befalls you. If a good example doesn’t exist, or you need more ammunition, then it’s time to talk about all the reasons why we should care about security in the first place. We care about our end users, and we care about their data. Usually because there a legislative compliance or standards requirement we have to meet. Other times it is because losing user trust and business reputation is much easier than regaining it. It is also true that budgets and schedules have a habit of getting destroyed by security vulnerabilities. Short of a full breach, even a public disclosure by an interested security researcher is enough to require a response. If you haven’t invested in security, your product will be at risk of being delayed, sometimes indefinitely, by a sudden vulnerability and the need to respond to it.
I know from experience, that once a security researcher discovers that your codebase is full of vulnerabilities, many more will descend upon you like vultures. A well-secured codebase may serve up a surprising vulnerability or two that must be responded to, but an insecure codebase can result in hundreds of vulnerabilities delivered to you over the course of months or years. The pain can be intense and unceasing. I’ve experienced this firsthand while working on product development teams, and in every case, the end result is an application security program to try to stop the bleeding. You can proactively build your program before you need it, or you can be forced into it in a trial-by-fire.
The good news is that the path to an effective application security program is well known. What not to do is also well known. I’ve seen many companies try to take shortcuts to reduce the perceived disruption to their development organization or to save money. In the end, the shortcuts merely delay progress. The most common shortcut is to buy an automation solution from a tool vendor and believe that the problem is solved. It’s an easy mistake to make since buying a tool feels like checking-the-box. The tool vendor probably promised you that it was a complete solution, and in many other technology areas buying and deploying a tool is all you need to do to solve major problems. Unfortunately, in application security, tools play more of a supporting role than a primary role and cannot, alone, replace the need for a full application security program.
In part two of this series, I’ll talk about what I’ve seen work and how you can successfully build a scalable program of your own.
Read Part Two of this series here.