Part 2 - From Gates to Responsibilities

This is part of a multi-part blog series, if you haven’t already, please check out the first post:

Part 1 - My First 100 Days in ProdSec at a Series E Startup

In my first blog post I discussed how I found Highspot and what attracted me to this company. I discussed the immediate challenges I faced as I scaled up my own knowledge and tried to rapidly snap to the new culture and demands of my role. If you haven’t read that already I recommend starting there and coming back.

In this post I’ll explore the shift in security thinking from security gates in which the security team is viewed as a bottleneck to software delivery, to making security everyone’s responsibility through training, culture, and support from the security team. This shift changes the security team’s relationship with the rest of the engineering team from an “enforcer” to an “enabler” which is better for everybody!

When I started in security I was lucky enough to be at Microsoft as they launched the Trustworthy Computing Initiative, nearly 20 years ago. At that time the best in class security guidance was to control the flow of risk through gates (not Bill) — a set of structures and checkpoints that were there to make sure vulnerabilities weren’t slipping through.

Want to suggest a new feature or change an existing one? Get a Threat Model from the security team.

Want to check in code? Get a code review from the security team.

Want to ship your software? Get a security assessment and sign-off from the security team.

I was one of the security leads, and it was my job to sign off on our product before we shipped the software to customers. Knowing that most developers and testers did not have deep security knowledge and that it was my job to find and remediate all the security vulnerabilities in that product was stressful! I did not sleep well in the weeks leading up to our ship date.

Even if these security gates worked, they were not efficient. They made the security team the bottleneck for anything security related. It shifted all the responsibility from the engineers building the software to the security team, which leads to a “guess and check” style security program that is sure to fail. Further, this was designed with Microsoft’s ancient 18 month waterfall process in mind. It would never work in today’s Agile development landscape.

Be that as it may, we are still trying to shoehorn these ideas into the modern development process, and we’re failing to be successful. Even with high-level concepts like shifting left we are often not giving engineers the tools, guidance, and training they need to have the agency to make great security decisions for their own features.

In order to give engineers agency over security they must be empowered to make decisions through training and feedback and they must be responsible for those decisions. The security team exists to support and guide those decisions, not to make them. This is the only model that I know of that will scale with the organization. Unless you plan on hiring one security engineer for every developer you hire, which would be neither reasonable nor successful.

Training

Application security awareness starts with a baseline set of security training. This is where developers learn their role and responsibility in building secure code, attack scenarios, and preferred remediation and defenses. Nobody knows your codebase better than your developers, arm them with the knowledge and passion to build secure software.

Highspot has one of the best new-hire onboarding processes I’ve seen. Your first week is spent learning about the industry and product, then you have a deep dive into the specifics of your job for the rest of the month. If you’re an engineer that means understanding the tools, process, and best practices we use. We use this as an opportunity to make sure every engineer has taken our Security 101 training in which we give each engineer the knowledge they need to understand risk, common vulnerabilities, and where to reach out for support.

After Security 101, engineers have the opportunity to increase their knowledge by participating in fortnightly security Office Hours. During our office hours we answer pressing questions from the engineers and when we run out of questions we deliver live training. This office hours training can be formal training that will be eventually rolled out to the rest of the engineering team or it can be more specific or narrowly scoped based on a new Highspot initiative or even a news article or breach from another company. It’s always important to learn from your mistakes, but it’s even better when you can learn from somebody else’s!

Support

The security team must provide support to the rest of the engineering team to be successful. This means being responsive to requests and doing the work needed to provide clear guidance when needed. Support can take many forms, from the training mentioned above to reviews, to library or framework choices, or recommended remediation.

I started at Highspot during the pandemic, which means that I didn’t have the opportunity to meet many of my colleagues face to face. Without that I found it difficult to build trust based relationships and without the relationships my interactions were more transactional. Transactional relationships can be fine in some situations, for example if I need to ask the IT team for a new account on a system somewhere I don’t need to leverage my relationship there. But to build an environment of security culture engineers have to feel comfortable asking questions and pointing out flaws.

Security can be scary, oftentimes I’ll find flaws in systems that will create work for others. If that request doesn’t come from a place of mutual trust, respect and goals people will quickly stop coming to my team for help. A huge part of being able to support the engineering team is for them to feel comfortable coming to the security team with questions or requests. Building those trust relationships early through frequent interactions will pay dividends later.

I accomplished this by having a lot of one on one zoom meetings, turning on my camera, speaking up, and asking questions during meetings.

Tooling for the long haul

In order to sail across the ocean safely we need the tools and automation to make security easy. This means implementing and integrating secure defaults, libraries, frameworks, tools, and automation. Each of these play a different role in the implementation of a secure system and detection when things slip through the cracks.

Defaults

Creating secure defaults for new systems drastically reduces the number of mistakes that are made. Having a set of secure defaults also gives you a place to update when mistakes slip through. Are you seeing a lot of unencrypted S3 buckets? Set encryption as a default setting whenever new S3 buckets are created. Are you finding a lot of OS related vulnerabilities in your environments? Create a secure golden image with secure and sane defaults set. Whenever an issue comes up ask yourself if that could have been mitigated by a better default setting somewhere. Make the change to mitigate the immediate issue and update the defaults so you’ll never see that issue again.

Libraries & Frameworks

When was the last time you saw a CSRF or Session Fixation issue? If your answer to that question is more than a year or two you can thank the major improvements that libraries and frameworks have made recently. Picking up to date libraries and frameworks and using them properly can drastically help increase the overall security while decreasing the overall impact security has on the velocity of the development process.

It is very important to use the libraries and frameworks appropriately, though. Many of these libraries ship with secure defaults, which is great, but they may not be drop-in replacements for insecure existing libraries. This means that sometimes an engineer may simply disable the security controls to get something working, rather than spending time understanding the new systems. This is where the security team can step in to help establish best practices and provide support and training. Additionally we have monitoring in place to detect many mistakes and do regular assessments to minimize the time these issues are live.

Tools

Depending on where tools are deployed and integrated into the software development process they can be more or less effective at mitigating risk, providing feedback, or possibly introducing developer frustration. Highspot uses a lot of tools for different purposes. The security team runs SAST and DAST tools frequently to look for newly introduced vulnerabilities. You may choose to have the security team run the tools periodically or to build them into the build or integration pipelines. Highspot chooses to have the security team run the tools so we can triage each issue and provide directed and specific remediation guidance. This helps drive down noise for the developers and keeps the CI/CD pipelines running quickly and smoothly. We do integrate linters with security rules into our CI/CD pipelines to make sure engineers are adhering to best practices.

Tools can be a great way, not only to help catch vulnerabilities, but also to keep security in the forefront of each engineer’s mind. Reminders from the linter or SAST/DAST tool can remind engineers that security is important and there is a level of code quality needed for acceptance.

Automation

I love the phrase “Automate it the second time.” If there are tasks that are done frequently they should be automated. This will help the security and engineering team scale and respond to new and unique challenges. We have integrated and automated most of our tools to work together, our dependency checker, and other tools are directed to our bug tracker for triage. Tools are integrated with Slack and email for visibility. When a vulnerability is fixed it is turned into a unit test to ensure regressions don’t creep in later. We will also automate the discovery of vulnerabilities whenever we can. Checking for things like TLS configurations, cookie flags and headers is a waste of time if done manually. Automate that, and integrate it into the tools you use!

Summary

A lot has changed in the 20 years I’ve been in security. We’re not having to convince each team that security is important as much any more and the way we think about working with each team has evolved as well. There is a lot more collaboration between security and the rest of engineering now and a lot of that has come from the focus on support, enablement, and tooling that makes the security team a partner in delivering great software instead of an impediment.

In the final post we’ll explore more of my own mistakes and assumptions in my first 100 days, and how I was able to shift my thinking to be able to meet the needs of a new team and a new company.

Continue Reading Part III Here