More and more enterprises are adopting a DevOps approach to developing software applications, creating continuous integration / continuous delivery (CI/CD) pipelines to streamline and speed up the app development process, with continuous testing (CT) to make sure it’s all being done correctly.
However, many companies are finding out the hard way that the DevOps process can potentially create security vulnerabilities in enterprise apps. This has led to the rise of “DevSecOps”, in which developers not only integrate security into apps and infrastructure design from the start, but automate certain security gates to make sure the CI/CD workflow isn’t slowed down.
This is particularly crucial in the financial service sector, where banks and financial institutions are undergoing digital transformation to keep up with competition from agile fintech start-ups, but absolutely cannot afford to skimp on security. At the RSA 2020 Asia-Pacific and Japan virtual conference in July, several financial services companies shared their DevSecOps experiences and offer tips on how enterprises in other sectors could also ensure their CI/CD pipelines aren’t creating future app security headaches.
DevOps security challenges
Like with any security posture, the place to start is to understand the security issues that can arise in a CI/CD pipeline, and why they arise to begin with.
Nachu Subramanian, Head of the DevOps Engineering Center of Excellence at OCBC Bank, says that key security challenge of DevOps is the fact that it speeds up the development process so much that the cybersecurity teams often struggle to keep up, especially when the DevOps team is bringing in new tools and new technology stacks.
“Also, when [the DevOps team is] trying to build up their CI/CD pipeline with various new components, the cybersecurity team needs to catch up with them and ensure that they are able to add security to all those DevOps components as well,” Subramanian says.
This is exacerbated by the fact that the DevOps teams often neglect security while setting up the CI/CD pipeline because the whole point is to accelerate app delivery. This is the benchmark by which DevOps justifies its existence, which means DevOps teams tend to put a priority on maintaining velocity over processes like security that might slow the pipeline down, Subramanian says.
Other challenges include the use of free and open source tools and third-party libraries (which makes life easier for developers but may have their own security issues), inadequate controls or processes for risk and compliance, and the fact that solution architects must keep up with a complex and ever-evolving security architecture where even a single new component or change can create potential app security problems.
Four pillars of DevSecOps
Subramanian presented four essential pillars of DevSecOps for enterprises to adopt for their DevOps journey, starting with secure architecture design, which requires security to be implemented at multiple architecture layers: campus, branch, data center, internet, edge and WAN.
Subramanian says the data center and internet architectures are especially crucial for financial institutions – data centers because of the need to securely transfer data to and from public and/or private clouds, and internet connectivity because of the data transfer between financial service providers and their online customers.
The other three pillars Subramanian offers are:
- Secure code and binary review: Developers should implement secure code reviews either while it’s being written or after it’s placed in a repository, followed by a binary review to catch any security threats that the code review might have missed (especially when using third party code) and a security review of free and open source tools
- Container and cloud security: ensure that any containers you use are secure, even during the runtime, and develop a plan to quarantine and shut down non-compliant containers gracefully without impacting the end users
- Continuous compliance: Enterprises must ensure that their DevOps team is complying with the above security checks at all times.
This last pillar is important because security is not something you install up front and then forget about, Subramanian says. “You need constant security reviews do make sure this is being done continuously. We have to keep our compliance standards and compliance policies updated all the time.”
Moreover, compliance reviews should be automated as much as possible, he adds. “For example, let’s say my compliance standard says that when I’m pushing my code to the release branch or to the master branch, I need two or more reviewers to be there. We shouldn’t be manually checking whether at least two people have already reviewed the code – we need some automated tools for reviewing this for ensuring compliance.”
Subramanian says further that all compliance standards and policies should be measurable and quantifiable. Another key element is near real time validation and verification to catch vulnerabilities on the fly and immediately report them.
Creating DevSecOps tracks
In a separate talk, Mohan Yelnadu, Group Head of Application Security at Prudential, talked about his company’s DevSecOps journey in similar terms, beginning with the specific attack vectors he was facing in terms of apps development.
Yelnadu explains that Prudential’s security focus starts at the coding stage, including custom code and open source libraries, noting that he expects the latter to become more prevalent over time.
Yelnadu also cites container security as a significant challenge to app security because there are several different layers that each have their own potential vulnerabilities.
“When [the app] is hosted in a container, the container itself has a base OS, and there are multiple components stacked on top of the base OS, then you have your application binary running on top,” he explains. “So vulnerabilities could exist any of those layers, as well as the container orchestration platform, whether it’s EKS or OpenShift or whatever.”
Because there’s no single tool or ‘silver bullet’ that covers all the vulnerabilities in different layers across this attack vector, Prudential has developed four different DevSecOps tracks that focus on different areas:
- Static application security testing (SAST) for custom code
- Software composition analysis (SCA) for open source libraries
- Dynamic application security testing (DAST) for both
- Container security (CSec) for all the container vectors, including the orchestration platform
Yelnadu also walked through the steps Prudential had to take to develop those tracks in the hopes that other enterprises could get a head start rather than start from scratch and figure everything out the hard way.
The obvious starting point: identify your organization’s requirements for each track based on your own technological landscape. “Analyst firms, consultants and tool vendors publish guidelines for establishing DevSecOps tracks, but they won’t always reflect your specific technological environment, so be aware of that,” he advises.
Next: determine which tool vendors fits in your environment, choose your tools accordingly and integrate them at all strategic locations in the software development lifecycle. This is especially important for medium and large organizations whose business units are spread across different geographies, as opposed to small businesses whose developers may be all in one location, Yelnadu says.
Also, he adds, create a process and publish it so the development teams understand how the tools can be used, and set up a documentation ecosystem such as FAQs and short tutorial videos to educate stakeholders.
When setting up a triage process, false positives are an obvious and useful criterion, as most automated tools tend to generate false positives, Yelnadu says.
Yelnadu also recommends to create metrics to measure and track improvements – perhaps onboarding metrics to start, and then issue-based metrics that track the security vulnerability issues that are discovered and determines how they should be prioritized for fixing.
One other piece of advice for managing your DevSecOps tracks: have a strategy to re-evaluate and – when necessary – replace your tools every few years. “For me, I do it two or three years – it depends on your risk appetite,” Yelnadu says. “The point is that a lot things are changing – for example, we set this up in 2019, and the world has changed since then. A lot of new attacks have come up, new exploitation techniques have been discovered. In such cases, if your tools in different tracks are not updated enough, they’re of no use.”
