Salesforce Migrates DevOps to the Cloud with CloudBees CI

Since its founding more than 20 years ago, Salesforce has emerged as one of the world’s top Software as a Service companies. Known for its market-leading customer relationship management (CRM) platform, San Francisco-based Salesforce has successfully broadened its offerings to include cloud-based solutions for customer service, marketing automation, analytics and application development.

Thousands of software developers and business users build powerful Salesforce-based enterprise applications using a platform called Force.com, part of the Salesforce Customer 360 Platform. Salesforce’s IT organization, through its business technology group, helps manage the development platform.

The group’s DevOps team plays a key role in the development process. “We make sure that developers are equipped with the latest technologies and tooling,” says Aaron Nassiry, DevOps engineer, with Salesforce’s business technology group. “Our goal is to let the developers focus on their core skillsets and features and get the immediate feedback they need.”

Migrate DevOps To The Cloud Challenge

Salesforce’s fast growth has encouraged more developers to create applications on the Force.com platform. Currently, the group works with about 1,200 developers across 60 to 70 teams, putting pressure on the DevOps team to keep developers supplied with the latest tools and techniques. “There are all kinds of tools coming into the market all the time,” says Nassiry. “Our job is to make sure we’re utilizing the right technologies and providing the right guidance to our scrum teams.”

One of Salesforce’s important software development tools is Jenkins. For years, engineers used CloudBees Jenkins Enterprise to automate and accelerate code integration and delivery. Initially, development work was hosted on the group’s own datacenter and used by about 70 development teams at the time.

Aging datacenter

But the on-premise datacenter environment wasn’t ideal. The underlying server infrastructure – composed mostly of Linux virtual machines (VMs) managed by Puppet – was more than five years old. Changes were increasingly difficult to make, sometimes taking up to a month. “You’d basically have to work through 100 different steps to make it work right,” says Venkat Kothapalli, deployment engineer with Salesforce.

Adding capacity was also tough. To keep up with demand, administrators were forced to add more VMs and that meant more management overhead. Other inefficiencies made scaling the system difficult. For example, development teams were stuck running static Jenkins agents that needed constant maintenance. And since the agents were running continuously, they consumed significant energy and other resources, keeping costs elevated.

The growing maintenance burden took a toll on administrators and potentially made it hard to recruit new team members. “That’s not how you attract the best talent,” says Nassiry. “Good developers, good administrators like to add value or do exciting work, so you can’t keep good people around if all they do is maintain 30 or 40 VMs. It’s always our goal to work smart and make sure our teams are doing meaningful work.”

Solution

Destination cloud

The situation began to turn around after Salesforce directed its IT operations to leave its on-premise environment and move to the public cloud. “It just didn’t make sense anymore for us to run a datacenter and do application hosting and everything else by ourselves,” says Nassiry.

Instead of “lifting and shifting” their datacenter to the cloud, the DevOps teams chose to adopt cloud native services – namely, Amazon Elastic Kubernetes Service (EKS), a managed service that makes it easy to deploy and manage containerized applications at scale. The choice was fortuitous because CloudBees – the DevOps solution the team had been using for software delivery automation and management – also offered Kubernetes integration and support through its new CloudBees CI solution. “It was like a match made in heaven,” says Nassiry.

Migrate DevOps To The Cloud Experience

At first, the prospect of migrating from its on-premise Jenkins environment to CloudBees CI on Amazon EKS seemed a little overwhelming for Nassiry, who didn’t have deep experience with AWS. But he quickly climbed the learning curve with the help of his teammates and timely support from CloudBees. “There were times that I was personally stuck and CloudBees was able to guide us to the right path,” he says. His advice: “Take the plunge. Don’t be afraid.”

Today the vast majority of Salesforce’s Force.com development community builds applications in the cloud using CloudBees CI on Amazon EKS. Moreover, as part of the move, the DevOps team also codified the entire CI/CD platform, simplifying management of software pipelines.

Crucially, the switch to CloudBees CI allowed the DevOps team to leverage efficient on-demand (versus static) agents, saving the time and cost of maintaining dozens of always-running agents. “This eliminated one of the headaches we had to deal with using traditional Jenkins,” says Nassiry.

Faster, less costly upgrades.

The switch to configuration as code with CloudBees CI on Amazon EKS has helped the DevOps team complete system upgrades significantly faster. Previously, when the team needed to update its 1,000-developer CloudBees platform, the job took about three weeks of manual testing, enlisting the efforts of four team members. Since moving to CloudBees CI, a single administrator takes just two to three hours to do the same upgrade. The efficiency boost has enabled the DevOps team to upgrade the platform multiple times a year, giving developers earlier access to the latest functionality, plugins and security features. “All of our plugins are configured as code,” says Nassiry. “We don’t need to go to the operation center and download plugins and install them. It’s all through code. We are much more confident now to do upgrades and it takes very little time for us.”

Driving innovation with Kubernetes

The team’s move to AWS has helped support the widespread adoption of modern containerized applications. “Kubernetes is our go-to platform for deploying new applications,” says Nassiry. “That’s a huge innovation which was made possible by moving to CloudBees CI on AWS.” Developers get easy access to new technologies and utilities by pulling Docker images into the organization’s security-scanned repository. “Developers can now leverage a lot of utilities out in the wild and securely bring them to our platform and use them right away,” says Nassiry. “They don’t need to go through the DevOps team, which accelerates innovation.”

Security and compliance

Like most major companies, Salesforce puts a premium on security and the DevOps group’s policies and processes reflect this priority. “Getting to production is important, but getting to production in a secure way is more important,” says Nassiry.
“It’s built into our culture.”

Security safeguards are embedded throughout the group’s cloud infrastructure, which leverages VPNs, two-factor authentication and security certificates among other measures. Every line of code is scanned for vulnerabilities. The fact that CloudBees easily integrates with all the leading security scanning tools and plugins helps enable faster development lifecycles.

What’s ahead

The DevOps team is looking at other security initiatives, including adding CloudBees Role-based Access Control (RBAC) to create an extra layer of security between teams, and bringing the code-scanning solution SonarQube into the pipeline.

Just ahead, the team plans to standardize its CI/CD workflow as part of an enterprise-wide pipeline for the entire business technology group. “The idea is to have the enterprise standard built into the organization so we can deliver CI/CD tooling and processes in a very fast and efficient way,” Nassiry says.

Focusing on what matters

Salesforce developers readily embraced the move to CloudBees CI on AWS. “It’s given them a lot of confidence in us, that we provide solutions and services of this caliber,” says Kothapalli.

“CloudBees has enabled us to move to that next level.” The cloud platform has also proven to be solid and reliable. “At any given time we have 20 or 30 builds running 24 by 7 and it’s all handled predictably in a very stable fashion with CloudBees CI,” says Nassiry. The platform’s dependability allows the DevOps team to spend more time on improving the developer’s experience. “With CloudBees CI up and running, you almost don’t have to monitor or worry about it. It helps the developers keep doing business as usual despite being remote or despite all the other interruptions they might face,” says Nassiry. “It means we can focus on building new features and new services to make the developer’s life easy within Salesforce.”

Results

Cost savings post DevOps transition

“Switching to CloudBees CI meant we no longer needed to maintain 35 to 40 static Jenkins agents. This has made tremendous sense in terms of time and cost savings,” says Aaron Nassiry, DevOps Engineer.

Productivity boost

“We used to have four people dedicated to one upgrade over a three-week period. That’s just not scalable and we couldn’t operate that way for long. After migrating to CloudBees CI on Amazon EKS, we can do the same upgrade with one person in just two to three hours. That’s a huge improvement,” says Aaron Nassiry, DevOps Engineer.

More frequent upgrades

“Before moving to CloudBees CI on Amazon EKS, we upgraded once in two years. That’s how painful it was in our old system. Now in the new system, we’ve upgraded three times in the last six months, giving our developers the latest plugins, latest changes and latest security features,” says Venkat Kothapalli, Deployment Engineer.

Attracting talent

“Good developers and good administrators like to add value and do exciting work, so you can’t keep good people around
if all they do is maintain 30 or 40 VMs. It’s always our goal to work smart and not just have our people do busy work. Moving from on-premise to the cloud helps us achieve that,” says Aaron Nassiry, DevOps Engineer.

Easier customization

“By running CloudBees CI in the cloud, we could move from a monolithic system to one that allows domains to be segregated on different masters, so that changes on one master don’t affect others. You can actually customize each of these masters to their own specific needs,” says Venkat Kothapalli, Deployment Engineer.

Business continuity

“With CloudBees CI up and running, you almost don’t have to monitor or worry about it. It helps the developers keep doing business as usual despite being remote or despite all the other interruptions they might have,” says Aaron Nassiry, DevOps Engineer.

Predictability

“Today, each developer gets the same Docker image every time they run a build, so developers always get a clean, predictable image. This helps with a lot of issues we were seeing before, where files would buildup in an agent or an agent became out of sync or some files were missing in a library,” says Aaron Nassiry, DevOps Engineer.

Timely support

“I want to give a shout out to the CloudBees team. There were times that I was personally stuck and CloudBees was able to guide us to the right path,” says Aaron Nassiry, DevOps Engineer.