CI/CD Pipelines for long term projects

Discover how you can supercharge your CI/CD pipeline with 91导航鈥檚 functionality, supporting slower-moving, longer projects. Long-term projects in Pipelines allow you to use CI/CD over extended periods and prevent long-lasting feature branches from impacting your releases. In this breakout session, Richard Owen, Senior Project Manager at 91导航, and Jonny Harris, Salesforce DevOps Engineer at Zurich Insurance Ltd, showcase:

What long-term projects in Pipelines are
The pain points that long-term projects in Pipelines resolve
How long-term projects in Pipelines work and the benefits they bring to teams
A live demo of long-term projects in Pipelines

Learn more:

Relevant videos:

Hi everybody. Thanks so much for coming on to this breakout session on CI/CD pipelines for long term projects. Please do feel free to drop any questions as you go in the chat. We do have some time set aside at the end for Q and A.

I'll try to get to as many of those questions as possible. So first, some introductions. I'm Richard. I've been the 91导航 Senior PM for three years now, and I lead a large teams group focused on making workflows easy and efficient for Salesforce teams of all shapes and sizes, but especially for the largest teams we work with today.

And in this role, I've been product manager for our pipeline functionality since its inception, working with over 400 teams ranging from teams of two to three, up to teams of more than 50 to streamline and improve end to end processes. And I'm obviously delighted to be joined here today by Johnny. So I've worked with Johnny over the last three years as he's been managing and elevating the Salesforce development process at Zurich, going from admin to developer to DevOps engineer. And over this time, we've worked together on a bunch of initiatives across 91导航 Pipelines, and it's helped give us a much better understanding of what teams need as they scale and grow to be successful.

So what are we going to talk about today? I'll start by giving you an overview of Pipelines projects. Where did it originate? What were the pain points that we've seen the whole idea of teams experiencing today?

Then I'll highlight how it works and some of the benefits that it brings to teams. I'll then pass over to Johnny to talk about how he's implemented this, the problems it solves, We'll see a live example of it in action as well. So with that, let's get started. So most Salesforce teams that we work with today have multiple different streams of work, which have different cadences.

Some of the small teams may be able to push everything through in their business as usual stream, but that is very much the exception rather than the rule. Most teams have a BAU stream and they may also have longer term projects. And these long term projects can last from a few weeks up to a year in duration. Teams may have multiple projects ongoing at any one time.

The most I've seen with any team is about 12 parallel projects. Importantly, these projects exist because there's functionality which needs to be held back from going to production until it's approved. And then it needs to be released either in slices or as a single broad block, a single product release going straight to production in one go. So what are the challenges that these teams face today?

So first off, they need to be able to isolate project work from their business as usual pipeline. Those changes shouldn't be allowed to go to production until they're ready to go. So being able to isolate them, keep them aside and then move them through when they're ready is really, really important. Secondly, they need to be able to iterate on existing work.

So if you've got a project which lasts several months, then you'll be able to work on work which is done earlier in the project. Because if you're branching from main, the project changes haven't made it back to main yet. They haven't been released to production. So we think about this in the context of the pipelines branching model.

So where all features were branched from main or master, if you've project work which isn't getting released for a while, and it's just sitting there in your staging or UAT environment, you've created the features, it's pushed it to the earlier environments, it hasn't gone through to main and pushed out to prod yet. It won't be there to build on when you're carrying on work in a later sprint, when you're creating that next feature branch. Carry that on for several months and it becomes really, really hard to build on a project release effectively. Plus you've just got a whole bunch of stuff sitting in earlier environments which isn't doing very much.

And then the flip side, this is true as well. If you're working on an isolated project stream, the branch you're working on will get really, really way behind your main branch and environment drift becomes really bad. You're likely to get a whole bunch of merge conflicts when you try and merge that together and release the product. And dealing with that close to a production release is no fun at all.

And then finally, simple PR volume is going to be an issue here as well. So imagine a six month project with hundreds of PRs in progress. Having those hanging around will get in the way of other PRs. It makes it harder to see what's in your BAU stream.

It creates a lot of noise for you. And on top of that, it just starts slowing down your Git repo that you're working with. So how do teams handle this before we developed Pipelines projects as a solution? So before this bug strategy was developed, teams had two main ways of dealing with longer running projects.

So firstly, and we see this mostly with smaller teams managing fairly simple projects, they could just load all those changes into a single sandbox we've got there, the long term project branch, paired with a sandbox, and keep building on top of this, building more and more stuff in there, and then release it when ready. Now this has a couple of main drawbacks. Most notably, it's hard for them to manage problems or things that need reversing if you've got no test environment to validate changes, or you've just got that one box, because it's really a one shot process. It's also hard to manage any Jira or ADO tickets which you're working with associated with this, you need to re associate them as soon as you create that PR going towards production.

Alternatively, you could have a second pipeline, the development project work, which ends in a branch which is referenced here. So this is similar to the case which we had previously, but instead of just being a single Dev Sandbox linked to a project branch here, that is linked to a separate sub pipeline which can include other testing environments, other dev sandboxes, to be able to iterate on that work. And then once you're ready to release it, you could create that PR from that project box into your main pipeline, and then release it through UAT and then production. Now the main drawbacks here, again, you've got a lack of ticket tracking, so you need to reassociate tickets once you create that PR.

But also, if you're using the reporting API, you're likely to miss metrics on project work. Because reporting metrics are linked to an individual pipeline and this doesn't work very well. It's quite a clunky process. So what was our solution here?

So we decided to create first class support for this sub pipeline process, but within a single pipeline. So we've created a new project object in pipelines, which has several key benefits. So first off, it allows you to create features from a private branch rather than the main branch, so that way you can iterate over changes quickly. And any changes going to production in the business as usual stream or from another project will get back propagated directly into your project environments.

So this way they take a closer understanding with BAU, it makes the eventual product release cleaner and easier. And on top of that, you deal with merge conflicts little and often as you go. This allows you to shift left. It prevents those conflicts from being a problem later on closer to production.

Then all tickets that you work on are featured in your projects, are going to be tracked through the system as are associating them with the features within the projects. And then those tickets are associated with the project release that you create. And this makes tracking changes so much easier and ensures that all appropriate tickets are covered as you push a release out. So this is quite similar to the release functionality that we have in pipelines.

We're just managing it at earlier stages of the process. And then finally, the overall PR volume is greatly reduced here. So rather than dealing with hundreds of PRs for a project, we consolidate them into the project branch and then we release that project in slices. So this reduces the PR volume, it makes the overhead far lower and easier to deal with.

It's much easier to see what's going on in the pipeline as a whole. And here's what it looks like. I've got our Git sandboxes here on the left and our main static environments, the integration, UAT and production, office environment, we've got this new project open environment which shows what branch is linked to. It shows the number of environments you have contained within that project.

Really one of those. One of these offers super easy strategy to show in a few minutes time. And this project can be joined to whichever stage of the pipeline it needs to take. So if needs to go straight in towards production, if you do that, if it needs to go from an earlier stage or into integration, it can be done that way as well.

And changes coming into production through the BAU stream will be back propagated to the project, which will help keep it up to date while you're working on it. So now let's dive into the project itself. So here we can see the project. This looks similar to that multi pipeline case that we've seen earlier on, but this is now all within the same pipeline.

So it's connected to the UAT branch, as you can see on the right. And then any features which develops in here, you develop the most part of the Dev Sandbox, you then push it through and then those features get collected up back into the project UAT branch, ready to be released towards production. And this helps you iterate on your existing work. If you've got a long project you can build up on top of that without those changes needing to go to prod first.

When the project release is ready to go, we can come back out to the main pipeline and see the tickets associated with that project which have built up. So here's where we can then create a release slice for the projects and start the process of pushing it towards production. So if you highlight the projects, you can see these are all the tickets which are associated with it. And there's where you can create the puller of static into UAT to move it through.

And once we've created that, then here's how it looks. So you can name the release, we show you what projects it's come from, and this has got all the tickets associated with it. You've got validation which is done as a block. It's a single PR validated as a block, almost like an early release sprint.

So the project name, margin it, along with any name you've given for a release, all tickets are attached. And if you need to release a second slice of that project, you can create a PR from it and it will just contain that extra work that you've done. So creating and releasing a project in slices becomes super straightforward. And with this, I'll now hand over to Johnny to talk a bit more about how he's implemented this at Zurich.

Where the project would be trying to push through, say, something like a new object and because the new feature branches were coming from main, the project branches wouldn't have those new objects in. So we had constantly just deploy the new objects, even though there might have been just a new field being added or a small change. We were constantly deploying objects, which caused quite a lot of pain points for the developers in the team. And then for me, the project to go live with with a lot of work to get it back into the BAU chain.

As Richard mentioned, you get come across all the merge conflicts, you've got outdated sandboxes, you've got outdated branches to try and work out what the right stuff is to deploy at the right time. So our kind of midterm solution, which you can go to the next slide as as we're working and discussing through, we we needed to find a way to to make it so when we merged project work to a feature branch, it had all the project work in. So it had an earlier point. We needed to make sure that the project, when it was going forward into the BAU chain, it went in as one deployment rather than lots of individual little bits going through all the time, which caused merge conflicts.

And we need it to be easy. I don't spend all my hours in my work day working through merge conflicts to try and make it get into the BAU pipeline. So we want it come across in one block nice and easy, nice and clean-cut. While those discussions are going on and Richard and team were scurling ideas away and trying to work on a solution, What we had to do at Zurich, if you go next time, Richard, we just for the interim, as Richard mentioned, we worked on having a split pipeline.

So that means, as Richard showed earlier, we had a BAU pipeline and then we had a project pipeline that was connected for a sandbox and the sandbox and the BAU would have the last branch of the project. Now it worked to an extent. It covered off some of the issues, especially around bringing through those objects at an earlier point in time. However, for myself, that meant I had extra pipelines to maintain.

So working across five orgs, suddenly you're doubling that up, tripling that up. You know, we were up to 10, potentially 10 pipelines at one time, which is hard to maintain. There's lots of manual steps to transfer across into the pipeline. And also we're unable to carry Jira stories across.

So when you were one of our projects, you're talking suddenly 100 Jira stories might be attached. We weren't going to sit there and manually attach them all again. So we lose that tracking bit going across. Go to next slide.

So once we've got long term projects in there, actually all those challenges melted away magically as if overnight. So we now have them back to having one single pipeline to manage both the projects and BAU work. So that's now less management for me to try and keep on top of all the different CI jobs and the pipelines and keep them in tune with each other. The project teams can now pull feature branches from their highest environment at an earlier rate.

And actually, big win there is seeing the benefits of the Delta comparison that 91导航 allows you to do for the Apex changes. So instead of having to deploy the whole Apex class at times, we're seeing just lines being deployed. So that also then brings in that less risk of moving through those changes. So we're not looking at a massive 50 line Apex class, we're looking at one line that's changing so we can trust the test.

We know it's what's going on. We know it's changing. And we're facing less merge conflicts and actually touch wood since we've had long term projects in the last project, I don't think I saw one single merge conflict, which is a massive win for me. Again, it's less focus on fixing merge conflicts and more focus on iterating and improving our processes.

And also just that last little win as well, the Jira stories are now being carried over. So it means now we we're actually tracking the Jira story all the way through end to end and we can keep an update with the business then aware where that story is at all times. Okay, so what I'll do now, I'll just share my screen and I'll just walk you through the kind of what we've got set up currently is Zurich, just in demo pipeline. Cool.

Rachel, just to check, can you see my screen? Yep. Great. Okay. So this is as you saw in Richard's slide, this is how it looks at the moment and this is what we're utilizing at Zurich.

So what we've got here is just a standard BAU pipeline that's working through just three environments. We got a dev environment into our UAT environment, into our production and off the side here can see our project work. So if I just click through into the project, you'll see here that I've got a currently a UX dev environment and it's got one of its own project environments inside. So this is its own sandbox that it works on.

Against this sandbox, a bit of Blue Peter magic, I've got the change ready to go. So what I'll do is I'll promote this change in. And what this will do now, it will deploy it to my branch, 91导航 summit, deploy it to the org. And what that will allow me to do is off the back of that, pull through that change into the BAU pipeline.

So say if you had five or six changes here, you deploy them all into the UAT environment that you project, you get testing, you all the sign off. As Richard mentioned earlier, that is one then one PR into the BAU pipeline. Now it would go across. A little refresh.

What you'll see then is that project work ready to be merged into the BAU. Which is there. So you see us now, project's going through, it's running some checks, we've got some approvals to be ticked that will then pull through those changes. And what is really great about the project work is just how easy it is to spin that up.

So again, in your normal pipeline, all it is is go up to add. It's just Create Project. And from here you can give it your project name. And then you can either select the branch or create a new one.

And again, the great thing here is coming from main, so you're keeping that common ancestor throughout the whole of the pipeline. So it helps with those merge conflicts. Yes, come to create that branch. And then it's just a case of create project And it's as easy as that.

So now that project shell has been created, ready to add in your CI jobs like you normally would in pipeline. That's all ready to go. Richard, back to you. Oh, well, thanks very much.

Thanks very much, Johnny. And we've got about about ten minutes left or so in the breakout session here. So happy to open the floor for if you've got any questions about this functionality or anything about projects, please do drop it in the chat and let us know. In the meantime, Johnny, how many how many of your products do you anticipate using this for over the course of the over the course of the next few months, over the course of the year?

How many how many in parallel do you have? So for us at Zurich, I I imagine what we've got at the moment, like like you see here, we've got our normal BAU chain of environments. And actually, if what we're going to start to do is apart from outside of incidents, we're then going to have a project for each of the changes going through. So they'll be kept separate until they are ready to come into the pipeline.

And then what that allows us to do is to keep a direct route to production for quick fixes, for hot fixes, for those incident changes without being clogged up by project work, which might sit halfway through and then stop it and stop moving forward. So I'll imagine us having while we could end up having five or six within one org at one time to be able to manage those changes as they go through. And I said earlier, have those slices going in when they're ready, when they're tested, when they're signed off, rather than waiting the six months to a project to an end in one big bang to try and iterate in and keep those small chunks going through.

Yeah. That makes sense. I see a question coming in from from Dylan. Yes. So do you feel as though this leads to big bang releases versus iterative updates?

What's your view on that? Yeah, so as I've just said, I think you can do both ways, right? Because I know that in some industries, small iterative changes is easier to do. In others like insurance, sometimes those iterative changes are harder because of the sign off you need a bit more red tape around it.

Both are possible with the pipelines, long term pipelines. I've I've done one project where it was smaller changes going through at a quicker rate, and we've done one where it's a big bang going live. So both work just as well as each other. Yeah.

That makes sense. I think the goal is to try and make those big bang releases where they are necessary as smooth and as painless as possible to try and reduce the drift that you have so that you're just dealing with that block of work as it is. One of the interesting things that we've seen as we're working with a lot of larger teams now is that best practice stops being so monolithic. Quite often when we're working with teams of five to 10, it was quite easy to say this is what good looks like.

But as we're dealing with a higher volume of changes going through, they all need to go into the common environments, like going into integration, going into UAT. There is a bottleneck which can come out if you're dealing with individual features. And so being able to consolidate elements into a smaller number of elements, you can validate in blocks more effectively, They may not need to stay there for a long time, but if you can push those through effectively and quickly, then it'll put you in a much better state in terms of being able to get those changes to production as easily as possible and speeding things up overall.

Just very quickly, Richard. Sorry. Yes. There's a point of proof of that. That change is now merged into that environment, into that branch.

I can now create my second release that will go alongside this, the first deployment going in. So you can see how they can cut up those deployments into bigger section or smaller chunks to move through. Yeah. That's cool.

And then, of course, if you did create multiple chunks of this and then have bring it to production, they can also be incorporated into a pipeline release in the same way as you would a group of features if they needed to go that way. Then we've got a question from Vijay here saying, How can hotfixes be handled in long term pipeline scenarios? In fact, that's a very relevant question. It's one that we've been thinking about at the moment.

Johnny, how are you dealing with the have you had to hotfix anything which is coming out of a project so far? Or has it so far been not a case that you would be looking at? We haven't had the case for that yet. Just because we have the the testing in place.

Like I said earlier, that stops the project going forward until certain testing is met and require and there's certain requirement. But, you know, for for me using it, if you've got a hot fix that has come from a project that you need to get back into that project pipeline, then going through the project yourself, you can move it for a quicker rate. So it's into that environment and pull it through into especially if there's nothing If there's nothing else in that higher branch, then you could just pull through that hotfix into BAU through to the production to fix that issue. But as I said earlier, what we're trying to do is keep that BAU pipelines as free as possible so that a hot fix can go through there.

Then you just once it goes past the pipe, the projects, you can back propagate it into the project and go back down inside the project shell itself. So this in my eyes, there's multiple ways to do it that both come to the same outcome. Yeah, I would agree with that. I think that's certainly certainly possible to be able to to create changes to the through the BAU workflow to be able to create hotfixes.

It would also be possible, and this is something which we're looking as a future enhancement here, to be able to more easily expose the ability to create a hotfix directly off a project release branch. So this would be similar to what you might do in a kind of more of a Gitflow based workflow. But you've created a release, and then you can create a hotfix off that to then merge back into it to be able to hotfix that up. That's something which can be done manually today.

It's something which we're looking to add improved support for in later releases of this of this functionality. And in fact, for some teams who are who have started adopting long term projects effectively as a way of implementing more of a Gitflow based workflow, that is what they're doing today. So it's a very relevant question. It's one that we're thinking about how to present the best user experience.

You're right. The approach is very similar to Gitflow. And it's effectively so the choice of about which branching strategy to adopt effectively it's determined by what you value most in terms of the process. If you want to maintain feature independence as much as possible, then keeping on branching from your master branch and being able to that through is the most effective way of doing it.

If you want to be able to run from that earlier place, be able to create to do the branch from also like a developer environment and then bring that back in and then push create release branch off that, then that's effectively the Gitflow workflow. It's very similar to what we're doing here in projects. And that that workflow can work for a bunch of teams as well. And we're intending to enhance this to to bring to allow better support for that.

Address Salesforce development integration with SAP development taking place in different swim lanes with codependencies. That one we may have to take away because I'm not sure I fully get the context of the question. If you can put a few more details about what SAP development is referring to there, then we can Yeah. So it depends on how that's linked in with the dependencies.

Typically, at this point, it's something which I think you need to manage side by side in terms of being able to make sure those changes go through effectively. But maybe that's one which needs to take away and come to you with an answer. We've got a couple of minutes, we've time for maybe one more question? Yes.

We'll still be on here for another couple of minutes until we get to the end of the session, but otherwise, just want to say, yeah, thank you very much, Johnny, for joining me today. It's a pleasure. Great to see this in action. And then you'll be able to go, everybody here, thank you for coming along to this breakout session.

We can head back to the main room ready for the panel to start. Thank you.

91导航

91导航 Agent

Compare & Deploy

CI/CD Pipelines