Jason Lantz, who joins to talk about CumulusCI – A Salesforce automation tool, is the Sr. Director, Release Engineering at Salesforce.org.
Sponsors
This episode is sponsored by FormAssembly.
FormAssembly and Salesforce eBook – Find out how FormAssembly helps Salesforce customers optimize their data collection.
This episode is sponsored by Clayton.
Go to Clayton website, request a demo or use the chat service by telling “salesforceway” code to receive free one month Secure Development Essentials.
Main Points
- Jason’s self-introduction
- Jason talks about what is CumulusCI
- What is the configuration layer in CumulusCI?
- The other three layers: key-chain, task, and flow layers
- How to understand CumulusCI from a beginner point of view?
- Does CumulusCI work for enterprise developers with repositories other than Github?
Links
Video Teaser on CumulusCI
Contact Me
If you enjoy this podcast, please reach me out to let me know your thoughts or simply say Hello. My (Xi’s) contact info:
Take care, until next time :)!
Transcript
AI-generated, unedited.
[00:01] I'd like to welcome our sponsor, Form Assembly. Form Assembly's all-in-one web form platform lets you create forms for just about any use case. From contact forms to donation forms, all while taking advantage of useful features such as notifications, e-signatures, and more. Not only that, but you can also connect data to systems you already use. FormAssembly integrates with Salesforce, Podio, PayPal, and many other common solutions. You can find out how Form Assemblies help Salesforce customers optimize their data collection in a free ebook that we've linked in today's show notes. Whatever your data collection needs are, you can be sure that Form Assembly keeps your data secure with encryption at rest and in transit on all plans. Plus compliance with GDPR, CCPA, and more regulations. At the end of the day, form assembly helps you save time, money, and effort while getting the maximum benefit out of the data you collect. And I'd remind you, when you support our sponsors, you support the show. Hey, everybody! This is Xi Xiao. This is yet another new episode
[01:31] of Salesforce Way Podcast. Today, I'm sitting with a new guest. His name is Jason Lantz. Hello, Jason. Hello. So I'm Jason Lance. I am currently the senior director of release engineering on Salesforce.org's products team. I have been with Salesforce.org from back when it was the Salesforce Foundation. When I started about six and a half years ago, you know we've since changed into Salesforce.org, and then last summer got acquired by Salesforce. So now officially part of Salesforce. And my job for the last six and a half years has been. Basically, figure out how to run our managed package development projects. So, I, you know, we're we're technically not an ISV because we're part of Salesforce, but I refer to us as ISV-like. So we operate very much like most ISVs do. We build managed packages on the platform. And you know my my job for the last six and a half years has been trying to figure out how do we do agile scalable team development using version control as the source of truth for everything, and yeah, it's been a really fun challenge and and over the years. That's cool. So I understand that one of the major products you are managing now is called Cumulus CI, right? What is that? Yeah, Cumulus CI I describe as a a framework for portable automation for Salesforce projects. And I think the portable
[03:01] automation idea is is really critical to what Cumulus CI is and kind of understanding it. You know, Cumulus CI itself doesn't have a user experience directly. Cumulus CI is a a framework for configuring the automation that you need for your project, and then we build kind of client applications on top of that framework. So. The first exposure that most people have to Cumulus CI, we you know Cumulus CI itself ships with our command line interface, which is CCI, and that's really just a client of this framework where projects have defined their configuration, and then this client can just go and run that configuration. So, all the configuration for Cumulus CI, you can actually see this in our open source NPSP repository, and all the customization that we built for that project in Cumulus CI. All of that config lives in a Cumulus CI YAML file. In the in the root of the project repository. So if you've used you know Circle CI or Travis CI and stuff like that, very similar concept. You're you're defining all the different types of automation that you. Okay. So rather than a bunch of shell scripts that are kind of procedurally written, Cumulus CI is a framework where you can define the automation and the orchestrations that are relevant. To generate usable environments of your project, you know the way the configuration layer of CumulusCI works is CumulusCI ships with a
[04:31] a global CumulusCI YAML file that defines all of the out of the box tasks and all the out of the box flows. And what what gets run, and then all the projects can use that by default, right? All projects do use that by default in Cumulus CI, but then the Cumulus CI YAML in your project repository is all the overrides of that default configuration for your project. So, for instance, with the QA org flow, by default, the the QA org flow calls a sequence of three other flows. It first calls the a flow named Dependencies that will handle getting all the dependencies handled for you. For instance, we build extension packages on top of NPSP and Eda, our two main kind of open source platform products for education and nonprofit. And when we build an extension of NPSP in Cumulus CI with a single line in the Cumulus CI YAML file, we can just say, "Hey, I depend on NPSP's GitHub repository," and the dependencies flow will. Handle going out, looking at our public GitHub repository, finding everything NPSP needs to install, finding its latest released version, and installing all of that into your scratch org for you. And it's just a single line in the config file to point at our repo, and then you inherit all of its dependencies. So it's kind of like a recursive package manager
[06:01] in every other language. And so, if you're building metadata that references NPSP, you have to have NPSP installed in your scratch orgs before you can deploy the project that you're working on. And so we build a really rich dependencies layer in Cumulus CI to do that. The second thing it does is runs a flow called Deploy Unmanaged, which deploys all the unmanaged code from the the source repository you're working on. Our dev, you know, when when we're developing the package, we're developing unmanaged code. And then later we deploy that into the packaging work to bundle it up. So, and then the third thing that it calls is a flow called Config QA, and that's the point that most projects customize, like a post deployment. Exactly. Yeah. So it's so it's really resolve the dependencies, deploy the code, and then configure it. Is is the kind of three step process of most of the flows in QMLCI, and that config QA flow has two steps in it by default. But say I wanted to add a dataset that was specific to my product, or we have tasks that come with Cumulus CI that actually your your your friend David Reed and and Salesforce Stack Exchange community moderator currently has has really taken on expanding out this framework that we built for capturing datasets using the Bulk API. And these are rich related datasets. So you define a mapping file, and then we have an extract dataset command
[07:31] that uses the bulk API to query all of that, all the objects and fields that are defined in the mapping down into a relational database. And then there's a load dataset command or task in Cumulus CI that takes that dataset and that mapping and loads it into a new org. So you know if if in order to get a testable environment. You kind of need to have seed data populated into it to to really be able to to log into a scratch org that was just generated and the automation run against it. So once you've captured that data set for your project, you in the Cumulus CI YAML just add a third step in the config QA flow. To call load dataset, and now anytime anybody comes into your project and runs the QA org flow, they're getting an org that has all the default stuff Cumulus CI does, but then a project specific dataset loaded into it for them automatically. Okay, you mentioned Jason. You mentioned that it's called task. This terminology, that's a Python functionality created by Python programming language that can do specific task. One of the things like import data into the work. So, using like this, yeah. So, there's Cumulus CI has four main components. We talked some about the configuration layer already. There is a keychain to Cumulus CI that allows you to create scratch orgs and connect to existing orgs like sandboxes or whatever. Okay,
[09:02] and then there is also the ability in the keychain to store third party service credentials, like your GitHub API credentials, if you want to interact, have automation that interacts with GitHub. And then the the third component of Cumulus CI is tasks, which are the basic unit of work of all automation. So all automation ultimately boils down to a task, and a task is implemented as a Python class. We, you know, Cumulus CI ships with a ton of tasks that we found do useful things for Salesforce projects, like interacting with the Metadata API, the Bulk API. We've got a framework for writing browser tests that's really rich. Actually, what what brought me to Helsinki last year was to attend the Robot Framework conference, which is the open source framework that we've integrated into Cumulus CI for browser testing. So there's all these different tasks that that we find are are useful when we're building up the set of automation for each of our projects, and we ship all of those with it. But the one thing I do want to emphasize is that we put a lot of effort into making sure that if you don't know Python, that's fine because each one of the tasks has a set of task-specific options that it accepts, and those options can be defined in the Cumulus CI YAML. So, and set in the Cumulus CI YAML. So, a good example is our metadata deploy task has an option called path. The default task
[10:32] called deploy in Cumulus CI by default will deploy the src directory if you are in metadata format, and but you can define a custom task. Say you have a dev config of just a set of unmanaged metadata that you've captured into a subfolder somewhere. You can define your own task in Cumulus CI YAML, reusing our deploy task, but then setting a different option for the path of where it should deploy. So you can define a task called deploy dev config, which we do in a lot of our projects, and that task is fully defined in YAML. You don't have to touch any Python code in order to do it. And then the the fourth component that we've talked about already is flows, which are just named orchestrations of a sequence of other flows and tasks. But ultimately, a flow reduces down into a list of tasks that are run. Hmm. Okay, I think I got the idea, but let let me act as a a new user for Cumulus CI. So I install the Cumulus CI. Installer in my computer, then I can run CCI, which would give me the command to to interact with the Cumulus CI, right? And by default, there's this global Cumulus CI YAML file that has predefined flows, tasks inside. If I want, I can go open the file and check what are the things defined inside. Those are the opinionated features already predefined by the authors and contributors.
[12:03] And you study that, you understand how it goes, and then in case you need tailored features, you can extend that with the local YAML file on on your project directory. So, and you don't really need to understand deeply the Python programming language because most of those tasks. Like a unit of work, you mentioned like a small wrapper for the features. You most of the time just need to tweak with the parameters of those tasks to get your job done, and you just define it in the YAML file, which is much easier than really using the coding part. So this is how I, as a newcomer, to start use the Cumulus CI. Of course, I need to read the documentation, right? Yeah. Well, and and and that I have to say that was a great concise description of Cumulus CI. I might have to to to borrow your your way of explaining it because yeah, that's that's exactly what it is. It's Most of the, you know, very few users of Humility AI are actually doing anything in Python. Now, the the one thing to point out, though, is you can write your own custom tasks in Python. And we provide a bunch of base classes in Python that you can subclass. And you know, a task in Cumulus CI is passed org credentials. Basically, the the keychain gets an OAuth connection to the org.
[13:33] So a task is passed an access token to get to all the APIs. Automatically. So, if you want to write automation, you don't have to think about authentication. Like that's getting passed to the task, and every every task just has a method called what underscore run underscore task, and you just put your logic in there what you want to do against the org. And we have a, for instance, a base class. I believe it's a base Salesforce API task. That comes with an authenticated instance of Simple Salesforce, which is a Python client library for accessing the REST API. And then we've also tweaked that a little bit to also support tooling. So, for instance, if you want to call Simple Salesforce in the run task method, it's self dot SF, and that is the API wrapper, or self dot tooling to interact with the tooling API. And then you're just writing your business logic of interacting with the API in Python. Really, pretty simple, but incredibly powerful. Because if if we don't have a task for you, you can write your own. You can store those task classes in your project repository, and then wire them into your Cumulus CI YAML. Andy, you can contribute back to the open source library if you want. Yes, definitely. Okay, that sounds really modularize the ways of implementing this framework. I see all different like thoughts are put into there. There are some modules that you can extend, and it's really like industry
[15:03] standard hooks inside the flow allow you to extend in different like a diversified manner. So it sounds quite convinced to me. At least I'd love to to study this framework, and also give me a lot of learning points as well. How these things are done in that project.
[15:25] This episode sponsored by Clayden. If you build on Salesforce, you know writing secure code can be really hard. Clayton makes it easy for you. Clayton plugs into your Git repos in seconds and automatically protects you from hundreds of security vulnerabilities, such as Crowd FLS violations, code injections, vulnerable third-party libraries, and more. Clayton works with GitHub, Bitbucket, GitLab, and Azure DevOps. Go to getclayton.com/salesforceway to get one month of secure development essentials for free. It's getclayden.com/salesforceway.
[16:10] You can also find the link in our show notes. Clayden build on Salesforce, stay secure.
[16:20] And Jason, you mentioned the community is for managed package, and it's heavily interacted with the GitHub. What if I'm not using the managed package? I'm not developing that. I'm just the enterprise single, you know, a customer, and I don't use GitHub. What should I do? Yeah. So that was as as you were talking, there were kind of two points I did want to make, which is, CumulusCI doesn't require you to build a managed package. In fact, when you initialize a project, it'll ask you, "Are you building a managed package, or are you just building unmanaged metadata that you want to deploy?" And so it works for both. It the other thing I do want to point out is I've talked a lot about open source. Where my heart is, but we use Cumulus CI for proprietary managed package projects that are developed in a private GitHub repository too. So you don't have to, you know, what you are developing doesn't have to be open source to use Cumulus CI. It works fine with with private repositories. I get asked the question. I've talked to a ton of people over the years at Dreamforce and Trailhead X and such about you know, and one of the questions that I think. A lot of people ask about Salesforce DX. Is like exactly this. I am developing in a single org. It's highly customized. How do I automate all of that with Salesforce DX? I am a huge proponent
[17:50] of the package model of development, and you know, t here is trails that kind of outline the org development versus the package development model, but the.
[18:04] The best the best way to approach that, you know, you're you're basically taking on a monumental task to completely automate the entire configuration of your production org. I tend to think you're better off to approach that incrementally and carve out discrete sets of functionality from your org. Kind of treat those as a package that is developed in its own GitHub repository. So, say for instance, you have HR functionality in your Salesforce org, and you want to hire a consultant to come in and work on your HR functionality. If you have that in a GitHub repository, and you've got some like fake sample data defined in Cumulus CI's automation for it, and everything, you can go give that consultant access to the GitHub repository, but never access to your sandbox or your production org. Because if they're just working on the HR functionality, they can develop that in scratch orgs. They can commit those changes back to the GitHub repository, create a feature branch and a pull request, and you can review it. You've got you know an audit trail of all those changes that are made, and they're never touching any of your production systems or any of your customer data. Um. So, and and the thing is, yeah, that doesn't automate your entire org. It's just getting you automation and a kind of standard, uh. Standard to the rest of the software development industry, practice of of working with with software development just
[19:34] for your HR product. But then, what you do is when you have a new release of that HR product, you then deploy that into your sandbox and you do your kind of UAT or integration testing there in the sandbox. Make sure it looks good. Then you run that same automation to deploy it into production. And what you've done is carved out a piece of the unmanaged soup of your org, and given that piece a better development process that's more scalable. And as you continue doing that incrementally, you start carving up little pieces of your org and building this process. You are improving the overall experience of doing development on it. Understand, and so that's the way that I would advise people. You know, especially if you've got a new, if you've got new functionality you want to build, start it that way. So, so earlier you had mentioned kind of starting as a new user with Cumulus CI. Kind of two things that I would encourage people to try are. Our public documentation for CumulusCI right now is cumulusci.readthedocs.io. We are working on updating that documentation. I guess I'll I'll give the safe harbor disclaimer on any of that. You know, this is all open source. It's it's all open source. All the tooling that my team builds. And we run it very much like an open source project. We try to be very open and transparent with the community. You know,
[21:05] it's not something you're purchasing as part of your MSA or anything like that through Salesforce. But it's it's tooling that we built and we wanted to make it available to the community. So if you already have Salesforce DX installed, you already have it connected to a Dev Hub and you already have the default Dev Hub username defined. Basically, to get Cumulus CI working, you have to install Cumulus CI on your computer, and we've got good installation instructions in the in the public docs for it. To do that, and then for doing some of the more fun stuff, like the the dependency stuff that I was talking about earlier. You need to set up the GitHub service in the keychain, and there's instructions on the the public docs about how to do that. That's a that's a one time thing. So you're just globally setting up your GitHub credentials for Cumulus CI to use, and and from there you're you're pretty much set up to start building projects. So the the demo that I typically do for people. Is make a local Git repository on your computer. So if we're talking command line, you know, mkdir the project name, cd into the project, git init. That's all you need to do, and then from there you run the command cci project init, and that will walk through and ask you questions about your project and what you're building, and and one of the questions in there is is what you're building extending some other Cumulus
[22:35] CI project. Um and because this is kind of Salesforce.org's tooling and we're we we we go ahead and include NPSP and EDA as options in that, so you can just hit like I think it's two for NPSP in that option, and then there's a third option where you can just give it the URL of your own GitHub repository. But I think a really interesting demo for most people is run that. Use a dependency on NPSP, and NPSP is a complex set of dependencies, which is why I think this is a good demo. Because to install NPSP, you have to have some unmanaged metadata deployed. Then you have to install the five underlying managed packages that make up kind of traditional NPSP, and then with NPSP or the Cumulus project that I mentioned earlier. We created a sixth manage package that extends all of those other fives, all those other five. So you know setting up an NPSP environment is not trivial. You know, but if you go through that CCI project init and you say I want to depend on NPSP. Once you're done with that, that entire project is set up. We bootstrap it for SF DX. We bootstrap it for Cumulus CI. We set the dependency in it, and from there you can run the command CCI flow run dev underscore org space minus minus org space dev. So the dev
[24:05] org flow exists in every Cumulus CI project, like I mentioned before, and also every Cumulus CI project has a set of standard scratch org definitions that are named in the project. So the on that command, the minus minus org dev is saying I want to use the the dev scratch org shape. And what that'll do is Cumulus CI will call out to SFDX. It'll spit up a scratch org, and then it will run all. It'll you know resolve the dependencies for NPSP, get NPSP installed into that scratch org. And build you a development environment where you can start building on top of NPSP, and that's the command that you use to do it. To open the org, it's cci.org browser dev, and that'll pop open the org in a browser. The other thing is when we create the org in SFDX, we set an alias on it. So the alias is the project name underscore underscore the org name. So for NPSP, if you create an NPSP org from our repository directly, it's NPSP underscore underscore dev is going to be the alias on that scratch org that it creates. So you can still use all the SFDC commands against it. It's just when when Cumulus CI is referring to an org. It's kind of namespaced in the GitHub repository that you're in, so you don't have to put the project name on your org name, because that makes your commands longer to type. So you can go into one repository, create a
[25:35] dev org that's called dev, go into another repository, create a dev org that's called dev to Cumulus CI, and Cumulus CI will recognize those as different when you go between repositories. That's cool. I got it. Excellent, Jason. I think we have spent a lot of time talking about this Cumulus CI tool. Thanks, Jason. So it's really great talking to you. See you next time then. Great. Thanks.
[26:06] Hi, I'm constantly looking for good guests. If you have any guest recommendation, please reach me out. I'll make sure they are joining to the show to share their knowledge. Otherwise, thanks for listening to the show. I'll see you next Thursday.
