9. Quality Engineering | Ari Luokkala

Our guest Ari is an experienced quality and automation engineer. He spent time sitting with me on educating how quality engineering can contribute in our Salesforce development, and what specific challenges are exposed in the context of Salesforce environment.

Main Points

1. Ari’s introduction and job
2. What Does a QE do in a project?
3. A typical setup in a Salesforce development project?
4. What are the benefits QA brings to the project?
5. Discussions on interesting tools such as Jenkins, CCI, Atlassian tools, Robot framework…
6. What challenges does Salesforce development introduce? (unstable Scratch org, creating UI testing for lightning UI…)

Links

• Jenkins
• AtLassian Toolkit
• CCI
• Robotframework

Transcript

AI-generated, unedited.

[00:00] Episode nine.

[00:05] In this episode, our guest Ari Luokkala gives us an introduction about quality engineering. So, what it is, what added value it brings in, and what specific challenges in the context of Salesforce development. So we set up a fictional interview session to go through all these points. Hopefully, after listening to this episode, you could have gained a good high-level picture and start using it in your Salesforce journey. Alas, don't forget to hit our website SalesforceWay.com to check the show notes and useful links. Shared from our

[00:51] talk.

[00:55] Hello everybody, Xi Xiao here from Salesforce Web Podcast. Today I'm very excited to have a very interesting topic that I'm get triggered. So the topic is about. Quality engineering, right? Yeah, and automation in Salesforce project. Okay, yeah. So it's like a introduction on quality engineer and how quality engineer could contribute in Salesforce project. That's basically the context. And I've my guest sitting with me with coffee and with tea together here. So his name is Ari Luokkala. Yeah, my name is Ari Luokkala. I'm 41 years old, of IT engineer from a Helsinki area, and I I work at F Secure as a senior quality engineer in in. Salesforce project. Okay, so actually, I wanted to have this conversation with you because I think the quality engineer is really important in the market. There are a lot of demanding on the quality assurance, quality engineer roles, and ask them to join in the project like full time. Yeah, especially nowadays, the emphasis is on automation. So automation specialist. So that's like a big job of the quality engineer. Yeah, nowadays it's more and more like that. Okay. So it's like complementing the manual testing. So. I think that the the best

[02:26] kind of approach is to have this intelligent manual testing exploratory combined with the automated regression testing. For doing the dull stuff, so to say. Okay.

[02:39] Speaking of Salesforce technology, I think it's a bit slower in the test automation comparing to like a front end web development or some other technology. So this is like a specific cloud-enabled environment, and the test automation wasn't so easy back in old times. Yeah, well, Salesforce is is a is a very versatile ecosystem. For example, there are two UIs. The Lightning and the classic UI, and so yeah, you have this third-party platform that you are that you are dependent on. So yeah, there's challenges, but it's doable. Okay. So you have working experience on the Salesforce test automation and and the quality engineering tasks. Yeah. At the moment, my experience on this field is like five months. So, but my background covers a lots of different automation project. But for Salesforce specific, this is the kind of the first. Okay, cool. So I want to pick your brain here. So I want to play a game. So in this whole episode, because in the audio session, it's hard to you know see the code and check how the the whole project process is is combined. So let's play the game, and so our audience. Can be easier to understand what we are talking about here. Okay, yeah. So I love games first of all, and I love to control the games. So here, let's see. So

[04:09] here's the scenario. We are in the company, and Salesforce is like a new setup here. I'm the environment owner or product owner, and I'm responsible for the whole environment. The environment was already set up, configured well, and we met the business requirement. And now business want us to extend it with the customization. So we get the budget. We get a technical Salesforce architect, a project manager, two code developers to join in the project, and we are about to kick off the project. You know, do the customization, Apex code, Lightning components. So. The technical architect tells me we need a quality engineer to join us. But me as the product owner, I don't know too much about the quality engineer. So you are here with me as the first session to convince me what benefit you can really bring me to the project. Why do we need to hire you either part time or full time in the project? What benefit you can create for us? Yeah. So that's that's a great question. Not not so easy to start now, right? You start now to convince me. Yeah. Yeah. So first of all, there is a good. Good practices to to establish these kind of

[05:39] continuous integration systems into projects. So whatever you do, it it will be continuously tested automatically. That's called continuous integration. Yeah. Okay. Yeah, and also if you have a test there, you can call it continuous testing because it's all all the time running the tests. Okay. When the the code changed. That that gives us a kind of confidence that, first of all, this test is is mostly focused on regression. So, so what we have should remain the same in behavior. Unless we have changed the application logic, so we use the automation to to ensure that the existing functional functionality is in place and works as as assumed. So quality engineer can be helpful for for setting up the CI pipeline. It might need you know kind of very much kind of different skills like you know in the test automation tools, and also have a system administration skills to to cope with the different, content integration system machines etc. And this is. That kind of gluing all the pieces together of deploying the code. So you mentioned the CI, which is the continuous integration, right? Yeah. And let's say in the project we have two developers, we have one architect. So the developers

[07:09] would use Git. To collaborate and submit the code into like a remote repository, which is like a the truth was the master of truth or piece of truth. Okay, and I think they can run the for example the unit tests by themselves. Right, right. So why do we need this continuous integration? The quality engineer brings in. First of all, in this Salesforce context, we can we have a possibility to to run the tests in a clean environment with a scratch orgs. So, with coding engineer can set up this. Of course, skilled developers can do also. So, set up this kind of multi brands pipeline, so that when a developer takes a new feature, develops, it will be automatically spawning off this this CI job. And of course, unit testing can be done from your laptop as well. But on on top of that, we can have the smoke test for the UI, for example, and some kind of more advanced test, and automatically give you feedback on that. Okay, that is really important that feedback loop. So let's say, because I'm not sure if all the audience can follow us here. So let's say we have the master branch right in the remote, and each time, for example, a developer need to introduce a new feature

[08:39] or fix a existing bug, they will branch off from the master. Exactly. Right. That's the way we. Okay. So let's say I have a feature one branch, and I start working on that branch. I update update the code, create new logic. I make multiple commits to that branch. So, what would happen in that branch? So, when when you take create a branch, it is possible that. Immediately, on that time, the content integration job is created. So it's kind of dynamic in nature. Okay. And then every after every comment you do, you push to the remote repository. It will automatically go through the changes with a deployment, a unit test, and additionally the smoke test. So that is triggered by the change. That is the best way to do it. So the developer does not necessarily need to run the unit test locally. That's true. Even if it's good to run them every run at least, right? Whenever the pusher commit, they're confident. The pusher commit to the remote repository, the branch, the bug or feature branch. Then automatically, the continuous integration environment would trigger the predefined. Testing tasks. Yes, and and the one benefit is that you know as a developer you can have a testing in the Salesforce world we

[10:09] we speak about orgs, so it's kind of instance of Salesforce. Right. So you can have you might have a. At your own customization or messed up or anything like that, you know. But currently we have technologies that we can do the testing in in really clean environment. So there is no just testing the code logic. Okay, so you mean we can spin up a clean environment, install the custom code, do the tests. And tear down the environment. Exactly, we can test. It's like a disposable environment. That's the scratch org. Yeah, that's the. This Salesforce context is the scratch org. Okay. So we spawn it up, deploy, running on the test, and when we're done, we just delete the org. Okay. And and if something is happened, you will be notified. You get the email notification. You can get the email. You can get the chat message. Okay. There are different notifications. Mechanism you can use in content integration so that. Do I get a phone call? I

[11:20] guess that's possible, but not so easy to implement. Okay, all right. So, so that's the feedback loop you mentioned, right? Yeah, because it's it's important to have this transparency. What is happening, and we know we get the feedback and we can adapt according to that. So developers is also released from you know running the unit test. They can just focus back on updating further the code. While later, when they check the email, they say, "Okay, the previous part failed." Then they can come back to check. Exactly, you're you're absolutely right about this. So it it. Gives you, it kind of frees you, frees you the time executing that stuff in the CI. That you can do something else, like further develop the code. Okay. So let's say then the developer has finished the feature, and is ready to push into the master branch. Yeah, is there something also happening in the master branch? Yeah, well, there's different ways ways to handle. I would use there some kind of quality gate. So first of all, for example, if you have Bitbucket, you can require the the CI build to be passed before you can merge. So Bitbucket is like a cloud service. It's like yeah, it's like a remote Git repository.

[12:50] Yeah, service. It's a Atlassian company. Yeah, source code version control system. By alassian, okay. Think it there is a similar one also called GitHub, right? Yeah, it is. Yeah, okay, that's true. Okay, so you can build these kind of quality gates, like not not allowing the merge if the build is not passing, etc. And additionally, with the UI test, you can make sure that that end to end tests work okay in the branch before merging to master. So, I mean, for unit testing, Salesforce developers really are familiar with. Yeah. The UI testing, do you mean, like a robot will click or the buttons for you, or what does that mean? Yeah, it's like. It's like mimicking the user behavior. It when it's you when she or he is using your application, opening the for in Salesforce context is like opening the browser. Okay, doing the user flow, user scenarios, and where where your application is installed and interacting the application and then verifying with the machine code like the expected results are. Visible in the UI, so, oh, that's interesting. Then, why do we need the unit test? Why don't we just put everything into this UI testing, then cover everything, right? Well, well, UI testing is costly, so so I mean it takes more time than

[14:20] unit testing, and okay, so I think it's it's cool to have these different layers of testing, though. Okay, so it means the unit test is fast. It's easy to create. Yeah, they're fast. As kind of kind of simple in nature. Okay, you can have them many. You can run run them parallel very fast, and they ensure the functions and methods works. Expected, but then you have the other side that okay you have all all in place and then you have execute a user flow so like acceptance, is it working as should be in the UI? Yeah, but but to answer the question, the UI. Testing is more more costly. Okay, so like a quality engineer needs to create the UI test, right? And also, I assume you need to keep it up to date. If the UI layout changes, yeah, you need to update the test as well. Exactly in Salesforce world, we have moving factors like Salesforce platform itself. It has the the annual releases like Spring, Summer, etc. Something might change there. Okay, and of course, then your your application may change. So, well, then there is um, you either it's a bug or you have to adapt the test case for new designed functionality or behavior. Okay, yeah, that helps me to understand as a product owner. We need to have this multi-layer testing to guarantee our.

[15:51] Code quality. Yes, right. Okay. Um. That sounds cool. Can I tell you one thing before we? Yes, yes, yes. So, so let's assume that you have passing the build in in your branch, on the pull request in the feature branch. Yeah. Okay. Then you merge the master. Okay. So we have kind of integrated branch. So where we have integrated your changes into the master, yeah, then should be run again this test. So it's like verify that the integration works. So it's like a we construct a building, and when we add a new block into the building, we always need to rerun everything to make sure that block really integrates into the building seamlessly. Yeah, that's what we are doing. Yeah, and that's that's automatically also in this case also freeing up your time. To to do something else while it's testing, and what is important is is that you you you scope the the UI test. Or whatever test you have to the right things, because as I said, it's costly. So you have to really think what is the most important stuff that you are like confident that if this is passed, we are we are fine. Okay, so let's say my project is also with only limited budget. Would you help us to do the manual test as well? Of course.

[17:22] Okay. That is that is something. I don't like that as much as automation, but I have realized that it is still something we have to do because of the complexity of the systems and there are so many ways things can be done. And then you have exploratory testing, so. You might find scenarios with your thinking with your human brains, how just like ad hoc idea. What if I do this? What if I do that? And that's something that that your script can do. So I think they're just both needed and complementing each other. Okay, I used to work with some security guys who join the project to help us to do random testing. So they do see the application from different angles, comparing to the developers. Do QE or quality engineers have a similar? You know, angles. Well, quality engineers should have the holistic view of the application, know every every feature and functionality, how it behaves, and especially. Well, if it's come like changing resourcing, like developers, then there is one cornerstone to, in addition to product owner, to stand for that. How the things should work, you know. It's quite really important. So maybe product owner knows a bit from the end user point of view, a higher level picture. Business requirements. Right, right. Yeah. I don't want to get

[18:52] into those technical details too much. Yes. So you can help, especially in the long term, to explain how the application should work. Yeah, and also I'm also and the engineers should also, if there are requirements. For the product, should should ensure that they are fulfilled, and overall, we get the right kind of product, and collaborate closely with the product owner. So. Yeah, that also depends on how long you're gonna stay in the product, right? Yes. If after the project the users are gone, with for example, if we are having consultants to do the development, they are gone after the project usually, and the QE could bring more benefit if you can stay longer. Let's say if you are like an internal employee, right? Yeah, for the product. Yes, because I I think it's it's. It's extremely important, in my opinion, to to know the product and functionalities, especially if you have a changing resources in development site. Okay, that's also maybe one of the reasons that in the job market the QA or QE is really demanded nowadays as a full time job. They really want hire people to the company internal employees to know the product since begin and inside out. Yeah, because I I truly believe that that will bring the best result.

[20:22] Okay, hopefully I have the budget to to hire you. I hope too. Okay, um, yeah. So we talked about the concept level how to help the project. I already understand what you can bring. You we have the test automation, we have the manual test. You already know everything from the higher level technical point of view. So. How do we really do those stuff? Do you have like preferred tools to you know fulfill these? Yes, I have. Well, I have. Lately, used a lot of robot framework, a generic test automation framework. Okay, and I have noticed lately that actually Salesforce has there is the other tool Cumulus CI. Cumulus CI, yeah, Cumulus CI is a Salesforce specific tool for content integration. Okay, I'll put that Cumulus, Cumulus, Cumulus CI. Yeah, I will put the link in our show notes so that our listeners could check what that is. Yeah, could you explain a little bit more? Cumulus CI is is Python based. Content integration deployment tool for Salesforce. So with with the tool you can, for example, deploy your code, run unit tests, and nowadays there is also integration to to Robert framework. So, in the Robot Framework, which is a more like generic test automation framework, you can use Kumulo CI. Functionalities like

[21:52] running tasks and so on, okay, so that you can fulfill the okay. Let's. I'm I'm kind of less technical than you as a product owner. I'm understanding the Cumulus CI is like a. Automation framework, right? It can help you to run multiple tasks that you define. Yeah, it's yeah, it is task based, task based, um, continuous integration and deployment tool. So not not. Well, you can run unit tests. But that's the level level where it gets. When you want to run like UI test end to end testing, then you have have to have another tool. Okay. But this is suitable for the for that level, and it it is extendable, so you can write your own tasks and. Combine tasks to flow, so it's very efficient and customizable tool. So in Cumulus CI, we have the so-called flow, which is combined by multiple tasks. Exactly. Okay, a flow is like a bigger task. Yeah, it's like a list of tasks. Let's. Give an example. Can I install my code into a clean environment with one button click or one command?

[23:12] There are multiple. Yes, you can with the Cumulus CI. You can if you build the flow right, you can do it with one command. Okay, let's say, and you can also run run unit test after that in one command. It just depends how you combine tasks to do to establish a flow. Okay, okay. Let's say, in the Salesforce installation, usually we have to you know upload the static resources. We need to deploy Apex code. Maybe we need to enable some permission sets for certain users. We even need to create some user or create some regular user data. And then finally, run the unit tests. Can we do all this in one flow? Well, yeah.

[24:01] That permission set is doable. I'm not sure about that cursor data. I think you can do because you know Cumulus CI can use three different APIs: bulk back API, and then there is the Salesforce Enterprise API, and what was the third tooling API? Okay. So there is three APIs in use, and it kind of limits what you can do natively with the Cumulus CI. And for example, in in your example, I'm not sure about really about that user user part. Is it doable? But the the Permissions set surely. Okay. For example. Okay. So it means I run the command, then the environment is ready for me, without regular user data. Can I? Yes. Okay. Yes. Then I manually create the data. Yeah. Things. Yeah, you need to populate the data. Okay. Somehow. Okay. Because I see from developer point of view, quite often they need to, you know. Install the the whole package into a new environment to do the test to to make sure everything runs as expected. Yeah, as you mentioned the scratch arc. Yeah. So if we can automate this installation process, definitely that already helped a lot to save the many time. Yeah. What what comes to the scratch arc, which by the way you can use also with Cumulus CI. So, deploy the stuff to the scratch org disposable environment. You can use these definition files to define

[25:31] what kind of features you have in the scratch org. Enabled, that's a scratch org out of box feature. That's true. Okay, so I understand the community CI could talk to. Maybe the API of Scratch Org. Does it also use the? We have this fantastic DX. Well, yeah, it kind of uses for Scratch Org part the the SF DX, yeah, command line tool. So it it can use the DX command too. Yeah, it internally uses it for Scratch Orgs. But other than that, I think the Cumulus CI is is not dependent on the SF DX tool as such. Okay, so you can work with the traditional developer edition static organization as well. Yes, you can. Okay, which is free for everybody. Yeah, everybody can, you know, go there and. Launch and developer. Okay, a decent org. The common CI is it open source? It's free to use. It is. It is open source, and it's. It's. There's a who are using that in the industry. Good question. I have no detailed information about that, but.

[26:45] It's very active, very positive. Okay, I'm subscribing that the feed from the repo and there. Okay, have lots of development there. That's that's good to know. Otherwise, you know, if it's a dead project, we are a bit. No, it's it's so so. Okay, okay, then I think our listeners and I deserve to really check what Cumulus CI is. Yeah. Okay, but the. Do you use some other CI tools like Jenkins or some cloud-based? Well, yeah. Kumul CI as such is is a tool capable to do this CI stuff from your laptop. But then you need to something to run it in remotely automatically. There, I would take for example. Well, there are lots of different continuous integration application servers, but I. For example, could take like Jenkins, okay, and what is trendy or how would I say like hot is is this pipelining? So you have the Jenkins pipeline as a code, and there you embed the uses of CCI inside the, the pipeline code. So what it means in practice is that you would just put the Jenkins to run tasks in the remote server automatically after change and just uses Kumulci to do what it needs to do done. In addition to deployment, I could use like Robot Framework to to do the smoke testing on the UI. Okay.

[28:16] So and those all would be run in the same pipeline in Jenkins in the remote server. Okay. Um, I think I've already got a lot of the concept level. Let's finally bring everything we just discussed into one picture to make sure our listeners and I understand it correctly. I I just kind of recap. Yes, you test me now. So we have the Jenkins. Yes, Jenkins server right. It is running on certain server. Yes, and that could automatically trigger the continuous CI. Yeah, to run some tasks. Yes, because you can put the Jenkins to talk with your. Version control system, for example, Bitbucket. Okay. So they can send, if you change, make a change, they can. It can be set up to send a web hook to Jenkins, which triggers the the job, which then in in uses the CI. That's what you said when that developer commit, make a commit to the remote. Feature branch, that triggers the Jenkins to run the task, which is the community CI task. Exactly. Okay. So now I click everything into one picture here now, and the community CI would run, for example, the unit tests. Oh yeah, it's possible to. Well, as I told you, there is this Jenkins pipeline script concept, and you can have as many stages as you want there. So you

[29:46] can definitely each stage. Yeah, do whatever. For example, if you use Kumul CI in the pipeline script, so you have a Kumul CI, it deploys the application. Next stage is run unit test. And pipeline scripts is is very flexible. So next thing you can run, for example, robot framework test as a next stage, or or like sequentially. Okay. Okay, skies and limits. Yeah. Okay. All right, and. Yeah, so the Jenkins is like sitting on top of everything to observe and monitor the changes. Yeah, and also giving you the feedback. Then giving ah send the email back to the developers. Yeah, and then community CI is triggered, and Robert framework what which does the UI testing is triggered. Yes, and then finally when everything passes. Then you also maybe get a a pass at least the icon a green icon says yeah that's what you use to get the green pipeline okay which is nice to see it sounds interesting so how long does it take to set this up? In the tools, practically, let's

[31:05] say if you join the project since begin, right? It depends how much you have time dedicated to these tasks. For you, yeah, and it depends how much you know already about this technology, okay. But if I use myself as an example, well, Salesforce was not familiar to me. I know I I knew Jenkins, Commerce CI was not familiar to me, SFDX was not familiar, Salesforce not so all related to Salesforce not familiar. Okay, but Jenkins and Robot was. So, I need I need it to to set up something. It takes some several weeks. We get initial skeleton for the pipeline, and that's the evolving process that continuous improvement to see what works and what not. And then finally, for example, I noticed that Cumulus CI has integration to robot frameworks, and I realized that we can utilize that in our testing and. So, for example, you can log into Salesforce without typing username and passwords, so on. So, if my project now hires you as a full-time QE, you definitely can do it faster, right? It's the second time to yeah, similar stuff because you you know the concept how you construct the pipeline. Okay, but to do and where. Sounds good. All right. I'd like to have you in the project. Oh well, thank you. All right. Let's keep it up. Sounds like

[32:35] a plan. Yeah, I really would like to have you join my team. Thank you. Okay. Good. So that's the end of the session. I think Ari and I had a chat about this. At least from my point of view, as product owner, it's not everything so clear. But I understand we in the project we need people who are contributing, who are building the block. But we also need people who maintain the block to help them to maintain at least to try to automate things as much as possible. Yeah, use like use use these. Remote machines to do some work for us and keep up the quality, you know, on a certain level, keep us keep us feed, but contin continuous way. Okay, yeah. Thanks. So yeah, definitely now Salesforce, the technology also, you know, empower us to use the DX, use the the Git repository to to store every everything, the code, the configuration, even the some test, the user data. Yeah, yeah, you can store there anything. Also documentation. So, so we're not bound to to like a developer org or something anymore. No, the choice is the source code. Exactly, that's the key. So that we have everything there. Okay, like the Jenkins jobs as a pipeline script is as code. Okay, you can it's mobile, so you can move it to different instances. Okay. Cool. Thanks, Ali. I think that's

[34:05] all for our session today. Thank you, Xi. So that's it. Bye bye.