Julian Joseph, who joins to talk about Salesforce Quality Engineer topic, is a Salesforce Quality Architect, Senior Salesforce QA.
Julian has tremendous experience in Quality Engineering, Test Automation, and the DevOps area.
Main Points
- Julian’s self-introduction
- What does Julian do in Salesforce
- What does the quality team do in Salesforce
- The importance of naming convention and quality mindset
- The balance between speed and quality
- Don’t ignore the customer experience
- Prioritize testing tasks
- Test automation
- Quality engineer role in the project
- Tools
Links
Contact Me
If you enjoy this podcast, please reach me out to let me know your thoughts or simply say Hello. My (Xi’s) contact info:
Take care, until next time :)!
Transcript
AI-generated, unedited.
[00:14] Hello, everybody, and welcome to the Salesforce Way podcast. I'm your host Andrew Davis, and I'm thrilled to have here with us today Julian Joseph. Julian, welcome. Hi, thanks for having me. So Julian, do you want to tell everybody a little bit about yourself and about your role today in the Salesforce world? Definitely. So I'm actually a senior quality engineer at Salesforce proper, but I actually got into the Salesforce world through way of a nonprofit. I started off at a small nonprofit called Liberty North Korea as an intern. I learned a little bit of Salesforce there. Came on to do actually a few more internships there, and finally transitioned to a partner of Salesforce, doing some support, and then into a quality engineering role, and found my way over here to Salesforce. Fantastic. Fun fact: Julian and I are both in San Diego, California. So yes. So Julian, I would love to know more, and others would love to know more, I'm sure, about what is the life of a quality engineer at Salesforce do like, and what do you do? And you're at the .org side of things, right? Yeah, so you know I mentioned coming from the nonprofit space, a nonprofit itself, and the partner I worked with was also a nonpro or worked with nonprofits rather, um called Classy, um had a integration with Salesforce, and that got me deeper into the Salesforce ecosystem.
[01:44] I'd say. But today, yeah, I'm at like I said, Salesforce.org specifically, as you mentioned, and that department or set of clouds—fundraising cloud, education cloud, and philanthropy cloud—those set of clouds, of course, serve nonprofit. Org specifically, and the way that we're set up right now is actually a managed, a set of mostly managed packages on top of the Salesforce platform. So in a lot of ways, we operate like an ISV, and similarly, my job, being quality engineer, I'd say, is similar to doing quality engineering at an ISV, and so testing things on top of the platform, testing new Apex. Or Visualforce or Lightning Web Component pages that are being created or updated, looking for bugs on that end. Not necessarily the same across the entire Salesforce company as a whole. Of course, a lot of our development is creating the platform itself, but I'd say there are a good number of teams, including my own, that work on top of that. And so, in terms of what that looks like on a daily basis and what that kind of feels like, I'm actually on a team with a bunch of other quality engineers, and then we are actually spread out, embedded on little Scrum teams working on our individual products. Sure. Go into a little bit more. You mentioned about you
[03:14] work with the rest of the quality team on you know quality strategy and how to you know figure out, I guess, what to test, what not to test. But say more about that. The activity of being on the quality team. I admit I am more from the developer side, so I think I need to broaden my perspective on. What's the mindset of quality engineers? What are you really bringing to the job? Yeah, great question. I like to think of quality engineers as actively combating bias. And that sounds a little bit philosophical. Maybe sounds a little bit maybe even broader outside of the technology space. But I think bias is something that we we all bring in. Whether or not we actively recognize it or are actively working against it, it's definitely there. And so, as a team, quality engineers themselves, of course, we're bringing in our own bias. But something I'm excited for is that my team is pretty diverse, and I think that is one of the strengths of any team. But I think especially a quality team where you're trying to tease out different scenarios or different ways of thinking that. Might be assumptions within a particular group that need to be challenged, and a lot of those assumptions are, I would say, what lead to either poor product design, poor feature design, and of course bugs. And so that's
[04:45] what we're actively trying to work against. And so in terms of the team itself, our team is a mix of both exploratory testers and auto automation testers as well. So on the exploratory side, sometimes you hear that more referred to as something to be more manual, manual testing. And yeah, indeed there is a lot of button clicking and. Of course, manual work that's involved there, but think of it more of exploratory tester. Really, a lot of what we're doing is less about just running through the the actions. Of course, we still need to do that from time to time, but a lot of what we're doing is trying to look for new ways to break something, new ways of thinking, assumptions that were. There that are being baked in by the development group that we need to call out and ask about. Hopefully, doing that as early in the process as possible because that's going to be when it's saving the most time, the most money. But yeah, really looking at it from all these different angles, and of course our automation testing group is creating these tests at scale, working with us to also they're doing their own set of exploratory learning, trying to figure out what makes the most sense to automate, what are the highest priority cases that need automation, how how can they help and support us, but how can they also help us eliminate our own bias as exploratory testers. And so, yeah, it's a very diverse group, diverse group of mindsets. I
[06:15] mentioned a little bit that my team is particularly diverse, and I'm happy to say that there are so many different people on my team. Whether it's a, we actually have a yoga instructor. Individuals with visual impairment. I myself identify as Black, gay, Christian. I have my own eating disorder, and just all those differences and things that you might actually not even think are going to influence the technology. I would say they come out in very clear ways. I think whether it's accessibility testing or Looking for you know different assumptions about what's this person's name going to be, all those things kind of come out. We try to tease them out as a team and talk about them and and bring them back to our individual Scrum teams and then again try to improve our each of our products.
[07:06] You and I have chatted before about the idea of what someone's name would be, and all of the biases and assumptions that go into, you know, just the field validation of what somebody's name would be. Could you give us some examples? Yeah, there's actually a great document that was sent to me by a good friend, Mark Baseman. He actually does Salesforce for Good section of the Salesforce Admin Podcast, or he used to do that. And this is article. Hopefully, we can link to it in some of the session notes. But it goes through. It's it's probably been seen by some of the people who are listening to this podcast. I think it was pretty popular back in the day. But it goes through just all of the potential biases for a given name. Things like people's names are case sensitive, or people's names are case insensitive, or people's names don't change, or perhaps my system will never have to deal with a name from China. Thinking a little bit about diversity there probably, or one of my favorites, people's names have an order to them. Picking any ordering scheme will automatically result in consistent ordering among all systems, as long as both use the same ordering scheme for the same name. So it's just like that's one of the more technical ones, but all of these things as I read through them and I think through them about names and bias. To me, I just kind of get a little bit of kick out of it because I think about it from perhaps a developer or creator
[08:36] standpoint. Maybe these are some assumptions that we are bringing in. And are kind of going through is of course that's true of course that's true that must be true, but I would say from a quality standpoint from a quality engineering standpoint, like I'm reading through those and my thought is nope nope that's not true that can't be true, you know just trying to in my mind. I think our minds are trained as quality engineers, or really anybody with a quality mindset, which could be anybody, trained to just question. Is this going to be true or not? Question all these assumptions. Probably because we've been burned by each of these individual cases before. And so it's like I can think of times when you know I assumed names would be case insensitive in a given scenario. Maybe that changes the ordering or some other feature, and then that's going to break, bring in some bug and. So it's it's hard to not read this and just think about think about where this bias is coming from, and then calling it out and wanting to at least question it, if not influence the product in a way that it's actually changed for the future. I love that. That really, really helps broaden my perspective on what quality engineers are are really doing, quality and quality controls are really doing. Challenging those assumptions. I'm familiar with some of the classic, you know, cognitive
[10:07] biases. I wonder if the we you don't have to. Answer this if you don't know, but there must be a name for this, like happy path bias, right? Where you think that. You know, with the developers, I'm very familiar with this mindset of developer. They're under a lot of pressure, right? They've got a lot of features to get done. They manage to just get something to work, and that alone was painful and a big struggle. And they're behind, and there's all this pressure to deliver, and so forth. So there's a there's a bias, time pressure bias, or whatever whatever it's called, where they just really want to get things out. They want to be as optimistic as possible. And then you can. I love the idea of having thinking of quality control as challenging the assumptions and looking at all of those edge cases. That that helps me a lot. Definitely. Yeah. You mentioned. Yeah. Any thoughts on that? Otherwise, I mean, I just love kind of bringing that up. Like what. This tendency for development teams to try to focus on that happy path and getting things out as quickly as possible, which I think is a very both real and understandable, and to even some degree necessary need. Like we need to get things out quickly. That's part of agile, but it's also just part of improving the world around us. If we want it to improve, if we if we're actually creating this with a sense of good in mind, which is of course very present, I would say, in Salesforce.org.
[11:37] We want to get it out there as quickly as possible. We want to create that good as quickly as possible. But at the same time, of course, we want to do that in a way that doesn't detriment, doesn't create a regression or pulling back and creating new issues of existing working functionality. And of course, we don't want that to get out new features that are going to be confusing or frustrating in their own way. And so. I think a lot of times it's easy to see almost these two careers or these two focuses at odds, like getting something out and also getting it out with high quality, because one is essentially trying to speed things up, the other is trying to slow things down a bit to a reasonable pace to one that's sustainable. But at the same time, I think we're all in this kind of working at the same goal of customer experience, trying to do things better for the customer, give them the best possible result. Whatever that means, and so I think there is there is a amount of synergy that can happen between. These between developers and quality engineers and others on the team who want to get things out, but it's it can be a little bit tricky if we feel like the goal is always to get it out quickly. And I can say this for myself, it can be tricky on my end to feel like the goal is always to get it out perfectly, without any bugs, because that's also never going to happen, and that's going to mean nothing's ever going to get out.
[13:10] You know, I I spend a lot of time in talking about DevOps related things, and DevOps, DevOps, DevOps, right? But it really is that balance between speed and quality, or speed and stability, and. To if if it becomes personal, like you versus the developers, and everybody's very defensive, and it's a personality conflict, then yeah, it's easy to to to feel really uncomfortable. But if you just think actually both of them are really necessary, and it's just important that we have you as a full time advocate for quality, and the developers as a full time advocate for you know building new stuff and getting it out there, it's important to have both of those voices and to balance and to figure out. You know, within that imperfect space, you mentioned about finding these issues as early as possible, where it's where it reduces the the net cost. I think you said reduces the. You know, it's a lot cheaper if you can catch these bugs and issues earlier, rather than having to redo the entire build, repackage everything. It's obviously a whole lot better than exposing customers to bugs, right? So, your mindset is very much on the customer experience, thinking about these names can't have Chinese characters in it. That that assumption. You're trying to look ahead into the future. What might customers experience? You mentioned a few of these things. Do you want to say anything more about the the mindset of thinking about the customer?
[14:40] Yeah. Yeah. I mean, ultimately, I think that customer experience is our north star. It's the way to figure out are we doing things. That are actually beneficial. If if we're just creating a shiny new product that has a lot of bells and whistles, or maybe it's even very elegant and feels just very simple and nice, you know whatever. Whatever we're creating, if we're creating it for ourselves, at the end of the day, that's that's nice and fun. And if you are the customer, then that's great. But most of the time, you're not the customer. The customer is somebody out there using your product in the real world with their challenges, whether it's. Some sort of accessibility challenge, or whether it's just basic, just trying to accomplish their goal and like, and finding different paths to do that. And so, the product that we create together, I think, needs to have that customer in mind first and foremost. So, what does that look like a little bit more practically? I would say. Looking and seeing what what the customer is doing with their product in the real world. Obviously, I think I hear about that very often with a lot of product manager podcasts and articles. Like get next to your customer, watch them do it. I do want to be. Realistic that sometimes that's just not possible, or it's not possible at scale for an entire team, and so finding other ways to bring in that customer experience,
[16:11] whether it's people who are doing that research and having them speak or share different interviews and things like that, just seeing what it's like to actually be a customer is so important. For myself, I feel very grateful that I got to do that at my time at my previous company, Classy, when I was a when I started off in technical support, and so I was doing this with Salesforce admins at nonprofits. On a daily basis, trying to figure out, you know, they wanted to connect our integration. They wanted to sync over this data. They really didn't want duplicates. Really, really did not want duplicates, and I'll never forget that. And you know, they were coming in with these questions, and and there were so many times where I would think to myself, especially early on, oh, this is really on them because they they shouldn't have clicked here, or. This duplicate that they're so worried about—it's not really a duplicate technically. Look, the email address is different. Like clearly, that's not a duplicate. But it took some time and some—I think—very gentle but kind guiding from my product manager at the time, Tim Gumpdo, and some others around me who just would. Work alongside me to try to figure out. Hey, this is actually this is really frustrating them, and those, those I'd say like longer days where it's like I wanted to figure this out for them. I started to realize, oh, it's not,
[17:41] it's not really on them to click in the right place all the time. It's not on them to. Decide or to understand my view of what a duplicate contact is, because in reality it's whatever they need to do their job at the end of the day, and it's whatever they need to do to do their job without feeling that unnecessary burden or pressure, or even sometimes I'd say shame from not knowing how to interact with our system, and so I think that's our. That should be our goal as a product team. It sounds very, I'd say, UXy to think about that customer piece, but I really think it should be the entire team's goal. Like when we create something, when we, as quality engineers, test it, we need to be thinking about what is this customer actually be going to be going through. What documentation are we putting out there alongside of our product to make it easier for them to use? What are the things that we think are obvious that are not so obvious? Can I bring this in front of somebody else who has? A different experience than I do, and might see this in a different way, and give me different thoughts back. Can I test this in a different way, in a different scenario, that is going to give me that is going to reduce some of my assumptions? Things like that, I think, are of the utmost importance.
[19:03] Fantastic. Um, and it's I love, you know, in in that. You both acknowledge the importance of giving every member of the development team some awareness of the customer experience, but as well that there is a cost in terms of time and so forth to do that, and it's always going to be a little bit imperfect. But that you need to, but it's critically important to figure out that that balance. It occurs to me that in the Salesforce world, so of course your customers are very very diverse, each individual. But you mentioned in the Salesforce world that every environment is very different, right? So there's a huge diversity of different environments, and that probably makes it very different from building, you know, some web app. Any thoughts on that? So yeah, Salesforce testing is definitely comes with its own difficulties, and I say it's hard, but I think it's hard for a good reason. We allow a lot of customization, and so every Salesforce instance—I almost think of every Salesforce instance as its own website. And when I say that, I mean when we are building something or testing it, we can't just think about, oh, this is our space. This is what we can define. We can control fully our Not just uptime, but our release cycle and everything that's
[20:33] going out. To some degree, there is that control within the Salesforce ecosystem, but in a lot of ways, everybody is customizing this in their own way. They're going to have their own uptime, their own release process to their customers. And I will say a lot of my own thinking because of the way that we're structured on my teams at .org. I'm thinking it more from an ISV perspective because we operate like an ISV in a lot of ways, creating managed packages, getting them out there to our Customers even pushing out up pushing out upgrades out to our customers, and so I think this does apply across the board in a lot of ways to both ISVs and then people are using just sandboxes and production is developing within their own space. But I'd say especially on the ISV side, we definitely have to think about. Are our customers using a sandbox or production? Are they using managed, or are they maybe trying to develop unmanaged with something that we're building, namespace or unnamespaced? What features do they have enabled? State country picklist, multi currency, person accounts, even even if this org was created through a trial, or whether or not they installed our packages, those. Seemingly subtle nuances can have pretty big differences in how we're testing, how it's going to be, how our clients are interact interacting with it. Most
[22:04] notably, I would say permissions, or as like I like to say, permissions, permissions, permissions. Just a reminder of how difficult permissions can be at times because they are so utterly important to. Keeping security within your Salesforce environment, who you're going to give access to this field or object with, but at the same time, they are one of our challenges when developing managed packages because we either need to include those permissions in a permission set, or tell you how to enable them, or more often do a little bit of both. And kind of finesse how we're getting that information out to you, so that you can control your own security, but you can also do it with the easiest, simplest process possible.
[22:57] It's so helpful for me to hear all of this. Your point of view greatly expanding my appreciation for the challenge of this role of quality engineering, especially on especially on Salesforce. And what's really coming through to me is the reality that none of us know in advance exactly every possibility that might happen. Even you know within whether you're within Salesforce or in a particular customer, it's impossible to fully predict all of the ramifications of enabling something or rolling out some new capability, or all of the possible ways that people might interact with that. So, some of the things that I'm hearing from you certainly feels like it's an ongoing learning challenge, right? You're always learning about how the system interacts, you know, with different parts of the systems interact and. You're very knowledgeable about how Salesforce works. So then, you know, you're not just a quality engineer who's blindly clicking around in fields because somebody told you to. You're really thinking about the system. How do you prioritize within all of that? You know, within all of these different questions and uncertainties, how do you use your energy in a way where you feel like you can make the maximum impact, catch the worst transgressions, and minimize the chances that you miss something? So. I was kind of saying some of the challenges like permissions,
[24:28] like what URLs we're using, test URLs, login URLs, what limits we have within the scratch org versus the sandbox versus production. Those are all going to be just challenges that often matrix out and multiply, right? They're going to be Different configurations that you can set within one org, and then you can do that again in a completely different kind of org. Set those same parameters and set set all of that up, and that's going to be a challenge to test every time in every different way. And there's, like you said, a cost to this. There's a finite amount of time that I need to be able to test all of these different configurations, even if it's just a field. There's going to be a name. There's help text. There's the type of the field. There's API name. There's all these things I could be thinking about on just one field. How do I know what I need to test, and how am I going to do that, and with within some reasonable amount of time? And so there are a few different strategies that I I like. One strategy that I learned from my previous director, her name is Jessica Ingrisalino, is called risk-based testing. And I love this when you have so many problems, so many really problems, yes, and so many different combinations, and it just—it's a seemingly infinite pile of work, or responsibilities, or tasks that you need to. Approach using risk-based testing. You can use
[25:58] that to focus in and rate these different scenarios, and you do that by adding in both probability and severity to get a composite risk score. And so, what that looks like practically is, let's say, I'm testing a feature that I recently tested on a on another team that I've worked closely with here at Salesforce, and that's migrating recurring donations to our new enhanced recurring donations. And so, I'm working with hundreds, tens of thousands, even hundreds of thousands of recurring donations, trying to figure out what fields could be null, what validation rules people might have. What various features these people, these customers might have enabled in their org, and then trying to test our migration to make sure that it performs in a way that is going to be beneficial to customers. And so I mentioned probability, severity, each of these. Different configurations or scenarios or even potential bugs that I see, I'm going to rate them on these two metrics. And so probability, I'm going to be looking at how likely is this thing to occur, and so. If I think a lot of people have a validation rule on, let's say half of our customers that might block recurring donations from migrating, I might give it a pretty high probability score. Or if I think it's very few, I might give it a low probability score. Or if I look
[27:28] at particular field being null, let's say the contact field on a recurring donation, that one's probably going to be filled in because you need to know who the donation is going to be going towards. So Thinking of that field being null seems pretty unlikely, and so I'd probably give it a pretty high probability score that I need to test it because it's probably not going to be null. I also need to factor in the severity for any of these scenarios, and so how bad is the fallout going to be if a bug gets out for this situation? And so if I'm Again, looking at let's say validation rules, there might not be that many validation rules that let's say block this migration in this particular way. However, if it does, if there are validation rules, I know that those can often completely block records from being created. And so I know the severity there could be pretty high, and this is something I could even massage with the development team or product managers. Maybe the fallout would be bad if these records didn't move, but maybe we could also. I know we're developing a feature that lets lets us do this again, and so maybe it's a very simple fix. So severity isn't as bad as it might seem. I can we can just turn off this rule. Maybe we have a very clear list of the errors, and we can do this again. And so, these these scores, which I like to just put on a one to ten,
[28:59] they definitely are pretty subjective. But again, it's a way of just focusing in and looking at what are the most The most high priority situations that I need to test, and so together these two scores they give you your composite risk. I usually just multiply the two together, but you can figure out whatever way works best for you. But I multiply the two. I take all my cases. I like to break them up in three groups. Red, yellow, green, and then now I actually not only have a set of tests that I can run in a let's say a particular order, I also have a way to talk about these tests with a given development team or PM. And help make decisions about what is the risk that we might introduce into the product. So, maybe you're able to give me based on our current schedule a half day to test. I can tell you, I can probably get through all the red and maybe half the yellow. But if you give me this full day, then I can get through all of them. And so, just being able to say, "Hey, these—I'm at least testing all the high-risk scenarios. I've done this work to figure it out." And if you give me more time, I can test more. But maybe we're okay with this lower level of risk. Maybe something we'll test, you know, after the fact, maybe after beta or after down the line a little bit. And so I'm giving some flexibility back to the team at large and the product managers who are on these tight deadlines and need to get things out. So that's that risk-based testing strategy is one that I particularly
[30:29] enjoy. That's very very helpful framework. The there's a similar consideration you mentioned up front that your team consists of a mix of exploratory testers and automation testers. When do you decide? It seems like you'd also need to make a decision for each scenario. Is this worth creating an automation test because that's going to be a lot more upfront work, but a lot less time to run? Or is this a scenario where you can just do one-time testing or exploratory testing? Definitely. Yeah, make that decision. I would say, yeah, the make the way we make those decisions. I think there are other factors that need to be brought in that are not just what is the highest risk, even though. That is one of the most important tools I would say to deciding whether or not a test should be automated. Ideally, you can automate all the highest risk tests, but in reality, I would say that's kind of like saying I am going to go through my backlog as a product manager in order of the highest priority. You could try to do that, but if you've tried to do that, it can be difficult, and you're probably losing out on a lot of different items and a lot of different features that can be combined to. Increase efficiency overall, and so similarly with testing, you know I might want to automate the scenario of syncing over a million recurring donations
[31:59] and migrating them, but realistically to run that over and over and over again, just not it might not be realistic. I might not have a space to do it in. It might be very expensive just to set up and to coordinate, and it might be a very fragile test. So at the end of the day. While I might want to automate it, I might identify it as a high risk. I just might not be able to do that test that often, and there are just other certain considerations that you're never. Going to at least not in this day and age fully extract I would say from an automated test, and so when I am running for example a regression scenario, I'm just going it's usually thought of I'm going through all of my old tests that I've already run before and I'm just making sure we haven't introduced any new bugs. Sure, that's that's happening, and sure that can be automated from a very linear standpoint most of the time with automated tests. However, at the same time, I'm usually not just going through all the tests and just looking at the result in the way I've written it out in that regression because. I know something has been merged in recently, this sprint or maybe a previous sprint. I'm remembering, oh, this one could be connected to this other feature. This recurring donation, let's say, feature might be connected to an update that we made on contacts, and that's not necessarily part of my aggression, but I want to focus more around it. I'm adjusting as I go along. And automated tests today. I think once we get AI good enough to be able
[33:29] to adjust and to think as creatively and complexly as we can, maybe we'll get there. But today, the typical automated test is. Not going to be able to adjust on the fly that easily, and so it's always going to take some level of human interaction, human knowledge about what's happened, to supplement some of that automation and tease out some of the more complex scenarios. But yeah, definitely in terms of like frameworks and figuring out what to automate, RBT I think is a good place to start. But I would also bring in the knowledge, of course, of your automation team of the framework they built. Usually, I would say it's a conversation also with the DevOps team or the release engineering team to figure out what makes sense, what can be automated efficiently. You know what? Where can we shave off? Where can we combine a few of our automation tickets together, even if they're not necessarily the top three highest risk items? You mentioned interacting with different teams, including the development teams, and the idea of automation tests. Of course, in my mind, triggers thinking about Apex unit tests and JavaScript unit tests and test driven development. And I've been a Advocate in the past of behavior driven development, but does that play into what you're doing? Do you work with the developers in those ways? And yeah, what's your way of thinking about that? Building definitely. So my thoughts on how
[34:59] to build testing in is. If you are already writing the requirement, it's probably a little too late. You should still do it, but you know if if you've already gotten past design and you haven't brought in a tester and you haven't talked about this as a team, you're actually missing out on a lot of opportunities to I say increase efficiency. One of the reasons being is usually your quality engineer on your team is going to have the most customer-centric mindset. It would be great to have a customer on your team who's always there talking to you and like giving you that feedback in a way that you can digest. But realistically. You the quality engineer is usually serving that purpose of I'm trying to think about this from as many customers as possible, trying to encapsulate it and bring up those risks. And so, I always like to think about what, where can a quality engineer be brought in in design? How can we influence and improve that process? And I think during design or just even the pre-design, where you're just talking about. This maybe a group of product managers or even some business individuals involved talking about the value of the product. Just getting some thoughts in of, hey, did we think about this from a customer perspective? If we really, if we want to improve this feature that we're talking about or add in this new feature, this is this is, I think, is how they're going to be interacting with it. That's how I'm going to interact with it when I get to testing.
[36:29] That's what I'm going to be bringing up as concerns. Why is this button here, or why is this duplicate feature exist even at times? There, that's those are the kinds of things that I think a quality engineer can be brought in and to talk about and to think about and to positively influence that process. A little bit later down the line, there are some other very specific practical tools that I think. Benefit development teams. I really am a big fan of BDD, behavior driven development, and using Gherkin format, given when then format. That's there's a ton of other frameworks and ways to write tickets and work items and acceptance criteria. This is just one of my particular favorites because it really does focus on that customer experience. BDD. If you're not familiar with it, it's actually an evolution of tester and development. But instead of just writing a test. Do this. Look for the result. Verify this, which is good and useful. It's actually looking at the behavior of the customer more. And so, what is this individual going to do? Given I am on this page, when I click this button. Then this will happen, and I would say you can even take a step further. And I expect this to happen, thinking about what is their mindset, and what
[37:59] what what are their expectations in this particular flow. So you mentioned Salesforce. org. You're basically working like an ISV, creating all these packages. You've got your own DevOps team and so forth. Does that flow into how you guys actually run your automated tests, and how how do you guys run? What are the technologies you use to? Yeah. So in terms of how that flows in, I'd say at least when I'm using that format. We are typically talking about it in the refinement and ticket creation process. So even before it gets to the point where it's necessarily being automated, or even tested, bringing in this given when then. This BDD scenarios, it's I would say really focused on when we're creating the ticket when we are trying to decide as a team are we going to do this work or how we're going to do this work.
[38:56] One example, and I can like kind of move on to like some of these frameworks and how they, how this actually plays out on the automation side. But to just give an example of, let's say you are trying to create a contact with address information, and you want that to flow over into an opportunity. So you might want to write out in the BDD scenario. Given I am in this situation, so a lot of people might first think, "Oh, given I am on the contact with an address." Then what's going to happen? But actually, a lot of a lot of this thinking is actually where is this customer coming into it? What environment are they in? Even what features do they have enabled that might influence this action? So I usually like to start with given I'm on a sandbox, and I then maybe I've even logged in, and then I navigate to the contact page, and then I click this button. And so the next step of that is the when. I I was saying then, but I should said when I click this button. So this is the customer in action. So given I'm in this situation, when I click this button, then now I'm looking at the results, and now I'm trying to figure out what those are. How we would automate that, and how we would even expand those tests. I would say first. Is nuance because it depends on what you might be changing at that point, what your concerns are, and that's where the QE should be raising
[40:26] up those concerns. But often you're thinking about what about in a different environment, what about with a different feature enabled, what about with what if I don't go to the contact page but I create it through anonymous Apex, or maybe I create through an API call, like all these like trying to come at it from all these different. Scenarios, and then asking that final then what am I expecting to happen? What kind of feedback do I get? If it's an error, does the error message make sense? All these kinds of questions are being brought in and creating each individual BDD scenario for that ticket or work item that you're creating. Now, when it comes to automating it and figuring out how can I do this at scale, we at Salesforce. org we actually use a couple tools to do this, which I just love. One is called Cumo CI. And another is Robot Framework, which is actually baked into Cumulus CI. It's our automation framework that runs in tandem with Cumulus CI. But Cumulus CI, you might have already heard about it on this podcast. A few of my colleagues, Jason Lance, David Reed, have already spoken about it some. So I won't go into it too too much in depth because definitely you should check out their podcast. But From a quality standpoint, what these tools are allowing us to do create these environments at scale. And so, I mentioned things like namespace, sandbox, production,
[41:57] state, country, picklist enabled, all these different settings and potentialities. Cumulus CI is actually allowing me as a quality engineer to easily create these environments. Turn them into effectively a one, one line, command line request. Hey, create me this environment with these exact settings. And then robot framework, that automation framework. When we do automate our tests, that's going to be running against all these different environments. And so now, as a QE, can focus a little bit less on just trying to figure out how to create my environment, which can be a whole, a whole difficult. It can be a whole day's worth of work. I would say in and of itself, like how do I create this scratch org with this setting? With this, you still need to figure some of that out. But once you've done it, you can kind of lock it in, and now that becomes one of my orgs that I can use over and over again. And then I can start working on my own automation or working with other automation engineers to, hey, if we automate this, now I can take this one test and run it against these ten different environments automatically, or with a few lines of code. And so it allows us to scale up very quickly with all the complexities that exist within Salesforce. That's amazing. That's fantastic. It's such a so nice to hear from the inside of Salesforce quality engineering how you guys, how you
[43:27] and your team think about these challenges. I guess taking us back to the beginning, really made a big impact on me when you said that quality engineering is about challenging our assumptions and identifying these biases and assumptions about how things should work, and instead taking this really broad point of view. But then just having that customer mindset, thinking from the point of view of the customer, being that advocate for the customer within the team, from the quality team, you know. Including appreciating even the the stress that customers feel, like you know the shame of oh I shouldn't have clicked that or something like that. There's really identifying, empathizing with the customer, and and figuring out how to make it work. And then it's just powerful how you think about this huge range of complex situations, this massive matrix of customer behaviors, the matrix of org configurations, and then. Sorting through that with this risk-based testing, like you mentioned, and then figuring out how to embed that in your processes and engineer it—that's why you're your quality engineer, right? Because you're you're looking at environment creation and automation and how to make the whole thing scalable. I've learned a ton, Julian. Really, thank you so so much for sharing all your insights on on the Salesforce Way podcast. Great. Any final tidbits? Anything you'd like to share for the audience? Yeah, so I definitely really appreciate
[44:57] that, and I do want to acknowledge that we've all been there. Quality engineers by no means are perfect. We have our own biases. I've definitely to myself said so many times, this doesn't need to be tested. This is definitely gonna work. We didn't change any code there, so why even check that? And then been burned by it down the line, and had to do a hot fix, or at the very least push a ticket back that I know I should have checked something else out. And so this is by no means like a a dig on anybody who said any of those things. I am definitely guilty. And we are. I think many of us are. We've probably done that as well. And so this is this is just an understanding that we've been in this situation. Quality engineers. And people with that quality mindset. The goal is really to eliminate and to remove bias whenever we can. But it's really much more about. Thinking in that way, it's not about necessarily just doing it. Hopefully, we can do it, but it's getting into that thought process of I need to, I need to come back against, I need to combat these assumptions had, and really realize there's something bigger than me that's more important. It's what the customer is experiencing. What I think they should be clicking, what I think they should be doing, is to some degree a moot point if they're not actually able to do it. And how can we empower them? How can we How can we design our systems in a way that actually respects them and puts them at the forefront of all of our design and all of our testing?
[46:29] That's awesome, Julian! Such a delight to have a chance to connect with you again and to hear your thoughts on quality engineering. You've really broadened my view a lot. Thanks so much on behalf of the Salesforce Way. I'm Andrew Davis, and thank you so much to Julian Joseph.
[46:46] Hi, I'm constantly looking for good guests. If you have any guest recommendation, please reach me out. I'll make sure they are joining to the show to share their knowledge. Otherwise, thanks for listening to the show. I'll see you next Thursday.
