Daniel Ballinger joins to talk about his Parallel Unit Testing diagnosing story.
Daniel is a Salesforce MVP, Senior Salesforce and .NET Developer and the author of tool SFDC Explorer.
Main Points
- What is parallel unit testing?
- What is the default running mode in the Salesforce Platform?
- The troubleshooting story Daniel is telling about the parallel unit testing
Links on Parallel Unit Testing
- Daniel’s Twitter
- Daniel’s LinkedIn
- Daniel’s blog post on Parallel Unit Testing
- Debug log parser and visualizer
- Daniel’s blog
Contact Me
If you enjoy this podcast, please reach me out to let me know your thoughts or simply say Hello. My (Xi’s) contact info:
Take care, until next time :)!
Transcript
AI-generated, unedited.
[00:00] Hello
[00:12] everybody. This is Xi Xiao. This is yet a new Salesforce Way podcast episode. Today I'm sitting with a new guest from the south of the world. His name is Daniel Ballinger. Hello. Hello, thanks for having me. So, Daniel, would you mind to introduce yourself? Yes, so Daniel Ballinger, I live in New Zealand. I've been doing Salesforce development work for eleven years, I think, since about two thousand fourteen. Been a Salesforce MVP for about five years. Do a range of things. A lot of work on the Salesforce Stack Exchange. I put out like the list of key prefix lookups. That's quite a popular resource. Yeah, sort of do a range of things with Salesforce. Cool, and I understand that you're also a .NET developer as well. Yes, yeah, and that's where my initial integration with Salesforce came from. I was talking to the APIs. And so I did a lot of work there, and I think one of my earlier tools was the Visual to Apex alternative, because again, they tied into the .NET. We needed to talk to Salesforce, but the Visual support wasn't quite all there. Okay, so your experience is still more on the Salesforce side than the yeah. I've I've still do .NET to the day. I do, but yeah, I've sort of it's sort of started off purely .Net, and then I started doing some APIs, and now I do sort of
[01:42] a strong mixture of both. Excellent, because it's really hard to wear two hats, you know, in two totally different technology stack. I I couldn't imagine how much time you use, or you know, really play with the two balls. Yeah, yeah. I mean, I don't know if I dive as deep into DotNet anymore, but I certainly do keep it keep it going. Excellent, excellent. So today I invited you to talk about the topic. It's the parallel unit testing, right? Because I read your blog post. You you blogged in details about how you tackle this parallel unit testing Salesforce. And the different steps you have tried, and then how you interact even with the Salesforce support. So you write down a lot of details, and really give people, you know, some some advices, and also inspiring people to really sometimes to dig into the details as well. Yeah, it's a topic. I mean, it's something that affects me every day. I I need to run my test suite every day. I want to look for regressions. I want to make sure I'm getting in the right path. I mean, that's why I have a test suite. I want to run the test. And so, it is something I care a lot about.
[02:57] And I'm interested in because I have to do it all the time. The faster I can do it, the more I can do it, and the more use I get out of it. Okay, so let's start with a little bit of introduction. So, what does it mean parallel unit testing? So, like long ago, when I started with Salesforce unit testing, your unit tests would run sequentially. They go one after another, so one would finish, the next one would start. I mean, that's that was fine, except of course. It takes a long time, and the more tests you have, the longer it takes. I mean, I've heard of people whose tests run into the hours to run them all sequentially. Maybe I am fortunate; mine takes about fifteen minutes. It's still a lot of time. Like you push the button, it's like, well, am I going to sit here waiting while it goes? Whereas, in contrast, if you Turn on. Like I can't remember exactly what release it was, but they introduced parallel unit testing, and so your tests. Well, they run in parallel. They run together. So a subs subset of them will start running all at the same time, and of course, if you're running a bunch together, you'll get through the total faster.
[04:11] Okay, there is a setting in Salesforce called Disable Parallel Testing, so you can force it to run one way or the other. But that's the general. The general idea. Okay, but by default, because I never really paid attention to that settings, that the tick box you mentioned, by default, what is the the running mode? It will run in parallel. It'll try and run as mini tests. Well, I say as mini tests. It will try and run quite a few tests in parallel. Now, my findings have been, it will actually run up to about 30 tests in parallel, which is quite a degree of parallelisation. Yeah. But is that the Salesforce platform decide how many unit tests are running in parallel, or is it? I believe it is. It's not like a documented feature about how many they actually will. But empirically, I've seen it run about 30 in parallel. Like if you sit and watch the test run. And you'll see how many are spinning at any one time. You see twenty, twenty-five, maybe up to thirty going at once. So, for example, if I open the the Dev Console and then I select all the unit tests and then click to run them, let's say asynchronously, then those unit tests are running in parallel, right? As you just mentioned, until I Untick that box in the setting. Well, yeah, there's a like an older, historic synchronous unit test
[05:41] method, okay, which goes back to the APIs. There's actually a different two different methods you call depending on how you want them to run. The synchronous one and that one is actually sort of a busy wait. You call the web method and it doesn't return until the test methods are finished. Now. Okay. Obviously, that's a bit of a problem. It's great if you want to run one or two test methods. It's not really the way you run your entire test suite. Yeah. Yeah. So the the dev console would keep like this loading bar and forever until everything finishes. Right. Yes. Yeah. It's yeah. Historically, the advantage of that was the test would run immediately, whereas it used to be if you put them in the parallel queue, they'd sort of you might take twenty, thirty seconds before it's got round to running it. Okay. It it wasn't as immediate. These days, I find it's pretty pretty rapid, pretty quick that it starts processing them. Yeah, because what I learned is that always when I try to run in the Dev Console, I always tick this asynchronous box before I running anything, then I get the the window back to my response immediately. And I didn't really see big difference, actually. Yeah, yeah. I mean, yeah. So I was sort of coming from when I think about it, like a decade ago, it used to be quite different. Okay. And then t here is other differences around like how you get the debug log back, but again, most people would never notice
[07:11] the difference. Okay, okay. But this has nothing to do with what we are talking about today, the parallel unit testing, right? There's parallels. There's parallels to it. It does. It does
[07:26] feed into the way you use the test. Typically, most people are going to want to run them asynchronous when you're running a large volume of them. Yeah. In another word, that does not decide whether the unit test would run in parallel or not. So these are two different things. Am I right? Ah, yeah. There are there is a degree of control over whether you want to set. And I don't think you can do parallel synchronous testing. Ah, okay. So synchronous will always be.
[08:00] Non-parallel. This serial. Yeah, asynchronous could
[08:05] be serial or parallel. Ah, that's interesting. I never paid attention to that. Yeah, and that's where that checkbox comes in as well, as to whether you how you want your asynchronous ones to run. Okay, I am going to do a test later, but I don't know. I never paid this attention, but it's good to know that in Salesforce we have this parallel testing and we have this. Serial testing, maybe. Yeah, I try to define the terms here. Okay, so now we set the ground. We know what we're talking about. Then. I understand that you try to run your solution unit test either in the parallel unit testing mode or the serial testing mode. And in the serial mode, it works perfectly. All the tests pass, right? Yes. Yeah. And they all pass, and it's great. And you think I've done a good job. Everything's right. But I want it to go faster. So you think, well, that's what parallel testing's for. So turn that on, and then. Some tests start failing, and and it's not like a typical failure. It's like a um, it's like a compile error almost. It says connection was cancelled here. It's not like your assertions failed. Like Salesforce just stopped the test running. It killed it.
[09:29] That's that was what I was seeing. So connection stopped. It doesn't really tell specifically what the reason was. Yeah, it's not like an exception you're typically expecting to deal with. Or it's not really even it's not even an exception you could catch. It's more of a system level something's gone wrong. Okay, and then after that, you decide to really dig into this these errors. Yeah, yeah. I really wanted to see if I could resolve that because I want my tests to run in parallel. I want the advantage of saving half the time running my tests if I can. I mean, okay. Well, that's like. The degree of parallelism you can get, like I say, if you could run two threads, in theory, optimally, you would save, you would do it in half the time. Right. And if you could run four threads, and then we start to do a bit of maths, it would take about an eighth of the time, or something. You get the idea. It's like, yeah, the more threads we get, I mean, there is a degree of diminishing returns as you add more threads. It's not necessarily going to get much faster.
[10:34] Okay, but that's supposed to be really a remarkable a number of time reducing, right? Even with two in parallel, that's already half of the time. Yeah, if I can half the time, that seems like something that's worth investigating. Okay, so how did you start? From where do you start to with your you know troubleshooting? Um. Well, I mean, because it's like an internal system area. The first thought was like, I'm going to raise this with Salesforce support. So I started that thing, and then I sort of started exploring other areas myself. In Winter eighteen, they added a new attribute for tests classes, our test methods, and it sounded like perfect because. I think actually no. I think it might be at the class level, not at the method level. So on the at as test attribute, you could say is parallel equals true. So I think the idea behind that was you would disable parallel testing for the entire org, but then you would go and cherry pick individual test classes that were okay to actually run in parallel. It's kind of a weird concept. It's like, don't you tell Salesforce don't run anything in parallel, but it's actually okay to run these ones in parallel. Oh, and that that kind of sounds like a nice idea. It's like these ones are parallel testing safe, and I think it introduces introduces a number of additional requirements on those tests, like type of DML and stuff you're allowed to do in
[12:05] those parallel ones. Okay, that sounds good. But I mean, I have like I should do a count, but there's I think 800 plus tests to run in my case, so I'd have to go through and attribute. Put that attribute on. I don't know how many of those, and I thought the reality was, there was only a subset of my tests that seemed to be failing. Although, admittedly, that. The failures changed per test run, so I thought, well, what if I could just say don't run this test in parallel? So I tried putting that attribute on a few classes, test classes, and it did nothing. Doesn't actually work that way. So you can say it's you can say it's parallel true. But you can't say it's parallel false. Okay, so you wanted to have the other way around, basically. Instead of say everything running in serial mode by default, you want everything running in parallel, while you handpick only the ones that failed and then put them into the serial mode. Right. Yeah, that seemed like a pretty simple solution because it's easy to see the ones that fail. Every time one fails, I say, "Well, let's take this one out of the." Yeah. Yeah, makes sense. It didn't work. It didn't work. No, they just continue to run in parallel. It just does nothing. Okay, but it's a good idea. You try to put the true to false and then try to see if it works.
[13:37] Okay, so if it doesn't work, then what did you do then? Um, so sort of bit of backwards and forwards with um, Salesforce support trying to get like, well, what's the underlying problem? And they they sort of take the simplest way out to start, which which is just to say, if parallel testing doesn't work, just don't use parallel testing. I was like, well, I knew parallel testing doesn't work. I want to know why it doesn't work and. So I had to sort of, I had to try and prove to them that something more was going on here.
[14:10] What I found was, I think at the time I had about seventy-two test classes which contained those eight hundred or so test methods. Okay. If I ran them in serial mode, I'm trying to remember. Well, what I found was. In my case, the parallel testing time was taking more or less the same amount of time as the synchronous. Maybe the synchronous was a little bit slower, but not a lot, which seemed very odd. Like I wasn't really getting the advantages of parallel testing you'd expect to get. That's interesting. So I sort of had to dig into why that was happening. I think I reached out around that time on Twitter to say, "Well, why are my parallel tests failing so much?" Some people thought the problem was around custom hierarchy settings. Have you used those? Is it part of like custom settings that you can define different tiers, one depend on another? Yes. Okay. Yes. So you can have. So you can have your settings, and then you can say, my settings are either global for the org, or they are at a profile level, or they are at a specific user level, and that's where the hierarchy comes in. Okay. But I use those in the package to control a lot of on-off toggles, and that's nice because you can have a global setting. Or you can say system admins get these settings, or this specific user gets these settings, and then that's. But the tests
[15:41] have to obviously work with those settings. And the belief was the DML operations around those custom settings were causing problems. Okay, but the problem was okay. I I need to isolate those from my test cases. So Salesforce had introduced something called the stubbing API. Okay. You know what stabbing is for testing? Yeah. Let's step back a little bit. Is that it's the people in the Twitter pointed you to this custom hierarchy setting instead of sales for support, right? This is just a potential reason. Yeah. The commu a community feedback was that that was something they'd seen the similar sort of problems of very slow or problems with parallel testing, and they they isolated their problem as. I see. The custom hierarchy settings were causing a big impact, and when they took them out of the test cases, they found the problem went away. So I thought, now you try to see if that's really the problem in your case. Yes, and. So yeah, so I went down the stubbing route where I basically isolated the code in my classes that uses the custom settings, and I made a mock that I could insert in during testing that would. Prevent any socket queries or other database interactions with those custom settings. So the mock
[17:11] you are using a stopping API, Salesforce stopping API. Okay, because you you mentioned that maybe some of our listeners doesn't know what what that is. So could you brief us what what is stopping API? Yeah, it's quite neat. So I think if a class is public, has a public constructor and public methods, or maybe I don't know if it works with protected methods, but you can provide an alternative implementation of that class during testing runtime. That will get used whenever you call those methods. Now, so what you typically do is you'd implement your your stubbed version, and then tell the .NET tell Apex whenever you whenever you ask for one of these
[18:01] classes. Or whenever you try and call methods on one of these classes, you're actually going to get the results back from the stub, and then that way you can control exactly how it handles those returns.
[18:16] And that was really useful. I see, I see. But most of our listeners might be familiar with this HTTP mocking API or this class, right? When you need to call out in the test, instead you just get the dummy data back instead of really making the call out. I think the stubbing API is the similar functionality, but much more powerful, right? Yeah, similar concept, just more at a class level than rather at the API callout level. I see, I see, I see. Okay, okay. And that was useful, and I tested it, and it did sort of remove some of the problems. But I think, in some ways, it just moved the problem elsewhere. So you mean you are using the stubbing API to eliminate the DML when you need to, how would I say, to touch the custom hierarchy settings with the yeah, basically. Prevented any interaction with the custom hierarchy settings when that mock was enabled. Okay. So that way, I could say, well, I've isolated this; it's not touching it at all. Because the code-based text went into the custom settings all over the place. Like, is this feature turned on? Is that feature turned on? Yeah. Yeah. Okay. And it does not help on your case after you. Not significantly. No, there was a. I think mine had another underlying cause. That was causing that
[19:47] was my problem. I mean, it was an interesting exercise, and I think it did speed up the tests a bit, but didn't get to the underlying problem. Okay. Did you did you keep those stopping implementations? Yes, it's still there. It's actually quite nice because it gives you quite nice level of control over what your settings are. Okay. There was no reason to pull it out. So then, what else you tried to proceed? So another one of my sort of a bit of a parallel. Another one of my interests is debug logs and how you can visualize what's going on in the debug log because a debug log, I think it records timestamps in the nanosecond scale, and it's really hard to get your head around like how long a nanosecond is. I think I think like. I, I took the all the events that were happening or the major events that were happening in a debug log, and I panned them out onto like a little timeline. Okay, so it should give you a sense of like, is this a big major event? Is this a small event? Where did this happen relative to the other things in the log? So you'll be looking for things, major things, DML trigger execution. Where did particular errors come up? Yeah. And then, so what I did was I ran all my tests, captured the debug logs, and then visualised the debug logs
[21:17] all together, stacked up. So you could see like this transaction did this, this transaction did that. And what I found was one particular DML operation and the setup of most of my tests was getting longer and longer and longer, until eventually the test started failing. So this was. This was not a major DML, but that gave me like a really good like. Here's a smoking gun. Here's something that's going clearly going worse with a degree of paralysis. Um, so it was a DML operation to insert only six records in a custom object. It's like that should only take a few seconds, seconds at the most. Yeah, and I think I found it was dragging out to like fifty-two seconds plus. Oh no! Three minutes or four minutes, by which stage the test started failing.
[22:15] But that was interesting because this was this particular custom setting. I'm trying to remember. It controlled something like an option that would go on each opportunity, like price type, pricing type, is being used on the opportunity, and because everything in the managed package depended on. Opportunity records. Just about every test case needed to create an opportunity and assign at the right price type.
[22:43] Um. So all my test classes were trying to insert these records. So that was that was sort of like where I got to at that point, and it was quite interesting to get there through the visualization. It made it quite easy to see. So what what is that visualization tool?
[23:04] It is the I distribute it through my employer. It's called the Fuse IT SFDC Explorer. Okay, so it's a free tool for everybody to free tool. Okay, trick it trick is it's .NET based, so pretty much Windows only at this moment. I've been looking at porting it to .NET Core, so it could run on other OSs as well. Okay, right now it's Windows only, but part of that tool is yeah, it'll take a debug log. Parse out the events. Draw a timeline. Yeah, yeah. It's it's beside the topic of this episode, but you know the visualization for debug log. Every now and then, it's really important, as you mentioned. A nanosecond, nobody have have a clue what that is. But then, when you put into the timeline, you know what things has happened. As you just discovered, for certain DML in the setup, just inserted six records. You just mentioned right, and then you took several minutes. That's definitely some smoking gun. You mentioned it. To to check about yeah and it's it's a lot easier when you can see them stacked together and to say oh here's the one that's stretching out okay I will get the link from you and then put to the show notes okay okay yeah um so. From there, I think I took that back to support to see what they're going to tell me is useful from that information. It's like I can say, "Here, look, this DML is breaking it. What's going on?" Trying to remember
[24:34] what support said, I think they did identify it as a problem, and then that was the reason the transactions were being terminated. Basically, they were all deadlocked, trying to wait for this database operation, and it's kind of weird. It's like, why are the test cases not isolating the DML operations? Why are they all dependent on each other? Indeed. One of those, one of the theories around that was the those particular records have an indexed field, and I think when I was inserting them, I was putting the same, like the test same test data in that index field. So I'd have price type one, price type two, price type three, and price type four, and every single test would have those same records. So I think the theory, the theory there was, we've got they're getting deadlocked because the index isn't isolated between the testing context. So something we tried to do was.
[25:46] To make every test setup put like a um unique stamp on the index value, it would insert okay, and then that way, in theory, the indexes between the test cases wouldn't conflict. So, like when test class one sets up, it would. I think I actually think I might have used like a nanosecond timestamp. As a as a prefix, okay, just make sure it's unique in that index. Yeah, every time the test run, it would generate a unique index that would be repeatable. But I don't think that worked either. Actually, it still still sort of fell over a bit.
[26:30] And then I think there was a lot of backwards and forwards with Salesforce support saying, "Well, why is your test isolation not working?"
[26:39] And I think the end result was we couldn't isolate it. We couldn't get it isolated, even with the unique index. It's still falling over there.
[26:51] So I mean that sort of brings me to like as far as I was able to push that. I think I went backwards and forth with R and D and sales support for a while, trying to push that forward, but that didn't really get too far. I think they are aware of the issue that the the isolation of test data and parallel testing isn't fully working. Hmm. So, because from your solution, your unit test, it doesn't sound like a really special case. It's like a general cases that you have that set up for a custom object. I would assume, right? And with some fields, one of fields is yeah, it's just. Yeah, I believe that's the problem. Well, it's it's hard to say because we made the index field unique, and it's still happening. But certainly. Just trying to repeatedly insert those so many standard records isn't working. Unfortunately, mocking wasn't really a solution here because. This needs to be in the database. Everything else depends on that relationship from the opportunity. Okay. I see. I see. So, until today, you don't have a solution for your problem, right? I've been thinking about other ways. I mean, this comes back to what we were talking about at the start. So. You run
[28:21] it. You can run it in serial, and you get one thread, and it just runs through them. Or you can run it in parallel. Now, initially, in parallel, you might say, "I just want two threads, and I should half my time." And then when we talked about how Salesforce actually runs it up to about thirty threads. Now, I don't know if you've done much parallel computing theory. Like in thirty threads, okay. Well, in thirty threads, the chance of a Contention is much higher than if it was at two threads, but Salesforce doesn't give you a way to control. How many threads it runs in parallel?s So if I could say yes, I want parallel testing. I only want two or three threads. Then my problem would probably go away. Yeah. And I would get overall better throughput. So I think I raised a Salesforce Ideas idea about the ability to control the degree of parallelization. Okay. I mean, that's maximum we can do if they one day really implement that. Yeah, that's something I've been pushing for. Though it's always hard pushing developer ideas to the Ideas Exchange, and this is another one of my soapboxes. It's like it's hard as developers to get ideas promoted. To the level that, like, say, maybe a general user or admin idea would work, because there's so many, so few developers to so many admins and users. Okay, we are minorities. Yeah, yeah. I have
[29:51] have other ideas. Okay. Um, I was working on a blog post, but I haven't actually put it out yet. Okay. Maybe by the time we publish, it's already out. I I should do it. I should. So many things to catch up on. The so. Usually, when you do asynchronous testing, and that turns into parallel execution, you say Salesforce run these tests. You give it a collection of tests. Typically, I would have been saying run all my tests. The alternative is you sort of spoon feed it a subset of tests, and then you wait till it's finished. And so there, we're trying to trade off the overhead of The control mechanism of passing in and monitoring the test execution versus not getting blocked. So I might say, run these five test classes, and then I monitor it. And when they're finished, I put in another five, and another five, and I just keep spoon feeding them in. And the idea is it won't get so deadlocked because it can't run. More than the five I've given it, and then the goal is between when I start and when I finish to get maximum throughput. And so by trying to keep the parallel testing, but reduce its reduce how many it can actually run in parallel. I'm sort of trying to work around the problem. I have had some success with that, but there is an
[31:21] overhead. On how quickly you can monitor it, and then how quick it puts in new tests. Okay. Yeah. Nobody wants to take control, you know, manually. Immediately start the next round of unit test. So I've got code in the Fuse ATSFTC Explorer that does does this. You can check a box, and it will monitor the test execution. Ah, but I think, and it monitors the test execution and tries to spoon feed it in. But yeah, just the advantages or disadvantages you get from. The overhead of by the time Salesforce notifies you via the streaming API that the test execution's finished and you queue up the next batch, whether you're going to see better throughput overall is questionable. Interesting. You're really spending time to think about how to granularly control. Well, it's like every time you run your tests, you got like fifteen minutes to sit there and think, "How can I make this faster?" Yeah, yeah. I am so happy to have you know gurus like you to really spend time on this, even if. You know the backend is the black box. You only go via the support to to to really dig into the details. But still, you use all the tools that you can use to to to try to accomplish the goal.
[32:50] Cool. So, those are the ideas you're gonna try in the near future, right? You don't know whether it works now or not. It's just the idea. Ah, yeah, I've been experimenting with it, and I'll put it out there because there's there's a number of people who use that tooling, that tool. So when the feature is there, they can test it and tell me, like, did you get better? Because obviously, everybody's test suites a little bit different. It'll be interesting to see what happens there. Excellent. So, thanks a lot to Daniel. I think this is a good storytelling. Even though at the end, some of us, the listeners, might expect, okay, you solved the problem. And together with sales for support, and it's a happy ending. But at the moment, still we have this things to dig into further. Yeah. Right. Yeah. Yeah. No, my quest quest still continues. I shall keep at it. Yeah. But thanks a lot to Daniel. I mean, this is like a invaluable experience you are sharing with us, and you really spent your time to to fix your problems as much as possible. Yeah. Yeah. Well, thanks for hearing me out. Maybe somebody else will have some other ideas they can share. Yeah, indeed. Explore those. Cool. All right, that's the end of our session. Thanks, Daniel. Let's talk. Next time. Thanks. Bye.
[34:13] Bye.
