Chris Peterson, who joins to talk about Apex language, is the Director of Product Management for Apex in Salesforce.
Sponsors
This episode is sponsored by Clayton.
Go to Clayton website, request a demo or use the chat service by telling “salesforceway” code to receive free one month Secure Development Essentials.
Main Points
- Is the Apex team currently working on the language expressiveness?
- Is it a good idea that Salesforce supports some good open-source libraries in the ecosystem?
- Is it valuable to build an Apex package management tool like npm for javascript?
- Shall Apex add features to distinguish various types of Apex tests, such as unit test, integration test, large volume data test, bulkification test, etc.?
- Apex team’s working effort to have better meta-data coverage
- Chris’ tips on writing good Apex tests
- Is better mocking in Apex test needed by developers?
- What is the new Apex transaction finalizer?
- Discuss several ideas to support the open-source community
Links
Video Teaser on Apex Language
Contact Me
If you enjoy this podcast, please reach me out to let me know your thoughts or simply say Hello. My (Xi’s) contact info:
Take care, until next time :)!
Transcript
AI-generated, unedited.
[00:00] This episode sponsored by Clayden. If you build on Salesforce, you know writing secure code can be really hard. Clayton makes it easy for you. Clayton plugs into your Git repos in seconds and automatically protects you from hundreds of security vulnerabilities, such as Crowd FLS violations, code injections, vulnerable third-party libraries, and more. Clayton works with GitHub, Bitbucket, GitLab, and Azure DevOps. Go to getclayden.com/salesforceway to get one month of secure development essentials for free. It's getclayden dot com slash SalesforceWay. You can also find the link in our show notes. Clayden built on Salesforce, stays secure.
[00:58] Hi, this is a continuation from the last episode.
[01:03] But still, you know. Talking about the Apex language, since we're here, people are saying that still it feels like a Java 1.1 from the syntax point of view. A lot of time we need to work with the map, work with that's not fair collection. It's more like Java 1.5. Okay, yeah, you know the point, right? You're right, and and. Part of that was during the compiler rollout, which was a four-year process. The success criteria for the compiler rollout was that the old and new compilers produced identical bytecode. Okay. So, trying to add new features to the language would have actually been counterproductive for getting the compiler to finish. When the compiler did finish, we rolled out switch, which was one that people had been asking for for twelve years, something like that. Yes, yes. But we needed to do more. Absolutely.
[02:03] And part of that's on me. In fact, a large part of that's on me. That the issues I've seen with Apex in the wild and the ones I've struggled with personally in my time being a customer and a partner, I thought security needed attention much more urgently. Language expressiveness is hugely important, but. If the language is highly expressive, if we added generics and did them as a first-class citizen, but you still had to have this copy and paste, describe-centric security model, how much are we really helping language expressiveness? If we don't give you effective baseline tools for things that you already have to put into your code, and do in a painful way. You still have this these blocks of copy and paste FLS checks. I don't think that that was the right priority, relatively. And I think generics are incredibly powerful. I I worked on the Financial Force open source libraries for a while, and not having generics made me want to rip out my hair a couple times. So this is not to discount the power of generics and language expressiveness features. It's to say that. I thought other things needed the attention more urgently. Asynchronous processing did not have effective feedback mechanisms, so we put out batch Apex error events so you could at least tell what records failed when there was an unhandled exception in a batch job. If there was a gap before, your choices were to guess or just be sad. Like that. That's not good enough. Transaction finalizers are trying to do the same thing for queueables and potentially future potentially
[03:33] scheduled. We'll see how that goes. But this this fire and forget with no feedback, paired with a platform that tends to be the system of record, was something that concerned me deeply. I understand. How can we be the system of record if we don't have a more robust system for handling failures and understanding what the failures were? And so. Yeah, between security and that, like I had to prioritize those two, and I, I couldn't see an argument for language expressiveness while those were still sitting there in need of attention. And now that we're working through more of these fundamental issues, and there's the state of security on the platform in Apex is not the end state of where I would like it to be. But it's no longer in a state that it needs urgent attention. It's something that we can continue to iterate on and continue to improve. But it's no longer the top pain point of every partner I talk to. Yeah, I heard you. Believe me, I heard you, and people also heard you. So we understand. I was thinking about one things to support from the Salesforce point of view to officially support some good open source libraries, so that it grow much healthier, and you know, get more people to use that. Is is this a good idea? Yes, so I actually just stepped up as a maintainer for the ES API library because the old maintainer left. So if you if you're not familiar with it, no, it's a Library that Salesforce Product Security put out as the original field and object level security recommended library
[05:04] because even Salesforce hated doing the manual field and object level checks through describes. So the Product Security Group put out a open source Apex library. And a lot of partner apps use it because it was the officially blessed tool by product and AppExchange security review for a while, and so I took it over as one of the maintainers along with my product security rep because it needed some love, and so definitely open source Apex is really important. I'm really excited about packaging too because it makes it a lot easier to handle versioning and dependencies on libraries. You're talking about the second generation library, or yeah, second generation packaging. Okay, the fact that it's going to have support for putting multiple packages in the same namespace means that as an ISV partner, you could put your library code and two consumers of it as three different packages. Whereas historically, you'd really end up copying and pasting your library code into both of those packages because they couldn't coexist in the same namespace, and you wouldn't want that library code to be global necessarily. So this this happened in a large number of partners that I've been at and seen code from, where the way you include a library is by copying and pasting it into each consumer. Exactly. So second generation packaging, I think, improves that experience substantially. But I think you're talking also about like open source Java libraries, not Apex as well, right? Not really. I don't care about Java at the moment. Mainly the Apex,
[06:35] but because when I touch in some other modern languages, most of them they have their package management tool like npm for JavaScript, right? Yeah. Haskell has its own. Closure has its own. But for Apex, is second generation packaging really the tool to manage the libraries? I think it's a big step forward, and I'm. It is. It is going to pay very close attention to the feedback we get as it rolls out, and also talk to the CLI team because I think for the first time in Salesforce history, we have a unified. CLI based approach to managing projects holistically instead of each feature team kind of having their own. Different metadata API representation without a common tool chain, yeah. And with second generation packaging being a lot more flexible in this area, I think there's a lot new new opportunities here. I don't have a specific vision in mind right now for what you're talking about, like npm imports or something like that. Yeah. Okay. But I do think it's really important. When I was on the AppExchange team, it was something I spent a lot of time talking about how. At some point, there needs to be a tool like this. Yeah, but I think that we're still building some of the the prerequisite elements right now, and.
[07:53] Towards the end of this year, next year, I'm guessing it'll be something that I spend a lot more thought on. I don't have a specific vision yet, but I think I will. People were pinning it on Twitter to remind you. Good. Okay. A lot of really interesting features have started that way. I mean, transaction finalizers spun out of a comment Dan Appleman made to Stephen Tam at Meet the Developers three years ago. So yeah, okay. Offhanded comments do sometimes turn into something material. So send them. I'm interested. Okay, cool. I wanted to also share some pains that I've had, and also some my past guests. They told me things to ask you. Oh, my favorite part. Yeah. Okay. Talk to developers, right? Yeah. People want to have annotation or a certain way to distinguish the unit test, integration test, or large data volume test. Do you think if that's a good idea? Yes, absolutely. Okay, okay. The testing system in Apex is something that I like. The foundational elements, the fact that you could have this testing infrastructure that guarantees everything is isolated, is wonderful. You heard me talk about how critical hammer is to our efforts, and so. I'm very selfishly incentivized to get people to write better tests because it helps my my product be more successful in the wild from Hammer. And we definitely haven't put as much investment into the test infrastructure lately as I would like because
[09:23] we've been focusing on asynchronous and stability and security. And I think it's time we need to take at least a we need to take a look at what we can do realistically without major roadmap disruption. And I think that that's the type of thing that is very reasonable. We're not making a major alteration to the runtime. We're not making a major alteration to the test siloing capability. We're just adding a tagging mechanism and trying to enrich the execution APIs. Yeah. I do have to go and argue with a couple people on my team who are really passionate about testing about if that beats out the relative priority of something else that we had looked at, looked at, which is parameterized tests. So inside of Salesforce, we have a lot of tests where we want to run the test on every permutation of some kind of thing. So you may or may not have heard of metadata completeness, the the effort to make sure that. Everything that's in Salesforce has a metadata API representation, so all of those preferences that are only available in the setup UI are properly exposed in the metadata API. Right, right, right. The community team and in Spring, no manual work, no manual work. Exactly, the community team just put out a true source controllable representation of Lightning communities in Spring. That's cool. So there's been a lot of work to try and close those gaps, and one of the ways that was implemented internally is the metadata team wrote a parameterized test inside Salesforce in our proprietary
[10:54] homegrown test system that said for every attribute of an org that can be changed, they're called org preferences internally. For every org pref that can be set by a customer. Generate a test failure if it does not have a metadata representation. So they just assign these thousands of test failures out to everybody and said, "Fix it," and. That same type of thing holds true in Apex test. For every type of field, I want to make sure that this test code passes. And you can do that today with a lot of copy and paste or a loop, but you get lower fidelity feedback in those tests. You'd write one test that handles all those cases, and then you get feedback on the first case that fails. You don't know if you broke ten cases or one. And so, parameterized tests. Since we've used them so much, I've seen a lot of value come out of them as a way of doing different permutation testing with higher fidelity results. Is one of the other things that we have been talking about doing in the relatively near term. That's cool. So that's the type of quality of life testing stuff that absolutely I am very interested in because it's not as intensely disruptive to other aspects of the platform and the roadmap as some other things that we've talked about. And I think that, especially when we find ones that are easy to implement and impactful, it's really hard to justify not just doing it. Hmm. Okay. I want to use this time since we're talking about Apex testing. What are the good strategies for effective,
[12:24] maintainable Apex unit tests? Could you teach us without using you know third-party libraries? That's a wild question because it is because because I know you hate the people use the I plus plus. Oh God, I don't want to talk about it anymore. Yeah, that's the thing. But we need to, you know, we need to. the the the senior guys to to teach us what are the good ways at least some some points right yeah I mean a lot of the the good testing practices in Apex aren't unique to Apex you look at good testing practices anywhere it's write tests that run quickly mock out dependencies when you can I think that that's actually one of the bigger ones is when you can avoid interacting with the database at scale your tests will run substantially faster. The slowest aspect of most Apex tests is how much time is spent writing to the database and rolling back transactions. So, when possible, write code such that you can do unit testing instead of integration testing through the full save order cycle. Write a handful of test methods that make sure that your trigger does what you expect it to, through DML operations, but then write all of the permutation and edge case testing that invokes the trigger handler directly without going through the save order cycle, so that you don't have that extra latency and overhead for database writes and rollbacks. Because the biggest issue I've seen at scale is
[13:55] people saying their tests take too long, and so they don't run them as often as they should. I think that goes back to your point about being able to tag tests as unit versus functional versus scale. Yeah, and so I think you're increasingly convincing me that that's something that's worth prioritizing. Along with this conversation goes on.
[14:14] But that and that's a really important one is making sure your tests run fast enough so that you can actually use that feedback. ScratchOrgs help set up a CI environment with ScratchOrgs, like. No brainer. It used to be a lot harder to give recommendations about how to run tests at scale. If you're not running your tests every day in a scratch org right now, you're missing out. That's that's kind of like table stakes as far as I'm concerned for good testing strategy. And with source control as the source of truth, instead of orgs that you have to meticulously maintain and handle sign up and handle cleanup on, it's just an order of magnitude easier than it used to be back in the day. I remember people I worked with writing incredibly complex metadata API clients that would try and list everything in the org and generate a destructive change manifest so that it could reset the org back to baseline. You don't have to do any of that anymore, so it's wildly easier. I I know things have changed a lot, positively. I I joined Salesforce for about two years. I've seen so many good tools coming up, like a Salesforce CLI, Scratch Org, these things. It was really a game changer. They bring it in line with a lot of more modern development methodology, and I think that it's really nice that I can sit here and not say use these Salesforce proprietary tools and techniques. Instead, it's. Use the same tools as good unit testing in any any language. If you're writing Ruby unit tests, you still try and avoid writing to the database. So it's
[15:45] the fact that I can say use an off-the-shelf CI process, and yeah, you're invoking RCli instead of a different environment sign-up tool. But so what? Your your build task is still a build task. It's just using a different implementation. The interface is still fundamentally the same. So a good CI cycle where you reset environments and run tests every day. Is so much easier, and that's the best practice everywhere. So, increasingly, as we come in line with these. I'm the wrong person to talk to because there are unit testing experts across the whole industry, and the more of their wisdom applies to Apex, the better. Because, but still, you know, one of the things is that the Apex doesn't have a really good mocking library. One of the good ones is the FF Lerp, right? It has mimic the Java. Was it Mosquito? What was the name? Yeah, it's based off Mosquito. Okay. Yeah. So, so what's your word here? Do we have a plan to improve the mocking part as well? Yes, so there are limits to the stub API, which is what it actually uses. So the FF the FF mocks. And the underlying stub API that uses an Apex are kind of joined at the hip. So Apex built a API to generate mocked subclasses of classes that aren't virtual because before that it was using code generation to try and. Create
[17:15] new classes that it could use as the mocks, and code gen as part of your build step is really a large amount of overhead. It adds extra code to your org into your package. It wasn't the right way to do it to make it scale. So the Apex team built the stub API, which is what they use now for generate mock objects. But it does have some constraints. I believe it doesn't work on inner classes today. There's some system types it doesn't work on, things like that. And I would love to prioritize those. I have one problem there, which is you are the first person who's mentioned it to me in the last year. Okay, if everybody piles on and tells me that better Apex mocks are what they need right now, you bet I'm going to take all those quotes. I'm going to put them into a presentation, and I'm going to say this is this is now risen high up on our roadmap. Yeah, I have thousands of listeners. I will ask them to shoot your message soon. Legitimately, one of the things that that. You know, I always believed that it would be true, but it shocked me how much how true it was. Was is just how much customer quotes impact decisions at Salesforce. If you're going to make an executive presentation, you don't have two or three quotes from customers who are really invested in the ecosystem. It's going to be dramatically less impactful than if you did. So, a quote from me in my own presentation is 10 is good as a quote from you or your listeners, and it very materially does impact
[18:45] the roadmap when we hear over and over again that something's a problem. We heard security was a problem over and over again. We went, I went to partner security conferences. I talked to the AppExchange security team. Every time I talked to a partner at Dreamforce or even anywhere else, the first thing was, when are we going to fix FLS and CRUD checks? And when I hear it every time, and my executives hear it every time, when I put that at the top of my roadmap above. Shiny new feature X. We're going to invest in infrastructure that doesn't have a big marketing impact, but makes the lives of our customers better. That's awesome. But I need to prove that last part of the sentence makes the lives of our customers better. Prove that better mocks is the thing that I can put there and say this is the quality of life improvement we need to make our customers happy. So it's not just me saying. Yeah, my gut feel says it's about right.
[19:42] All right. Before I should go, I want to take the time to ask a question, which is a timing question. It's about the Spring twenty feature. One feature is the Apex Finalizer. I know you you have talked about it, but it's a really interesting feature. Could you tell us what it is about? Yes. So at its most fundamental level, it is a callback when a queueable job. Finishes, and that could be success or failure. So it is limited to queueable right now. I would like to expand where it rolls out to, but certain contexts are more challenging than others. Lightning controllers, for example, because they're compounding multiple actions together, adding extra overhead there could have material performance consequences. So we we need to tread carefully. So it's it's not a no that we're not going to expand it to other places. It's that Queueable was the easiest place where there was the least amount of concerns. And I thought the most need for it, because a queueable job can't really convey specific information about its failure conditions, other than by throwing an unhandled exception, and then you get two hundred and fifty five characters of the message in a sync Apex job. That's not really a good enough level of fidelity, and you'd also have to enqueue something else to look at the queueable status to note that it failed, and so we wanted to do something that increased the robustness of queueables, which is why we decided to make it a callback when the queueable. Either succeeds or fails, so if the queueable job gax even the finalizer
[21:12] will still run, and so self retry was something that. You couldn't really do with queueables unless you had a scheduled job that was a watchdog that tried to watch for failures and reenqueue. With a transaction finalizer, you can reenqueue the queueable job that failed as a retry mechanism if it was a transient failure. If you hit a timeout in a remote API, retry it three times and see what happens. It also works really well for logging, because the way the finalizer works is you create your own class that implements the finalizer interface, and then you call one method system attach finalizer, which says this is the callback for this transaction. But it doesn't serialize the state of the finalizer as you call that method. It just marks that object in memory as the callback, and you can mutate the state of your finalizer throughout the entirety of the callable execution. And the state of that finalizer is then brought into the new finalizer context where it runs. So what that means in a more concrete example is you could create a list of strings on your finalizer. And as your cuble is running, continually append new log strings to that. So, I'm about to make a callout at X seconds into the cuble execution. I've made the callout. The result was success. It took 23 seconds. A queueable job has now completed. All of its strings into this buffer, and then when the queueable job completes, at that point
[22:42] the finalizer serializes. We use our binary serializer, the same one we use for like queueable job state itself. The queueable job context is closed. So we either commit or rollback on the database side. We get rid of the Apex context, which means all of your static variables and all of your limits go away. Then we create a brand new Apex context and new database context, and deserialize your finalizer into it. So whatever your finalizer state was, ends up being a state in the finalizer when it runs. You mentioned even if the execution GAX, still the finalize would be kicked. Yeah, because if you want to do a retry, we need it to work on unhandled exception. We need to work on limit exception. We need it to work on GAX. Hmm. There there are certain GAX that are so fatal, like the database crashed that we can't recover from. But. Those are few and far between compared to a null pointer exception, some Java code that can either be retried or logged or something like that, depending on what your needs are. And it also it has other use cases too. Like you can't mix setup and non-setup DML operations in the same database operation. So by virtue of the finalizer execution and being a clean database context that doesn't have anything already written on it, because your parent queueable job is either completed or rolled back, you can do data DML in your queueable and setup DML in your finalizer. I had this issue for the user
[24:12] object. Yeah, exactly. Yeah, now you don't have to enqueue another queueable. You can just clean up the final final state. I've reparented the ownership of all these records, and now I need to add a group member to properly do the sharing recalculation. Cool. It's a queueable and a finalizer, so it's still one request. It still is one thing, and you don't have this orchestration challenge that you might otherwise. Cool. They were built to be really low level. So, Kevin Porman, who's a developer evangelist, has already updated his promises library to use cqueues or to use finalizers, so that when you have a promise in your cqueueable chain fail. It can properly retain that state, and so I'm hoping that there will be more open source frameworks. Going back to our earlier conversation, using the finalizer interface to do things that were desired but hard to do in a framework level before. So a logging framework, a retry framework, promises layer of abstraction on top of cubbles, etc. I'm hoping we'll have that type of framework around it as open source tools that you can pull into a project. More so than I'm hoping people use finalizers directly because I think it does fit itself better as a feature that enables new types of frameworks than it does as something that you're interacting with directly yourself all the time, which is a lot like the stub API and FF mocks. Yeah, for you and for your team, you're building the Lego
[25:42] pieces, and it's up to the communities to really how to use them in the creative ways, right? You have seen this multiple times before already. And the other thing is, performance is part of the API contract for the Apex Standard Library, and so there's a lot of places where I'd love to go and add extra functionality to something that exists today, but if I do that, I probably break somebody. And I also can't really take pull requests from the community into the Apex Standard Library the same way as an open source project. Yeah. And so I think our time with a finite capacity is spent better building new foundational APIs that can be utilized effectively in multiple ways, rather than trying to build overly prescriptive frameworks. And I do realize that per your original, your earlier point, we're going to need to work on making it easier to pull in open source libraries to make that model really scale up. Yeah, and and also for having a healthy community. Yeah, and also I remember you mentioned that you are also one of the open source library contributor. But does still want Salesforce give more incentives to the open source library? I don't know whether money wise or whatever effort wise or MVP titles whatever. Yeah, definitely. I did just somebody from the community team reached out to me to ask how substantial an open source contribution was when they were reviewing MVP nominations. So. I think that maybe more of that is already happening than you realize. But okay, true.
[27:13] Actually, somebody on my team just yesterday opened a pull request to the PMD tool to try and add support for the with security enforced clause to one of the security rules that's there. Okay, I'll ask Rob about this. It's yeah, I don't think he's seen it yet because we just opened it yesterday and there was a test failure that we had to fix. But at the end of the day, yes, we need to do more on that front. We're limited on how much we can do ourselves, and so I'm most interested in making it so that open source tools have the support they need. Like getting Apex Apex language server support into PMD was something that required an engineer at Salesforce to help with. Because there isn't a lot of documentation around it, so it needed help that only we could give for that initial push to make it an integrated feature. And then after that, Robert has done an absolutely wonderful job helping maintain and curate the list of rules. Some of which have come from Salesforce in terms of the product security group contributing some. I think this is the second minor rule change that's come from the Apex team. So my goal is to keep contributing, but especially focus on the areas that only we can do something to help and empower the community in that way, and make us available for the parts where we have a unique capability to help more so than others. Thanks, Chris. I think this is excellent point for us to round up. Wonderful. Great.
[28:40] Hi, I'm constantly looking for good guests. If you have any guest recommendation, please reach me out. I'll make sure they are joining to the show to share their knowledge. Otherwise, thanks for listening to the show. I'll see you next Thursday.