Kevin Jones, who joins to talk about his open-source project ApexLink, is a Senior Architect in FinancialForce.
Sponsor
This episode is sponsored by Copado.
Copado is the first Salesforce Delivery Management solution and the #1 native DevOps platform for Salesforce.
Copado DevOps 360 is the first Salesforce Delivery Management application, enabling you to measure the performance of your Salesforce implementation.
Main Points
- Kevin’s self-introduction
- What is ApexLink, Kevin’s open source project?
- Is ApexLink similar to the PMD?
- Is ApexLink similar to the Illuminated Cloud?
- Why does Kevin use Scala language to create ApexLink?
- How does ApexLink work?
Links
Video Teaser
Contact Me
If you enjoy this podcast, please reach me out to let me know your thoughts or simply say Hello. My (Xi’s) contact info:
Take care, until next time :)!
Transcript
AI-generated, unedited.
[00:00] This episode sponsored by Capado. Capado is the first Salesforce delivery management solution and the number one native DevOps platform for Salesforce. Capado makes the jobs of Salesforce admins and developers fun and easy, even in the largest, most complex works. Plan and collaborate on work, then track your changes right on the user story with an admin-friendly Git interface. Automate development and testing, and track metrics so that you can target improvements. Capato DevOps Three Sixty brings advanced analytics such as value stream mapping, executive summaries, and rich interactive dashboards. These insights help you find inefficiencies and delight your users by delivering innovation faster than ever. Take the pain out of the Salesforce development process and make Salesforce development fun again with Capado.
[01:11] Hi everybody. This is Xi Xiao. This is yet another new episode of Salesforce Way podcast. Today I'm sitting with the guest. His name is Kevin Jones. Hello, Kevin. Hey there. How are you doing? And good, and good on Sunday, a sunny day, and we both are locked in the in the room, right? Oh, yes, definitely locked at home at this point.
[01:32] So, so Kevin, would you like to introduce yourself? Yeah, so I'm Kevin Jones. I work for Financial Force. I'm a senior architect there. I guess I've been working on Salesforce for about four years. In fact, since I joined Financial Force, I didn't really have any experience before. So you you are UK based, am I right? Yeah, so I'm based in the UK, and I kind of, I guess, in my role is kind of a fairly general architecture role, assisting the product architects and the product teams that we have at Financial Force. How big is Financial Force? Actually, I heard there are many developer teams inside and many managed package as a product. Yeah, there certainly are. So there is twenty plus developer teams, roughly. I mean, it varies. We form teams and you know move them around as required. The number of packages is actually very difficult to put a full figure on because there are a lot of extension packages around Financial Force. So they're essentially seven core products, and depending on which core product you're looking at, there'll be some number of extension packages, and they do vary widely. Okay, we will talk about a bit shortly about your role as a general architect in such a big ISV company. You know, a lot of people, including me, are really interested. You know, like what do you do daily and what what's your role? But let's let's park that aside for a short period. So let's focus on our main
[03:02] topic today. Is your open source tool? It's called Apex Link. Am I right? Yeah, yeah. So I've been working on this for for quite some time and kind of tweeting about little developments on it for a little bit. Is it also something you do for your work? So then you open source it, or is it really your side project? So it's Apex Link is really a side project. It's it is open source. You can go to GitHub and get it. And we do use it internally as well. So there's kind of use cases around around it that are kind of important in terms of financial force. But largely, you know, I want it to be seen, or you know, its main value is in allowing better tooling for the platform. I think in the open source world. Okay, so it's a new tooling. So, could you brief me what is it about? So, if you if you go back a little bit, I think when I when I started working on on Salesforce, one of the things I noticed was the the tooling that we had available to us didn't really work so well with the scale of the packages that we we deal with. So, some of our packages are extremely large. You know, kind of. It's difficult to put put numbers, but say perhaps up to a million lines of source code in a single package, and the tooling. You know that we have available made certain things particularly difficult. I probably point to refactoring as being one of the hardest things
[04:33] to do on on very large code bases. So, if you want to change the design in some way, typically you'll have to make numerous changes across the codebase, and then you have to deploy, and the deploy takes quite some time, obviously, particularly on large packages. You know, we can be looking at an hour and a half to do a deploy to work out whether, you know, take the worst example, whether you've missed a semicolon off the end of a line. You know that's the that's that's the really horrible outcome. You know is yeah I've waited an hour and a half to be told that I've missed my. So, what I really wanted was something that could give much quicker feedback. So, what it's about is looking at, largely looking at a package is what we look at mostly, and trying to work out very quickly whether it's in a good shape to do the deploy, so that I can modify many files, redo a design, and know it's got a pretty good chance of being able to deploy without having to wait the time period that it was going to take. So you mentioned it's a local tooling. So it means you we don't need to deploy to the Salesforce platform and get the feedback. Everything happens in your local laptop. Yeah, yeah, exactly. It does all of the work. You know, it's kind of a library. So, so basically, it just runs on your local machine, analyzes the package. Mostly, what it's doing is analysing the classes in the package. You know that's really the focus of what it's trying to achieve. But in doing that, you have to look at other metadata types. So, you know, we look at custom objects,
[06:03] we look at labels, platform events, flows, all the things that you can reference out of Apex. We need to know they exist, and you know what fields are on the custom objects, so that we can validate that you're actually referencing a field that exists, as opposed to You know something that you've just misspelt, say, you know. So largely it's Apex, but it's Apex plus the metadata that goes around that as well. Hmm. I see. That's always the the case because it's not just about the code, the Apex. There's this custom object, maybe custom metadata types or the other. Or even flow or process builder, right? Around your package. So if you are missing those metadata, then your package is not self-contained. You can't really get all the dependencies ready. Yeah, I think this has traditionally always been the big problem with Apex code analysis. Is it's not just about the Apex code. You do need to have access to all of these things, which of course means you have to parse them and understand them to a certain degree. So you know, Visual Force pages is another good example where you've got page dot, and then the name of the the page is in the Apex code and. We need to check whether that's a real page or it's not, you know, and whether you are calling it correctly. Often, you'll find with the the other metadata types, we don't need quite the same level of knowledge about them. So, in the case of Visualforce pages, very simple analysis is does the page exist or not, as opposed to you know actually understanding what the control of the page might be doing using. So, t here is a bit of flex in those areas in some
[07:33] of the areas. As to what you need to know, probably the hardest of the the set to be to be straightforward about it is the custom objects, custom metadata, and platform events. So that set obviously contribute towards the schema. So they're they're all visible through the schema mechanisms, so you have to know quite a lot about them, and they intersect with the platform types in a lot of different places. So getting that part of right is is fairly tricky, I think. Hmm. So when you say it's a local tooling, I immediately think of the PMD, one of the local tools. I talked in the past with author Robert Sausman. He told me that he extract a Apex compiler from the old Force. dot com IDE, and then he used that as like the foundation. So what is what is this Apex link here? So what's a kind of difference? I want to see this angle. So I took a slightly different tack early on when I was starting to look at this problem. It wasn't obvious how you you did this with the. Salesforce compiler jar, so people didn't really understand what Robert managed to achieve. So I'd already started down the path of writing or putting together a parser using Antler. So Antler is a parser generator tool that's pretty well used out there, and then
[09:03] when I looked at it, I thought, well, I could switch. But the problem with switching was essentially I was using this jar that was coming with really no understanding of what was going on inside it. You know, there was no so that meant there was no ability to change anything. And from my point of view, being able to look at changes in the language was something I was quite interested in. So instead of using the jar, essentially I rolled my own version. So if you go and look inside Apex Link, you'll find the grammar for for Apex that I use. It's it's moved forward, you know, quite. Quite a number of times we've had to improve it. Probably the most obvious case recently has been the new switch statement. You know, add that in. But largely, it seems to be about right. One of the challenges around this area actually is that there is no formal grammar for Apex. You know, it's just not written down anywhere. So we've got reasonable guides about how you use it, but from a you know a tooling developer's point of view, it's actually quite difficult to know whether your grammar is good. I think all I can say is I know it works with the samples that I work with. You know, you know, it's it's capable of parsing everything and doesn't seem to make mistakes in how it does that. So, it'd be interesting to see how that develops. There's there's a slight little side angle here. I think one. One area of the language that I think most developers would really love to see some improvement is is support for something like generics or templates. So at the moment you've got essentially
[10:33] three generic types that are commonly used: you know your list, your set, and your map. And if you're a Java programmer, of course, you can write your own. And the ability to be able to write your own collection types or other kinds of generic type would be so valuable to us. And so it's always been in the back of my mind that if you're going to do that, you're going to need your own parser because, you know, that's not really part of Apex. As it is, you need to change the language a little bit. Hmm. This reminded me in the past I had a guest, Scott Wells, the Illuminated Cloud author. So he spent a lot of time to, you know, use regular expression to really parse the language, the Apex language, and then build everything on top of that. So, are you doing something similar here as well? Yeah, so it's probably pretty similar. I don't know what Scott has used with Illuminated Cloud. So the the grammar that you give to Antler, essentially, it turns at least part of it into kind of regular expression kind of matches for you. I don't really work at that level. It's kind of one level abstracted above that. So you write the grammar, and it'll be things like I'll declare. I don't know if you want an if statement. You know, an if statement is constructed from the keyword if, and then there's going to be some expression which should evaluate to true or false, and then you expect to see an opening brace, and then a block, and then a closing brace, and that's the level that the grammar is kind of written at. So you know, it's kind of one level up, and then of course you have
[12:03] to define what do you mean by an expression, and then what do you mean by a block of code, and You know, a block of code is fairly straightforward. It's a list of statements, usually. So then you go in and say, "Well, okay," you end up recursive there because you know one of the statements could be an if statement, of course. So the grammar just lays it out in in kind of relatively simple terms. They're not too bad to read, actually. If you ever want to go in and just kind of understand how Apex constructs, they're a little more difficult to write because there's a few more concerns. But in terms of oh, you just want to go and read the grammar, then there's one available to have a look at. And Antler turns that, from my point of view, it turns it into essentially a Java implementation for me. It's auto generates, co generates the Java implementation, and I just ask it to parse Apex files, and it gives me a tree which represents the source code that you've given it. Hmm. So when you say it's a tree, for example, if I have one Apex class, then you do start on the top. The root is the class name, right? And then it contains multiple methods under the tree as the second layer. And maybe each method contains input, output, like inside of it different types of like string, integer, all those kind of. Is it called abstract tree, a AST? Yeah, exactly.
[13:33] Abstract syntax tree is usually the term used for for that tree. I actually use two. So Antler will generate you an AST automatically. You know, it's part of its code generation. I tend to do most of my work in Scala, so I recode the tree into a Scala version of the tree. Which is it allows me to slightly abstract from what the powerizer generates. Sometimes the powerizer generates too much detail. And actually, I want a little less detail. I'm not concerned with some of the subtle differences, so I kind of uplevel it just slightly, and at the same time, move it from being something which was designed to be in Java to something that works in Scala, which is convenient for me. So you say. You said it's written in Scala language, right? Yeah, and it's open source. Yeah, in the GitHub. So does it mean if I check the source code, I'm able to see how you interpret the Apex class files, and then you generate the AST? From the code, if I check that into details, yep. If you go in there, you'll find that there is a there's a scalar class which essentially is an if statement, if you like, and you know you can go in and see how that gets processed. And see what what kind of validation we do on a if statement, you know, or any other kind of statement. So there are obviously Apex Link is a kind of work in progress, as these things always are. So some of the levels
[15:04] inside it aren't as complete as you know. They would be in a final version, so mostly what I've been working on actually has been expression evaluation. So the expression evaluation part of it is pretty complete. And the say the evaluation, the validation of the expressions. Largely, what that's doing is making sure that things like method calls actually go to a method, or if you reference a field, that the field exists, and we know what type it is. You know, the the the that level that it works. There's a level above that, which is statement validation, which is if statements, switch statements, and so on. That's not so complete. But the layer above that is really type validation. It's kind of saying, do we understand type references in general, and that's pretty complete. So there's a bit of a mixed picture, but you can you can get a sense of the kind of work that's needed. Hmm. Why did you pick Scala language to write this tool? Is it because you are more familiar with it, or is it better for this purpose? Um. Actually, I wasn't that familiar when I started, but um, I've kind of got a bit of a background. Yeah, exactly. I've kind of learnt the language as I went along. So I've been yeah. Oh, that's cool. Yeah, learn one language to to write another one. I try and understand another one. It was really that. I kind of got a bit of a background with functional languages, and I knew that they were very good at processing AST-like structures. They're just a very natural; they tend to have really good facilities for that. And being an
[16:34] open source, if you like, a part-time project, I knew that you know my time to commit into this was limited. So using a language which was really going to help me seemed like a very attractive choice. So we've kind of stuck with it all the way through. I mean, Scala itself is is interesting if you're into languages, just because it's kind of on the cusp of what a modern language looks like. So it's quite of interesting for me to look at it and see what kind of features it has. It does compile down, obviously, to run on on the JVM. You know, so it produces a jar just as it would if I'd written it in Java. So from that point of view, the usability is very high on it as well, which is quite nice. Hmm. Yeah. Just for some of our listeners, maybe if you want to extend your programming languages, Scala. I know it's a JVM-based language, right? It has a kind of a interesting combination. It has object-oriented, like the old Java. It also has the functional. Part so you can choose either side and even you can mix them together. I don't know if that's a good paradigm, but definitely a lot of people are using on either side. But it's kind of funny that you know I've been around a little while on on object and functional languages and played on both sides, and of course now they're converging, which I think is great. I really love to see that, and actually we see that a little bit in Apex. People have done. You know, functional libraries for Apex, which I never would have thought possible. It's really quite imaginative work, but it's really nice to to play. I think with both paradigms simultaneously
[18:04] and try and get the best out of both of them. Hmm. Yeah. So I want your help to like drill down into this Apex link. How does it really work? But of course, it's audio format. We can't look at the code. Can you just give me like the steps? How do you? Get the Apex, and what do you do? And then, what's the final product coming out of this tool? Yeah, so I've kind of gone round a few loops of design iterations. I am going to give you the the very latest, which I think is probably the most sensible. And I am not sure why I didn't get there to start with. But so, the top level abstraction in the library is a kind of simulated org, which may sound a bit weird. But when you're trying to replicate something like Salesforce deployment, which is kind of what I'm doing, then it makes sense for me to kind of copy what Salesforce do. It means it's a much more natural mapping. I can kind of think about what actually happens on an org and what I should do. And I've got this kind of parallel. So the top level abstraction is you create an org. Now the org itself, if you like, because it's simulated, is just an in-memory structure. So inside the org, you're going to have a set of packages, just as you would have on a normal org, and you deploy the package onto the org just as you normally do. Of course, it's all in memory, so you know we don't have any persistent state there. I focus quite a lot on packages because you know my background's with an ISV, and everything we do is about packages. But there
[19:34] is an unmanaged package in there as well, so I kind of. Looked at it and thought, well, how do I deal with unmanaged code? Because we have that on ours, and said, well, actually, from my point of view, it's best if I think of that as being a special package type, and and it's just the unmanaged package that you can deploy to. So you can do both managed package work with namespaces and unmanaged, and both seem to to work fine. So you use a local computer memory. You allocated some memory just to simulate. The Salesforce work, yeah, that can host further different type of packages, and then on top of that, you can do the work. Yeah, obviously my my simulation is very thin. You know, it's really only the bit that I'm interested in. Of course, you know, it's not doing any of the execution side, but but in terms of keeping me in sync with my thinking, how it should work, it seems to be a natural way of trying to look at the problem. Okay. I think when I've looked at it in the past, I've looked at it more like a classic tooling problem. So the the origin of Apex Link is to think of linker in traditional toolchains. So I don't know if you're not familiar with that. You know, back when you were writing C or perhaps C plus plus, you create object files and then you'd link them all together. So this notion of actually comparing things against each other. But actually, I don't think that you know have kind of evolved from there and said, well, actually, I think I'd be better off, you know, centering around the idea of orgs and packages, essentially.
[21:04] Okay. So there's a. That's the first step, right? Yeah, I think it's an iteration, isn't it? Of I don't really understand how to do this. Nobody seems to have done it before. You know, you're going to get it wrong a few times, and you might have to rethink how your design looks. So, so the current version is centered around this org package model. Once you get below that, that's really just the high level kind of concept. Obviously, within a package, you've got to start thinking: what do I actually have? You know, we we've talked already that you have quite a lot of metadata types that appear in a package. But because I'm centered around just understanding Apex, the thing that's really important to me is the notion of a type declaration. So when I talk about type declarations, I mean classes or enums or interfaces. You know the the classic types that are created by Salesforce. But there are other kinds of type declaration out there, like the type declaration in the schema created by a custom object, which is a kind of different thing. But they all come under this kind of banner of I'm declaring some type that is visible to the Apex language. So inside the package, what you get is essentially an index of all the available types in that package. So they're coming from the Apex classes primarily, but also from custom objects, platform events, custom metadata, and they all become visible. Inside that scope, so you can go to the package and say, "What what do you have inside you?" And that's largely how the cross validation works. Because obviously, if you reference a type
[22:35] in an Apex class, I need to be able to go and find it, and you know, I'll find it by looking in the package and this index of what's available in that package or not available. So does it mean that you need to scan through all the files inside the package? For example, if I install a package, I understand that there are different folders like object folder, application folder, process builder folder, and you need to go through all these files to check what the types you need to index. Yeah, exactly. So the almost the first part of the construction of a package. Is to give it a directory, and at the moment I have support for a couple. I have the old Metadata API package structure it'll support, and also the new SF DX. So it'll it'll auto identify an SF DX folder structure and construct a package around that. But the very first thing it does is it scans all of the the content in that directory structure in order to work out what's available. When you construct a package, you tend to have to do it in specific orders. So a good example probably is you know I want to construct the labels first because the classes can depend on the labels, not the other way round. So doing the scan of all of the metadata means I know what's there and I can pick out oh you've got these three labels files I'm going to load those first and then I'll load the classes later. You know so there's a dependency order built into how you load the metadata essentially. I see. I see. That's that's clear to me. It's a step two. Yeah.
[24:05] Yeah. So I guess there's a another kind of layer here, really, to understand, which is this is often the most difficult layer. Is is how do you actually do the validation on the Apex classes? And and I probably want to point you at one specific problem, which kind of. Illustrates some of the difficulty. So, if you go to Java or any kind of Java-like language, or even .NET languages, mutual dependency is something that was brought into these languages. So, I can have a class say A and a class B, and A can reference some method in B, and B can reference some method in A. And you look at that problem. Actually, from a programmer's point of view, this is really nice because it's very flexible. You didn't used to be able to do this with the older languages, but you can these days. So, if you want to validate, you know, which one do you validate first? You validate A or you validate B? You've got to make a choice, haven't you? And it's like that doesn't work. So, a lot of the next layer of complexity comes in in trying to work out how to make how to do validation in that environment. And I have a very simple kind of solution, which is the kind of two pass approach. So first thing that we do within the package is load the class and pass it in order to understand what's in that particular class file. So you know, as we talked about earlier, the the properties, the fields, the methods, the inner classes. We need to know about all of that stuff, and we load all of the classes in that way in the package. Do a one pass across them, which is just simply load it up
[25:35] and get a basic understanding, and then on the second pass, it does validation. So it'll run through the same set of classes and validate them. But of course, when you're doing validation, you've already loaded the other class. So there's a kind of ability then to reference other things. So if A references B, well, B's already been loaded, so we can do that. And similarly, B's referencing A. It doesn't matter what order you're doing it in. So a lot of the mechanics in the package is about this two-layer validation. First, powers everything. Then run a load of checks over it to make sure everything is correct and. The output from the tool essentially is the result of that second pass of validation, which is a set of issues related to the classes. So if it notices something is wrong, it will then flag it, and that will give you error messages. Okay, so it's a two-step again in in in this phase, right? Yeah. So I understand what is the parse. You know, you need to get all these properties inside the the classes, for example. But you mentioned the circular dependencies in Apex, which is similar to Java. But is that the only reason you do this validation? Because I try, I still try to understand: Is it really mandatory to have a two-step here? Yeah, I think well, it's difficult to imagine how you could resolve the dependencies between classes without putting two steps in, or at least you know a minimum
[27:05] of two because of the mutual dependence problem. You know, you've essentially got to load. All of the classes to know whether they exist and whether they're valid, and then then you can start looking at detailed, you know, issues around them. There is quite a lot of complexity here in. Drawing the line between these two passes that I'm describing, because it's one of those cases where the algorithms you use as part of the validation influences where the line is drawn. So you have to be careful in, for instance, in the first pass, not to try and use an algorithm which requires that something else is loaded because it may not be loaded at that point. So you kind of, when you're coding this, you have to be kind of aware of whether other types of metadata are visible to you at that point or not, and constrain the algorithms that you're using depending on what data you have available. It's a, okay, it's probably one of the harder parts of of kind of tying all of this together. Some examples might help, actually, just to be a bit more specific. So, let's say we've got a class, and I want to know what fields are available on that class. And you think, oh well, that's a simple thing because you can just iterate; you can see them all. I've loaded them all. On the power tree, but of course, the fields available are also coming from any inheritance that you might have. So, if you've extended some other class, then that field potentially is visible on your class as well. So, actually, you you almost get into this. Dependency
[28:36] of across classes problem straight away there because you don't know that the class that you are inheriting from has actually been loaded yet, depending on what load order you did it, and of course, so you can have slightly different algorithms in terms of trying to chase these things down. So, to give a concrete example, you know, if I load a class A and it extends a class B. Then I could, you know, code in that I should automatically load B. But there's nothing stopping, you know, also having mutual dependency as well as inheritance at the same time. So trying to resolve that becomes quite a tricky problem. So doing two pass just makes the whole thing a little easier from that point of view. I can see
[29:19] that. Hi, this conversation continues on the next episode. See you next week.
