Jonathan Gillespie, who joins to talk about his Nebula Logger open-source project, is a Lead Software Engineer at Salesforce.
Main Points
- The Nebula Logger in general
- Different modules and features in the architecture design
- The new Plugin feature
- Real-time platform event monitor component
- Configuration layer and custom hierarchy setting
- Open source learnings
Links
- Jonathan’s LinkedIn
- Nebula Logger Repo
- The architecture diagram
- Jonathan’s Nebula Logger blog thread in Joys of Apex
- James Simone’s episodes in the show
Video Teaser
Contact Me
If you enjoy this podcast, please reach me out to let me know your thoughts or simply say Hello. My (Xi’s) contact info:
Take care, until next time :)!
Transcript
AI-generated, unedited.
[00:09] Hey everybody! This is Xi Xiao. This is yet a new episode of Salesforce Way podcast. Today I have a new guest with me. His name is Jonathan Gillespie. Hello, Jonathan. Hey Xi, how are you? I'm good. I'm good. Would you mind to introduce yourself quickly? Sure. I'm Jonathan Gillespie. I work right now as a lead software engineer at Salesforce, and I've been in the ecosystem since around 2011 or 2012 in kind of different roles. I started out originally as more of a business analyst doing Salesforce admin work, then moved on to doing Apex and Visualforce development, and then eventually moved on to doing consulting as a technical architect for about five years up until now, and now I'm here at Salesforce. Hmm. You enjoy coding, right? I do. Yep. Over the last six years or so, I've also been contributing heavily in the open source world for different Salesforce projects, including working on Nebula Logger. Yeah, Nebula Logger. That's the the topic we want to discuss here. So you started to create it since when? Back around two thousand sixteen or two thousand seventeen, I was moving between jobs, starting my third Salesforce job, and wanted to just learn more about Apex, and just the platform in general. And so one of my friends and colleagues, James Simone, and I were also just wanted to you know keep in touch, and so we worked together on some open source work. That was originally supposed to be more of an overall
[01:39] framework for Salesforce. Ended up moving away from focusing on some aspects of it, like the trigger handler framework and other details. But it seemed like the ecosystem for Salesforce was still really missing a robust logging solution. So I took the section of the code that was doing logging and moved it to be its own separate project. That's now Nebula Logger. So I had a look at your repository. So you spend a lot of effort to create good documentation instructions how to use the tool, without really needing to dig into the code details. And also, in case we want to study how you structure the code, you have this architect diagram to show how you modularize the different components inside your repo, right? That's right. Yep. As as new features have been added over the years, the code base itself has gotten pretty large. So trying to organize it has been something I'm I'm constantly working on because I think there's a lot of people that want to see the actual source code. The idea of logging, it's something that there's sensitive data being processed and people just want to understand how logging is handled and what risks, if any, there are. So just trying to have the code be clean and easy to read has been one of the goals of the project I've had. Okay. So I'll use that architect diagram as a reference here. I have it open on my screen, and maybe you can have it open on your screen as well, in case you don't remember everything. And for our listeners, you don't need to you know to check that. But of course, I have it attached
[03:09] in our show notes in case you want to have a look later. But here we just quickly go through the high-level things because it's really neat, clean. I can see that things are modularized. But there are things I'm really interested in. Let's just talk those things through. The first part is the logger engine layer. I have it on my screen here. That's the first module. It clearly has the feature saying that it supports the declarative tools. It supports Lightning Web Component, Aura Component, and of course, it supports the Apex. So from these three different sources, we can log. Use your Nebula Logger tool, right? That's right. Yep. And so the logger engine layer is really where you interact with the Nebula logger. So if you're doing Apex development, or you're building Lightning components, or you're you're building flows, logger engine layer is really the the only part of Nebula logger you have to interact with. There's a whole bunch more that you'll get. We can of course talk through those details. But really, I wanted it to be easy for a few easy entry points for people to start using it without necessarily having to worry about the rest of what Nebula logger does for you. Hmm. But you definitely had the preference or the order to build this feature up, right? Maybe start from Apex as the beginning. That's right. Yeah. Originally, it was just very small logging class for Apex that was years ago. Just writing directly to a custom object. And then,
[04:39] as new features have come out, both from Salesforce and just other people have asked for new features to be added, I've expanded that to include Flow was one of the first ones I believe. I think I did Flow first or Process Builder, and then expanded out to Aura and Lightning Web Components as well. Okay. For the flow and the process builder, you use the invocable Apex class. That's right. That's the bridge between the declaratives and the the engine that does the logging in the Apex side. That's right. Yep, I think invocable actions are are really powerful, especially with how evolved flows have become. You can do a lot of stuff directly in flow, and then you can very easily supplement it to like this by just adding Apex actions or Apex invocable actions. So yeah, so it it serves as kind of the middle ground between or the middle layer between your flow and the rest of Nebula Logger. I'm not so familiar with the declarative tools. But when we lock the information, do we only lock, for example, we create or update some records those like information, or do we really lock the errors that was thrown out from the flow and the process builders? I'm nothing. There's definitely for every team, it's going to be a little bit different as far as what your system's doing and what's meaningful to your team. But personally, I do think that you should consider logging more than just errors. You don't necessarily need to know only errors. You also sometimes need to know the other context of what's happening
[06:09] in that same transaction. And one feature of Nebula Logger is. Typically, if you're using like an Apex view system debug, it's not very performant, and there's not really a way to disable some of it. Nebula Logger though has a custom hierarchy setting, which allows you add either the entire org level, a profile level, or user level. To control what logging is enabled for that particular person, so I think it makes because you had the ability to disable that at runtime, just simply updating configuration. I encourage people to always try and think about logging more than just errors, because when you're trying to do some very thorough troubleshooting, if you're having some widespread issue, or you just want to monitor some some new feature that you rolled out. Uh, not everything errors aren't don't tell you necessarily the full picture. Right, right. So I think having having a strategy for other logging levels helps. Hmm. I mean, for me, because I'm not so familiar with declarative tools, and I'm I'm not so trusting that it's efficient, especially when the loop is is involved. I'm happy to really log more information. But my my question was more like, are we able to log the errors into the backside? Yes. So I think it was a year or two ago. Flow Engine now has the ability to log what they call Flow Fault Errors or to handle Flow Fault Errors. So for each node inside your flow, if there's some issue that happens at
[07:39] a particular point, you can add a Fault Connector. Is I believe what they call it. And from there, you can basically—it's a way to handle errors within flow. So when you combine that with Nebula Logger, it gives you a way to log any errors that happen inside of your flow. Okay, that's a good information. And then you also have the Lightning Web Component and Aura components supported. That's right. Yep. How do you do that? So in order to do that, I I basically added a Lightning Web Component that you can now embed inside of either your own Lightning Web Components or or components. And that way, it's when you're doing JavaScript development, you can also call just another JavaScript class, which is the Logger class. And that lets you just not have to necessarily interact with Apex directly. The component itself handles that for you. You're just focused on calling some JavaScript functions. And then that component does. Then on the backend, it handles talking to the rest of Nebula Logger's logger engine. Ah, so it feels like it's a API inside JavaScript. You can just call that. Exactly. But that still means that in order to use your logger, you have to attach this component into every and each of the pages. In the front end, that's, yep. And we just had a recent change to it. Traditionally, or originally, the way this was built, you would embed the logger component inside of your markup. That still functions and is around
[09:10] today. The new way that just came out, I think. Last month or the month before, is you can now import the logger component as a JavaScript module. It's a little bit cleaner, a little bit simpler, and it works for more lifecycle events like connected callback inside of a Lightning Web Component. Okay, so Salesforce is still improving the features on that side, right? Yeah. Let's shift onto the Apex side, which I'm most interested in. So in Apex, you have this feature allow us to log the information with the different ways. For example, we can do the DML. Then we can even use the REST API to do the logging, and you also have the platform event to support it. Different ways. Why do you do that? There's just like anything in Salesforce, especially in Apex, everything has limits, and there's trade-offs with each of those different save methods. Originally, when I first started trying to build first logger years ago, a lot of none of those options existed. The only way I had to do it was just trying to write to a custom object. But the way that Apex behaves, or the way that the platform handles transactions, behaves writing directly to a log object is not a great solution, especially when you're trying to deal with Apex exceptions. So, for example, just any Salesforce object, be it a standard object or S object. If you do something like
[10:40] you insert an account and then you insert some related contacts, if something fails on those contacts. Not only will the contacts be rolled back, but everything else in that transaction will be rolled back, including accounts. Same thing applies if you're trying to use a log custom object and write to it directly. If you end up having exceptions in that transaction, then your logging data is rolled back, so you don't actually get to store some of the logging. You can get around it some by doing try catch blocks. And some other things, but it's it's not the most robust way to log when you're expecting there to be actual exceptions thrown. Ah, so if I want, I can just. Try catch it without throwing it up. I hide these errors in in order to write the logger, but yeah, but still my upper level Apex code still want to see what's going on underneath, right? Okay, okay, and and that's the reason you started to explore the platform event. That's right. Platform events, I think, are probably the most critical aspect or the architecture for any logger on Salesforce, Nebula Logger included. That's because they have two distinct behaviors that make it really nice for doing logging. First and foremost, is they have the ability to, well, just Step back a second. Platform events are basically a very special type of s objects in Salesforce. It's something you configure. I don't really want to say the term custom objects because they don't behave like a traditional
[12:10] custom object. But there are special objects in Salesforce, and so and the idea behind them is they use a kind of a publish and subscribe model, pub sub, and so kind of behind the scenes, there's stuff that's happening asynchronously. The first behavior, though, that's really important for logging purposes is. With platform events, you can configure the object to use this feature called publish immediate. And what that means is that when you publish or save a platform event, if it has publish immediate enabled on it, even if there's later an exception in that transaction, the platform event will still save. So using that as part of the core engine, you can log data, save it to a platform event, and then still throw an exception in Apex, and your platform event won't be rolled back. So you have to tick that tick box in order the platform does not roll it back. I think so. I can't remember if it's the default behavior or not, but it's yeah something you configure on the object. Yeah, that's a good information. I mean, I assume because it's a platform event, it's kind of like sit outside of the core part. So, I assumed by default it should be not rolled back. Okay, that's it. Might be the default behavior, but there is the other behavior of it's treated more like a normal S object in that if there are errors, then your platform events will be rolled back. So. But publishing media is is really critical for logging.
[13:41] Okay, makes sense. So it means in your repository there's a subscriber side as well to to listen to that and then write into the the log object. That's right, and that's where the second module on the architecture diagram comes into play. It's called the log management layer. So, logger engine, yeah. The logger, the log logger engine layer is basically responsible for generating the platform events and exposing ways to do that via flow, Apex, and Lightning components. The log management layer basically listens to those platform events. And then transforms them into custom object data that you can then report on and do whatever you want as far as the Salesforce platform is concerned. Okay. Since you started talking about the second module, the log management layer, let's let's quickly go through that. So. I can see that you have created multiple custom object, the lock underline underline see the lock entry. Those things are easy to understand, right? For each execution, I would assume there's one lock record, and that there could be multiple lock entries attached to it. But you also have other object. What? Why do you need those things? Yeah, so the like you said for sure the core objects are log and log entry. Log entry is anytime you add some message that becomes a log entry. Every message and a single transaction becomes a single log, so you end up with one log, multiple entries.
[15:12] There are three other objects though. Logger scenario. And then two objects related to tagging system. Both scenarios and tagging share some have some common functionalities or concepts behind them. The idea behind both of them is that you're enriching your logging data. So, for example, a lot of people like to use the tagging or labeling system as a way to just kind of add additional strings to that are meaningful in your code. Those end up being additional records, just like you would have in Gmail or any email client where you can add labels to your your emails. Same idea, you're just adding extra information onto every single entry. So tagging is is really just about adding in these additional words that are meaningful to a person looking at the entry and provide extra context in that sense. Scenarios have similar idea of you are still providing basically a string value and that is used to enrich the logging data. The difference though with scenarios are more about identifying particular code paths and adding additional controls to those code paths. Yeah, I got it. I mean, before discussing with you, I was confused. Why do I need all these things? I want to get rid of them. You know, just have the call to start with. But when you said this, I do have worked with customers that have multiple teams on different clouds, even sales clouds, marketing clouds, whatever clouds.
[16:42] They are. Handlely totally different things. It's nice to have this scenario or the tag things to separate them. And maybe can I use these different tags to even do reporting and assign the logs to different people? You right now you can't do automatic assignment based on tags. That's something that I think a few people have asked in the past. Not not supported right now, but maybe in the future. You absolutely can use it for reporting though. That was really the original goal of Nebula Logger when I started it. Is You can use like standard Salesforce provides basically the system. Debug method inside of Apex that shows up inside Developer Console and Apex log object, but none of that data is reportable, and so it's very hard to make make it actionable. Right. So all of the data that Nebula Logger generates, ultimately the the main goal is to try and get it into custom objects so that you can then report on it. You can set up your own email alerts, Slack alerts, those kinds of things. However, you want to use that data. You know, for using this Salesforce platform out of box features, that part is like a mysterious thing for me. I know it's powerful, but I never had the chance to really hands on everything. But now, when you are doing this, I guess you definitely had the chance to practice those things. How to think about how to use them, you know, based on your tag, your scenarios, how to automate to help the users, right? That's a great experience, you know. Yeah, I mean,
[18:12] I learned a lot from working on this project, and it's it's a way that I I use it almost as an excuse to try new Salesforce features as well. Indeed, indeed. So yeah, it's it's fun in that regard. Okay. And on this module, I also saw you have the trigger handlers on each and every one of these objects. So why do you do that? A couple reasons. One is there are a few features that Nebula Logger handles for you as far as just setting additional fields and things like that. And the way I do some of that is through the trigger handler framework that's built in. But more over the last year or two, it's evolved more to also provide a way for people to further customize Nebula Logger. So one of the challenges when I first started out the project is. Other people really wanted to, especially because then it was it was a much less mature product at the time. A lot of people want to modify Nebula Logger, either add new fields or slightly change customization, or do other integrations with it. And so, about a year and a half ago, I spent time further expanding the trigger framework that it uses to basically provide plugins. So now you can create a custom metadata record. There's a custom metadata type called Logger Plugin, and in it you basically are saying when maybe your logger runs, also run either this flow or this Apex class that I've developed.
[19:43] And so it gives you a way to. Extend Nebula Logger without having to modify the core package, so you can still do upgrades, but you are able to supplement it and add new functionality into it. That's a really nice thing, and I think it evolves along the time because you just mentioned people would like to have their own feature, but they have to customize it based on your code. It's kind of like a forked version. But once once they customize it, then they they lose the chance to get the future update from your end. But now you have this plugging feature; people can plug their code in there. Whether it's a virus, so it's a good feature. That's up to them, right? That's right. But but that's a good. Way of abstracting things to level up your thinking about coding, right? You have to plan these things. Absolutely, yeah. I mean, it was getting this built. I went through two different versions of what I call the the plugin framework. The original implementation was much simpler, but it and did not provide some flexibility that I wanted. So did a redesign of it, and it's been same version for but I think about a year or so now. So I consider pretty stable. And yeah, I mean it's it's basically classic dependency injection of you're configuring your own classes and flows. You tell Nebula Logger. Where it is or what you want to run, and it runs it or adds it at runtime. So really powerful stuff. I do also provide
[21:13] a couple of official plugins, but anybody can make their own plugin. So I have plugins for things like Slack and. Integrating with or archiving data into a big object and some other extra functionality that its functionality I don't think should necessarily be part of the core package or at least not yet. So it also gives me a way to explore new features. That I'm not 100 sure belong in the core package. So I'll develop things as a plugin first. People can test it out, and then at some point, if I feel like everybody may benefit from it, I roll it into the main package. So it it helps as far as just kind of experimentation as well. That that sounds really like a good plan. I've seen some open source projects that just go way too big. I can't really use them. Right? I just need to slice them down into small pieces to use them. Well, I I really admire what you just said. You know, try it out in the plugins, and then later merge it back to the the core part. And on top of the module we discussed here, there are also other interesting parts I want quickly discuss. One is you have the data cleanup part, which is basically I assume is the batch Apex, which will regularly delete the old data in the log. Yep, that's right. So this was another one of the challenges. Again, everything we're doing on Salesforce, we always have to consider platform limitations. Data storage is one of them. I think data storage on Salesforce is one of the most expensive things,
[22:43] as far as how much it costs for for what you're getting. Yeah. So being able to report on data in Salesforce is incredibly powerful, especially because this is a tool being used by admins and developers. We we know how to use this data and how to interact with objects. But it does eat up our customer storage space, so I introduced a couple years ago batch job and some related fields that provide a way to automatically delete old logs. Out of the box, what you have to do is you basically just set up a scheduled job that's included to run once a day, and by default, any log or log entry you generate will stay around for two weeks, and then that batch job will automatically delete it. You can also customize that though per with the custom hierarchy setting, logger settings. So at the profile level, the user level, or even the organization level, if you want to have it something other than 14 days, it's just simply updating a value and saving it, and then every logger will will update the retention date. Yeah, there's a lot of features there, and I I appreciate what you did. It saves developers time. We do need this data cleaning up. Everybody needs it, right? So thank you. You have written that there already for us. And you mentioned it's there is a way to configure the features. I think that's the next one we're going to talk, which is a configuration layer. You have a entire layer to control the feature, how setting up your tool, how it runs. But
[24:13] still, there's one more thing I want to discuss is the UI components. You mentioned when you was guest writing a blog in James Simoes' blog, that Joyce of Apex. Yeah, in the Joyce of Apex. There is a real time was a platform event viewer page. You can check the information. What what what's that about? So it's the component you're talking to shows up on the diagram is called log in log entry event stream, and so platform events again they they use a publish and subscribe model. So the idea is Salesforce is just putting them onto what's called the event bus. And then either internally with things like Apex and LWCs, or externally with external systems. Different systems can subscribe to those platform events, and so in real time or near real time. As those events are being saved, you can monitor them, and so this log entry event stream component is basically a way for admins and developers to watch and monitor in real time those platform events. As well as you can filter along the way, so just kind of a nice way to keep an eye out, and it's basically just gonna keep as new data comes in, it'll just keep scrolling through that new information. Really helpful if you're trying to actively monitor for one particular issue or type of entry. So it's just a nice little way for for keeping an eye on things. Yeah, definitely. Sometimes when I debug some,
[25:44] I don't know, it's curable something, I do want to see how it's moving on along the time. Well, especially with cubles, because or matchables, or really anything that's asynchronous, those are just a little bit harder to troubleshoot by the nature of being asynchronous. So yeah, this this helps because it doesn't matter if it's coming from multiple transactions or not. Just as soon as that log entry is generated, you'll see it there. Right. Right. The one last module I want to discuss is the configuration layer, which is the most confusing for me. I don't want to have so many configuration parts. You have one, two, three, four, five, six. Yes, six different component configurations. So, why do we need so many? It's a lot, and it really all depends on which features you're using. You may not necessarily need to worry about a lot of them. Okay. The main ones to note in there are the one we've been talking about is the logger settings, custom hierarchy setting. That's one of the most important objects, and that is what provides the ability to customize things per user. I think the second most important one in there is the it's a custom metadata metadata type called Logger Parameter. Okay. The idea behind this is this is where I store feature flags essentially that apply for the entire system. So there are certain features that every logger has that either query and process additional data in your org or set additional fields for you. For orgs that have already a lot of complex code in their
[27:14] system, and they want Nebula Logger to be really lightweight in terms of CPU usage and heap size usage, you can turn off some of these features, and that basically helps scale down how much it's how many resources it's using. Okay, that's a really advanced feature. But out of the box, by default, things just run smoothly for me, right? I don't need to comfy. Yeah, okay. the The idea is at least for system admins, as soon as you install it, you don't have to do anything. There's a lot of the things I would recommend doing as far as reviewing and configuring things yourselves, but it's intended to work as soon as you install it. Hmm. One thing you just mentioned is this. What's it called? A hierarchy use. Uh. What's the custom settings? Yep. Custom hierarchy settings. A custom hierarchy settings. What does that mean? You said it can configure per user. So they're another special type of of a Salesforce object. You don't create them under Object Manager Manager, but you do create them under Setup, and I think they're just called Custom Settings there. And so the idea of a custom setting is. Really great for these kinds of use cases. You want to have something configured a little bit differently for different users and profiles. So when you create a custom hierarchy setting, Salesforce gives you some fields on that that are unique to custom hierarchy settings, and they're called the setup owner type and the setup owner ID for it.
[28:44] And so, what you can do in there is you can create three different levels of records. One is organization wide, so that's something that applies to your entire org. The second level is at the profile level, so you can have additional records that are tied to specific profiles. So you can configure things to work one way for system admins. Versus a custom profile versus your sales users, your service users, however you want to divide it up at the profile level, and then the third most granular level is you can associate a custom hierarchy setting record with a particular user. So you can say for this very specific user, use these other field values. And for example, on Nebula Logger, some of the fields that I have on this object are things like is logging enabled. And what logging level is being used for this user? Are we logging at the finest logging level? Are we logging at info, warn, fine, whatever? There's also controls in there for like what save method is used and other details. So all of that, as far as Nebula Logger is concerned, can be customized per user. And so common use cases for it are. If we're rolling out a new feature to a particular group of users, we may have finest logging level set up for just those users that will be using this new feature because we want to know a lot more, especially when we launch a new feature. For other people elsewhere in the org that perhaps aren't using that feature or just. We don't need to know their interactions as
[30:14] much. We could do something like only log errors for them, so it gives you a little bit more control over what's being logged for any particular person. Okay, so for example, if in my organization there are certain profile. Let's call it. I don't know key users. I just want to lock the information from them. Their behaviors in the environment. I can do that with these these settings. Exactly. That's a nice and powerful feature. It's nice, and I think it helps in two ways. Of, you don't always need to track. Sometimes just tracking errors is enough. I think it depends on where you're at in your project and and what you expect to happen. And so by switching it from finest to perhaps just either warnings and errors for certain users, you're also helping to reduce how much data storage is being used by Nebula Logger because you're just not storing certain entries. But then, at the same time, when you do need that extra information, it's very easy to increase it for just certain people instead of drastically increasing your data storage for everyone. So it's really great at targeting particular areas. I don't always need it, but whenever I need it, then it's really handy, right? Exactly. Okay. Yeah, everything's clear on your diagram. So I've, I've even by you know discussing with you about these features, I've learned a lot. I don't even need to read into the code. But as a developer, reading the code is the best way to learn. And as as you mentioned, when you
[31:44] create the plugin framework, you went back and forth to options. I think those details, the devil is in the details. So you spend a lot of time, right? And this is just the final version, the cleanest version. But you spend the effort to rolling up different versions. That's right. Yeah, I mean, everybody. The idea is trying to make it scalable so people can customize it, but still provide a lot of base features. Just kind of an ongoing challenge with the project. But I think it's it's found a pretty good happy medium as far as providing a lot of capabilities and extendability via plugins. Before letting you go, Jonathan, I want to ask your experience on the open source area. So, in general, what things you have learned, and do you regret doing that?
[32:33] No regrets. Definitely some late nights that you know I I was expecting to spend fifteen minutes on something and then said it was four hours. But no regrets. I really enjoy working on open source. I think it's really helped me learn over the years. That's really why I started originally, was just trying to improve my own skills. But by making things open source or by contributing to other people's open source projects, it helps everybody out. Everybody is able to benefit from these tools. Anybody can take Nebula Logger's code and do whatever you want with it. If you don't like how many configurations options are in there, you can fork it and remove a bunch of them. Like. The power of open source, I think, just can't be understated. So I'm very happy to be involved in it. I'd like to hopefully get involved in even more projects in the future. But I think it's it's a great thing for people to do, and I think everybody has a. Ability to try and get involved, whether it's coding or documentation or testing or just telling people that you want a new feature. I sometimes will think of features, but I don't necessarily want to add them until I know somebody's going to use them. Right. So if you just want somebody to contribute or have a feature in some open source tool, I think just engaging with that project is a great way to contribute. So I think no matter what role you want to play, there's some great way to to give back to the open source community. Thank you. Did this open source experience help you to get the current job in Salesforce? Ah, kind of yes. Um, so James Simone and I have been working together for. This is our third company
[34:04] we worked at at the same time. Okay. So James, through some of his open source work, got initially recruited, and then they started implementing Nebula Logger for the team. And then a couple months later, he got me recruited as well. So we're now working together internally and using Nebula Logger on a day to day basis. Thank you, Jonathan. It was really great to talking with you today. Thanks for having me. See you later. Bye bye. Bye.
