Scott Lee, who joins to talk about Salesforce Integration Best Practices, is CEO of Elega Corporation, Senior Salesforce Developer, Game Builder, and PluralSight Author.
Sponsors
This episode is sponsored by FormAssembly.
FormAssembly and Salesforce eBook – Find out how FormAssembly helps Salesforce customers optimize their data collection.
Main Points
- Scott is a Pluralsight author producing Salesforce courses
- When shall we consider Salesforce integration?
- The options to integrate a large volume of data
- Discussion on Mulesoft and Heroku for Integration
- Various Salesforce APIs involved in the Salesforce Integration
Links
Video Teaser
Contact Me
If you enjoy this podcast, please reach me out to let me know your thoughts or simply say Hello. My (Xi’s) contact info:
Take care, until next time :)!
Transcript
AI-generated, unedited.
[00:01] I'd like to welcome our sponsor, Form Assembly. Form Assembly's all-in-one web form platform lets you create forms for just about any use case. From contact forms to donation forms, all while taking advantage of useful features such as notifications, e-signatures, and more. Not only that, but you can also connect data to systems you already use. Form Assembly integrates with Salesforce, Podio, PayPal, and many other common solutions. You can find out how form assemblies help Salesforce customers optimize their data collection in a free ebook that we've linked in today's show notes. Whatever your data collection needs are, you can be sure that Form Assembly keeps your data secure with encryption at rest and in transit on all plans. Plus compliance with GDPR, CCPA, and more regulations. At the end of the day, form simply helps you save time, money, and effort while getting the maximum benefit out of the data you collect. And that reminds you: when you support our sponsors, you support the show.
[01:29] Hey everybody! This is Xi Xiao. This is the new episode of Salesforce Way Podcast. Today I'm sitting with the new guest Scott Lee. Hello, Scott. Hello. Hey, how are you doing there? Doing good. How are you? Perfect. I'm really excited to have the conversation with you. So we had some LinkedIn chat in the very beginning, and I didn't know you were also a Pluralsight author. Am I right? I am. I'm a Pluralsight author for for Salesforce courses, and you know. A few different certifications, but I'm also a video game developer and a huge nerd all around. That's so cool. I want to call out the Pluralsight author part is because when I check your Pluralsight profile, you have already produced the six courses started back in 2017. So that's a huge achievement to me. Yeah, so yeah, I started in twenty seventeen, and lately it's been mostly admin stuff. At the time that we're recording this right now, and I am, and and right now during this recording, I also have one going on for doing integrations with Python, and so that's kind of the one I am working on now. So hopefully, maybe by the time some people are listening to this, that'll be out as well. Okay. Yeah, definitely. We will put the link, or let's say I will put your profile to our show
[03:00] notes, so that everybody who click the profile can get all the courses. You know, in the profile site. There you go. Courses. Yeah, good. And I know you are also game developer. In addition to that, what do you do in the Salesforce ecosystem? Yeah. So with Salesforce, yeah, I'm a I'm a senior developer, and I do a lot of like these days. It's a lot of development operations. So you know, I do CI pipelines with Salesforce and automated tests, and and those types of things. I also do a lot of training and a lot of sort of mentorship with the other developers, the other contractors that we work with, and those types of things. So it's basically that, and then a whole lot of learning on the side and and creating those those courses. That's cool. Your next course is about Salesforce integration, and I invited you here to spend thirty minutes to just talking about integrations. We want to cover the basics here on this topic. Um. So I know integration is definitely the hot topic in Salesforce. You can spend your whole career specialized in the integrations, and Salesforce has acquired a Microsoft just for the integration, so called integration cloud nowadays. But I know you have good experience. You have good stories to tell us. You think these fundamentals to decide whether or not customers should go with integrations with Salesforce or not. There are some techniques or points we should pay attention
[04:30] to even before jumping to the pool. Am I right? That's right. I have I have stories of victory and I have horror stories as well. So that's good. We want to learn from the mistakes. Yes, as many of us as many of us do. Right. Um. Yeah. So you know, with integrations, the very first question that I feel like people should ask is. Is this even the right solution? Because there, so like, kind of from what I've seen, and and maybe this is anecdotal, but I feel like most companies they take one of two approaches with Salesforce. You know, Salesforce is. Traditionally, a CRM platform, right? And we all know that. But the way that I also describe Salesforce to folks is that it's really a generalized business application platform. If you need to do any kind of CRUD operations—create, read, update, delete. Then you can really do just about anything you want. You just create some custom objects. You write your logic, and you you know if you want to create a custom view for that, go go for it. And all of a sudden, you you have a business application, whatever that application needs to be. So there are companies that. Use the platform to run their entire business, right? So when you go beyond CRM, you suddenly have a whole different set of concerns. So if you're putting your your HR system on there, your billing system, your accounting system, operations
[06:00] related things that that may pertain to your business or your organization, you have very different challenges at that point, but. You know, so there's there's companies that do that, but then there's other instances in which companies might say, "Well, it's it's CRM, but we now we want to take the CRM system and let's let's tie it to all these other things that we already have, right?" And maybe I'm not exactly sure which one's more common. It's probably the first one where. You you want to you want to tie together all these systems, but then when you do that, where do you start, right? And and and are you making the right decisions in doing that? So it sounds to me, we need a Salesforce architect sitting inside the picture to give us advises or share the wisdoms before we say. It's a good idea or not to make the integration to tie these systems together, and if so, how? That's yeah, that's right. If you have an architect, certainly the thing is, is a lot of folks find themselves in situations where there is no architect. You know, the company is not going to pay for an architect. The the the demand for Salesforce architects is so insanely high that you know you're not going to be able to find one in whatever city you're working in. So you know, I mean, I can just tell you here in the Houston, Texas area where I work, we have had a lot of trouble finding really good Salesforce
[07:30] talent out there. So, so a lot of the time, there are many cases in which the developers find that they are suddenly having to come up with a very complex solution from scratch themselves, having to do things that they've never had to do before. Right, and I was in that position certainly at a few different points in my career. Hmm. Could you share some stories about the mistakes either you or the customers had in the past? So they decided to go to do certain kind of integration, but it ended up not a good idea. Do you have such a example? Yeah, so I would say the the first thing that I want to mention is that you know as as we all know, as many of us listening to this know, there is sort of the Salesforce way of doing things, and um and maybe that's pun intended for for your your site, right? But uh, but but there's the Salesforce way, and then there is. Like traditional development, there's .NET and JavaScript, Node, and those types of things, and so when you have developers who come in from that outside stack and they. Come on to Salesforce. The trouble with Salesforce, while while also being kind of a strength at the same time, is that Salesforce looks deceptively like all these other things. You know, that you look at the database and it looks exactly like any other relational database, but it's not. Apex looks exactly like Java or C Sharp, but it's not, and so on and so forth,
[09:01] and so. So, so the I would say that's the number one thing to be aware of is that when you're working with others and you have that Salesforce experience. It's very important to communicate to them that there are t here is different concerns that they need to think about. So, what a lot of developers coming in from the outside will try to do is they'll say, "Oh, well, you know, Visual Force this looks very similar to like ASP. dot net or or so one of these other declarative component frameworks that maybe I am used to using. I'll I'll ah I'll I'll snap in Angular or React to this." And you need to say, "Whoa, whoa, whoa! Not so fast." It's not that you can't do it; you certainly could, but there's a lot of things you might want to think about first, right?
[09:49] And and so for many of us, even if we are not at the senior level or the management level, the project managers and so on, even a lot of us entry level or mid level development folks, we can still have a very strong influence on advising on those best practices. So you know, like that—that's a big thing I would say is don't just assume that. You know the either senior developers or others that you're working with are automatically going to go look at Lightning Web Components or look at how things are meant to be done on the platform. A lot of especially like you know you get consulting companies, contractors would come in. They have a job to do. They got to get stuff done right and. And it's very tempting for them to say, "Oh, this looks like everything else. Let's just you know slap it together with these existing frameworks." So, anyway, that's the first thing. And so, and and tied in with that is, okay, if you if you build out a custom view, a lot of these developers are used to, well, let's just go hit the REST endpoints. So you know we got to insert some records. Let's call the Salesforce REST API, and it's like okay, but are you thinking about the API governor limits because we're limited on those per 24 hour rolling period? And you know, are you thinking about the org as a whole? Because with everything that you do with these integrations, you always have to be thinking, what is my API call consumption? And
[11:19] so, a lot of developers who are coming in for the first time, that they're not thinking about those things. And every integration solution that you build, or or anything that you come in to customize with, you have to keep that in mind, and you have to be sort of. Judicious or very sort of strict in your decisions in that area, especially if you deal with a very large amount of data, because a lot of the time when you build an integration. You're building for anywhere from tens of thousands to hundreds of thousands, maybe even millions of records that you have to process and interact with, or have some kind of a synchronization between two systems. And when you have that kind of a volume, everything that you decide in terms of how that communication happens with the platform matters, right? So those are the the first sort of warning signals. Hmm. In the past, at the very beginning of this podcast, had an episode about how to use React, use Angular in Salesforce, just the functioning of the front end part. But indeed, the REST API is something we didn't touch at all. That's the best option, you know, for the front end to communicate through the backend. But if we use the Lightning Web Component, we don't have that restriction anymore, right? We don't have this REST API calling governor limit. Which is more native to work to Salesforce, but the reason, of course, I understand the mainstream front-end developer is
[12:49] easier to find in the market. So you want to pull out a team. Of the developers, you can't easily, as you said in Texas, say you can't really find that many people. Then what can you do? Then you pull out Java or .NET developers. But then again, the REST API, did you really think about that? At least you need to raise a red concern at the beginning, right? What's the volume of the consumption of the REST API? Does it hit the governor limit or not? That's right. Yeah, and and in some cases, it's just explaining that as a simple concept because, um, you know, again, they just they see everything on the surface and and they're they're not thinking about. Taking a look at this whole concept of governor limits and those types of things, which you know you have you have two basic sets of limits. You have the the limits that you're always worrying about when you're writing triggers or creating workflow and and or or flows and things like that. But those are always in sort of the the database transactions on the platform, which are incredibly important, but Just on the API side alone, you have to be thinking about that. So, yeah, REST API, bulk API, streaming API, with all of these, they all have sort of different sets of limitations and intended use cases behind them that you have to be aware of. And when you're working with those other developers, those those folks who are coming in from from the .NET background or whatever it may be. Yeah, you just you have to sort of educate them on
[14:19] that. If if you're the Salesforce person, I guess it's like it's kind of like when we talk about IT security. Like, you know, security is everyone's responsibility, and when you're a developer on Salesforce, it's also. Always your responsibility to educate about Salesforce. That's a good analogy. You mentioned about the large volume of data. I can sense, and also in the past with some other guests as well. Playing with the small or even million size of data is totally different game with a really large, like over millions of records. So when you are attacking that amount of data, it's serious that you need to have argued to plan this integration at the very beginning. Do you have similar experience about playing with large volume data? I I do and well. So there's a few different tools that you think about when you think about massive data volumes. So. Salesforce generally will give us some things that they have devised that they intend for folks to use, that are certainly extremely helpful. Those things usually come at some kind of a premium, and I've found that with whoever I work with. That's usually the biggest obstacle is what is the price for some of these solutions. So, so one of those examples would be these days would be MuleSoft, and MuleSoft
[15:50] is incredible software because they basically take. In some cases, even massive data volume integrations, and make them easy to do by having these nice declarative point and click tools for for setting up how those things work. And they they build connectors which are very well designed and those types of things. But again, comes at a premium, right? And so a lot of companies they're looking to save money, especially with the pandemic going on. They're looking at how do we cut costs, or how do we at least face face the uncertainty ahead. So, so they're not going to buy MuleSoft. They also might might not buy something like. Heroku Connect, which is another sort of high volume data solution, where if you want to replicate a number of objects in your Salesforce org over to an outside database, that is a great tool. That allows you to do that. You simply select which objects you want, you click load, and it will run the process. and And and with all of that, you didn't have to write any code. You didn't have to deal with any scaling problems. But, again, a lot of companies just won't be going with these tools, and so the first thing I would say is. If your company or or whoever you're working with, or if or if you're the decision maker, certainly take a look at those tools first.
[17:20] There's a reason that Salesforce bought and acquired MuleSoft. It's because it is right in line with their vision of reducing the amount of code that you have to write. Sort of democratizing that development process and so on. So always take a look at those tools first because they will save you, in some cases, hundreds of hours of development time.
[17:43] But say that that is not an option. Then what do you do? And that is where it gets gets fun.
[17:56] At least, to me, most of the customers I have worked with—they haven't had the chance to really purchase Microsoft. I don't know whether it's because it's a high price, or just the other tools they're more familiar with, or the developers are more familiar with. Yeah, yeah, and I haven't worked with MuleSoft either. I I would like to. I've I've spoken with a few folks who have used it, and most of whoever I've spoken with, they say that it's like the best integration platform they've ever seen. But you know, take take that with however much. Uh, you know, skepticism or whatever you would like. Yeah, right. Uh, you know, everything, every tool that you use will inevitably have some kind of a pain point. I think, but anyway. Um, so yeah, all that to say. Definitely look at at the tools that are that are out there to to help. But if you're looking to possibly save money, especially money in the short term, because hey, in the long term, these tools are probably going to save a lot of organizations' funds. But if you have to, if you have to roll your own for whatever reason, right? Whether it's costs or whether it's the capabilities of the tools at your disposal and those types of things, what do you do? And Salesforce, even for that, has said, "Oh yeah, we have an answer for that too. We have Heroku." And Heroku provides that that platform to build on, and so on. And you know, with Heroku
[19:26] versus AWS versus Microsoft Azure App Service, or or like AW or Azure Functions, as they're called, I've used these different services now. And the thing I'll say is, I think Heroku again comes at a little bit of a premium, but. For the little bit of additional cost that companies would pay for Heroku, it's it's like the luxury developer experience. It is a great tool, a great platform. Now that I've had the experience of building for AWS and for App Service for with Microsoft. Heroku is so incredibly easy because what they do is they have a build pack where if you if you write something in Python or JavaScript or people have done Ruby on Rails and all these other languages, you basically upload your code through like a Git push with version control, and it it literally handles the rest. It will take whatever dependencies you've declared. Like with Python, if you have a requirements.txt file, it'll it'll take those, install those for you, and it and it just runs and it just works. Whereas with AWS, there's all these like. Little caveats, little gotchas, where it's like, ah, well, you know, when you use the AWS Lambda function, it's it's a Linux runtime, which means that some of your libraries that worked on your Windows machine, those may not work here, and you might have to compile them in like a Docker container, and
[20:56] it becomes this huge sort of chore, right? So. So that's that's the so so now we're working our way down the hierarchy, which is start with something like MuleSoft or Heroku Connect if you can. If you can't do that, build something on Heroku because that's what Salesforce intends for you to do, and it it is the just a a great experience for developers. Any sort of CPU bound application, it it's a great platform. And then after that, if you then need to step your way all the way down to something like AWS. Or Azure, then go ahead and and move on to those other platforms. But that is sort of the hierarchy of things I would request if I were coming in. What what I know now as a developer, and I'm talking with management and and the organization about you know what their their budgets and things like that. It's like if you have the money to spend, that's the the sort of list that that you go down and you move down. So, when you get to that point where you finally have to write some code, how do you structure that code, and what languages do you pick, and those types of things? So, like the the course I'm working on now, that's exactly why I'm making this course is because.
[22:11] You know, I've mentioned there's people who come in from the .NET and the JavaScript background. I am like I'm on the other side, like a lot of other Salesforce folks, where they had very little technology experience before coming into Salesforce. Salesforce has this great tool set where you can get so many things done. Without having to learn these insane computer science fundamentals and so on, and so I had to I had to do that journey myself. Where I said, "I'm starting with Salesforce. I have to go and I have to learn C sharp or Python." And how? Where do you even start with that? And how is that different than Salesforce? And if all you've known is Salesforce, now you're going. Now you have the opposite problem that I said earlier. In, in what we're talking about, where, um, once you leave Salesforce, now you have all these different options at at your disposal. You're not thinking about governor limits anymore. You're not necessarily now. It all becomes important because as as you learn from Salesforce. Some of those principles with Salesforce will definitely apply to building good solutions elsewhere. So bulkification is a great example of that. You you take a look at how you're forced to write Apex code, and then you realize, oh, I can actually use those same patterns to do pretty performant code in these other languages as well. Understand, and also this is one of the target for this podcast. It's really gap between the mainstream development, the software engineering concepts,
[23:42] and with the Salesforce ecosystems. There is this gap that people tend to live inside the Salesforce box. We don't really know what's happening outside of there. Of course, there are definitely good points in Salesforce we can bring out, and on the other side, vice versa, we also have a lot of things to learn. And I also want to ask you, you know, there are so many different APIs in Salesforce, like the traditional REST API, SOAP API. We also have this a bit new, like a streaming APIs. So, do you have experience of using these? Like, how to pick? What are the concerns when we do the integrations? Yeah, so t here is t here is a few different categories of application that you'll probably build, and how you pick an API depends on what kind of application you are building. And so, if you have uh sort of a transactional um system where you have maybe a view and you need to insert through like a data entry pattern one record at a time, or you or you're updating individual records, REST or SOAP work very well for that. And and the thing I would say about REST and SOAP is most modern systems are have gone the REST route. Um, you know, SOAP is it? It's older. It requires a bit more setup to to use properly. Whereas with REST,
[25:12] what REST really is is you're just you're just hitting HTTP endpoints, and SOAP is very similar. Except with SOAP. So, for example, if if you're writing C Sharp or .Net to to use so properly, you want to go get like the Whizel file out of Salesforce, and then you have to store that locally, and then you're kind of referencing like this like local sort of version of like how Salesforce works. It's It's very, in my opinion, that's a very antiquated way to do things these days. So, I, other than legacy systems, I personally don't recommend using SOAP unless something you have already uses SOAP. Then it makes sense. At least that's my that's my sort of personal preference on the matter. So if you're not doing sort of individual records, you're not doing individual transactions. Then you need to think about the other APIs because if you're dealing with larger. Data volumes. You have bulk, and you have the streaming API. And when we're talking about the streaming API, we have a few different things within that. But so bulk is really that is your ideal option for ETL or extract, transform, load operations because if you need to do tens of thousands of records and and load those in, then bulk is optimized for that purpose. You can do a huge number of records within a given transaction, but.
[26:42] With streaming, streaming is different because we we care more about doing things in real time, and. At first glance, streaming sounds sort of similar to REST because if you go and you read about the streaming API, it's actually HTTP driven, and we're sending individual messages, whether it be. The the the field values of records or platform events or change data capture events, whatever the case may be, it sounds very similar to to getting a message from a REST endpoint, but it's not, and the reason is is because when we're talking about streams. We're talking about more of the publisher subscriber model and building listeners that receive those messages. And so, streaming. What what some may not realize at first glance is that streaming is actually ideal in some cases for very large data volumes. Because why wait to do an ETL transaction when you could just always be receiving the latest information in a stream? As so as things occur in real time, you can get those messages, and and that's that's a big bonus because at a certain point, ETL processes can either take way too long to run, or
[28:13] they they break at a certain point with a certain amount of volume, depending on what sort of outside solution you're running and those types of things. So so anyways so. Yeah, I need to back up a little bit because now we are hitting a blind point to me, so which are not so familiar. I know the ETL tool is meant to. Set between the two integrated systems. It's a simple case, so it's a middleware. So when you pull the data out of system A, you kind of do the transforming. You kind of calculate the data, and then format in another format. And then the next step is to feed it into the system B, which is the load action in the ETL. That's right. Do you mean the streaming ways that we don't actually need to handle this transforming part? Is that what you mean? Well, so the the thing is, is you're still going to do a transform and a load potentially, right? But. You're not doing that on a massive volume in a single sort of batch job. The difference is is that you have individual. So in most cases, you have individual events or individual records that are changing. And as those change, those may be triggering sort of the criteria or the the indication for, oh, we need to perform some sort of
[29:43] action on this data. And so, if that's the case, you can calculate as those events happen. You know, it's like you really want to handle things as they happen. If you can, is really the the the ideal. That's a good point. What I understand the streaming versus the ETL, to me, it's like I watch the YouTube videos. I open the browser, go to YouTube, then I watch the full length of video. I don't really in the old traditional way download the video as one gigabyte of size and wait for thirty minutes and then watch the video. Now the the browser download the video and I can watch it seamlessly. That's right. Okay, that's right, and and that that's exactly what's happening is you're getting streamed video content as opposed to I have to download the entire thing. And indeed, that's what kind of that's what the internet kind of went through is we before we had video streaming platforms, you had to download videos back in the day. If if you may recall, yeah. Hmm. I think a lot of developers, including me, having the challenge is that we are not familiar with this new public subscriber model that's recently introduced in Salesforce. So of course, we fall back to the ETL. I think it's the thing for us to learn and get more familiar. Like you just introduced us and get to understand what are the context best feeding for the streaming or best feeding for the ETL. That's right. And with streaming, there's there's there's
[31:13] trade offs just like just like anything else, right? So like the the concern with streaming over ETL might be. Well, what happens if the communication pipe to receive my messages or to receive my stream? What if that breaks down? How do I handle that? And so, like with with the most of what the streaming API is as it exists today with Salesforce, they give you a seventy two hour period to replay those events if you miss them. But but then you know it's like to that point, it's like okay, well I can replay the events if I miss them. But now I whatever my process is, if I missed messages, I now have to build something to handle. Those miss messages. I have to have something that figures out. Okay, this is where we we started a blackout period, and then this is where we we now are able to pick it up and and keep going. And and maybe maybe the the the communication was out for some limited time window. And we have a gap to fill, but again, it's like if you're building a custom solution or custom code to deal with that, you you have to sort of solution that now, and that is something that you would not have to do with like a bulk ETL job because bulk ETL is a little bit simpler in that way. So anyway, yeah, there's there's definitely other things to consider. I see. Even if you get that advantage of oh well, where well now we are near real time with streaming, right? Excellent, so Scott, where can people find you? Do you have Twitter,
[32:43] LinkedIn information? Yeah, I'm on Twitter at Scott Lee. I was I was an early adopter of Twitter, so I'm able to just have my name as my handle. I think I I think I got on to Twitter in two thousand seven or something. And I just got very, very lucky. So you can find me on Twitter at Scott Lee. My company website is elegacorp.com. That's e l e g a c o r p.com. And and of course, you can find my courses on Pluralsight. Yeah, I will put your LinkedIn, Twitter, your company information, and also of course the Pluralsight your profile, which links to all the courses you have and also the upcoming new Salesforce integrations. So great, thanks Scott, thanks for coming to the show to educate people and myself included. It's really nice chatting with you. Awesome! Thank you so much for having me on.
[33:45] Oh, oh, oh, oh, oh, oh, oh, oh, oh
