Cold open: who Robert Cioffi is and why this story matters

SPEAKER_01 0:00

This is the How to MSP podcast. I'm your host, Andrew Moore.

SPEAKER_00 0:06

And it lives with you, it stays with you. Even though it's five years later, I still, I still, you know, I get that anxious feeling, but this is a coping mechanism for me. It's cathartic to be able to talk about it. And if I get one person that reaches out to me and said, you know what, I'm taking some additional steps to better my company, that's well worth what happened to me.

SPEAKER_01 0:28

All right. Welcome to the How to MSP podcast. My name is Andrew Moore. I am your host from RidgeView Advisors. And I am joined today with a very special guest that I'm super excited about. Uh Robert Chulfi with Progressive Computing. He is the CTO and co-founder of the organization. They are an MSP. And uh he's got a remarkably interesting story that I am really excited about. Uh Becky Teal from Huntress introduced us. And I know from talking to folks, he's in the channel doing a lot of things uh with different organizations, not just his MSP. So Robert, thank you for joining us today.

SPEAKER_00 1:05

I appreciate you being on the It's my honor and privilege to be here, uh Andrew, and uh anything I could do to get the story uh spread out a little wider into the community, the better. Uh so I just appreciate the opportunity to speak to your audience.

SPEAKER_01 1:21

Awesome. Awesome. Um so a couple of questions. First we'll start with um and I'll let you uh uh go into all of it. Uh where are you at physically? Where does the podcast find you today? And then tell us a little bit about who you are and uh what you do and kind of why people should uh should care about your opinions and thoughts on things and in the channel, what have you been up to? Let's talk about that.

SPEAKER_00 1:41

They should care about my opinions because I have them. Welcome to social media, right? Yeah, how is that for entitlement? Uh no, I mean, uh you know, I think I have a uh a pretty good story to tell. Uh it's a pretty awful story, but uh we'll get into those details later. But a little bit about uh me and us. Uh Progressive Computing was a company that I co-founded in 1993 with my uh current business partner, we're 50-50 owners. Uh we started out way back in the long ago, before time in 1993, of doing application development. We were building uh uh character-based uh DOS software. I mean, the stuff wasn't even networked. It was these were independent D-base type databases. If anybody remembers Clipper, uh, reach out to me. I bet you got a lot of gray in your hair like I do. Um but our business morphed over those uh many years, and it it follows the typical trajectory of what IT looked like. We got into networking, NT came, you know, Novell, NT, uh then uh cloud, cybersecurity, somewhere around 2008. We were a little bit early on the MSP uh uh transformation. Uh but we're a uh traditional MSP based out of Yonkers, New York. Uh if you know your geography, that is just north of New York City. We border the Bronx. Bronx border is about maybe five miles that way, um, in case you're wondering uh where it is. Um uh but we were, you know, it's a city of 200,000 people, but I have probably 25 million neighbors in the you know 50 mile radius. Um yeah, and just you know, got very involved in the IT MSP community a decade and a half ago, uh got very involved in peer groups. I facilitate two uh peer groups as an IT Nation involved uh owner peer group, um, facilitator. Um and just uh just you know, I love the community, and uh community is a word that I'll repeat often as I tell my story uh about a huge success factor in us navigating uh through uh what essentially should have been a business ending event.

SPEAKER_01 3:54

And and one of the things that we talked about briefly before we move on is um you had mentioned there's this flag for those who are not watching and that are maybe listening to us. So there's this really beautiful American flag back behind you in your officer. Can you give everybody a quick background on that? Because I know that there was a story there.

SPEAKER_00 4:12

Aaron Powell Um Well, it's uh it's a flag that I've had in my possession for uh decades. I mean, since I was a teenager, it it um it came into my possession uh and I've never known what to do with it. I've you know, obviously other than hanging it outside a home, but I always felt like I don't want it to get weathered. Uh you know, I wanted it to be someplace that I could sort of display it. And then once COVID hit, uh behind me are these uh blinds and shades and everything went video, so I needed something to cut back on the glare. So I was like, all right, let me put this up as a temporary, and it it stuck. Uh the number of calls that started out 10 minutes late because people wanted to know or comment about the flag was a very big surprise to me. So um, you know, my parents are off the boat from Italy. They came to this country with nothing, literally quite nothing in their pockets other than a trunk full of clothes. And they put four sons through college. And uh, my three brothers are all extremely uh successful. I'd like to think that I've uh follow followed in their footsteps a bit. Uh, but it's, you know, I'm just a very patriotic individual who loves this country, and uh it just for me felt like the natural, easy thing to do to put up an American flag behind me.

SPEAKER_01 5:30

Aaron Powell I think that's amazing. And I've I love that I love that there is still a level of true patriotism that runs through the uh the country in a way that really describes what this country is about.

SPEAKER_00 5:45

And it's gonna be I don't want to get political here, but I think there's a lot more of that than media would uh uh suggest is present. I think it's a lot more prevalent than what we're being told.

SPEAKER_01 5:56

I I I would agree with that too. And I think that there are a lot of people who are proud of this country and sometimes that maybe don't feel like they can can share that. And I love it when when people do feel like they can they can and they have a great story behind it. So I applaud that. I appreciate that very much. Um so let's get into this a little bit. So um Becky and I were talking at the end of the podcast, she said, I want you to talk to Robert, he's got a very interesting story. And she's like, I'm not gonna tell you what the story is. She's like, but you guys should have a conversation. And so I reached out to you and then we we started talking, and you told me about your experience,

The nightmare every MSP owner already carries

SPEAKER_01 6:30

and I was absolutely entranced because of my situation. I think one of the things that we saw, and I'll I'll preface this a little bit when we had RMSP, especially in the post-pandemic era, right? Probably starting around uh 2020 and and and past that, with the a number of cyber incidents that were not only happening to our clients, but we were seeing happen in the industry in total, there was a real concern around our my my COO's group and other operators that we knew. It was like we are one event away potentially from taking all this sweat equity and all of this work and all the things that we did from having a worthless business and a bad reputation as operators because we couldn't protect our clients. And so it was a terrifying thing that would really drive a lot of the decisions that we made, you know, and how we built security and what we tried to do to protect our clients and ourselves. But it was always lingering in the back of our head and it gave us, it gave me a lot of anxiety. And so I've never met someone uh at this level that had experienced something like that and come out of the other side of it in any capacity. And so when you began to tell me your story, I was fascinated by it. So could you kind of start from the beginning and explain to our listeners what happened to you and where you're coming from and what what brought us to this conversation today?

SPEAKER_00 7:55

Um Sure. Uh uh unfortunately we don't have hours to get through all of the details. So uh you and the audience will forgive me if I'm uh you know leaving some cliffhangers or not painting a full picture. I've learned that I need to kind of navigate through this uh very quickly. Um brevity is not my forte, Andrew. So I I tend to be a little uh I tend to get deep into the story sometimes. Um but essentially what has happened to us is exactly the nightmare scenario scenario that you painted, right? A lot of MSPs will um uh have anxiety or have anxiety over this notion that, hey, one day a threat actor could potentially own all of my customers simultaneously. Um and uh you know that would be a really catastrophic, terrible thing to happen. Uh let me um let me start with invoking the terror and anxiety that you feel when one of your customers becomes ransomware, right? Every MSP that I know that I've ever spoken to can raise their hand to say, yep, that happened to me, and man, we sweat bullets. I lost sleep, I was maybe even in the bathroom puking for a few minutes, you know, just thinking about like how awful this is. Um, the the stress level, the the anger, the the blaming, the um the long uh very hours over many, many days uh to put Humpty

July 2, 2021: 2,500 endpoints in 90 minutes

SPEAKER_00 9:30

Dumpty back together again, right? Now take that feeling and multiply it not just by the number of customers that you have in total, uh, but maybe maybe put an exponent on that as well. Because what happened to us was on July 2nd of 2021, uh a threat actor had gained access to our RMM tool, Caseya VSA, uh, and they broke through a zero-day exploit, uploaded uh Reval Sode No Kibi ransomware in a little package called agent.exe, uh, and then instructed every device under um control and management within that RMM tool to install that uh program. And since most computers had an SSD drive those in you know five years ago um uh and were you know good performing machines, that ransomware took about 90 minutes to install on every single endpoint that we managed. How many endpoints? This was 80 clients in 200 physical locations in four time zones, right? Um so that was initially what had happened, right? So we're sitting there, we're getting reports of something happening, a lot of confusion. That's the story in terms of the metrics and the numbers of exactly what happened and the the numerical impact.

SPEAKER_01 11:02

What so when the agent was deployed, yeah, was it readily was it apparent that the payload had actually done something, or was it just like we know that this has been deployed and we're waiting for what the next shoe to drop? Like how did that happen for you?

SPEAKER_00 11:16

What happened was, and I can tell you like you know where I was and you know exactly how I found out, uh, but we had essentially found out um uh after the damage was done, right? Because as ransomware is encrypting a machine, you don't know that it's doing it, right? It's not gonna uh you know raise its hand and announce, hey, I'm starting to encrypt your machine. Uh it's gonna do so quietly behind the scenes uh until it is complete. So the reports that we were getting in the early afternoon of Friday, July 2nd, 2021, uh were reports of things, you know, things not working, right? Or um systems being inaccessible. Um and that was the very first telltale signs that you know something widespread was going on, um, especially since our phone was ringing off the hook coming in from just about every one of our customers. Um and that's I think you know, where the emotion, if I if I can uh paint that picture for you, I was sitting in my kitchen uh at about uh noon, well, it was about noon that day. It was remember, this is Friday, July 2nd, right? Right before, you know, talking about the American flag here, right before uh 4th of July day weekend. Um, and where where are most people's heads on a day like that, right? They're probably thinking, hey, it's about noon, I'm checking out for the day, I'm taking off early. The weather here in New York was spectacular. Uh, you know, we can't always say that here in the Northeast. Um, so everybody was kind of checking out for the day, and um, I was doing something very unusual at that hour. I was eating lunch. I know that's crazy uh to think

The hallway, and the client list read in revenue order

SPEAKER_00 13:00

that a business owner or an MSP would uh eat lunch at noon. But there I was in my kitchen. It was about 12, maybe 20, 25 at that point. Uh I was chatting with somebody, you know, joking around, just like kind of thinking, hey man, we're done for the day. Ticket counts are dropping. Um, and I saw my director of operations uh come up a set of stairs. Uh I sit on the second floor of the building. He came up the stairs, was walking down the hall, and I could just tell looking at him that something was was really, really wrong. Um, in fact, when I looked at him, I thought uh he looked like somebody who was coming to deliver some really terrible news. Uh in fact, I had commented to somebody uh who was in the kitchen with me, uh, and I said the words, holy shit, I think somebody just died looking at Jay, you know, coming down the hall. So I wasn't sure whether I was getting up to console him or whether I was bracing for him pack that he had something terrible to tell me. It was the latter, and thankfully nobody did die. Uh, but it certainly felt like I died that day. Um, so I met Jay right outside the kitchen, and uh he was pal, you know, he's Filipino, right? So he's got a dark complexion naturally, but he was white as a ghost. He was sweating, and we have a great professional relationship, but he would not look me in the eye. Um, so I knew that I had to like like pull it out of him. Like I knew something really terrible was happening, and I'm like, Jay, like what's wrong? And he was just pursed lips, shaking his head, and he looks up at me and he said, Every one of our clients is ransomware. And I apologize uh for those that are listening. Uh you might hear me pause, and it's because I do need to choke back some of the emotions sometimes uh as I relive through this. But um uh it was it was uh completely surreal, Andrew, for me to hear these words like like it was absurd. Like, what do you mean all of our customers are ransomware? Right. And in fact, that is exactly the distaste that I had in my mind. Like, what like what is this garbage you're handing me? Like, that's not a reality. What do you mean? Like I threw it back to him and I said, Jay, what do you mean by that? Like, like which one? And he goes, No, Robert, it's all of them. And still, I tried to reject his reality and insert my own. Uh, at which point he was insisting that it's, you know, that it was all of them. Now I'm gonna pause in the story here for a second because something happened in a split second or two or five. We're now standing in a hallway on in the middle of our office, and there are cubicles and other offices sort of around us. And now I can hear and even see what's happening with other people. And I'm hearing, and these are snippets of conversations. Um, so here we go. Uh, I don't know what's happening. Um, uh, please calm down. Um, I think something system-wide's going on. Um, hold on, let me see if I can get a technician. I could see people standing, you know, from their seats on the phone now. Um, I can see some confused looks. I can almost feel their um uh their discomfort of in the conversation they're in. And it's and it's a hundred percent. I mean, I I had my eyes and ears on about five or six different people, and that was what they were, they were all on the phone saying something similar to that regard, right? Trying to calm people down. I don't know. Let me get some answers, let me get back, uh, let me uh get right back to you. Um, so at that point, I kind of knew, like, okay, maybe there's some truth to what Jay is telling me here. And I said, I pushed on him again. I said, Jay, okay, you got to tell me which ones. Now he did something that I don't think uh we have a little bit of a debate whether or not uh he did this purposely uh or he did it subconsciously. Uh, but he started to list customer names after I insisted on him telling me. And uh it's a little bit of a rhetorical conver uh a little bit of a rhetorical question here since the audience can't respond, but I want you to think about what order he might have uh given me that client list in. All right. Andrew, you care to take a guess?

SPEAKER_01 17:34

Aaron Powell My guess would be so there would be two different ways I would I would probably start with our biggest accounts. I would I would start with the ones that were I'm like, oh, it would be the it would be the oh fuck factor. It would be like, oh, you okay, you don't believe me? Like let me talk to you about the the nasty ones, right? I wouldn't start with the small ones. I'd be like, it's you know, this one, this one, this one, and it just for dramatic effect. Like I would say this is where we're at.

SPEAKER_00 17:56

It was uh essentially, I'll say revenue order. He got to about number 10 and I begged him to stop. Um each one of them felt like I was like each name felt like it was landing on me like a slab of stone and pancaking me into the ground as I heard each name. I just sank lower and lower and lower.

SPEAKER_01 18:19

Uh I still have to be like, how did you not just like like I would have literally walked over to the trash can and thrown up? And then I probably would have stood there for a second and been like, is it is it okay for me to just leave? Like I like there would have been that fight or flight thing that would have kicked in and I just would have been like, I I have to I have to walk away from here for a second. Like I what how did you guys just like throw cold water on each other, slap each other in the face? Like, did you just go into a room by yourself and scream? Like what where what how did you guys move in move past sort of action? How did that work?

SPEAKER_00 19:00

Um, well, uh I've been in a I've been in a serious car accident once. Um, and if you've experienced any sort of physical trauma in your life, there is a there is that shock value, right? And so here I was, now my brain felt like it was expanding a hundred times its size, trying to push out of my skull uh because of all the thoughts swarming in it, right? And the emotions and logic that were fighting each other about what to do, what to think, what, what, you know, do I crawl into a hole? Do I, do I just leave for the day and pretend nothing happened? I mean, those are thoughts that kind of went through my head. I'm not gonna lie about that, right? Like, do I just abandon ship? Like, because the thing's on fire. Um, uh, some of those thoughts came a little bit later, but my initial reaction was uh I did go into this sort of command and control mode of, okay, we we've we've got to assess what the hell is really happening here. Because I'm not sure if the words that I'm hearing are true to reality. Right. And I think that a little bit of denial was was was prevalent there, right? I was a little bit in denial of still about what I was hearing, even though it seemed to be validated by converse snippets of conversations that I'm either witnessing or hearing in the office. So I grabbed Jay and we came exactly where I'm sitting right now with exactly what you see in my office here if you're if you're watching, um, with the same exact three screens that I have right here, right now. Um I've even been the same computer, who knows? Um and so I just started um going into like go mode in terms of a technician. Like I I need information, I need, I need data, I need, I need to see things for myself. I don't believe what I'm hearing. I need validation uh with data. Um and so at this point now, Jay is standing like literally right over my left shoulder, and my phone is ringing, my cell phone is going off, Teams is blowing up, my text messages are blowing up. I've got now my two chief uh uh technical people um on our team. At this business at this point of our business, we're in business 28 years at this point, and Mark and Scott had been with us at the time, and they're still with us today, I'm proud to say. At the time, uh they were with us for 23 years. So these are guys that I like, they are the fabric of our DNA. Trust them implicitly, explicitly. Um, so I had one on uh speaker phone and one on you know my cell phone, and we're trying to like dig through this. Um, and that's when I had a complete validation of um of the ransomware incident when I'm sitting at my desktop and I can see all the icons on my primary screen start to flip. From their typical Word, Excel, or other familiar icons, and they turn into white boxes. So you got cryptoed with your clients. Correct. Except for one particular file, right? The readme.txt file. That was unencrypted. And of course, I opened that up immediately and knew that we were, we were toast, right? So uh of course, because we had used the, you know, we we

Watching the icons turn white

SPEAKER_00 22:30

eat our own tacos here, right? The same tools that we deploy to our customers is what we use to manage our own systems. Um we can debate whether that's a good practice or not. But remember, this is uh five years ago, and um, you know, most, you know, um there there are things that we've learned all learned from those days that uh uh have changed our perspective on things. Uh but that's the moment, Andrew, that you were starting to describe earlier, where I put my head in my hands and I I froze. I didn't, I didn't know what to do. There was a good five or 10 minute period of time where I had contemplated things like, do I just go home? Do I power off my cell cell phone? Do I dismiss everybody? Am I gonna need to fire everybody? Right. Um, what do I say to my clients? Right. So uh let me put it this way as a as an MSP owner, most of us started this business as technicians, just like I did, right? Um, and so I'm gonna write a business book one day on business type personalities, and it's only gonna have one personality. I know you know these other systems and all these other personality types. My business book only has one personality type, um, and it's called Fixer, right? And it's only a one-page business book because in a paragraph or two, I can tell you what that personality looks like. There's a problem, I can fix it. And I'm not talking about just IT, like all the stuff, all the trades in my house, I self-learned, I do them all. Uh, I can fix basic things on a car. Uh uh that's I think that's also what made me, I'll I'll say, a pretty good business person is because I just look at business problems as a problem to fix, right? Just apply logic and reasoning and uh a little bit of that grit, right, of just being determined to be able to fix things. The fixer was broken here, Andrew. I had no idea what the hell to do next. I didn't know who to call, what to say, what to think, what to feel. I was staring into the abyss, contemplating my own mortality. Um I watched 28 years worth of sweat equity, blood, sweat, tears, and toil evaporate before my eyes, right? We had a multimillion dollar valued business whose valuation not only just went to zero, but might have even gone negative, thinking about all of the lawsuits that might be coming our way. Um, I thought about recalling my daughter home from college that she would not be able to finish because I wouldn't have the funds to pay for that. I might lose my house. I would have to fire all of my employees. I mean, these were the dark thoughts that started to swarm in my head altogether. I mean, any one of those demons is hard enough to battle, but 20 of them showed up at my doorstep all simultaneously. Um, so it it took a while for me to snap out of it. And thankfully Jay was with me here uh because he did and said something uh incredibly um I'll say it was silly, right? Uh, but it was exactly the silly words that I need. He saw me faltering. He saw the guy that like put so much effort into building this organization and was the go-to answer guy for everything. And he just simply put his hand on my shoulder and said, Come on, Robert, we got this. And my first reaction was, What the fuck do you mean we got this? Right? Like, are you fucking kidding me? Like, we got what? Like, we just got screwed, like to the nth degree here. No, we don't have this. Now, this was what I was thinking on the inside, right? But I looked at him and said, Son of a bitch, he's right. Like, we do have this. I have no idea how the hell we're gonna fix this, but we're gonna fix it. So let's go back to what you said earlier about fight or flight. I thought about flight, but I chose to fight. And that's that's what's next.

SPEAKER_01 26:44

I love that because I used to talk to my um the owner of our company um whenever things would get tough or whatever, and it just I I'm a big uh pop media guy too. I like to watch shows and movies and stuff, and I'm a fan of Star Trek, and there was this Picard quote that I really liked, which is one impossible thing at a time. Right? Yes. One impossible thing at a time. And so when we would sit there and be like, oh my God, there's like all these things that are in front of us, I would sit down and I'd be like, okay, dude, one impossible thing at a time. What are we gonna focus on first, right? So as as I'm very kindred spirit with you in being a fixer, like I am very, very, very similar into your personality. I like to know how everything works, to take things apart and put them back together and do my own home improvement projects and all those things. So um I love that

The first move was the core values on the wall

SPEAKER_01 27:30

that's where your mind went, which was, okay, we're gonna fix this. So what did you do? What was step one? Then what would like how did how did you start putting one foot in front of the other? Like how did you come out of the fog and how did you start getting people involved? And like what w what led you out uh, you know, both short term and long term to start, you know, rebuilding the business uh from that day?

SPEAKER_00 27:50

Well, I'm a huge trekie too, so we can geek out that on that some other time. Um but uh I'm also a big history buff, so I started to think about uh some inspiration from historical figures. And I thought about a quote from Winston Churchill uh during World War II in the face of annihilation, uh, and he famously quipped, When you're going through hell, keep going. Uh so I felt like I was standing at the gates of hell, right? If I can quote Dante, right? Uh abandon all hope, ye who enter here. Or in an, you know, there's the Italian version of it, which I'll skip for now. Um, but that's that's where I felt like I I can't go backwards, sideways, I can't stay where I'm sitting. I got to push those gates of hell open and start walking through. Um, so uh just sort of accelerating, we gathered the whole team into the conference room. And as people were filing in, my mind was still a blank, Andrew. I'm like, what am I going to say to these people? What am I going to tell my team members? People who I would consider to be my friends that we've worked together for so long. And then even beyond that, what the hell am I going to say to my customers? Um, and it was interesting because at that point we were about uh eight years into running EOS here. So for those who don't know what EOS is, just a system for running your business, uh, but one of the um components of EOS is to define what your core values are. Um so our core values were posted up in our conference room, and by some miracle, by some grace of God, I looked up at them and said, that's the answer right there. Uh, it is our culture. It is the, like, is it, I believe it's Peter Drucker who said that uh uh culture eats strategy for breakfast, right? I had no tactical plan. I had no strategic vision for how I was going to come over this, but I knew that I had an amazing culture here. And when I say I, what I'm really mean is I'm misspeaking here. We had an amazing culture here. So our four core values are you know, team together, we get it done, um uh uh commitment determined to do what it takes, uh um uh respectful uh always, uh, and um humble confidence service without ego. So I knew that those four core values were gonna be paramount to our survival. And so I started off talking to the team about look, you know, acknowledging what's happening here. This is really awful, but I want you to think about those core values repeatedly and keep them in your head in every conversation, in every email, and every text message, every ticket, every site visit between now and the next, I don't know how long, but this is what's going to get us through. So while that alone wasn't enough, it was enough for that moment to be able to pull the team together and snap them out of that, you know, tremendous feeling of dread and woe that we were all collectively feeling. Um, I needed to give them some sort of hope and semblance that, you know, uh one of the two leaders here, the top, you know, two people, had the strength and confidence to instill in them a sense of we've got this, right? Just as Jay said to me. Um, and but here's the reason why we have this. We are not going to fail because of who we are.

SPEAKER_01 31:26

Yeah. And then as you started to move through that process of getting your team back and re-engaged, just real quickly, like, how did you understand this is the vulnerability? I mean, because we use so many tools. Like, for all you knew, it could have been the antivirus solution, it could have been anything, it could have been somebody that hacked your environment, like through it could have been one of your former employees. Like at that point, you didn't know what you know today. Where did you go forensically? Like, how did you start to really say, well, now I need to turn this on, and then this is what I need to do? And then how did that lead to them figuring out what happened, how to remediate

Forensics, cyber insurance, and setting the recovery point

SPEAKER_01 32:00

it, and and then how you guys were able to start rebuilding? Like, what did what did that look like?

SPEAKER_00 32:05

Aaron Powell Yeah. So for the next 48 hours, right? Remember, this is a Friday afternoon, July f uh Monday, July 5th was a national holiday. So we had that three-day quote-unquote weekend.

SPEAKER_01 32:17

Can we let me put a pin in that for one second to keep your thought there? Because I want to I wanted to mention this earlier. For those who are not familiar with how these types of events work, usually they're before some sort of like holiday or an internal event that the hackers are aware is going to be potentially disrupted to your business. So a lot of the times you'll see these sort of things happen like right before Christmas or right before Thanksgiving or right before July 4th or something, because they know you're going to be short-stacked, you're not prepared for it, right? Like you're, you're, you're probably taking it easy. Um and so I just want to make sure that people really get a feel for like that this wasn't like just a randomly chosen day or that they were trying to make a point because it was the nation's birthday. It was because this was a strategic advantage for HAC, right? 100%.

SPEAKER_00 33:02

Yeah. Um 100% on that. Um so um and again, there's a lot of details here that I have to kind of just step through quickly. But we had uh notified uh our cyber liability insurance uh company, which by the way, our policy went into effect that Monday, and this was a Friday. Um well, we had a policy prior to that, but it was a garbage policy. So we had now a policy uh in force. Um I still would urge everybody to read their policy in clear details and maybe get some expert, you know, third-party expert eyes on it because you never know what's in your auto insurance policy until you have an auto accident. And then you learn what it really covers and what it doesn't. So um, and I know nobody wakes up every, you know, uh on a day on any random day and says, Oh, let me learn everything there is to know about this policy. Um, but it's really important to make sure that you do know what's in it. Now, um, the important thing that did cover us was the uh uh breach attorneys, right? Breach is a hard is a terrible word. I tell people don't use it, but for some reason uh they have breach counsel or breach coaches is the the name for it. So essentially a team of lawyers was assigned to us, um, and we were able to work with some uh uh some technology partners, uh, one of which was Huntress. Uh, you mentioned Becky Teal on your previous episode. Uh she and I had been friends and known each other for a while prior to that, but she's also part of the story. Uh, she got me connected with a number of people with inside Huntress, uh, because they were very instrumental in doing a lot of forensic analysis work. Uh Kaseya did as well independently. So by Sunday morning, we had uh two different independent um um uh forensics analysis on our log files, uh, and as well as broad understanding of what had happened to other companies as well. And I'll boil it down to this while Reval Soden O'Kibi, who was the uh the main operator behind this style of attack, is well known for like the double extortion of exfiltrating data. Um, up until this point, they were uh well known for pulling data out and then encrypting you. Uh, this was a classic smash and grab, meaning they exploited um the VSA vulnerability and they pushed ransomware out to as many computers as they could simultaneously with the hope of catching enough things in the broad, wide phishing net that they cast. There was an estimate of over 1 million systems worldwide that were affected by that attack. Um, but we knew by Sunday morning, actually, we knew by Saturday afternoon by Huntress's analysis, but confirmed by Kaseya by Sunday morning that at 1049 a.m. the threat actor had gained access to our system, uploaded agent exe, ran the script, and by 12:30 p.m. ish, so in about 90 minutes-ish, um, 100% of our endpoints were encrypted. Right. Now that timeline varied a little differently based on different company victims, but that's essentially the timeline that had happened for us. So we knew that we also now had an RPO, right? A recovery point objective of um uh of somewhere prior to 1049 a.m. So what we decided to do was just to simply pick 8 a.m. that morning to give ourselves enough of a a cushion there. Um now we had an amazing uh disaster recovery uh vendor we still use today. Uh we had about, like I said, 250 servers that were fully protected by this technology, and we were fairly confident

Not a ransomware problem, a math problem

SPEAKER_00 37:00

that we would be able to do full restores. Um but A, um, well, let me back this up. What we had now, Andrew, was not a ransomware problem. What we had was a math problem. I can take your computer and recover it and restore it either by formatting it, if there's no, you know, data needs to be salvaged on it, uh, or if I have a backup of on it, I can, you know, a backup of it, I can restore that, right? Um, and if I had, you know, all the time in the world, I could probably get you back up and running in a couple of hours, let's say, just, you know, spitballing here. Uh, I could do the same thing for a server or a customer, right? I've got 25 computers, I've got two servers, let's throw four engineers on this, and before the day is over, they'll be fully restored. But think about the math that I had 80 customers, 200 physical locations, four time zones, how the insert expletive am I going to get all this work done? And now I want you to think about this. Who do I do first? And that answer is not so much about who you select first, but by selecting who you do first, you're essentially telling everybody else, you're not that important to me. Right. Pecking order. Right? If you got selected first, oh, I must be their best customer. And if you're told, mm-hmm, we can't get there till Wednesday or Thursday, I must be not important to you. Right. So trust me, the psychological uh impact of those decisions was not lost on us.

SPEAKER_01 38:40

So I'm a I'm an ops guy. So my first thought would be, okay, we now have our recovery point. We have a pretty good idea. Obviously, like we probably turned Kissai off. So we're flying a little blind, even though we're so but let's just say that, okay, so we figured out that we're gonna do some recovery, like we're gonna start with our own systems, we're gonna get functional, we're gonna make sure that we're not, you know, that we're we're turtle shelled in some way. Like we we start there, then we start trying to figure out how to get in, you know, now we've got our base of operations, we're gonna reach out. Did you did it occur to you like that you were like, do I do I maybe I know some people from up peer groups? Maybe like, who do I reach out to? Like who isn't impacted? And like, and did it worry you that maybe by reaching out that you had the air of of being

Community shows up: 27 companies in 17 days

SPEAKER_01 39:30

pwned on you, and maybe you didn't want like maybe you were embarrassed to reach out to people. Like, how did that start to work in your mind? Like the like what do I do to start dealing with this capacity issue and how did I move forward with it?

SPEAKER_00 39:41

Like, so here's where's where the word community now slips in quietly and is the real champion or hero of this story, right? Some people have characterized either me, myself, or our company as a hero or uh as being heroic, but I'm placing that label on the entire uh MSP ecosystem and and community. Community really, to me, and here's the punchline is community is the missing element to your cybersecurity stack, right? It's the one thing that you haven't installed is community. Uh or maybe you have, but many MSPs have not. Um and so what happened was it took one of our peer group members, a friend of mine, Kevin, up in um Massachusetts, Cape Cod. Um, you know, we're both Northeasterners, so you know, we we punch first and then uh hug and make up late later. Um Kevin's an awesome uh individual. And he put out a message to our peer group at the time, because my company participates in a group, we still do in the same group. Um uh he put out a message to them saying that if we don't help out these guys at progressive computing, they're gonna die. And so he basically said, hotels are waiting for you in Yonkers. Uh, book them now, get on flights. I will pay for your flight. Like Kevin was like really urging people to get here. Now, I had spoken to Kevin and initially, you know, because of uh something maybe we might characterize as a little bit of Italian bravado, said, no, no, no, Kevin, I like I got this. Like, we got this, right? There goes that we got this line again. Uh, we got this. Like, I don't need your help. Like, I was being too prideful and maybe a little stubborn, uh, and not maybe realizing that the full magnitude of that math problem that I had urged, but Kevin saw the math problem much more clearly than me. So within uh like literally about 24 hours of that, he uh and two other individuals, somebody from Kansas and somebody from Santa Barbara, California, were standing in our office. These are three business owners, and they collectively had brought six engineers with them. Uh, and like with like no an like a one-way ticket here with like an unlimited uh hotel reservation. Like they were willing to stay here for as long as possible. Uh, all in all, we had 27 companies uh come to our aid in doing direct delivery of services. And I'm not counting the vendors that helped us, right? Like the Huntresses and the Axiants and the Kinectwises and the PAX8s of the world. And forgive me, I'm probably missing one or two names. But we had 27 logoed companies coming to help us to help restore our systems over the next 17 calendar days.

SPEAKER_01 42:46

Well, and that that's amazing for the actual MSP community, and that like I think one of the most important things that people should remember is that that is potentially a real deterrent to people trying to perpetrate one of these crimes against MSPs in general, because the idea is like, well, we'll band together, right? Like we'll like there's almost a deterrent factor against it, not just from a community standpoint to help each other, which is unique in our space, I believe, than in other industries. But I feel like it's a real opportunity for us to be proud of what we showed to attackers at the time, which was like, yeah, I mean, you mess with us and we're gonna get it fixed and we're gonna fix as quickly as we're gonna do it.

SPEAKER_00 43:26

Well, at least if you if you mess with a certain uh MSP, right? And and I don't mean this disparagingly to those MSPs who might be sort of living in a little bit of a vacuum or a little bit of a bubble, right? Their main community connection might be Reddit, right? Uh or some forums. Like you really got to get out to conferences. You really got to meet your neighbor. Like the guy down the street that you call your competitor isn't really your competitor. He's a friend, right? Or she's a friend. Um make good connections with those. People because in a time of need, right? And be a true go giver with them, right? Uh don't don't stab each other in the back. Make friends in the industry. Because without those community connections, we would have been dead. My business would have probably dissolved before the end of that calendar year.

SPEAKER_01 44:20

So talk to me a little bit about let's let's wrap up the event and then move into we'll call it, I'm gonna call it post-mortem, but I'm I don't uh that's probably the wrong word. The uh the path forward and what the phoenix that came out of it, not just you know your business, but the the opportunity that you saw to help the community. Let's talk about how this thing wrapped up, how how you managed to move forward, what client stuck with you, did they catch the guy? Tell me a little bit about the end of the story here, and then I'll talk about how you you came out of it.

SPEAKER_00 44:52

We definitely took a big revenue haircut. Uh the way I characterize it is I lost a small MSP, right? Um uh and if I know my numbers correctly, if you're probably in that, you know, average sized MSP, um maybe a maybe a little bit smaller than you, maybe maybe half your size. That's probably what I lost in revenue. Um uh it was a it was a big, big hit that took place over about 12 months. Uh but I mean we're we're over double our size in that five years since then. So I mean, uh that just speaks to our ability to rebound and um uh all of the customers that did stick with us. Um, I think, you know, it's it's a testament to the relationship that we had with them. Uh but we did uh manage to get to about a 95% restoration for about 95% of all the endpoints that we managed in those first 17 calendar days. Remember, I said calendar days. We were putting in 18 hours a day running, you know, we were running like lunatics. Uh we had technicians showing up, like, where are you from, buddy? I just met you. Like, here's a bag of USB sticks and some instructions and an external hard drive, like, go. Like, you know, um, we had a war room set up where Jay and I were like kind of like operating like uh as uh uh commanders or generals in a war room, like, you know, constant people filing in and out, and like it was the nerve center for our operation. It was exhausting mentally, physically, emotionally. Uh, but every day was a bit brighter. That was uh Kyle Hansloven, CEO of Huntress, got in my ear uh on that Friday afternoon. Uh and he told me, he said, I know this is your darkest day, but just hold on to that fight. And I promise you that every day is just going to be a slightly bit uh brighter. And those words stuck with me, and I still, you know, appreciate and I tell him all the time how much I loved uh his encouragement. Um, so that's kind of what got us through by certainly by the middle to the end of July, we were uh somewhat holish. I mean, you know, look, there was still a lot of fires popping up here and there. Um you can imagine all the little crap that was still broken, right? Yeah. Printing to the wrong printer tray, missing icons, missing pieces software, settings don't work. Like it was just a lot of crap broke. But but we had most organizations just about mostly operational pretty, pretty quickly.

SPEAKER_01 47:28

Aaron Powell Well, and that's the part that like the other part of it that really sucks too is you spend years and years and years building up positive capital with your clients, right? And then it gets burnt on this. And so that, you know, I'm sure that any little issue that was outside of the problem, right? Like any technology problem that normally they would have been considered of, or you know, he missed an SLA or the guy didn't do a callback or something, or the project that you were doing for him didn't go quite right. Like normally you could lean on the good graces of your reputation and what you've done to help him in the past. And uh a lot of that got burned to the ground, I'm sure.

SPEAKER_00 48:07

It got burned, but it was also like, you know, when you have something that goes sideways somewhere, uh a CEO gets all pissed off about something, you throw a bunch of great resources, or you throw a great resource at it for like an hour or two, and then everything is all happy and joy joy again. But you can't do that when there's a thousand of those requests on your doorstep, right? Yeah. Yeah. Um so it took, you know, that's I think what um affected us uh a bit in terms of some of the losses that we felt that had come later when people decided to go elsewhere. Yeah. Um I think it was accommodating. That lack of trust or feeling like we did something wrong even though we didn't.

SPEAKER_01 48:47

Yeah.

SPEAKER_00 48:47

Um Yeah.

SPEAKER_01 48:48

That's and that I get that, because that's one of those, it's like, and I'm sure that even though you got all of the good stuff done that, you know, as your contracts expired or whatever, people had an opportunity to consider where there are other

Facing the threat actor in federal court

SPEAKER_01 49:00

they've been like, well, you know, we'll just let's shop around a little just to make sure.

SPEAKER_00 49:04

And then before you know, yeah.

SPEAKER_01 49:06

Yeah, I g I get that. So did they ever catch the guy that did it? Like were the groups that did it?

SPEAKER_00 49:12

Yaroslav Vysinski uh at the time was like a 22-year-old Ukrainian national uh living in the Ukraine. Uh and uh he was crossing the border from uh the Ukraine into Poland, going back to university uh in October, I think it was October 6th of 2021. Um little uh uh be known to him, uh the FBI had tabs on him. They were already in their systems tracking this guy. They knew he was uh one of the primary affiliates uh in charge of this attack. And you know, if you're not aware that ransomware is is almost like a franchise business, right? Oh, I want to open up a couple Dunkin' Donuts. Well, you can go do that, right? Uh uh uh Revil so no kibi is the Dunkin' Donuts. I just own these three Dunkin' Donuts uh like you know in this five-mile radius. Uh so he's this uh affiliate operator. Um they uh called Interpol and uh Polish authorities, apprehended him at the border, and then within five months had him extradited to the United States. Uh I actually spoke directly to the FBI agent who was there uh to not only greet him the first time as soon as they uh had apprehended him, but also had collected him on extradition day to uh bring him back to uh the United States. And they brought him to Texas, the great state of Texas. I can think of no other better place in the world for a uh um a threat actor, cyber criminal to be extradited to. Uh and Yaroslav was eventually sentenced uh because he pled guilty right away. Uh he was sentenced to a total of 14 years, which included his incarceration uh in Poland for those first five months. He's now in uh FCI Danbury, so that's Federal Corrections Institute, Danbury, Connecticut, which is about 45 minutes from uh where I'm sitting right now. Uh and as I am told, once his sentence is up, he will be immediately deported uh back to the Ukraine.

SPEAKER_01 51:18

So did you ever have a chance to meet him, be a part of the trial, any of that sort of stuff, or is it did you kind of keep your distance?

SPEAKER_00 51:25

Yeah, interestingly enough, that uh as I, you know, I filed an IC3 report as soon as the event took place, and you know, I was listening to all those voices in my head of all the conferences that I had been to about things I should do. Uh we had spoken to various FBI agents throughout this. They were more in data collection mode, like tell us what happened, you know. Uh they just wanted to know things. And I was, you know, like I just felt I was doing my, you know, my patriotic duty of like, yes, let me tell you as much as I possibly can. Um and because of that, I got registered in something called the VNS, the victim notification system, and I started getting letters and communications about the trial. Uh, and I ended up deciding to go to Dallas, Texas, to federal court uh to uh provide a victim impact statement. So that's something that I did um uh in uh January of I'm getting my dates crossed now of last year. And uh I for about 20 minutes I spoke to Judge Scholler, who oversaw the case, as well as the two assistant U.S. attorneys. There were FBI agents in the room, uh defense counsel, as well as uh Yaroslav Wysinski in a very, I must say, very tastefully orange jumpsuit with these beautiful chrome bracelets both around his ankles and his wrists. Um so yeah, I mean, and listen, I he he was he was a boy and still is. He's got you know that very cherubic, very young man looking face. Like he looks to me still like a high schooler. Um and so that's the face of who attacks us, not the not the guy with the shrouded hooded hoodie where you can't see where their face is obscured, and if they reveal their face like you might see some grotesque monster under it. No, that that that's not the face of threat actors. Yeah. Threat actors look like you, Andrew, they look like me, they look like this kid, they look like average everyday people, right? And by the way, not to be uh well to make sure that I'm not sounding sexist, they're also female, right? Yeah. There's a lot of women doing this stuff too.

SPEAKER_01 53:39

Yeah, and and it's you know, all ages, anywhere in the world, it's readily available to them. And to your point, the ability to get on the dark web or to interact with some of these, it really is a business model that is very sophisticated if people don't understand it, with um full market opportunities, kits that can be bought and repurposed to exploit things that are and with AI it's happening faster and faster. And I think MSPs really need to be very thoughtful of how they're protecting their business. And to your point, like you know, making sure that they're integrated into a community that helps them in the event that something were to go sideways. And so I know that you have kind of moved past the event and and your your business is now doing very well. Like what are what have you built? Talk to us a little bit about what you built, what came after this uh

Why he built MSP911 and CyberRISE

SPEAKER_01 54:30

that really kind of allowed you to tell your story and help others and protect the community a little bit.

SPEAKER_00 54:34

So I got I got very introspective. Thank you for that lead-in. Uh I I I got very introspective of August of that year, so about a month after the attack. Um and I was actually attending, I decided to go, even though it was only a month after. I decided about not going, but then something pulled me to go to a community event. Um uh uh our evolved community, the very community that helped us. Uh my team was doing a fabulous job, so I felt like, you know, I I could I could be on the phone and uh still uh still be there. Um, and let's not get ourselves.

SPEAKER_01 55:10

Let me pause for a second there.

SPEAKER_00 55:12

Yeah.

SPEAKER_01 55:13

If people who are not in peer groups don't understand, there is something that only happens in a peer group that you cannot explain, which is other people going through the same shit as you in this industry every day, and that gives you this camaraderie, this brotherhood, this sisterhood that people don't get unless they're in one. So I don't know that you have to make a a good reason why you didn't go. I'm sure it was really important to your soul and the importance of being able to do that. I felt like I needed to be there. Yeah. Yeah, for sure. I get that. So I and I just want the listeners to know like I I think it's important that people find that community. And so I I agree with getting in front of those folks and being a part of that.

SPEAKER_00 55:52

And but because the statistics say that there are 50,000 MSPs, I don't know, globally or in this country, I don't know what statistic to believe, but it's quite a number of them. And having attended so many different conferences and peer groups, I know that they're not all there, right? So I was thinking about all the people that I don't know that are not at these events and what would they do if they had suffered something like this? Um, and how would the community even know? How would they know who to reach out to for help? And I'm thinking about all their customers and their employees and the associated families and businesses that would get torn down a rubble. How could I help? Um, and so it uh I had a flash of a thought of registering msp911.org. So I immediately registered that domain name. It was available. I was a little shocked, um, but I registered it with this idea that I could be the big giant break glass in case of emergency, or the I need help big giant red button, um, or like the fire alarm, right? Like uh a resource that MSPs could go to uh should they find themselves hyperventilating like I was, right? Not knowing what to do. Because as smart as I am and as prepared as I am, and and as and those words could be used to everybody on my team, we still made a lot of mistakes that day. And there was still a lot of confusion and um you know chaos here. Like there was a lot of negative emotions fighting all of the things that help prepare you for battle. Um, you can't be prepared for battle unless you're you've actually experienced it. Uh so that's a thing that I wanted to do. Now I'm gonna fast forward to that because it's a multi-year journey there, but we have now rebranded ourselves as cyberrise.org. So it's cyberrise.org. It's uh all one word. There's a double R in the middle. MSP 911 is one of the three pillars of what why we exist and what we do. We are a full nonprofit, um, that I, as one of the three um uh board members, founding board members, cannot take a penny out of it, can't make a dime on it. Even if I somebody asked me to go speak somewhere, I gotta buy my own plane ticket. I can't, I can't put it through the organization, nor do I want to. Uh, but the idea of MSP 911 is that if you're suffering an event, you can call that phone number or send an email or use the web form to say, I need help. And somebody like me, and half the people that are helping with this right now are former victims like me who know what you're going through and can provide those coaching um and um assistance type of um uh services to you to help you step through that that process so that you are not going through it alone. Uh we just um at PAX 8 Beyond uh in uh Salt Lake City in early June, uh Matt, uh Lee, uh Jason Slagle, and I, that we are the three founders of Cyber Eyes, had uh case 50, a gentleman by the name of Michael, who was running an organization serving only dental practices uh that at the beginning of this year had gone through, I'm sorry, at the end of last year had gone through a very horrific, similar event. And he has said multiple times, I would not have survived if it weren't for you guys. Like you guys were like angels. You came out of nowhere, right? I think a friend of a friend connected him to us, and and it's like it's a miracle to him, in his own words, that his business is still alive because we rendered assistance to him.

SPEAKER_01 59:42

No, I that's awesome because I was at that PAXI conference and I didn't know of you or that you were there. And so the fact that this exists is fantastic, and we need to get the word out to everybody because you know, I it I'm sure that there's things that in the event of a breach, um, you know, that it's awesome to be able to reach out. But I'm I'm guessing there's probably just some information out there about like how to be better prepared and what to think about and like not just like we're doing that as well, yes.

SPEAKER_00 1:00:12

Yeah.

SPEAKER_01 1:00:13

Yeah. Having an incident command structure and knowing who's in charge during one of these issues and some of that stuff is just some to your point earlier, just like some basic stuff that like you don't even think about, like you just kind of go into React mode, but if you've already got some plans and some ideas, and I know that you know Becky talks about doing tabletop exercises when it comes to client breaches. Like, but if you're to your point earlier, if you're not eating your own dog food, if you don't know how to manage something within your own environment, I think it it it really behooves you to get prepared for that even prior to knowing an organization like you guys are out there after the fact.

SPEAKER_00 1:00:47

So I want to highlight something really quick. You were kind of alluding earlier that like the threat actors don't know who they're messing with when they're m messing with MSPs. This was the core tenant of what I saw as the underlying thought behind MSP 911 is that I don't like emotionally, I wanted to get on an airplane with, you know, fully armed and go seek my attackers, right? That's what the emotional Robert was thinking. Um I actually had a few friends that were like, we got I we got you, brother. I got the ammunition, let's go. Like let's go. I'm like, yeah, it's uh it's a wild fantasy. That's not gonna happen. Right. Um but I did think, well, how can I fight these guys? They're smarter than I am. They have uh they're they're technically smarter than me. They have more resources than I have, uh, they are much more motivated than I am. What can I do to fight them? And this is how I'm fighting them is by being the tide that lifts all boats, by educating the community, driving the awareness, connecting community uh together, and providing this resource that helps someone who is in that oh shit moment, and we could potentially turn it around for them. And we did.

SPEAKER_01 1:01:58

I love that. And so if they if anybody runs into an issue, they can simply go to the website and reach out to you, or they can, you know, um, you know, we'll we'll have information like your LinkedIn profile will be listed here too, so they can reach out to you too.

SPEAKER_00 1:02:12

There's a simple web form, you fill it out, it opens up, uh generates a ticket in our um uh incident response platform. We all get notified and you know, we take the call from there.

SPEAKER_01 1:02:23

I love that. And then you're at a lot of the conferences. I'm guessing you're gonna be at IT Nation in Orlando later this year, and you're probably at sounds like you went to PAX 8, and so you're gonna be around here in the channel.

SPEAKER_00 1:02:34

Aaron Powell I try to get out to as much that I think will benefit our business uh as much as possible. So and hey, look, you know what I talked about, Evolve. I mean, they're obviously owned by Kinectwise. Uh I just want to be completely, you know, agnostic here. There are a lot of other great peer groups out there. It doesn't have to be Evolve. Uh I love Evolve. Uh you might love something else. Just get thee to a peer group, right? I think Shakespeare said that. Get thee, no, something else, but uh, or he said get thee to a nunnery, I think is what he said. But I'm saying get thee to a peer group. Like, I don't care which one you pick, like go to one. Right. Get yourself involved in one. Plug in, please, please.

SPEAKER_01 1:03:15

Yeah, and and I the the podcast we just released recently with Becky, you know, I and I I stressed this, and this is a big topic that we talked about with her, is if you've got partners, like if you're working like you call them vendors, like I like to call them partners. If you don't treat your vendors like you would treat, like you want your clients to treat you, if you're not treating them that way, if you're not going to these shows, if you're not reaching out to them and creating personal relationships with them, not just when you need something or you're dissatisfied, but you're going to the show, you're sitting in the booth, you're having a 15-minute conversation, you're lining up a business review for after the show, you're having drinks with them later, right? If you're not doing those things, then you don't really know what resources they have available to you. You don't understand how important their relationship can be to your business. And it's the it's not just being at the peer group, but it's being integrated into the channel because there are a lot of things that are happening out there that if you're not aware of it, um, are really going to be beneficial to your business and don't cost you a penny. They don't cost you a dime. And you should be out there. And even peer groups are fairly inexpensive when you when you look at it. So I I encourage it.

SPEAKER_00 1:04:22

Trevor Burrus, Jr.: For the value that you get out of them, I think there's tremendous value. Um, you know, and it's be a go-giver, right? If you haven't read the book The Go Giver by Bob Berg, I would highly uh implore you to read that book. Um, but um it's more about like I try to go to these things. I mean, I know they're trying to sell me, but I'm like, how do I help you? Right. I'm dealing with uh a vendor just uh a solution partner just coming out of um their skunk works, right? Um I think I'm still under NDA, so I can't say who it is. But like I'm one of 10 design partners and I'm looking at this like they're saying, we want to help you. And I'm like, no, I want to help you because the more I help you get stronger, then I know that it comes back to me later. So that's the mentality that you have to have, is I'm not there for me, me, me, me, me. I'm there to like, how do I help? Right. Yeah.

SPEAKER_01 1:05:13

Yeah. I love that. It's uh it's the the right attitude to have. And so um I this has been a fascinating conversation. I know it's uh been a busy day, and I don't want to keep you any longer. We got a couple of questions I want to ask you before we let you go, but I just uh I appreciate you sharing with us, and I think it's been a fantastic um journey to hear that you came out of well, I'm not gonna say unscathed, but w having learned something incredibly important and not only done that brought the resources that you have to bear on the channel to make this community a better place. And so I think we're all better off for Andrew.

SPEAKER_00 1:05:46

I I might I might joke around a little bit, I might, you know, seem kind of jovial about it, uh but please understand, and and anybody listening or watching should understand that the PTSD that I have is real. Uh I have not sought professional help, but uh I have spoken to professionals about what I went through and they have told me 100% you have PTSD. I have also spoken to other victims just like us and it's real and it lives with you, it stays with you. Even though it's five years later, I still I still, you know, I get that anxious feeling, but this is a coping mechanism for me. It's cathartic to be able to talk about it. And if I get one person that reaches out to me and said, you know what, I I'm taking some additional

Closing five: books, bands, and the worst sales call

SPEAKER_00 1:06:30

steps to better my company, that's well worth what happened to me.

SPEAKER_01 1:06:35

Well, and I I'll finish by saying I'm gonna butcher this, but the antidote anecdote to the antidote to anxiety is action. And so if you are anxious, if you were uncomfortable, get out there, do something about it, make a change. Um so I think that's great. Um You mentioned the go-giver. Let's let's get into the questions real quick and I'll let you I'll cut you loose. What is your what is your favorite business book? What's the one book or books that you were like, you gotta read this? This is amazing. This is gonna help the business.

SPEAKER_00 1:07:04

I I have so many, and it was so difficult for me to think about what the winner is here. So I'm gonna pick on one that's definitely in my top 10. I'm not saying this is my top, but um how to win friends and influence people, especially if you're uh in mentoring somebody else or you're leading a company. Uh I would say give that book to uh a 20 to 30-something year old, like you know, someone who's early in their career. Um, even those that haven't read it is just tremendous value. I didn't read that book, I think, until I was about 40. Um, so I think that's an amazing book.

SPEAKER_01 1:07:40

That was Dale Carnegie, right?

SPEAKER_00 1:07:42

Dale Carnegie, correct.

SPEAKER_01 1:07:43

Yeah, great book. Great book. Awesome. Um, and then what's your favorite curse word? What do you what do you like what do you like to use? Like what do you feel comfortable saying on the podcast?

SPEAKER_00 1:07:52

Um well, I I used a couple of them already, but uh I'm gonna go to my my heritage, my my my uh my my DNA here being Italian. The word is cazzo, and it means so many different it it's the F word, right, in Italian. And and it's used in all the different ways that you can think uh it is it's it's just like our F-word in the various ways you can use it. But it's Italian, so it sounds more passionate and it's more powerful. So uh but if my mother heard me say it, she'd slap me.

SPEAKER_01 1:08:23

So well, and in the most Italian way possible, too. For those who were who were just listening, you used your hand very like emphatically when you used the word.

SPEAKER_00 1:08:32

So I love that you were very like doesn't everybody speak with their hands? And it's uh C-A-Z-Z-O, right? It's a that hard Z. So Yeah. That's a good one.

SPEAKER_01 1:08:43

That's a first. Like we've we've we've had um we've had some British cursing on here before, but not not and I mean as in they do speak English, but you know if you're you've heard British slang, they've got some very you there's a unique podcast. Specific words, yeah. Yeah. Um favorite band. So if you're into music, like who who you're on a desert island, band or artist that you're like, this is my go-to.

SPEAKER_00 1:09:08

Uh this is also a really, really tough choice, but I'm gonna go with Pink Floyd. Okay. My all-time favorite album is not only my favorite song of all time, but my favorite album is Wish You Were Here. Uh, it is what motivated me to learn how to play acoustic guitar. Now, for all of you fans out there, trust me, I play acoustic guitar badly, so you don't want to hear me play. Um, but it is the one song that I can play without having to look at a music sheet. Um I'm proud to say that, but I'm still sort of at the beginner level of uh uh playing. I just uh that album and uh between Wish You Were Here um and some of their other albums, I just it uh Dark Side of the Moon, like I can get on an airplane and get completely lost in that music. It is uh it's I I just find it takes me to a different place.

SPEAKER_01 1:10:04

You're you of the almost 20 podcasts that we've done this year, you're like the fourth person that's either picked David Gilmore or Pink Floyd as their favorite artist. So and I'm either I've got a demographic, either I've got a bunch of Live guys that are that are now like you know having that conversation, but uh it seems to be pretty prevalent. I remember the first time I heard the wall, and I just was like my mind melted, and I was just like, this what is this? Um love it. That's great. Um Okay. Worst sales call, business meeting, personnel meeting like that you've ever had. Maybe something that's a little funny, but something they're just like, dude, this this was crazy. You gotta hear this.

SPEAKER_00 1:10:42

Well, this this is a story. It goes way back in time when we were still doing hourly break fix work. Uh our MSP services were starting uh uh to come up, but we were still doing a lot of breakfix. And I had um um uh this woman uh call me and she was complaining about um all of the IT people that she had just gone through. And that probably should have been my first hint, by the way. Um uh they're not returning my calls, this guy disappeared for two weeks, and and on and on and on for literally about 20 minutes. And I'm just sitting there like playing therapist, uh, listening to her complain about all of the bad work, the terrible things that they've done. Um, and she just wants to find somebody who's really a true professional that's just gonna get things fixed. And um, you know, so I finally was able to cut in edgewise and like, okay, so let me tell you a little bit about us and how we work. And one of the things I started to step through was our our rate and our, you know, some of our billing practices just so that, you know, just to drive some general awareness. Um, and she was like, What? Like I think at the time it was like we were charging 150 an hour. Um, I might be off on that, but that was probably about the rate. Uh and she said, What? How do you dare charge me $150 an hour? And I said, Well, how much were your three previous guys charging you? And they were like, Well, they were all between $40 and $50 an hour. And I said, Well, there's your problem right there. Right. And I can tell you that she just never called me back.

SPEAKER_01 1:12:10

So anyway. That's a that's a I love that conversation. Like they're there, it's funny. I uh one of our previous guests um from Ashton Tech, Travis, would tell me this story. And he would um one of the questions they ask is in the in the pre-sales process, they're like, How many MSPs have you had in the last three years? And if the answer is over is three or more, yeah, then you know that there might it be yeah, it might not be the MSP.

SPEAKER_00 1:12:40

It might be the Yeah, exactly. Well, it's I I mean, what I also told her was like you're focused on the rate, you gotta focus on the invoice amount at the bottom because what we might do in two or three hours at a buck fifty an hour, they they're taking eight hours to do. And by the way, they're doing a shitty job and we're gonna do it professionally. Like, I I I'm not sure what you want. Like Yeah.

SPEAKER_01 1:13:02

I had a very similar conversation with some of the client, and they were like, you know, how much are you gonna charge us? I think we were like, well, you're gonna be about three grand a month or what any kind of free time. I was like, what are you paying now? And he's like, Well, I'm paying $1,200. And I'm like, Do you like the MSP that you're working with? He's like, I like them, but they don't do this other stuff. And I was like, Well, look, man, show me your bill. I looked at it and I was like, Do they have like an all-you-can-eat service offering? He's like, Yeah, I think they do. And I'm like, probably gonna be about three grand a month. And he's like, Yeah. And I was like, Man, just go back to them and tell them you want the upgraded plan. You don't need to change. I'd love to take your business, but man, like, you know, you just pay more. Like sometimes paying more is the answer, right? It's just get get a get a better MSP.

SPEAKER_00 1:13:40

I've learned that as well. You want, you want, you want performance, you gotta pay for it, right? Yeah. Um you want quality, it comes at a price.

SPEAKER_01 1:13:48

We tell people we're like, we may not be the cheapest people, but you know, we're we're gonna do a better job. Um, so uh last question is MSP owner, operations person, person on the channel, parker, vendor, who is somebody that has a story that needs to be told that we should be talking to? Is there somebody you can talk to me?

SPEAKER_00 1:14:06

So again, another one of those things where I've got like such a big catalog of people I could point you towards, but um I thought about Dustin Bolander. Um, he is out of Austin, Texas. I don't even want to tell you his company name because he's running multiple companies. Okay. Um he's um, you know, he's a bit younger than me. I don't think he has any gray hair on his head yet. Um, so I hate him for that. So, Dustin, I hope you're listening because I hate you for that. No, I'm kidding. Dustin is an awesome guy, awesome friend. We talk regularly. Uh I think he's got some uniquely insight uh fullness about or insights into the cybersecurity space. Okay. So he's not only an MSP operator focusing on legal, but uh he's also uh got a tremendous wealth of knowledge about cybersecurity and you know, all of the things behind the scenes that you wish you knew, you would talk to Dustin and he will tell you.

SPEAKER_01 1:15:08

In my former life, I before MSP, I worked on a grant from Homeland Security back early, like when it first got formed, like early 2002, 2003, I think, maybe it was right after it came out, and um when they created Homeland Security. And uh we started as this bioterrorism defense group inside of the the city of Houston under a federal grant. And man, I learned so much about incident response practices and like before cybersecurity was like kind of a thing. And it would like how do you get together, how do you talk, how do you communicate, how do you build processes? And so it's come so far that I'm beyond knowing all the ins and outs of it anymore, but I really still nerd out on it because it's just such a special part of like how I learned IT uh as an engineer. And so I would I would love to talk to somebody like that. I'd like to put an intro. Yeah.

SPEAKER_00 1:15:55

So he owns an MSP, he's got an insurance company, he's got his hands in a few other things. That's the other thing I really like about him. He's a true visionary. Um and but he's a visionary that executes, and I think that's a rare uh breed. Trevor Burrus, Jr.

SPEAKER_01 1:16:07

Well, he's I'm in Austin too, so if he's here, I'd love to meet him. So I'm meeting people in my own backyard. But uh hey Robert, I can't thank you enough for your time today. You've been more than generous. I appreciate everything that you are doing for the channel and uh sharing your story here with us, and I look forward to seeing you in person uh at IT Nation at the absolute minimum.

SPEAKER_00 1:16:27

Can't I can't wait to meet you in person, Andrew. Yeah, it'll be it'll be great.

SPEAKER_01 1:16:30

All right, thanks. Have a great weekend, sir.

SPEAKER_00 1:16:33

Thank you.