Video: Takeaways from Major Software Supply Chain Attacks | Duration: 5100s | Summary: Takeaways from Major Software Supply Chain Attacks

Transcript for "Takeaways from Major Software Supply Chain Attacks": Hi, everyone. Welcome to our webinar today on takeaways from major software supply chain attacks. We're really excited to have, Patrick Smith, our principal dev rel engineer at ChainGuard show today. But, before I actually invite him to kick off the session, there's a couple of things I want to point out. So, first of all, you'll see, number of resources under docs and definitely encourage you to take advantage of those. We have a, free chain guard libraries offer going on right now until, June 30 so that you can try out our libraries. And we also have some other, sessions or webinars that you may be interested in. So, definitely take a look at those links, check them out, and, sign up if you're interested. And, throughout the session, as you're listening to, Patrick share, if you have any questions, please make sure to drop them in our q and a. And, at the end of the session, we're we're going to go through each one of them to make sure your questions get answered. So, right now, we're gonna have, Patrick share. Thank you. So today, we're gonna be talking about recent open source attacks, what to know and what to do. We're gonna dig into the different attacks. This is our agenda. We're gonna dig into the the the different attacks. We're gonna focus more on the mechanisms, behind the attacks because by next month, we're gonna probably have a whole other set of attacks to talk about. So, hopefully, we're gonna learn some stuff that I think will also apply to the next round of attacks. But, you know, we'll also dig into some of the specifics. And we will, then introduce ChainGuard libraries, which is a new product from Changard that is basically designed to take on some of this malware issue head on. And we'll talk a little dig in a little bit because I think it can be a little counterintuitive. I've seen when people if the light turns on with people with SendGrid libraries, and they're like, oh, wait. It's not just scanning. It's actually preventing stuff. So I'll try to explain that a couple of different ways, but it's kind of a new little bit of a new thing there. So so we're gonna try to, really dig in there. And then I'm also gonna, cover some takeaways, that are go beyond any specific product vendor or whatever, but things that should best practices that, that they're they're not just I think, you know, six months ago, twelve months ago, I might have said, these are things that, oh, you should do. I think nowadays, there are actually things that you have to do because, things have just sped up so much and the threat environment has just gotten so intense. So first of all, let's just outline the problem here in very high level. So, you know, open source is it's been this huge accelerant. It was sort of the AI before AI, if you imagine some, you know, something that's kind of putting jet fuel in the tank. When I started coding, it was like, wait a minute. You can just pull, stuff that really smart people have built that solves any problem and just put click it together like Legos. And it it just kind of has sent, these ecosystems and what people can do with, coding, with, you know, systems in production, through the roof. You know? And, basically, you know, there have been some since the recent attacks, there have been some folks who are like, we're just not gonna use open source anymore, which is kind of an interesting stance. I think that it's simply not worth it because, ultimately, you you know, doing everything yourself is probably not the way and actually will probably leave you just as vulnerable. We could tackle that a little bit in the q and a maybe. But, but, you know, you wanna use the rocket fuel. But the problem is lately that the thing that sped us up so much and given us all these superpowers is actually causing, a huge issues. And, pretty much all the recent attacks with maybe only, like, one or two exceptions have basically, targeted the supply chain and come to you from various upstreams. Okay? So we'll return to this theme. But, a couple more drilling a little further down, what are the, what are some of the themes that we're gonna hit over and over again on this? One is that recent attacks have been targeting up upstream maintainers. Okay? So that is the new attack surface that, when you pull from any of these ecosystems, unfortunately, you may be pulling from, someone who's been compromised. And that's just reality right now. And maintainers are overwhelmed. They're not keeping up. We're digging to some of these, attacks in a second. But some of these folks are not naive. They're actually security professionals, and they're becoming victims of the supply chain attacks. And then everyone downstream of them also becomes victims. Second, is that there's a new this is kind of a fancy term for something, but it's a ghost, ghost artifacts, ghost attacks. The this is the new this is the new zero day, basically. So what is a ghost artifact or a ghost attack? It basically is set is the situation where, you say you you're downloading something from PyPI. You're like, oh, well, I wonder what I just downloaded. So you go to the GitHub repository and you look through the code and you're like, oh, looks pretty good to me. The problem there is that, what you downloaded from PyPI, Maven Central, NPM, wherever is not what is reflected in the source. So that's where most of these attacks are happening in the distance between the source. They're happening in bills. They're happening in upstream. They're happening in, pulling in upstream dependencies. They're happening in distribution. Okay? And then finally, attacks are shifting left. So that basically means that, attackers are going with the easy targets. They're going with the soft targets. And they found that what the soft targets are are, a, CICD, and b, developer workstations. Okay? So, those you know, you can't just guard your production system. You have to also guard everything, to the left of the production system. So it's a whole new sort of battleground. Alright. Now let's get into some more of the specific attacks that have happened. So we've we've moved forward and now this last month has seen I mean, every month has seen some major supply chain attacks. Last year, I was doing a meme calendar with one big attack a month. Now I'd say we're down to one big attack a week and the attacks are bigger. We'll talk a little bit of maybe, about why that is. I may wait for the q and a for that. But there are some specific reasons why this is speeding up and why it's not actually maybe likely to get worse before it gets better. But we can address that a little bit later. But, but we'll first talk about two attacks that are probably, I would say there's three big attacks, that are behind if you think about March. And we're gonna talk about two of them under one, you know, in sort of one breath here because they you'd share a similar mechanism, which is taking over CICD, CICD, continuous integration, continuous delivery. And so, you know, these are the modern bill systems that take us from source to production. Unfortunately, you know, they're super necessary. And in fact, you know, I'd say, if you're not using them, you may have work worse problems than getting your CICD attack in some ways. So now it's similar to open source. It's a little bit, it's a hard situation to be in because you almost have to use it, but it's a surface. So what happened with Trivy? First of all, what is Trivy? Trivy is a, it's a Go, package, or it's a tool built in Go. And essentially, what it is, it's a very, comprehensive scanner for, for, well, I was gonna say CVEs, but, actually, it covers a lot. You can scan Kubernetes clusters. You can scan, you could scan, container images. This is something we recommend very often to folks at Chainguard. We use Trivy. We also use Bryte. You know, we we use a lot of different scanners, but at its Aqua Security, these are professionals, who security is their bread and butter. These are not naive folks. In March, their, their CICD was taken over. Their repository was compromised. They, the attacker, pushed or updated 76 of 77 release tags, redirecting them to the malware. And in addition, there was a second round of compromise at the same time. Basically, meant that they pushed out a new version of of sort of a a malware, or, you know, compromised trivia. So this actually, had led immediately almost immediately led to somewhat up other more major attacks. I mean, some of the fallout here was this, you know, hundreds of organizations, real reputational damage. And one of the consequences was the next, CICD attack, which is, Light LLM. This is a, a AI portal and AI gateway. So really, they hold the key keys to a whole ecosystem. So basically, you upload your your API credentials to this to this service, and then you can use it to access, various LLMs, power agents, so on. So the fallout from this one was also enormous. And interesting on this, it actually is downstream of the trivia attack because, that was the initial ingress into the CICD system was actually a Trivia compromise. And that's kind of a theme here is that we're seeing these attacks sort of chain one to the other. And the fallout is almost hard to calculate because they one attack almost changed pretty neatly into new attacks. Okay? So just to talk about CICD compromise, this can take a few different forms. So for example, in the case of the light LLM attack, that would have been a compromise of an upstream. Right? So your dependencies were actually compromised. When your dependencies are compromised, then you are compromised. In the case of, in other cases, it can be, a misconfiguration. We actually had, I I'm remember, it's called a a hacker bot claw or something like that. And it's a new one I was just reading about today. And, basically, it's a automated bot that goes around looking for misconfigurations and targets, repository, like, high, high, visibility repositories and checks it for common misconfigurations when it finds something that compromises the account. That's another common mechanism. And another common common mechanism is, getting a hold of a long lived key from, you know, developer workstation or some other, location. So, so I would say that is maybe the top level thing to look out for. The biggest trend right now is the CDICD softness. It's been involved in almost everything. But let's get into Axios. So Axios, this is an interesting one. It's an attack that, that, the mechanism was social engineering. So I've got a scary looking picture of Kevin Mitnick. Kevin Mitnick was a a talented technical hacker, but also a social extremely talented social engineer, in the nineties. So one of the most famous social engineers of all time. But, but, basically, what happened here is, a maintainer, was they basically created a whole universe for him to live in where they brought him, this is Sapphire Sweets, North Korean actor, group that, that brought, this guy onto a meeting, a Teams meeting, and, leveraged compromise of his workstation. They actually then did this, they took advantage of a a post install hook. This is actually also a fairly common theme in the JavaScript ecosystems that pre install and post install hooks tend to be pretty bad. And that installed a rat. This is I put a a gift of pizza RAT. Sorry if that grosses anyone out though. I kind of like pizza RAT. But a RAT is basically a remote access Trojan. It's honestly it's more of a detail because it's just the mechanism. Once they had the the hard mechanism is to actually compromise the device, get access to the device. Once you do that, it's, you know, it doesn't really matter the details of what they actually set up to do what they did. But but yeah. So they basically only changed one line in the manifest and that in that, in it went to install a library that didn't really exist. But in the post install hook for that, it installed the wrap. So, the takeaway there is it's another mechanism is social engineering, but also elements of other compromise are are there. So so what the the takeaway there is developer workstations are now becoming an increasing target. Now let's talk briefly about, DeepSeek. So this is, I could have picked probably one of literally thousands of examples here. So I wouldn't say this one is the one to remember specifically. This is actually January 2025. But I honestly, it's one of the bigger examples of this, category of typosquatting or slop squatting as a mechanism. But it is, it's one of the bigger ones. But but, you know so just to give you an, an idea, in our dataset for how, what categories of attacks, or you know, so what percentage are are typistquatting, what percentage are target CICD, etcetera. 92% in a dataset that we've used back severance act collection for, in the PyPI Python ecosystem. 92% were typosquats. But each of those individual attacks is usually relatively small, and has relatively limited fallout. But together, collectively, they represent a huge, a huge number of attacks, altogether. There's more of these attacks, but each one has, like, kind of a smaller fingerprint. What's the mechanism in a typosquat attack? Forward. The mechanism in a typosquat attack is basically that you, type really fast. You know? You, you look for names that are very similar to a to a popular package. This is a you know, or something that someone's likely to type into their, you know, developer workstation. Common type of squads might be like, if if for deep seek, it was that they added an extra e. So there was deep seek, and then another one was deep seek, and they just put an I at the end. Like, you kinda, like, fat finger the k in the I. And they basically, what you do then is you add a you add malware to that. So when people hit install, when people npm install, when people install from Maven, then you get the malicious package. Okay? There's now a new variant, which go back to about 2023, called flop squatting. And flop squatting is basically takes advantage of, you know, if you're a human being, you're more likely to mistype something. The mechanism is a little funny and different with LLMs because LLMs tend to kind of look for logical patterns, and then they'll hallucinate names of of libraries. Because they're making educated guesses often about what the library name is. Often, it'll be something like what the cli is called. It's more simplified client name. They'll assume there's a package called that. Often, that's not the reality. There's a few different categories like that. But, basically, you take advantage of the LMS guessing, the LMS hallucinating, and you register those packages and bam. Now the LMS is installing stuff that it has supposed to install. And then finally, let's talk here about, Shaihulud. Now this was probably one if you have heard of any of these, you probably have heard of Shaihulud. Shaihulud targets developer, developers and may it targets developer workstations in order to compromise maintainers, then those maintainers that, again, themselves become vectors of attack. Okay. So it's a worm. It's actually named after a sandworm from Dune. Okay. If you're wondering why it has a weird name like that, Shaihilud. And it has evolved. Actually, three major times, though, also, there are many smaller copycats. So the original one was, you know, identity exfiltration, tote you know, token compromise, that kind of thing, which is pretty common with these. And there's a new variant. It's SHA one Halud, which, targets a little further upstream. It is actually more focused on the, GitHub actions ecosystem and the GitHub upstream. It's kind of I have to give it to them. It's kinda funny. They actually wrote this in their their code, sha one halud, and they it has also has a, library, that they have or a a module in there called truffle truffle sniffer or truffle hound or something. They actually named things pretty funny, the attackers. But SHA one, it's it kinda makes sense because SHA one is what's used for, commit hashes in git. So it's sort of a reference to the fact that it targets the git ecosystem. So I don't know. I maybe, my brain is melted from from doing too much coding, but but I I thought it was kind of interesting that they named it that. And, it's technically quite sophisticated and it's quite vindictive. It's, it actually has gone from, just exfiltrating information to actually trying to, like, cover their tracks slash, increase the cost of remediation slash just vindictively do damage. But, basically, they they detonate a, information deletion bomb or whatever you're gonna call that. It just deletes a whole lot of it's kind of a r m r f kind of bomb. So once it's cut off from its telemetry, it it then proceeds to just delete everything it can find. So that's a little rough. There's a bonus one here, sandworm mode. I won't even go too deep into it. But, hang on a sec. Yeah. So the bus radius here is actually hard to calculate. I would say, to some extent, it was the whole JS ecosystem, hundreds of organizations. We know that, from Trust Wallet, something like, $9,000,000 was stolen, $8.08 point something million dollars was stolen. That's just what we know about. I would estimate that it was far more than that because most people don't disclose when things like that happen. Hundreds of organizations affected. And, now that it's known that this kind of attack can happen, we're seeing all sorts of copycats. So it's one of these attacks where it's almost hard to calculate the, the fallout, and we're gonna be dealing with it and various copycats for a long time. And then Sanwar mode, that's our little bonus one. This one's even hard to get into, but it targets because it's it's, has a lot of complex mechanisms and it's very sophisticated. Many of these are sort of self evolving, but it targets developer, workstations and specifically your AI tool chain. So your so your, your agentic tooling. And then it actually has a sort of a multistage payload. So it waits. It does, different stages of attack. So it's actually fairly sophisticated and a fairly recent attack. What do all these attacks have in common? What's the commonality here? And it's basically that they target the supply chain. Okay? So and all of us here I mean, I'm I'm assuming most of us here in this room, you know, maybe there are a few curious folks or people who are training or students who eventually will be in this position. Most of us have users. Most of us have people who are relying on us for our code to be secure. We have reputations to maintain. We have businesses. People that are relying on us in our infrastructure. If that infrastructure is compromised through the supply chain or for through any means. But the thing is the supply chain, the upstream of us, that's to some extent, it's out of our control, in the normal case. Right? We're trusting these upstream maintainers. We're trusting these upstream folks. So we're in this sort of difficult position where on the one hand, folks are relying on us. We are on upstream for a whole lot of folks, and then we are relying on upstreams to be secure. And they're increasingly not. Okay? So that's the basic problem we're in here is this, supply chain conundrum. Okay? So let's introduce our proposed solution to this. It's is it a magic bullet for absolutely everything? No. I'll explain some of the areas it's not. But, however, the in in by some measures, the protection here is, 98% or or actually higher. That's Python and 99 something percent for, for for JavaScript. And I'll explain the mechanisms. We'll get deeper into it. But what is ChangeGuard libraries fundamentally? So ChangeGuard libraries is, I'll kind of explain the high level here and we'll get more into more details. But the, essentially, what we do is we take source, from say we go to PyPI, Maven Central, NPM. We find a a package. We take we we take the source link. So we find the original source. So that's what people assume they're downloading. They're assuming they're downloading something built from the actual source, which may or may not be true. We take that thing that people are mostly thinking that they're downloading, the source, And we pull that into our secure, highly operationally secure salsa three, fact chain guard factory, which is this, you know, this huge system for building all of open source every day. We pull it in. We build it securely. You know, it it's a difficult technical challenge because we can't necessarily trust any of this upstream. So we have to isolate everything, build it in isolation, kind of look at it very carefully. And then we build but we build from that source that's that that observable source. And then because of that, anything that happens in those targets that I was talking about. So if a maintainer has their laptop compromised and their API key is stolen, if a maintainer's CICD is is hacked, if any of these, types of mechanisms, happen, then or, you know, for example, if a upstream is compromised. So the developer is running privy in CICD. If developer is pulling in a package that that itself gets compromised. That won't you will actually still be working with a clean library because we we kind of did an end run around it. We pulled the source and then everything in the developer's build process no longer matters. Okay? So if their CCI CD got compromised, doesn't matter. You're just reading about it. You're reading your newspaper in the morning. You're drinking a coffee. You're not going into Slack and starting an incident response because it's you just actually using the clean source built in our system. Okay? So I'll try to explain that a few different ways. I always see when it clicks with different people. So and I will say, I like Tangerine libraries because it's not just scanning. It's not just giving you information. It's not just, oh, we're speeding you up 10%, 20%, 30%. We're actually deleting whole categories of attack. And actually, they're the categories that are mattering most for these major supply chain attacks that happened in March, and have been happening for the last, in increasingly the last three years. So let's talk about maintainers for a second. I love open source maintainers. I am an open source maintainer. Sometimes I wear Hawaiian shirts too. Sometimes I make weird fashion choices. But, you know, my some of my best friends are maintainers. I love going to these conferences. I don't I don't mean that. Like, I mean, I'm I was I'm setting up something here, but, like, I do mean that literally. Like, I I I love open source. I love maintainers. Maintainers are altruistic. These ecosystems are amazing. I love Boss. With that being said, they're simply not prepared for what's going on right now. These these ecosystems are under constant threat, and any any kind of target, even the long tail now with AI, not just the biggest packages, those medium popularity packages, low popularity packages, any of them are targets now because the activation cost of attacking these folks is so low. The attackers are just, well, being like, oh, you know, I'm the maintainer of this package. Like, let's penetration test it. Let let's harden it. And Claude will do anything you want as long as you frame it like that. So or, you know, not to pick on Claude. Chat g b t, codecs, whatever you're using. We'll all basically do the same thing. So the containers are their their infrastructure is not prepared to keep up with any of this stuff, and they're under constant threat. And they're just normal people. That's kind of what I'm suggesting with this Hawaiian cert thing is that these folks are ultimately like, you're on the line to protect your customers. These folks are not on the line to protect your customers. And if if their thing gets attacked, it'll be embarrassing for them. They're gonna have a bad day. But, ultimately, a lot of these things are passion products. They're not necessarily involve money or they're consort some consortial Linux foundation thing or something like that. It's not the same as having, users that you rely on. Okay? So I don't know. That's my, you know, love maintainers, but can you rely on them in 2026 in the way that we used to? Basically, the answer is is not so much. So that that's maintainers. What about the repositories? I would say, you know, I also love PyPI. I love you know, NPM has a lot of problems, but it's a great ecosystem. Maven Central, these are, like they're great pieces of infrastructure. They've modernized a lot. Now we have trusted publishing. However, that's another big however. It's not it's also not their job to to prevent malware. Okay? And and we're kind of the whole world has been built to kind of assume you can PIP install something and just trust what's on there. And it's it's empirically no longer true. And, you know, the PyPI folks will say to you, PyPI has never been compromised. And that's kind of true because they don't count any specific maintainer getting compromised as a compromise of their thing. They just kind of pass whatever through. There's like they're like, it's not our responsibility. However, that's kind of if you get hacked because you pulled something bad from PyPI, that's kind of, like, not a good it's not a great consolation prize to know that it wasn't PyPI that was compromised directly. It was the maintainer. Because PyPI still passed on something that that got you compromised. Kind of a corollary of that is after the fact is not soon enough. Okay? You cannot, just and and this is kind of what, NPM, PyPI, and stuff were kinda based on. And And the system worked really well for a long time is that if someone uploaded something bad, a volunteer would eventually come around and be like, hey, this is compromised. That's great. I love that volunteers do that, and that's kind of what's made everything work so far. And it when it happens, it minimizes the damage. And often things get caught these days within twenty four hours, forty eight hours, whatever, sometimes longer. Sometimes within just a few hours, Axios. Axios was online for two hours to to compromise. And how many people downloaded in that time? 600,000. 600,000 people. And and any of those could become a supply chain attack too. Maintainers were downloading that. So, you know, unfortunately, doing it after the fact is is too late, unfortunately. And then the CVs so and, you know, one of the big defensive oh, what about the CVE system? What about this? I mean, the CVE system is another amazing piece of infrastructure that served us very well for a long time. It's still critical to patch CVEs, almost immediately. And in fact, it's actually more important than ever because, you know, the CVE, the time to the time between disclosure and exploitation of a of a known vulnerability is now negative. In I think the the average is forty four days. Twenty eight percent get compromised within twenty four hours. And some of them are getting compromised almost immediately or actually before they disclose it, which is crazy. That's all according to Mandiant. So you do have to remediate CVEs. That's still important, but it's orthogonal to what we're talking about here. The CVE system and making sure that you don't have known vulnerabilities doesn't help if the upstream package gets compromised. So malware is an orthogonal vector. It's a diff it's a attack that happens on a different axis. It's something that you actually have to, worry about and consider as a separate threat, independent of CVEs and known vulnerabilities. Doesn't mean you don't have to worry about CVEs, unfortunately. We and chain guard containers, I think, is the solution to that problem. Chain guard libraries, I think, is the solution to the malware problem. These are two different orthogonal problems. Okay? So what does Cangar libraries basically bring you? And how do we solve the problem? Or what state are you gonna be in after you, you are using Cangar libraries? So first of all, Cangar libraries is proactive rather than reactive. Okay? Because we're sort of deleting these specific categories of, CICD compromise, leak, upstream, maintainer, compromise, you know, dependency, compromise, all these supply chain threats. We're just deleting these categories. So, look, speed doesn't come into it. And so if it falls within that category, you're just simply protected. Okay? So that's one way to keep up with the speed of the stuff that's happening right now is to just delete categories of threat and be protected though from them from the start. So it's proactive rather than reactive. And I would also add to that that it's not just giving you information. It's actually fixing a problem. It's trusted. So what does that mean? I would say there's two two, sort of variance to this too. I'd say, on the one hand, we're we're I to my knowledge, the only SALSA, well, for containers, we're SALSA level three. The libraries, I believe, were at SALSA level two and working toward level three. That means extremely operationally secure. That means we operate in zero trust environments where we attest to every, every artifact that we build. That means it's, cryptic the artifact gets cryptographically, some metadata gets cryptographically attached to the artifacts that speaks to that artifacts provenance, the circumstances under which it was built, who's built by, when it was built. And then at every stage in our build and distribution process, including when it gets put into your hands, you can check it yourself. You get to then verify the provenance of that artifact. That's key. If you're not checking the your, I would say, look into zero trust, look into Salsa, look into sigstore yourself for all of your, build pipelines. But that the next best thing is to use something like our, chain guard libraries that does come with attached, SBOMs, provenance, cryptographic signatures. Very important. I'd also just say that we're operationally trusted. Like, our necks are on the line in a way that PIPIs or or maintainers are not. So you can basically rely on operational security because that's our business model. Okay? And then, of course, our our coverage is, is very good and growing. So we're we're aiming for covering almost all of these ecosystems, and we're getting very close. So, now with ChangeGuard repositories, you can actually, hook up your artifact manager, you know, to, to our ChangeGuard repositories, or or, you you know, you could just use libraries directly and fall back to IPI if you want, but we do add some extra protection if you use repositories. Our coverage is getting very good. So chances are for all your workflows, you we already have it built, and we have to whatever the freshest thing is in there, you can also add the customer to make requests to make sure that, you know, what you need is in there. Alright. Oops. So here's the ecosystems that we cover. So basically, right now, and you may some of you, if you follow Dan Lawrence, he just actually teased you know, he had a survey saying, oh, which new ecosystem are you gonna go to? I don't have any special insight into that, but more ecosystems are on the way. But right now, we've, securely rebuilt, three ecosystems. So that is Python, and, you know, this is compatible with PIP, UV, whatever you're used to using. And then, of course, also, Java and JavaScript. Okay? So those are the ecosystems we're covering right now. If you're specifically interested people have reached out a lot about Go. That could be the next one. Again, I don't have any specific information on that. But, but you we're looking at a lot of different ecosystems, and we're gonna be adding ecosystems this time. So who's been using libraries? Who's kind of raving about libraries? Okta has talked about libraries. And basically speaking to the idea that, like, that I was talking to you about earlier, which is that, ultimately, it's not these upstreams that are responsible in the way that, us folks who are building products that putting into people's hands are responsible. So if you're if you're in that category and you have use end users, if you have downstream review, then strongly consider ChainCloud libraries. And we use Okta to secure us, so it's great when folks, like, when when organizations, like, that we respect like Okta are using ChainCloud libraries. And, so let's talk get into some takeaways. Few more silly memes on the way out. So this is a little more general, and we're gonna get into some of these. And we're gonna be a little more general and basically be like, be you know, we think you should use Fingerhut libraries, but, like, you know, I might say that one or two more times, but but what else can you do to secure your, your supply chain and to push your operational security a little further left. Okay? So one of the takeaways here is, that, you know, I've hit this I've hit this note a little bit, but it's worth hitting it again, is that attackers are going to look for the, the easiest way to compromise you. Okay? So they're not that means they're probably not gonna walk through the front door. What does that mean? They're not just gonna go to the what you feel like is the, the, you know, what you would think, which is your production system and just attack that directly. Why not compromise a developer's laptop, and grab the a the API keys and, hey, now we've set we have a compromise on our hands. Why not attack the CICD, and grab a key? So, so one of the takeaways for this is that basically that you have to treat, all your basically, your full, operational environment, and developer workstations, build processes as, first class citizens that you would secure just like you would secure web application, just like you you would secure your cloud infrastructure or your other production infrastructure. So least privilege change management, all that good stuff. What else what can you do about developers getting compromised in is that sort of, like you know, we outlined that sort of, like, Kevin Mitnick situation that, Axios, Sapphire Sleet situation where the, like, a literal actor has been hired to you know, this is actually what's happening out there. A literal actor is hired or someone who's a genius of talking to other human beings is calling up your one of your employees and and trying to get keys or or jumping on a team's call. You can't just expect employees to you know, that's a person who's a specialist. You should do phishing training, but the reality is that, that's that's probably not sufficient. So what is better is to structurally make it very difficult for your employees to, to give away the keys to the kingdom as easily as it was in the past. Okay? That means doing things like using u b keys. Okay? So hardware keys is is one of the ways to go here. Okay? And and looking for that kind of situation where it's more becomes more structurally difficult. We talked I teased this a little bit. I talked about it. I I wonder if you get the idea a little bit, but this is Predator. This is the Arnold Schwarzenegger movie, so he's running along. So that's my idea of a ghost attack. It's kind of like a ghost. But, ghost attacks. This is the really the thing to look for. Because this is kind of the mechanism that underlies a whole lot of what I was talking about. And it basically, it's worth saying one more time is if you download an artifact and you look at the source, that doesn't mean those two things are the same thing. And the delta between those two things, so we have Sam from the, or, Pam from the office kind of being like, oh, corporate needs you to tell the difference between these two things. That's not easy. That's not an easy task to know the difference between, oh, I have this artifact on my machine. I'm running this artifact versus, oh, here's the source. I would say it's, you know, you could try. Maybe you could tell there's a difference. A lot of times it's very subtle what the difference is, between these two artifacts are or it's not a practical matter that you could actually evaluate those two things. Okay? So what you need to do rather than be like, oh, after the fact, I downloaded something. Should I check if it's bad? You have to collapse that. Okay? You have to basically be like, I am so certain that I'm working from an artifact that was built from that source. Okay? Because so much can happen in CICD with developer compromise for upstreams, in distribution, from source to getting to you. Okay? That's the danger zone. So so that's one of the big thing. If you take away one thing from this, I would say take away that. The source code is not the artifact or you have to ensure that it is, which is not the easiest, proposition. Long lived keys are very, very bad. We have the Anakin Skywalker meme or whatever here. Like, you remove the static keys. Right? Basically, you want to be using infrastructure that replaces long lived tokens, long lived keys. You've gotta start to thinking about these as, like, toxic artifacts. Okay? So much of this compromise is either precipitated or made worse by the fact that these long lived keys are lying around. Like, if you wonder why why did some of these attacks, like, double tap all the time, it's usually because, oh, they did it x y z compromise. And then while they did that compromise, they realized, oh, yeah. Well, I could grab the key while I'm at it. Then over the weekend, they're like, well, I got the key. I may as well use it. So they do a second attack. So so keys are really need to be replaced by a short lived token, short lived keys. Great project to look into for for doing this in your own operational security is sigstore, which is basically will help you sort of power, a, to use, keyless signing, short lived keys, all that good stuff. They also have transparent ledgers. I love the six store project. And then, we also have something we call OktoSTS. It's something to look into. It's not a project of the same size as a six store. It's it doesn't do absolutely everything, but it's a bot that will, in your CIC, swap out your long lead keys for short lived tokens. Okay? You take a look at OktaSTS. I'm kinda let's kinda pull it together here. So you wanna verify the process, not just the product. What does that mean? The product once you look at the product and you're trying to examine the product after the fact, oh, is this the right package? Oh, let me scan it. Oh, I downloaded this tar. Let me look at it. That's too late. It's already on your developer workstation. It's already in your CICD. It's already running, you know, like, pin files or whatever in your, in your CICD. Like, but, to to do that is to kinda be like, oh, you know, I'm gonna drink some whiskey. Wonder if it's poison. Well, there's only one way to find out. Let's drink it. And then, oh, I died. So I guess it was poison. The reality is if you're scanning the artifact and it's sitting on your hard drive, that it requires a very sophisticated setup, like sandbox setup, maybe even air gap setup to examine these artifacts securely. And so don't you can't, once you have it on your drive, it's too late. So what you wanna do is try to get it from a source where it's kind of more structurally difficult or almost impossible for, for that artifact to be compromised. Or at least, let's call it statistically way less likely, a two percent chance from a 100% chance. Okay? Which is what Changer libraries does. So until June 30, Chancard libraries is actually free. So, I mean, I'm tempted to do in my, like, radio announcer voice, like, June 30, June 30. It's like, get your Chancard libraries now. But, like, you should check it out, because I do think it's it's it's solving the right problem. It's gonna that next month is probably gonna be as crazy as last month. Honestly, you know, we might go in more into this in a in a q and a, but, you know, things are gonna probably gonna get worse before they get better. I do hope they will get better eventually that the ecosystems are gonna catch up. But, but not any time not in the next month, not in the next three months, probably not in the next six months, twelve months. And and, I would say in terms of who's solving this problem right now and actually making categories of things go away, it's it's I don't know of anyone who's who's working in this space, who's doing something like this. So I would say, check out Chingart libraries. Try to wrap your head around the mechanism of okay. We grabbed source, so now these these areas of concern are actually no longer an area of concern. Because it it is a new category. It's different. It's not just scanning or whatever. It's not just, doing something after the fact. It's not CV remediation. But we actually also have CV remediation in chain guard libraries, which is awesome. It's actually kind of a new mechanism. And I think it's worth wrapping your head around it, and understanding what it does and the value it can bring to you and your organization to secure developer workstations, to secure your, CICD, to secure your organization against attacks like Trivia, Axios, WhiteLlm, UltraLinux, ULO, all those. Okay, everyone. Thank you guys for joining our talk. We're actually going to kick off some q and a right now. So, if you have any questions, please make sure to put them in the chat. I see that we have a couple in, right now, but let's I'm gonna give it just a minute to see if we have any more, and then, we will definitely take those questions. Okay. So, I'm actually gonna start us off. Patrick, it looks like we have a few questions that have come in, so far. So Cool. First off, wanted to, start off with, why are there more and more of these supply chain attacks, and do you think, they will slow down? Yeah. I kind of have a blog post on this called, something like 2026 a year of AI assisted attacks. But I'd say there's kind of two parts of the question. The one sort of pre AI answer answer is that there's been, like, a long running secular trend, which is basically just saying, like, a static trend that is consistent over time where and let's call it the software eating the world trend. Basically, it's like since, I don't know, 2010 or whatever or whatever whenever year you wanna peg it, More and more critical systems have been going, being baked into modern software, including open source. And that means because it's a juicy target, you know, say, a hospital moves and uses starts using, open source infrastructure or military or the bank a bank. That basically means they're juicier and juicier targets. And because that is a you know, that, that cloudification, or whatever you wanna call it, software reading the world trends, it it happens over time and more and more things migrate. It's things are more and more juicy targets. That being said, I would say that's not most of what we're seeing now. I'd say most of what we're seeing now since 2023 is a big leap from AI capability. So I would basically say if you make a Venn diagram with, like, people who are interested in making an attack, you know, they wanna do it. They have free time. What inclination? Whatever. And people who are, like, technically capable of making an attack, That used to be a fairly small sliver of folks, you know, but now it's a much larger sliver. And that basically is mainly because it's so much easier to do an attack now if you are using a, modern you know, not honestly, not even a modern agentic framework or whatever, like, your cloud codes or your codexes. A lot of these attacks are happening just with, like, chat g b t or whatever. But basically, AI is, like, bringing up the floor and making it easier for people. We would have called script kiddies or even totally nontechnical folks to pull off a large segment. Okay. And then another question, that came in is, what if I just stop using open source libraries altogether and block all these external repositories to reduce the risk? You know, it's an I've kind of it it really interesting question because I think there's there's kind of like a a simple version of a question. I like a galaxy brain personal question. It's kind of like that that meme you see with, like, the the simple guy and the Jedi. And, like, I would said on on the face of it, it seems ridiculous to be like, oh, let's just not use it anymore. And then but then you were like, oh, maybe it's a kind of a maybe on one end, you're like, well, maybe it's so bad to use these things that maybe we just need to, like, shut everything down, and not use these things. So let me take the question, like, really seriously and not be like, oh, just just need your reaction. So recently, there was a company, cal.com, that was basically, like, we're gonna actually go not be open source anymore. We're not gonna, like, release our software open source. We're gonna go closed source. They were criticized a lot. Basically, I would say that move is not gonna help them at all, basically, because AI can also attack your can evaluate your production infrastructure. It doesn't really need to see the source, and it's only a very mild help for it to see the source. And, actually, now they're getting very good at analyzing binaries, analyzing endpoints, and so on. So I would say closing that off is helping them extremely marginally if at all. They're also basically still dependent on there's basically no practical way you can not use open source as an upstream. Right? I mean, even, like, the NSA is running, you know, Linux work workflows. They don't have their own fork of Linux or whatever or, like, reinvention of Linux that they made, like, not invented here. And that's another way of interpreting the question would be like, oh, let's Vibe code everything. Let's rebuild everything from scratch. Think of anything that's, like, the more serious, thing to to to answer, but I would say, it's also not possible because, a, there's so much stuff that's basically not Vibe codable. Like, are you you're gonna be rebuild Linux? I mean, cool project if you can pull it off, but, like, basically, I would say functionally impossible. And then, and then you're basically going to be creating crappy versions of sorry to use that word, but, like, bad worst versions of all these open source things that themselves were gonna have problems. So if you're a juicy enough target, people will learn how to reverse engineer and and attack your Vibe coded Chromium versions. And they're probably gonna have more bugs than your upstreams. So I would basically say, take take this I like to take it seriously as a question, but, basically, I would say it's just an extremely bad idea and don't do it. But if you try it, let us know how it works out. And then, we have one final question, which is what would you consider, proactive best practice for development tools? Yeah. That's an interesting one. I would say development tools are probably I would say they're like the new battlefront or whatever. It used to be all about your prod system. Right? And mostly people didn't you know, and you oh, gotta prevent SQL injection. Gotta make sure, you know, I don't have critical vulnerabilities in my system. But I would say, the attackers are very aware that developer workstations are weak point. And, you know, just to take just to be very practical for a sec, like, let's spin a story. You know? I mean, we a lot of, modern systems, they notice, say, for example, pin dependencies. So you're like, oh, I'm only gonna put into my production system anything that's, like, less than a week old. Right? So you're gonna be like, I'm gonna wait a week then I'll put it into production. I'm not going to just run on latest in prod, which is we absolutely recommend. That being said, someone at your organization is evaluating the next version and checking it out and making making sure that works. Right? Probably, you have a staging or something like that that is set up to to evaluate the next thing that comes out. So so that means that your developers are pulling down the new latest and greatest and also the stuff that might have just been supply chain attack. So that's just one example. But, also, you know, developer workstations just aren't treated as harden. Also, developers are running a lot of, tools that are, for lack of a better word, much more janky than what's in production, such as, you know, I mean, love plug code, but it's, like, it's not really a hardened piece of software. You know? It'll just run whatever load it to context whatever markdown file you have in the in the repository. Right? I mean, that's how it's created by design. So I basically say, what do you do? I would say, a chain guard libraries is great because it helps with both, your production system, your CICD, all of those systems, and your developer workstation. Right? Because if your developer is downloading Axios to evaluate it for the next week, then they are they if they pull it from Chingart libraries, they're gonna be safe. Okay? You have to make sure they use it, so that's another issue. I would also just say with the modern AI workflows, like, look at skills. Like, we we're we're coming out with Chingarn, agent skills, which is basically hardened skills that, like so if your if your developers are pulling in agent skills from the open web, something like, somewhere between 620% of those are, like, actively malicious, and I would there's another, like, 20 or 25% that are, like, have way too many permissions. They're, like, god skills or whatever. They have way too many permissions. They're basically bad. So and do you have an idea of how many, of those kinds of skills your developers are running? No. Basically, nobody does right now. The infra isn't there. It's all fresh. It's new. Skills are only a few years old. So check out Kingard agent skills. Check out libraries. Try to figure out what your developers are actually doing on their workstations because suddenly, it matters a lot more than it used to. And I guess that's my long winded answer to that. Cool. Yep. Thank you so much, Patrick, for sharing your insights, and it looks like that's all the questions we have today. So, to close the group out, I actually want to share, two upcoming events that you may be interested in joining. We hope to see you there. This first one that you see here is actually with our customer, and they're gonna be sharing about how they're against AI threats. So we would love to join you. This is happening on, May 20, which is coming up soon. And another event that we'll be hosting is a learning lab on how to build safely with AI. And we have our very own Erica from our DevRel team who's going to be, running that session. Hope to see you there, and thank you guys for joining us again.