Name: Malware Unpacked: The Trivy/TeamPCP Attack, How It Happened, and What to Do Now
Uploaded: 2026-04-02T18:26:05.637Z
Duration: 37 min 9 s
Description: Malware Unpacked: The Trivy/TeamPCP Attack, How It Happened, and What to Do Now

Transcript for "Malware Unpacked: The Trivy/TeamPCP Attack, How It Happened, and What to Do Now": Hey, everyone. Thank you for joining us. I'm Ross Gordon, product marketing here at Chainguard. Super excited to be kicking us off. Let's let's jump in. I see folks continuing to trickle in, but, again, excited to be here and talk about what's going on. Obviously, the crazy two weeks March, with the trivia and TeamPCP attack. So I wanna jump in with a quick high level for those who may or may not be, in the loop. So to start, here's kind of the rough timeline that we're working at. We started all with Hacker Barqaw, and and Dan and Reid will go into more of the specifics throughout. But it's, you know, round after round after round, another open source tool has been continued to get breached and expand the blast rate radius, which we'll we'll talk a lot about today. And I think the scary thing, most and foremost, is the attackers behind it, TeamPCP, have basically said it's the year of the supply chain and that we're gonna be busy for a very long time. So looking forward to, you know, being back here in a little bit to take on some questions. And if you have questions that pop up over the course of the webinar, feel free to drop them in the q and a, and we'll definitely save some time at the end to get to those. But without any further ado, Reid, our VP of product, and Dan, our CEO and founder, please come on to the stage, and and let's kick this off. Thanks, Ross. Hi, everyone. Appreciate you all taking time out of your day. I know you are all super busy as we have been the last couple weeks. I'm gonna wait a minute for Dan to join. So as Ross said, I'm Reid on our product team, and then we've got Dan, CEO, founder of Chainguard. So, Dan, I was gonna go through and ask you a handful of questions. I think to start so Ross talked high level about the timeline. Let's just I'm assuming everyone here is very familiar with Trivy, but in case they don't, Trivy, open source vulnerability scanner, who uses Trivy? Why do they use it? And. Trivia is a free vulnerability scanner. It's made by Aqua Security. A lot of open source projects use it. A lot of companies use it. It's one of the better ones out there. I think people know it's widely used, but we're we're talking really widely used. I mean, I think hundreds of thousands of contributors, like millions of Docker Hub polls. Right? Yeah. It's it was one of the earliest ones that, like, really supported container vulnerability scanning. It's got infrastructure as code stuff now. It can do a lot. It's a powerful tool, and a lot of people use it. It's been around for a while. So okay. We've we've established trivia is a big deal. Can you go a little bit into what the exploit was? What happened in this particular attack? Yeah. So it's super subtle because it was, like, a supply chain attack that then led to another supply chain attack, and GitHub actions. were involved and then exploited and then also compromised and then exploited again. So it's really complicated to actually kind of explain what happened because it was a bunch of different things to the same systems in the same terms over and over again. But it started with the Hacker Barclaw attack. I don't even really know where that name came from. I think it was, like, the fake name for the account or something like that. But they exploited a misconfiguration inside of GitHub actions for a bunch of open source projects. GitHub actions are, like, you know, shell scripts that you can set up to run when something happens in a repository. Like, a pull request gets created or an issue gets created or something like that. For open source projects, you're kinda, like, taking untrusted inputs, letting untrusted people from the Internet do things to your project, and GitHub actions then runs code from untrusted people. It's kind of a a security challenge and a security nightmare in a lot of ways, but it's how things have to work. The way Trivy had some of their GitHub actions set up, this is incredibly common. This has happened to a lot of projects over the years, but one of the variables in there, that they were using was the branch name. So somebody sent a pull request, from a branch. But you get to control a branch name, and that was used inside of the shell script, to do stuff. It was a classic shell escape issue where somebody made a very carefully crafted branch name that exploited the script that then ran inside of it to steal a bunch of credentials from that environment. This got caught. They did it through a bunch of repositories. People noticed we were looking at branch names and things like that, and a bunch of projects got caught. There was a headline about it. Everyone said they rotated their credentials. Not much happened after that. Then a few weeks later, it turned out that the trivia project had not properly rotated all of the credentials that they thought they had, and the attackers had gained another foothold in that initial one. It's a common thing. The first thing an attacker does when they get a foothold is get another foothold and then get another foothold, for in case that first one eventually gets cut off. They have a way back in. So they use that to compromise the trivia binary itself. They put malware into the binary of trivia that was published on a GitHub page, the Docker images that everyone is using to download trivia, and the GitHub action for Trivia itself. So the first one was a compromise of the GitHub actions Trivia used to develop the project. The second one was a compromise of a GitHub action that Trivia project was publishing to put malware in there. The way GitHub actions is designed is that that spreads instantly. Right? They changed the tags. Everything was written over. So anyone that was using that trivia GitHub action, the very next run pulled in malware. That that has a huge blast radius, as you can imagine. It's this free tool that does security. So a lot of projects that care about security have this tool installed. In the very next run, there's malware in there. And then that did the same thing. That malware stole more secrets from all of those projects. And all this all these secrets we're gonna send, they were encrypted so only the attackers could read them, and they're gonna send to some random IP address that was, you know, tunneled somewhere else. We don't even really know where they went. But the attackers have posted something like terabytes of data, was stolen in the hours of that this thing was live hundreds or thousands of projects. And. well, Go ahead. I think there's two interesting things there. One, there were two slightly different GitHub Actions exploits. And. two, you were able to steal creds that the initial trivia action didn't have access to. Right? It was, any of your own creds. So, like, Yeah. it it's this cascading, Yes. Yeah. transitory dependency. Right? So it's going fractal kind of at this point because every. one of these leads to a bunch of more credentials that can be used to put malware and a bunch more things to steal a bunch more credentials. Credentials don't really do anything on their own. Like, at some point, attackers usually start and turn to ransomware or something else like that to, you know, actually get something useful out of these attacks. But this one's been patient, and they just kinda keep doing more of these to get more credentials to eventually target higher and higher, you know, numbers of companies or projects or organizations or something like that. Trivy finally got them out after the third time. And that same week, there was malware published to PyPI inside of the LightOwn project to a JavaScript project, Axios, and, again, doing the same thing, just using that malware to steal credentials from where it's running. There's a question here. Could it be a state actor? I I honestly have no idea. At first, it looked like a ransomware gang. There's a bunch of leaked Deepgram chats. It's hard to know what's real. Like, you know, there's not some real person that we can easily verify that's posting authenticated messages about what they're doing. So there's lot of rumor, a lot of speculation here. There's one post saying there was just way too much data for them to even go extort everyone. So they were gonna, you know, sell off some of the data to other groups to kind of, like, crowdsource ransomware and stuff like this. But it is starting to spread. Real companies are getting hit. Data's leaked. You know, Merkor had their whole, database compromised. Other companies have been hit in the wake. And I'm sure there are more open source projects that haven't noticed this either yet and other companies that haven't noticed this in all their CI pipelines yet that are just the next link in one of these whenever the attackers get around to it. I wanna hit on that. Could it be a state actor question? Because I had a question later. So, one, we know that at least team PCP is the name of the group that claimed credit for this. I I don't think we know who they are. I saw John Holquist from Google had referenced there was a specific North Korean actor, UNC ten sixty nine. They actually don't think they're related, but they use the exact same token that Team PCP stole. So I think Team PCP harvest a bunch of tokens. They're using them, and then it seems like they're probably just dumping these tokens for anyone to use just to kinda be jerks about it. Yeah. The motivations behind these are never easy to understand or predict. Right? Some people do it for fun. Some people do it to show they can. Some people do it with real bad motives behind it. Some people are doing it to make money. Nation states are doing it for other reasons. We're not gonna know for a very long time, and we might not ever know. We still don't know who Jia Tan was that did the XC utils attack. Right? The the other interesting thing, I think it was Trend Micro, did an analysis of the actual team PCP code, and they were looking at comments. And they said based on the comments, we actually think these are native English speakers, which I thought was super interesting. So, I I mean, that's the like, we don't know for sure, but it it seems like probably these guys are unrelated to state actors. But, also, I like, I don't know if it matters. Right? Like, does do you really care if, like, it was North Korea or China or, like, some kid that happened to steal all your tokens? Right? Yeah. Well, you shouldn't have to care, but I I'm not a lawyer, but, unfortunately, the securities laws and cybersecurity laws in a lot of different countries kinda mean you do have to know or care. CSOs and people dealing with this, they. do have to try to figure out who it is. These are really limits and changes your options in how you do the remediation. There are sanctions in place. Like, you're not supposed to deal with nation state actors. If you don't know if it is or not, you have to your best to figure it out. And if you're wrong, like, there there's strict penalties. Okay. The the last one on the the specific attacker, what I thought was funny was, like, after stealing the creds, they seem pretty brazen. Like, there were a bunch of repo names, like pwned and massive x fill. It's, like, almost like they were trying to troll people. And, like, even those those namings didn't matter. Like, they no one caught those eventually, but right away. I I thought that was pretty crazy. They weren't trying to be subtle and sneaky here. Yeah. It was just making a statement. Yeah. Okay. So, we don't know who it was. We talked a little bit about the attack. If we go back actually, no. So one of the things that's come up a lot is, shopping as a solution. I think this is a limitation that like, three, four, five. Like, it's been flagged forever. It seems obvious. Like, why do you think we are still talking about it? Is there anything particularly hard about. fixing it, or are we just, like, the industry's lazy? Like, what do you think? Yeah. It it it's annoying. Yeah. Like, it there's a bunch of reasons here. And I think fundamentally and I posted about this on LinkedIn. I think fundamentally GitHub actions are very hard to do correctly. That's why the shell injection, the initial one was possible. That's why the cause of or the the fix for a lot of these is, like, you were holding out wrong. At this scale, with this. many people using it and disconnected of a web, it it it's impossible for everyone to do things correctly all of the time. The fact that the default is to not pin by SHAZ is a huge issue. But also the fact that it's possible to change a tag later is a huge issue. Right? Package managers have come a long way. They've been around forever. Things like NPM and PyPI have learned a lot as they've developed. None of them ever let you move a tag after it has been published. They make it hard to even remove a tag and yank them and stuff like that because of the potential for breaking builds. That's just the default in GitHub that you use these mutable things that can move around. SHA pinning isn't even perfect. It doesn't even get you all of what you're looking for here. A lot of the GitHub actions are composite actions or, other ways where they actually just wrap something else. And even if you've pinned the SHA, that thing can just move underneath it. Yeah. So, yeah, you might think you pinned everything and nothing can change. And if it's a composite action that wraps a Docker image that has a latest tag behind it, you haven't pinned anything. You just pinned the you have an immutable reference to a mutable pointer at the end of the day. And if you look really in-depth into how the trivia action itself was compromised before it was published, that's exactly what happened. It swapped a normal action to a composite one where they controlled what was gonna happen behind the scenes. And it also took advantage of this other issue in GitHub, that we reported to them years ago called imposter commits. But it takes advantage of the fact that all of Git repos in the GitHub database, are kinda stored as one unified object in the tree. So if you click fork on a repo, they don't make a whole new copy of all of those commits. All of the forks are still kind of stored in one unified database. And so the the malware looked like actions slash checkout and then a SHA. Any reasonable person would think that SHA was somewhere in the actions slash checkout but it doesn't have to come from that repository. And if you go look at it in the web browser, there's this tiny little yellow box at the top that says, hey. That doesn't actually exist here. But if you do that inside of a GitHub action, that doesn't show up at all. So if you go look at it on these things, you you have to be able to reverse SHA two five six digesting your head to have any sense of knowing what's behind these things or if they're safe to use. That's not possible yet. So, yeah, shop pinning is like, you should still do it, but it doesn't actually solve the problem. So I wanna go back. You talked about how the attack evolved. Light LLM, creds were harvested. Did they do anything in particular that made Light LLM more more vulnerable? I think this was just random who happened to get hit. Right? They had trivia in some workflow in a repository, and they had creds there. I think it was a Python published token or something like that that got stolen, so the malware was pushed right to PyPI. They actually, did some things that made it a little bit harder. There were some GitHub tokens in there too, but they had stuff like branch protection and everything turned on. So the attackers weren't able to stick malware directly into the LiteLRM repository itself, but they were able to stick malware right into PyPI. And then I think, everyone's talking about Axios. Like, I don't think there's any evidence that, it was related to tPCP or the same attack at all. Right? I don't know. I there's been speculation back and forth both ways. Yeah. It's these things take a while to shake out, and everyone's rushing to get answers out, and half are right and half are wrong. We'll know more soon. Like, we mentioned malware a couple times, and I think it's it's good to just go back and point out. There was no actual malicious source code in any of these attacks. Right? Yeah. I'll push straight to the repositories. It's. straight into the binaries. It's harder to find there. It's harder to tell. Stuff you you you could put stuff into the Git repository, but it's usually easier to see, easier to catch unless it's hidden well. There's not a log built in there in the Git commits. Like, is there a common pattern with Teams that we're able to respond more quickly? Like, one, obviously, if you're using Chainguard, Trivia Image, you are protected. But, like, aside from that, which would be our obvious recommendation, like, what other things did you see people do that allowed them to respond more quickly? Yeah. There's a bunch of different ways you you harden yourself against this kind of thing. And you have to accept that nothing is perfect at the end of the day too. And if you're running code from other people in your environment at scale, at some point, of it is going to be compromised no matter what you do. You can drive down those odds by pinning and stuff like that and vetting things ahead of time, but you can't you're never gonna catch it all. There's a lot of tools out there that, like, look for malware and try to detect it in these binaries and look for anomalous patterns and stuff. There's, like, 30 different tools out there that do this. And, the light all on one, none of them caught. It was some researcher who installed it, and it, like, made his computer run out of memory, so he saw what was going on. It kinda reminded me of the XCUtils one. So I I don't think you could just sit there and scan and look for malware. And, you know, we try that hard to ourselves, but you're never gonna be perfect. It's a multiplayer game, and attackers have access to the same malware scanners you do. So if I were them, I'd be checking and making sure my malware didn't show up in those scanners. And then. we'll get better next time, and then the attackers are gonna get better next time. But it's like a defense in-depth thing and just general security edging. If you don't have root keys to AWS sitting in your GitHub account, that can't be leaked. If you are running that code in a server and prod and you apply principle of least privilege and can control and block lateral movement by securing your your the rest of your infrastructure. It's gonna be easier to remediate and the effects of it are less. I I think I'm hearing, like, one, assume you're gonna be compromised. Right? So, like, you you can't be, overly confident about this and then, have plans in place to respond really quickly when it does and then know how to, like, mass, rotate all creds very, very quickly so it doesn't take you hours to do. Yeah. Right? If you really need to have creds, make sure you have playbooks and stuff to be able to rotate them. Yep. If you have any two options, like run every malware scanner in the world and then trust what's sitting there and put my root keys next to it or don't do any malware scanning and don't have any keys in here, like, I'd pick the second one every single time. And I I mean, you touched on this, but I think it's important. Like, any malware scanner can kind of, by definition, only catch the last attack. Right? That's what they're trained on. And then, you mentioned this, and I think it was really good. Attackers use those malware scanners. So, like, any attack that comes out is going to, by its nature, bypass the malware scanners because why would you release a new attack vector if you can tell that it's already caught by the malware scanner? So it's just like, scanning is never going to it it's a, like, a minimum bar you should do, but it's never gonna protect you against null attacks. Yeah. The novel attacks is the big one. Yeah. Like, if if somebody's not trying it all and they just have a base 64 string with a curl pipe to bash to some, you know, dot r u domain, sure. Your malware scanner is gonna catch that. And there's probably attacks like that that happen all the time. But somebody somebody smart, somebody patient, somebody good is gonna be able to figure out a way around that. Okay. We saw Light LLM. We saw Telmux. I know Axios probably not related or maybe not related. Like, I anticipate there's gonna be a bunch more of these exploits. Like, do you have guesses? What do you think is is hot take. Are you sure? Yes. Yeah. I don't I don't think we're done yet. I think we're still in the growth phase of this. Like, the first couple waves of these, the attackers have too much data to go through. That's what they're talking about on the Twitter account. They haven't even finished finding all of the credentials from the first phase. They're just grabbing the top ones and putting more around and adding to that pile. It's gonna take a while before this slows down. So on that note about not having time to go through all the creds, everyone's obviously talking about AI empowering attackers. So, like, one, I can have this database and send Claude and, like, ask me to pull the most interesting ones that will give me the biggest blast radius. So this reminds me a lot. Prior to Chainguard, I was in the bot detection space. And, like, every time there was some public example of, like, a bot bypassing a CAPTCHA, everyone would say, okay. Like, defense is dead. There's nothing we can do. The bots have gotten too good. We always catch up. Is there anything about, like, So to the point now where I can't even pass a captcha. well, so the funny thing, this this is, like, a a detour, but, like, what people think of as a captcha has evolved. So, like, first, they were text that got really terrible, then they became images, Then, like, the actual act of solving the caption didn't mean you'd even pass. It was just a way to collect data and get mouse movements. Then, like, bots can perfectly imitate mouse movement now, so we're collecting, like, browser emulation stats and things. So it's like the the actual tool changes. Like, I'm curious. Is there anything about, like, the newer AI coding tools that make you think that attackers have, like, a permanent advantage here? I hope it's not permanent, but they definitely have an advantage here. Right? The only bottleneck historically for breaches has been patience and time by people willing to take that risk of, you know, committing a crime. Thankfully, most people on the Internet and most people in the world are good and aren't gonna go risk committing some crime to do something like this. So that number is relatively capped. So they have a certain amount of time every day, and they focus on specific targets. But AI is let like, it's letting developers write more code. It's it's letting attackers spend more time trying to get into more systems faster. You know, I love the agentic red team demos from all these amazing companies at RSA that, can, you know, do red teams and pen testing on companies a lot more effectively, but that same thing can happen. Attackers can do that same thing too. And that just means the the number of these is going to go up and security teams are gonna have to go even faster. So I hope it's not a permanent advantage, and I hope security teams get these same tools and practices in place to get rid of entire classes of these issues. Yeah. I think that attackers often just try new things faster because if you're, you know, somewhat if you're a criminal organization or if you're an individual, like, you don't have rules that you have to follow. So it's generally easier for you to adopt a new tool than a security team who has all of these rules in place. So I think we need to just more quickly figure out how to adapt. This reminds me of there was a sneaker bot that became, like, really popular that was, like, a SaaS sneaker bot that everyone was using, and it, like, stemmed back to one guy that was pissed off at this one particular site because he couldn't get the sneakers he wanted. So he just spent hours because his motivation was so unique to him to build that tool, and then it it spiraled and became a a tool that everyone else uses. Your, like arch nemesis, the sneaker guy. yeah. Yeah. Like, I honestly like, someone with a very particular and very unique grudge is often, like, I have a very particular set of skills. Yeah. yes. Yeah. Exactly. Oh, okay. Well, so going on that, so everyone here, I I think, is like a well, probably not everyone. Most people are defenders here. Like, what do you recommend people do to to, like, actively attempt to stay ahead of attackers who are iterating on on new threat vectors quickly? Yeah. Whack a mole isn't gonna keep working. You. have to whack a mole now while there are, know, moleheads popping up everywhere, and you you gotta deal with it. But you're gonna be on that treadmill forever. Get just get the secrets out of your environment. Rotate them right now quickly, but then get them out of your environment. Figure out how to set up OIDC. Figure out how to authenticate to everything from your build systems without that. We have an open source tool, Octo STS, you can set up. You know, we did this years ago. We were uncomfortable with, the state of GitHub security and built our own tools so we could talk to it without having to have secrets everywhere. And then, get a really good understanding of what is in your supply chain and where it's coming from and pin as much as you possibly can. But at the same time, set up ways to update all of it as much as you can too. Pinning and never updating and things like cool down policies and stuff are they're good ideas, but then you're just setting yourself up for failure when there's a vulnerability found in the next Log four j and something like that. So you're kinda trapped in this middle of not wanting to update too quickly and wanting to have control over updates while also needing to be able to update when something big happens. And the only way you can know if you can update is to do it regularly. It's like, you know, Yeah. eating your wheats every morning. And and I think getting to a 100% is really important there because attackers are going to find the one tiny odd weakness that you have, like, the one repo that has creds in it. You have to be really thorough. This reminds that's the hard part with open source. Even if you get rid of them for all of yours, yeah. you can't go see who else has them in your supply chain, and your supply chain is much, much, much bigger than your source code right now. Okay. That's a good transition because my other question, we saw a bunch of open source software compromised. I think there's people that are saying, okay. Well, I am I am now nervous about open source software. There's like does this lower your trust in OSS at all? Like, I I have a specific thought, but I'm I'm curious what yours it I I almost think that question doesn't matter. Like, there there's no going back. Open source is everywhere. Yeah. By and large, it's better than proprietary software. Right? It it vendors are getting hit the same way. Open source projects are. The only problem is you can't see it as easily, and you don't know what they're doing. You know, we could go watch the the Trovely maintainers dealing with this live in real time. They were notified because their users saw it, and the users helped in the investigation, quickly identifying where this stuff was introduced. Open source is under attack right now. It's, you know, it's it's attackers are looking at it. It'll come out stronger in the end. It's gonna be fine. We're gonna figure it all out. You know, you can't just go take it out of your supply chain. You have no other option. And if you did try to do something, it would end up worse. Yeah. I think that's a really good point that my my thought is imagine if we had to wait on Microsoft, whoever won big slow company to do the patch versus the benefit of open source is you can have tens of millions of people, as you said, watching to find the exploits, but also patching them more quickly. I I actually think it's, your your response time is gonna be much faster. And when I I often think of security as, like, any vault, like, whatever it is, like, a a gate physical security is always gonna be bypassed. The important thing is how quickly can you respond when there is an exploit. And I I actually think open source is generally faster to respond. Yeah. And then as you're looking at bringing us in, as you're looking at setting this all up, your point, every single gate is gonna be bypassed. You need enough gates, and you need enough tooling to notice that the first one has been bypassed before they get to the next one. That's the only way security works. So I think one more. So if you were using the Chainguard trivia image or if you were pulling our light our Python libraries, you weren't impacted. Obviously, people should do that. But, like, what what's a lesson that anyone here could take from, like, steps they could directly take today on their end to, like, similarly check upstream packages for Provenant? Like, what would you recommend people do? Yeah. The the checking and the provenance and all of that stuff is strange, because especially if you look at, Lite LLM, they had trusted publishing setup. They did the right things, but the way these tools are all rolled out is the attacker just turned it off and then published something without using trusted publishing. Yeah. These things are hard to really rely on until they're at a 100% adoption across the Internet. You know, I think, like, I look at TLS as a great example of this. Like, how many years and decades did it take before browsers were confident just saying, no. You can't go to that web page when the cert doesn't match. Right? And without events like this, really force it and can convince people to take action and the clients the open source clients and the package managers and the users to all be comfortable with making tough decisions that things are gonna break in the short term. As an industry, we're not gonna justify where all of that stuff just automatically works even though we have the tools and the technology and know how to do that. So people yeah. It's really it's understand your supply chain and be ready to rotate as quickly as you can if you have to. Cool. So I've seen a bunch of awesome questions. So let's shift. I think Ross is gonna jump back on and help us just field questions. Yeah. Or, Dan, anything that jumps out to you that's most interesting to start with? I'm reading through the So some of these were go ahead, Ross. I was gonna say the first, first chunk is really about getting a little bit deeper into what happened with Trivy. So curious, Dan, if you could speak a little bit about, like, GitHub, immutable actions, and if that would have been a potential solve. And with the Trivy, image specifically, was it an older version, or was it only newer builds? So kinda. probably a. question there. Jonas. Yeah. Was it an older one or only newer? Yeah. It was all of them. It or no. It was weird. It was, like, '76 out of '78 or something. Like, it yeah. Like, four out of five dentists recommend this toothpaste brand. So I yeah. I don't know if the script just failed or some automation flake or something, but two out of the 78 tags got lucky. Yeah. Then Lily had a bunch of great ones here about GitHub itself. Yeah. Like, GitHub only mentions Penny as an afterthought. Yeah. Dependabot doesn't even pin that well. Yeah. It's hard to, like, tie those tags back to the versions. It really there are usability issues with the shop pinning. GitHub immutable actions. Yeah. GitHub does have immutable releases now and actions work off of releases. But, like, from a client, if you're gonna go with that, you can't tell inside of your YAML file if it's an immutable one or not, which kind of hurts the usability in any real way of relying on it. Awesome. A similar one is, like like, PyPI has two factor auth now. NPM has two factor auth for publishing. But they it's somewhat controversial, I disagree with a lot of their decisions, but I understand why they made them. But they won't show you on the when you go browse which package is turned on two factor auth. So it's there. People can use it. You just can't, as a consumer, say, I'm only gonna use things that are published immutably, or I'm only gonna use things that have two factor auth turned on. And the argument for the other side is it makes the other ones bigger attack vectors, and, you know, attackers are just gonna go look for the ones without that logo and try to steal their passwords or something. So it's a toss-up. I get it. I disagree. But, yeah, this stuff is all hard for users at the end. There are tools, though, if you're publishing something to protect your users. Another another one from Lily, which, would be good to talk about is around CDs and zero days. So, there's a little bit of a confusion here around, if so let me just read it. So attacks have often had a delay of a few days even getting a CD that other security tooling can even reference. So is this fallout of these attacks being a zero day and not having time to disclose and the delay in getting the attackers out, or is it something else? Is it not really to do a zero day? It's not really a zero day. I it it is and it isn't. Like, there's a lot of debate over whether malware should be, you know, even be tracked in the CV system or if the CV system is only for, you know, accidental vulnerabilities. Is a program that, by design, steals passwords and sends them to the Internet a vulnerability, or is it, you know, operating. as intent? It's a bit definitional, and it doesn't really make a difference at the end of the day. But, yeah, you know, this is the case where the zero day policies also get kind of, controversial. Right? Google, you know, for years has had a hard stance around public disclosure, and they report things privately. They they have a set timeline on when they're gonna go public with it. Unless there's evidence of active exploits right now, in which case they think, you know, the principle of these terms is to tell everyone immediately. Even though it will give attackers something to do, it also gives everyone the ability to protect themselves. So there's trade offs at every level. Reporting this privately, no. Like, this was actively being exploited, and the best thing to do is tell the world. So everyone that's paying attention at least can take steps to protect themselves, even if it makes it harder for the defenders at the same time. We talked about this a little bit, but, given Tribune has been compromised and it's used by so many libraries and so many firms, use it to scan bones, what are the odds that there could be hundreds of libraries and firms impacted by this? And, obviously, the downstream from Trivia and Telenex and, CheckMarks kicks as well. So why are we not seeing anything in the mainstream media? Why is this not a much bigger deal than it currently feels like? are now finally. Yeah. I I posted something. Like, why isn't anyone talking about this yet? Like, Carpathi tweeted about it. Elon tweeted about it last week. Like, press is covering this now. I know it's a busy week with RSA, and I'll use agent agentic security products for the reporters to cover them first week. But, yeah, it is it is finally getting there. And, yeah, I think hundreds is, I I'd say hundreds to low thousands is my guess for people affected at the end of the day. Another question from Raj is, was there a tool or, like, what gave Trivia the initial clue if something is not quite right here and needing to kick off an investigation? Yeah. I'm not sure the exact, like, sequence of events as they first found out about this. There were people reporting it on their issue tracker right away when they noticed something was off. They probably had stuff flagged internally all around the same time. Like, those first few hours of an instant response are not fun for anyone, and the exact sequence of events is always a little bit fuzzy. Awesome. One question I have just as we are waiting for more to trickle in is as you're thinking about all of the entire open source ecosystem, we've seen it bop from actions, right, GitHub actions to libraries to containers even? And, like, how are you thinking about this game of whack a mole where if you secure one thing, you still have to secure the next thing if it's not if it's not protected? Yeah. You gotta do it all. Like, yeah, I hate when security industry and folks kinda look at it as, you need a 100% for it to be worth doing anything. You know? Like, oh, even if we fix this, there are still other gaps. Great. Still fix that, and then go fix the other ones. Don't do nothing. Yeah. There's one good one here from, Matthias. Renovate has a preset that pins to commit shot and adds a comment with the version at the end of the file. Yet that's what a lot of people do. That's actually how the trivia malware was hidden. You know, that comment means nothing, and somebody silently changed the SHA, without changing that comment. So if you rely on that, you're kinda stuck. I think Docker and OCI actually did a much better job here where it's it's little known, but in the spec, you can pull a Docker image by tag and digest. And the correct behavior for clients is to pull from the tag and then check with the digest match. And, you know, that actually lets you do both at the same time. So you can do, like, a Docker image at v 1.1.1 colon sha and or and mix and match them, and then it checks that that tag actually matches that digest at full time. Awesome. I'm not seeing that many other questions. Oh, one just came in from Nicholas. How does Chainguard protect from malicious commits and open source repos to end up in Vanguard hard hardened images or in our case libraries and also actions too. Yeah. This comes up a lot. We try to, we we try a lot here. You know? And scanning inside of the source code is hard. We try there's not a ton of training data because, we talked about before, there aren't many of these where the commits actually end up in the repos themselves. It has happened occasionally, usually right into the binaries. But we have an open source tool called malcontent that then works at that level. It works on the on the binaries that are built and looks at what behavior those are doing. And we it's super noisy. We'd have to tune and tweak it a lot to get rid of false positives and try to keep the signal to noise ratio up high. But because we build every single version of every piece of software that we have, we run it in diff mode. So if it's an SSH library in Python that ever since the history of time has been opening your SSH directory, it probably makes sense for that thing to be opening your SSH directory and reading your keys because that's the purpose of it, and it's done that every single release. Cool. We don't flag that. But if it's a, you know, string padding library in JavaScript that has never done anything other than string manipulation and all of a sudden it starts reading, your Etsy password directory, you can tell that from inside of the binary. You can tell that these syscalls exist. It's really hard to obfuscate that kind of thing. So we flag at that level. Awesome. Any final questions? If not, I might wrap this up here. Reid, any any others from you that popped up that we wanna cover? No. I don't think so. I think we covered them all. Really appreciate everyone who showed up. Thanks for all the time. If questions come up later, like, reach out. We're happy to answer and follow ups. And thanks everyone for spending some time with us today. K. for spending time with us. If you're interested in learning more about Chainguard, two ways to get protected. One, we are running a Chainguard libraries and actions and trivia image promotion right now. So on the libraries and action side, you get three months free. So you can lean in there and learn about, how you can gain access to our secure by default libraries. And on the trivia image side, you can get a drop in replacement for the trivia trivia image you're using now. Fifth version also available. Leave these links up as folks start to resume the rest of your day. And if you have any questions, by all means, you can always reach out on Chainguard dot dev. See you all. Thanks all. Have a great day. Take care.