Name: Go Beyond the Distro for Better Security and Performance
Uploaded: 2025-04-09T18:16:03.828Z
Duration: 52 min 28 s
Description: Go Beyond the Distro for Better Security and Performance

Transcript for "Go Beyond the Distro for Better Security and Performance": This is oh, man. Alright. Hello. Hello. Hello, everyone. Welcome. We just had a nice little dry run for the first couple of minutes. Getting used to a whole new, podcasting, video podcasting platform. I think it's first time for all of us here. So, I'll introduce myself, Dustin Kirkland, VP of engineering, lead the engineering team at Chainguard. We've invited two very interesting distinguished guests here to talk about, the past, present, future, history and and, maybe what's to come around the Linux distro. Eric Chang, founder of Oblique, John O'Bacon, good friend, former colleague at Canonical, and founder of StateShift. John O, you wanna tell us a little bit about about yourself and, what's what you do with StateShift? Yeah. So thank you, Justin, for for having us here today. Good to hang out with you as well, Eric. So, you know, I've been really passionate about building user and developer ecosystems for my whole career. So I run a company called StateShift. We work with a ton of different companies from big companies like Comcast and IBM, Dynatrace through to lots of early stage companies, and we basically help them to build movements around their around their products and services. So we, you know, we take all the things we've learned from working with a couple hundred different companies and distill it down into stuff that people can apply and get results faster. So And then I'm Eric. I am a cofounder of a company called Oblique. We're an identity directory sort of passionate about fixing all of the terribleness in terms of corporate access. So I'm more on the security side. And before, Oblique was working at Google for about six or seven years in the security space, working on Google's internal, Linux distribution as well. And then before that was working at CoreOS, a very a startup that I'm very fond of, that built a immutable container optimized Linux distribution. Awesome. Well, let's rewind way back. What was your first Linux distribution and what were you trying to build or trying to do with it? Yeah. So I, you know, I didn't really get to Linux before I was in college. Right? I was doing a CS degree and was like, how do I get Windows off of this machine and have some fun? So, you know, I'm dating myself a little bit there, but, like, in one direction at least, but was mostly just installing Ubuntu on devices and having fun with that. And then later on, you know, got introduced to things like Fedora and so on and so forth. But, yeah, I guess mine is a little bit mundane in that capacity. I wasn't, like, you know, booting, flashing things onto, my PC as a kid or something. Jono? Yeah. For me, I, my brother came to stay with us for a couple of weeks, when I was living at home with my parents back in 1998. I was moaning about Windows, and he said you should use Linux. It's it's amazing. It's Unix for your PC. I didn't really know what Unix was at the time, if I'm being honest. So he installed Slackware 96 for me on my computer and literally installed it, replaced Windows, and then, left a post it note on the screen that had their username and password, and it just said dark star login. I had no idea what the hell was going on, and he left. So I worked at a bookshop, got a book about about Linux and how it worked, and, and was hooked. And the thing that really blew my mind was how people all over the world kinda come together to to build this thing online, which of course back in the late nineties was was particularly cool. So, and, you know, came for the software, stayed for the people. So That dark star, login, was that a Grateful Dead reference? I don't I think that was the default. I think that was just the default host name with Slackware, if I remember right. Unless I actually never really thought about it. Maybe that was him. I doubt it. I don't think he's a Grateful Dead fan. So Oh, well, in the interest of completeness, I'll I'll share mine as well. It was 1997. I was tasked with building, basically bringing in an ecommerce site to life, for a job I had in in college. I bought a book, PHP three from the local Barnes and Noble. Tucked in the back cover was a Red Hat five dot one CD, and I installed it, immediately and found it familiar enough. The the moment for me was the fact that, was when I realized that I could download the source code for the binaries running on that machine that I had just installed, make a change, and replace it, and that was just mind blowing. I had been around Windows before that, and it was just so hard to do anything with Windows. It was never quite right, and it was broken, and it was buggy. And the idea that you could just fix it yourself was like, I It's just we install your bootloader. Yeah. Totally. Go for it. If you want to, you know, scratch your itch. John, you spent a lot of time around communities, building communities, helping people build communities. Communities often go through updates and upgrades of their distro over over time. I'm curious, about some of the, you know, the issues that you've seen and maybe opportunities to improve, that, in your experience. Yeah. It's it's been kind of interesting watching this whole story unfold because I remember when I got started with, you know, with Slackware and and Red Hat back then, you know, it was, like, packaging for the, quote, unquote, end user was was complex, and upgrades were a pain, and they often broke. And I remember when I first used Debian, what blew my mind was, like, kind of that dependency tree, and dependency assurance. And it's been cool kinda seeing how things have evolved over the years. You know, I think, traditionally, it was always, like, lack of predictability. Often there'd be fragmentation of packages. There'd be poor communication around changes to packages. I mean, yeah, you could go find it, but you'd have to dig pretty deep. You know, upgrades, like, doing distro upgrades were an absolute nightmare. And there was the kind of the balance between kind of security and stability. And and I think for for a long time, it was it was difficult. And I think there was always this kind of, like, tension between, you know, yeah, you can get a package in Debian, but it's much more involved for a maintainer to do that. Like, becoming a Debian maintainer was, you know, seal team six for nerds. Right? So I I think it's been fascinating kind of watching the role of the Linux distro especially as we've kind of moved into containerization in the cloud and how distros have had to kind of adapt and adjust. So Yeah. What do you think, Eric? Just updates, upgrades over time, where are the pain points? It's funny that you hold up like Deviant as that because I was really, the security liaison for G Linux, which is, Google's rolling release based off of Debian testing. And what's really interesting there is just how unreliable we found the, like, the Debian dependencies to be. Right? Like, Debian is, like, I depend on this package at this maybe version if you're lucky, but it's, like, Debian is very much a point in time release. And Yeah. And that's really interesting because it's, like, if you try to do a security patch, you're screwed because, you know, you get a make file blah, you know, that, like, explodes because some, you know, symbol changed over here. So I I think also particularly with Debian, there's this really interesting post by, Michael Stapleberg who's at Google but wasn't working on g Linux, talking about how he's winding down his Debian development. And a lot of it was the fact that it's like, it's there's no really consistencies throughout the ways that Debian thinks are packages. A lot of that's, like, tribal knowledge more than that is. Yeah. Like, Debian comes in and says something, and he was finding it frustrating to like my my favorite story about this is like bin versus user bin. Right? Like, Devian was trying to switch over to everything under user bin but individual packages would just be like, I don't feel like it. I'm gonna keep shipping in bin. And what you see is that, like, there's just this mess on on that side of, like, there's not a whole lot of consistency even though people downstream absolutely do kinda want that. So I I actually kind of have opposite experience away. It's like, it's actually quite challenging on on that side. I wonder, Eric, whether my experience there was basically a taller than Danny DeVito competition where things were so bad for you. Sorry. That's why the other thing you said about point in time was, you know, everybody will be like, oh, well, you if you want something stable, you should use Debian stable. But Debian stable was just prehistorically old. There was no middle ground, right, really between testing and stable in a realistic fashion. So Yes. On that note, I I think that's kind of worth an interesting little debate, right? The rolling distro approach you mentioned you mentioned Debian testing versus the Debian stable approach. By Debian stable that could easily be an Ubuntu LTS or a RHEL or SLES or any other major release. We could go back thirty, forty, fifty years in history. The history of UNIX has long been pick a moment in time, snapshot the entire world, harden that to a point where you can release it, and then give it a blessing and a version number, and and then don't change it unless you have to. And when you do change it, minor, you know, small, inconsequent or a small non breaking changes as possible, and then keep that running for five, ten, twelve, fifteen years. There's a lot to unpack there, but, maybe Eric, the the given your G Linux background and and CoreOS approach, maybe take the over or under on rolling distros, and then I'm curious how how, Jono feels about, you know, the the flip side, the the, you know, the hard and stable long term release distros. Yeah. I think I mean, there's, like, the technological side of, like, rolling distros. Right? Which is, you know, I have my complaints about certain package managers, but, like, I think NextOS is a good example of this where it's like NextOS has perfect information about the exact shawl that have all the dependencies and stuff like that. And, like, their monorepo packages just goes through the suffering of, like, you try to insert a security fix at the beginning of the release, everything breaks, and there's this long period of, like, trying to fix it. And you see this in any large software, space, like, the monorepo for Google has this where, you know, getting things rolled out has to go through all its qualifications. So I think that in a horrible way, it's kind of inherently fraught. Right? Like, you just cannot get this for free. And, you know, since I'm CoreOS, CoreOS didn't have a package manager, and that was wonderful in some ways and awful in others. Right? Like, it was amazing because CoreOS could be sort of this independent thing that upgraded, you know, irrespective of what your apps depended on with some maybe things around like kernel compatibility and whatever. But, yeah, I think it's just inherently really challenging problem. Right? There's no free free lunch here. Yep. Yeah. What do you think? I agree I agree with you, Eric. And, you know, taking a step back from just the the engineering, to me, it's I think a lot of this just depends on kind of what you're here to do and what you want to do. Right? Like, I I I think what's been fascinating to me in looking at kind of the role of Linux in the world is I would argue that in the last ten years especially, the pace of change with software has gotten faster and faster and faster. There's more people building more things. Engineering is more readily available and accessible to more people. There's more I feel like there's more investment in going into into into engineering. So I think when I, you know, when we kind of become crusty old men and compare, like, the old days of of Linux, it was just such a radically different environment, with a radically different set of needs. So, like, deploying a stable operating system kinda made sense. But Right. Then and I think if you're doing something that's relatively monolithic, then it's it's fine to do that today. But to me, what always excited me about the concept of a role in Distro was, like, the dream is real. Right? Is that I I have this one thing that's deployed, that's always kind of up to date with what I need that's serving the use case that I've got. And, you know, we were experimenting with this. What what was it? Was it gutsy, Dustin, back at Canonical? Like this kind of shining tunnel on the hill. Seven dot 10 as I recall. Yeah. It was it was we we always had this concept of a rolling release. I forget what what the what the code name was that it never really fully manifested. So, you know, to me, they're like, it's it's a it's a, you know, it's it's a great vision, but, you know, there are way smarter people than me, you two included, to determine, like, is that actually fully viable, and is it serving, like, the modern needs of of of engineering teams today? Yeah. I think, I think the timeliness, you you mentioned, like, what's special about this moment in time in history, UNIX has been around since 1969. You know? It's it's been around forever. Linux since, you know, '91 with a kernel and call it mid nineties, '90 '4, '90 '5 with the first distros popping up, Red Hat and Debian, Slackware, and others. So why are we still talking about this? And, you know, I think it I think it is worth calling out the fact that, the pace of innovation has dramatically increased, and the state of technology, both the, you know, the scale of the clouds and super clouds, the state of virtualization, the state of, various, computer architectures. We can't, you know, not have a podcast without mentioning GPUs and AI and all of the, like, just drastic, requirements around that technology on latest and greatest, which you might not get in a traditional distro for another two, three, four, five years until the next, you know, major release gets cut. And then you'll get a a refreshed version of those Python libraries and Java libraries and all the tools. But by that point, it's it's like you you were talking about. It's just ancient. It's antiquated. It's it's history. And so this is where I think the reinterest in this old concept of the the rolling distro is something that that is available on a on a testing basis from almost every other distro out there. But the idea of actually taking that and hardening that and baking that into something that is designed and intended to be run securely in production at scale, that's, I think that's what we're really here to talk about. Eric, you've got some experience with this, I think. Well, I guess I was going to have a slightly more cynical take on this, which was that we had a vision for this. Dustin, you and I kinda met in the Kubernetes space a little bit a long time ago, and we kinda had a vision for how this worked. And it just turns out Kubernetes is, like, way, way, way too complex for, like, a lot of companies to adopt. But I have friends out of the Core West whose entire job is that they show up to companies who have adopted Kubernetes, gotten, oh my God, and now you have to hire a very expensive person to figure out how to like get this under control. And so I think that, you know, mature production environments are like squeaky clean, right? Like they have figured this out, they have figured out how to use containers everywhere, how to divorce, you know, the operating system from this. You know, you might even be mature enough that you're not just shoving all of Debian into a container and then running it there, you're like being particular about the things that you consume. But the reality is that that's just like not There's a vast majority of the market that like has not actually figured out how to do this kind of adoption either because, you know, it's really hard to plumb, you know, system resources like GPUs into containers or you just have this mental model and everything around, you know, billing and that kind of stuff works in a VM space, but it doesn't work in containers. So I I think that, you know, Kubernetes kinda has something to answer here, which is, like, it is still fundamentally extremely complex. Right? Like, nobody in their right mind is doing anything other than paying, you know, Amazon or Google for a Kubernetes cluster these days. And I think that that just introduces a bar of complexity where it's like, yeah, we have this vision, but is it actually practical for someone, particularly if you're like a legacy company with huge amounts of infrastructure to adopt in a way that gives you this perfectly magical, you know, your operating system can update without impacting your workloads. Yeah. It also seems like the the the big question in my mind is, is the modality of a rolling distro a model in which human beings want to consume technology? I'd like to think that it is because it's a very simple way of deploying technology in a way that should solve problems. I mean, obviously, like, when we figure out the modality of doing anything, the interface, then we can figure out what is the engineering that's required to be able to deliver that effectively. But there is an open question in my mind around, you know, what is the right way to consume this? Because I agree with you, Eric. Like, for all of the great benefits of of of of cloud native and modern technology, it's a freaking nightmare to deploy for most people in the much the same way that one of the things we hear from developers all the time with our customers is writing software is way less fun than it used to be because it used to be just about writing code. But now it's about tests and CICD and all this, like, legwork that you have to do wrapped around, like, just writing code and having people use your code and the joy of seeing people use your code. So that that would be I'd be curious, what you think on this, Dustin. Is that the model that people want to consume software for modern technology? Yeah. I think the right answer is it fades your way into the background and it's not something you even think about. You have to think about whether you've actively chosen you know, a a traditional distro or a rolling distro. Like, the the last time I really thought about how the transistors in RF in my phone actually works is realms to basically never. You you know, I think if we if we can get to that point where you just need to write some software and in order to do that, you start with a, a set of software tools, IDE, a secure software supply chain, deploy that, maybe you wrap that up and run it in a container environment. Eric, the the Kubernetes point, very salient. Right? The the whole idea of a container at least creates that boundary where you can bundle in all the things that you need that are separate from the rest of the system and can move at a different pace and can physically move around from one machine to another, one VM to another. That that those layers, that's, you know, part of this, this moment in time where we're at where we can actually start thinking about it a a different way. But, yeah, Jono, the right answer, I would hope is that you can just forget about that, and that's a solved that's a solved problem. That's what I would think. This reminds me of I remember when when we were building Ubuntu and, you know, we built this new user interface, Unity, for Ubuntu, which was fairly controversial in in pockets of the Linux world. And I remember one of our critics one time talking to him at some event somewhere, and he said, you know, you you're dumbing down the Linux desktop. And he said, if you had your way, there'd just be one button on the screen and you'd press it and do exactly what you want. And I was like, yes. That's exactly what we want. Like, the whole point of this is to make the tech blend into the background. So, I mean, again, you two are the experts when it comes to the actual deployment and engineering here, but, like, to me, this would make eminent eminent sense to to be the right model. I think also talking from the security side, you know, like, what the what the distro wants or sorry about this. Like, fundamentally, Linux is really messy because outside of very specific, like, containerized spaces, there's no really clear isolation between workloads and systems. Right? If you install a package, it just gets installed. It gets installed in the same place you do whatever. You know, if you if that requires a new version of Python, you get a new version of Python for everything. And what the problem that I see in that fundamentally is that, like, the team that is incentivized to keep the Distro rolling, namely, like, the security team and the SREs and whoever owns that platform, is totally distinct from that. But many or sorry. Distinct from, like, the application developers who just want their app to work. And I don't think Linux in general has hit this maturity level where none of the distros, like, have a good sense of, like, I wanna install this but just for an application. Again, like, I think NextOS is basically my only good example, but that comes with, like, its its own set of problems where now you have 20 versions of a particular shared library. So I I think, like, a a lot of things and and coming from the security space working with other operating systems like macOS or Android or so on and so forth, like, there are just basic things in terms of the separation between user privileges and system privileges that don't really have clear parallels in Linux. The security perspective is just like a rolling disk server would be great, but how do you do that without totally nuking somebody's app who wants to stay stable? Yeah. I'm glad you brought up both Android as a near cousin of Linux distro, macOS as maybe a different type of cousin of a traditional Linux, Unix distro. In both places, I do think there there's a lot of learnings to be had in terms of, you know, how applications are developed for and packaged. I'm curious, taking us back to containers. Do you think the that sort of container boundary helps, Eric, or do you think it has to be done at a package level outside of the containers, or is there some, some some blend of the two, to solve the the security problems there? I mean, this this might be because we were in the more of the Red Hat space, but, like, everyone was trying to build these seven years ago or eight years ago. Right? Like, Flatpak and Silverblue came out of this where it was, like, applications and desktop applications that ship with namespaces by default such that like, my my favorite part about, like, Flatpak is a way of installing desktop applications. So you can install something like, you know, Spotify on your distribution and it is by default containerized. Even if you, like, go and open a file in there, it goes into a system prompt that is unique from that. But, like, those never really took off. And I also think there's this fundamental question of, like, my background, you know, at least at Google is on, like, a a desktop distribution, which is fundamentally super, super different than, you know, a server space. So I think definitely, yeah, I think we see even more containerization like options being put out there, right? Like, you know, Linux has no end of tools that kinda allow you to produce sandbox isolated applications, right? Like you can list all of the LSMs. Right? Like AppArmor, SE Linux, LabLock, all the new EBC app based ones. Right? I think the main thing is that, there's very few opinionated distros that actually say this is what an application is. So In server space, that's containers. I will keep ranting about desktops as long as you let me, Dustin, but I'd love for you to continue to talk also about the I'm curious from both of you. What do you think are the non negotiable principles, be a desktop or a server? If we're just talking about a Linux distro, and you can take it to whatever use case you want, what are the non negotiables? What are the principles that, you know, have to be there? And then where are the places that, you know, you think there's opportunity for differentiation and, perhaps, you know, hyper optimization for, for for particular users? Jono, you got thoughts on this one? Yeah. I mean, when it comes to the principles, I I guess I'd probably break that down into, honestly, like, principles that would apply to any and all software. You know? Things like I think the software should be secure. It should be open. It should be maintained. These things, I think, would apply to any certainly, o open source software, but any software in general. Like, we want a regular stream of security updates that are patching things as quickly as possible. And I think, you know, also just the key principle, and this is, again, kinda go back to one thing that Eric said earlier on. I think this is an area where Debian honestly failed was was being able to deliver the best of the current landscape of technology as quickly as possible in a way in a way that's secure and effective. You know, you got either secure or you got up to date with basically the two different options. I'm gonna have a bunch of Debian people screaming at me after this. But to me, those are the fundamental principle. To me, that's what's always been so magical about about Linux in general is is, you know, it kinda, again, goes back to what we what we talked about earlier, Dustin, when you said, you know, being able to to to to use a binary, get the source code, modify the source code, and build it. I think the reason why that was so mesmerizing for so many of us was this is a completely different ballgame. Like, we're in Thunderdome now compared to, you know, what it used to be. So to me, those are the principles that always have to be maintained. And I think there's always been a healthy skepticism from some people that Linux basically gets translated and turned into Windows, you know, for the for the for the purpose of the of of convenience. Like, oh, to make this easy and accessible for everybody, we compromise on those fundamentals. And I don't I don't feel like we need to necessarily compromise on them. I'll jump in there and say, I I think that's right. I've heard that more times than I can count. But I think just, you know, Mac, and macOS, I think, have shown us that we didn't need to compromise. You know, the success of the Mac desktop is not as, you know, quote unquote easy as the the the Windows desktop, but guess what? People are smart. They figured it out, and a whole portion of the world, the desktop world, has come around to that. The only thing I would say there, though, Dustin, is I think part of the reason why that happened is because Apple could be, opinionated. And I think in the Linux desktop space, much as I love it, is you are punished for being opinionated and and innovating. We we tried this with Unity in Ubuntu, and people got very reasonable complaints to have about Unity. But we tried to be opinionated and focused and and to differentiate. Things like, you know, installing applications, like just dragging the app over on Mac, it was a game changer, of course, back in those days. And I I think that the collaborative model of open source sometimes doesn't isn't set up for, like, big bold opinionated moves. That's something in that capacity. Like, on the security side, Mac Mac and Apple will break you super like, they'll be like, hi. We're deprecating this API because, you know, we have things that, you know, we feel that has insecure usage or something is abusing this. Right? And Good luck. Yeah. Good luck. Hi. You're broken one day, and it's your day to scramble. Right? And I think that, like, that's kind of unfathomable. I I've done a lot of this with, like, the Kubernetes, our back system rolling that out of, like, it is there's an expectation, I think, in rightfully in in spaces that your software doesn't change and your software, you know, your we don't break you. But security is fundamentally about breaking people, which is, like, kinda hard and terrible to say, but it is fundamentally about saying, like, okay. We don't we wanna stop this. How do we break this in, hopefully, the least disruptive way, but, ultimately, how do we break it? And I I don't think that there's a huge appetite for things that are not truly crazy, like, you know, there's a backdoor in your SSH kind of stuff. Yeah. On that note, let's, pivot to just talking security, for a little bit. Where, where do you see some of the challenges and opportunities around securing? And here, I think I'd like to take it to, like, applications. Applications that are being developed first party, maybe for internal use or applications that are being developed for, further redistribution, proprietary, open source, whichever way you want to take it. Building on top of a secure OS, I think is part of why we're here talking today. But what are some of those considerations just around security and and what do we need from an operating system, from that perspective? I'm still kinda shocked that no Linux operating system has the ability to say, like, only allow software to run that I installed through a centralized repo. Right? Like, if I have, you know, on Windows or Mac OS, there are pretty clear policies that you can have saying, like, you have you ever run something on Mac you know, on your MacBook? And it's like, are you sure you wanna, you know, run this because it's from the Internet? Like, you know, that's not a great experience, but I think it speaks to, like, where modern operating systems are going in terms of security. And there's no really good parallel on the Linux side. Right? Like, you absolutely can validate packages or sign by the package repo. But as soon as stuff runs on your executes, it's kinda all bets are off. This is typically complicated with interpreters. Right? Like, if you curl pipe batch something, there's basically no Linux distribution that will stop you from doing that. And if I'm a centralized security team, that becomes really, really challenging on my side of, like, I want a policy that says only install things through this PyPy repo or only install something from this, you know, the you know, only install devs that are signed by my centralized security. And, yeah, I'm I'm actually kind of surprised. I've tried this at at Google and maybe failed, so I'm not so surprised, but I'm, like, surprised that those type of conversations don't really exist in the space. Yeah. This is actually a real problem, when we were building Ubuntu because a lot of people would be posting, binaries onto the into the community forum and people running these things and doing some pretty significant levels of damage when they were running them as well. Yep. We saw that all the time. Yeah. Yeah. I'm still I figured it was like yeah. A lot of things, like, if you double click a dev, it'll just run, which is like as a security person is terrifying. I understand that that's the wonderful world of Linux makes us think that that's a free thing. But yeah, as a security person, you're always like, Oh, God, how do I turn that off? Yeah. It's buried in the details, but I mean, that's one of the big fundamental problems with any Dev or RPM. When they get installed, there are scripts that run before and after that, that typically execute as root, and there's no way to force or force drop those privileges. And if you're doing that on your desktop or doing that on a server, it's momentarily you're trusting whoever wrote whoever packaged that software, not just who wrote the software, but whoever packaged it. You're temporarily trusting them with full root access to your system for the length of time that that that's getting, installed. Very much the problem that Flatpaks and Snaps and and so forth were are designed to solve, but for lots of reasons, it's it's still a a a pretty thorny, pretty hairy problem there. I have a question for you guys. I think it would it would be safe to say that flat packs and snaps have not really taken off. What's your thinking on why? I mean, I have my own opinion, but I'll be curious for what you guys think about that. I I think that the things that a so every every service you buy in existence, right, everything you you sorry. If you're like a company and you buy something, the management things that the security team uses, those knobs, are not charged. Right? It's like, okay. You're gonna go use GitHub for all your things, and then you, like, pay them to have an organization with all these settings that security goes in and actually tunes the thing. And I think that there's just no demand for it. Right? Like, in what world is the security team trying to secure a Linux distribution? And if they are, they're probably just, you know, kind of using Puppet for everything and doing their best. But I think that that's the core thing is that the yeah. That and and also I think Flapack also is I forget where the registry was running. It's been a while, but, yeah, I think it was pretty limited in terms of like what kind of applications you could install as well. Do people use Linux for their desktop apps or do they use it for this cool command line tool that you released on GitHub? Yeah. I'll give you a hot take on that one. As a developer and maintainer who's authored a bunch of open source packages and then created packages out of the code that I've written in DEV format, RPM format, APK format, Flatpak, and Snap. The thing about the latter two that drove me nuts as the upstream author and packager of of Bangcode is that I do it and I'd create the baroque I'd have to go and learn the packaging system and I'd create the baroque rules for that packaging system and I think, sweet, I'm done. And now I'll go back to just writing and releasing my code. And then guess what? The packaging system changes in some way, and now it's my responsibility as the, like, volunteer upstream maintainer of this to now go and educate myself on the way that that super complicated packaging system has changed again. And after the second or third time of that, I'm like, forget about it. Someone else can pack if she cares, someone else can go and package it. For that reason, I have a personal preference for the simplest, dumbest packaging systems as as possible. And I'm on the I'm on the APK, route at this point, which is what, you know, Chainguard uses. It's just a tarball. It's literally just laying down new copies of binaries, signed copies on a file system, has taken away a bunch of that pre enced post enced logic, that stuff that has to run as root. That's left as a separate exercise, which will typically handle at the container packaging level, which is a different we we basically separated that to to two different layers. For better or worse, I mean, I think we can debate the merits of each of those. What it's allowed us to do is create much more appliance like experiences with configuration passed in by end users, but the thing that gets installed is much more like, I would compare it much more like a a Chrome OS, style of it's just an appliance. And when you need to when you need to upgrade, you don't upgrade in place, you replace that with a new thing. Yeah. It's interesting you said that, Justin. That was kinda my take in it as well was, I think one of the problems that we often face in the tech world is our tech community is so filled with such smart technical people that sometimes things just get overly complicated because if if we think it, we can build it. And I think that it always I'm sure there's some law around you know, Moore's law, there's one of those laws around, you know, simplicity is probably the best option, for most people, but that can sometimes fly in the face of, like, sophistication. And that's always been kind of my take on on flat packs and and snaps as well. I think also it you know, there's a reason that NPM is used. That's a reason that PyPI is used and these because they're dramatically simpler than basically anything that the Linux systems are gonna provide and it's will work on any list the standard distribution that you have. And so I think that whether or not my view of like what the perfect is, it's like this is a reality of the world that all of the language specific package managers are far, far simpler and easier to set up and will work on any Linux distribution versus having to do something specific for every single one. That's a fantastic point. Again, back to my package maintainer days, the idea of, the difficulty in trying to jam, a bunch of JAR files into a dev or grab NPM or or or Python wheels or Python modules and try to insert that into a dev or an RPM. It was actually it was pretty fruitless labor in that there's already a package manager called PIP, called NPM, called Maven, called Jim, you know, that that is exactly what the upstream developer expects and wants to use to install their software, and then trying to put that into another packaging system with its own dependency solver, was, I I think, a pretty fruitless effort. The approach that we've taken at ChainGuard is actually to embrace those and to start actually rebuilding the artifacts. And so we've just started rebuilding, JARs from source and making those JARs perfectly compatible with Maven, which is the way people expect to to to Maven install, some Java libraries and dependencies. And now we're working on the same thing for Python and PIP and, NPM as well. Should we take a couple of questions from, the q and a? I see a handful of them have, have popped up. Yeah. Let's do it. I also see some names I recognize in there, like Justin Garrison. So I'm you know, if you have any spicy hot questions here, please do ask. Okay. I'll, I'll grab a couple off of, off the q and a, I guess, starting with the first in, first out. What's the most secure OS, according to your experience? Anyone's got a hot take on that one? IOS, and it's not even close. Right? Like, in terms of, like, things you would actually use, I think iOS has been, like, far and wide understood to be, like, if you're trying to get hacked by a nation state, that's what you go to. In terms of Linux, I'll say CoreOS just for just for, you know, my my Linux is for the thing. Yeah. Yeah. Let's say the same. Yeah. I think I think that one, I'd have to dive deep into what are you optimizing for and who are you optimizing who are you securing it against. And I think that the more and more specific you can get to what you're building, then I think you could get into some pretty interesting, you know, tailored, hardened OSes that are great for a particular purpose. The generalized sense, I think that would be a bit of a religious war at this point. Let's see. There was a question around Apple's willingness to break backwards compatibility is such a is that such a terrible thing to push things forward faster, and more elegantly? So, Eric, I think you brought that one up that Apple will just straight up break you if they if they need to or they want to or they have to. Yeah. I think that I mean, I I think there are some prompts in our sort of notes about, like, what open source security is. And having done a lot of this, you know, I think what people in open source think security is is building frameworks that people can use for security. What doing security at a very large company where you're trying not to get hacked is is rolling those controls out, right, and making sure that, hi, you know, you can't, you know, like, we're gonna break you or here's the way that we're gonna turn this off and and so on and so forth. You know, like, the fact that open source can't even agree on a single packaging mechanism, I think is, like, way, way beyond the pale in terms of, like, we're gonna turn these things off or that kind of stuff. And I think that, it's just very fundamentally hard because it's like, that's not what these open source Linux distros do, and arguably, that's not their job. Right? You're a volunteer. You're not being paid by a large corporation to, like, ramp up the security at the expense of potentially users. Yeah. I I would say in that one, To the point that Eric was just making, like, I think it it comes down to, like, what are you actually trying to do here? You know? Like, at the end of the day, you know, technology is technology. And sometimes, like, if you wanna keep moving forward, you have to break from the from the past in some capacity. So to me, that's fine so long as, one, it's well communicated. It it it it's it happens when only when it's absolutely necessary, that there's plenty of, like, heads up, you know, warnings and notice and all the rest of it. Like, otherwise, what we get into is I don't know what percentage of the Internet is still running Windows XP. You know? It's like I think we we get into these, like, antiquated systems that people will just keep running and running and running, and the and and I think that then poses further security issues. Like, if you're running ancient software because you never wanna break backwards compatibility, then you create a less secure world. So, you know, I and I I I hope to try Apple will make the right choices around that. I don't know. I defer to Eric and you, Dustin, about whether they do make the right choices or not. You know? Right. I just love that point. Yeah. Absolutely. Yeah. Yeah. I think we've come a long way in, understanding that that it's not acceptable to, you know, expect twenty, thirty years of uptime of a given system. Not to say that it doesn't exist in certain places, but I think it's well understood that that's just, you know, purely not acceptable, anywhere except for maybe the Voyager two probe, still flying through space. There comes a point where, yeah, we're we you you can't stay on that old OS. We've gotta upgrade that thing. Right? Let's, let's dig into this one about DevOps. What's the impact of a rolling minimized, distro on security in in the DevOps team? That that's a place where, you know, that that many of us depend on our DevOps and SRE, teams to keep us up every single day. Asking them or requiring them to support this. What are the advantages and disadvantages there? I mean, it's like table stakes. Right? Can you update your operating system? Can you apply patches? And the way you do that can be mirrored ad. Right? You can run Kubernetes. You can do whatever. You can have dependent bot showing up and saying your container is out of date. But, like, I think that there's not really, like, a, yeah, what is the impact? It's like, this is gonna happen regardless, you know, both if you're in a mature environment. So I think, like, minimal distributions are super, positive impact on that, but, you know, if you have to support something that has very specific kernel, requirements or on host package, you know, dependencies, this is why you pay Red Hat a bunch of money. Right? Is to have them resolve this. Yeah. It seems to me like where a lot of the pain in DevOps is, it's a lack of predictability and a lack of control in many cases. And I think this is kinda butted heads with the traditional distro model where a lot of DevOps teams would have to be pretty reactive and go into firefighting mode to do that. And that's why I think, you know, obviously, we've seen the state of DevOps shift and change over the years, you know, containerization, minimal base images, the mutable infrastructure, declarative configuration. Like, these are all kind of things I think have been, have been put in place to kinda deal with that to a reasonable degree. But I I I sometimes wonder how much of the how much of that is a Band Aid and how much of that is an actual solution. And to me, what's fascinating about kind of the rolling model is what are the roles of those pieces within the rolling distro model? Because, again, go kinda going back to the concept that that is a great way to consume software, it seems like from our conversation and on the chat as well. The idea of, like, it's an appliance. It's a thing that I just deploy and that this the technology kinda goes away. Again, to me, it's like a great shining tower on the hill, but then the day to day of how of how you like, what does your manifest look like? And and what, like, how are you like, what does your configuration structure look like? I'm not entirely sure how those pieces fit together. You know? Yeah. The, first of all, I think the minimization part, it's a great place to start. I think everyone's definition of what does minimal mean varies widely, very widely. It was some years ago when I was working on Ubuntu and had put a whole bunch of effort into minimizing the Ubuntu minimal image. This was largely coming from downstream users of Ubuntu inside of Docker images. And at one point, we had gotten the Ubuntu minimal Docker image down to 20 or 30 megabytes compressed, and that still wasn't good enough. There were still people complaining about that. People complaining about that running it inside of user bin Docker, which compressed was over a hundred megabytes. So it was a hundred megabytes to run user bin Docker, and we could fit a whole OS distro into 20 or 30. It just it, I think, demonstrated the point that minimal is different to to everyone. All that said, yes. Get rid of anything that you can you can get rid of, and then, you know, you you will have a much more, secure base starting point. You'll eliminate some some attack surface area. You'll eliminate CVEs you'll never see because of that. But bear in mind that that's gonna differ widely based on, you know, who's who's who's calling, who's calling, what goes in, and what what's not in. The piece about the rolling distro and DevOps and SRE teams, I'll tell you with quite a bit of direct experience with the types of customers customers that are coming to Chainguard, many of those have, deliberately opted into a world where they will deal with micro, micro breakages on a semi regular basis or a more regular basis, rather than the the major breaking change that happens once every year or two. So, you know, you could take us, an LTS or long term stable Red Hat Distro or something like that and say, yeah. We love it because it never breaks, and that's true, until you get to a major upgrade, in which case absolutely everything breaks. And I've worked at in financial services and banks where, you know, you go through that upgrade every couple of years, and you basically plan nine months of pause the road map while we roll out a new a new OS. So we've kinda taken approach of, like, you know, I'd rather wake wake up every day and stub my toe than ever break my leg and end up in, like, nine months of rehab. No. What we do at Chang Guard is try to invest in the the CICD and the the quality assurance of that, so we'll minimize those toe stubs, as much as possible and never get us into a situation where, you know, you're dealing with a a a broken leg. There's another question in in here around, air gapped environments and whether that's essential for secure builds or not. Eric, maybe maybe this is one that that you've got some expertise on from, from your world building secure distros. Oh, man. I mean, so I mean, there's actually it's kind of similar to the next question around, like, don't forget the most secure system is the one that isn't used at all. Like, what's the trade off? I don't know. John put it kinda really well, which is like it depends. Right? I I think that this is why it is so difficult for open source, projects to have strong security opinions because there's a vast difference in the maturity level and the security posture of organizations that need to take it up. So if if you work for the NSA, right, you're gonna show up and you're gonna go sit in a windowless room where you check-in your phone and have very specific expectations about, you know, everything that happens. And in that world, yeah, sure. Maybe, you know, you have some air gapped environment or, you know, if you're running a hot wallet and a cold wallet, right, like, as, you know, certain as certain, cryptocurrency, companies figured out, like, those those are the kind of spaces where you definitely need air gaps. But it's just really hard to say for any reasonable company or, like, individual company because I I think most people just want sort of what is a reasonable setup, which is, like, someone didn't build this in their laptop and upload it to a repo. Right? PyPy has done some cool work around authenticating GitHub actions so they ensure that a GitHub action is the thing that pushed the code. For me, you know, these kind of spaces are, like, totally sufficient for the vast majority of users and a vast majority of threat models. I I don't think you need very few people would ever need to go to the whole entirely air gapped. But, yeah. I mean, compared to what is also just the real question. Like, have you seen the computer controlled software these days? Yeah. I've always thought of security as like an analog slider where you can move it to one direction far to the right where it's hyper secure. It's so secure that it's unplugged from the wall and while you're at it, go ahead and turn the computer off if you really want to secure it. Yeah. Then the slider all the way into the other direction is wide open. Anyone can do anything and it's super, it's really usable. And then but the, you know, the the the art is finding the the that slider and placing it in that analog perfect place for a given use case or workload. Getting that right and getting that right over and over is, you know, very much, I think, our jobs as as software developers with security in mind. To me, this also kinda shines an interesting light in in the structure of companies and how they do work because, you know, what I've seen, I'm sure you guys have seen this as well, is that the concept of security in in in software engineering, I think, is it is I mean, there's many different definitions of it. Right? And then you get, like, software engineers who don't consider themselves to be security engineers, but they're writing code that needs to be secure. And then you got security engineers who consider themselves to be security engineers who consider them more security engineers than software developers. So you see this, like, blending of different roles. And where I find this particularly interesting is then when you start feeding in other teams like product teams and marketing teams and other folks. And one of the things we always say to the companies that we work with is never ever ever put a lawyer in front of anything creative because their job is to mitigate risks. So they will suck all the fun out of everything. But but I think we face this interesting challenge where you wanna deploy software and services to the Internet. You need to make sure that it's secure. But if you lead with a security posture, then, yes, you'll be ultra secure, but then you'll probably have to innovate less quickly because by definition, you're gonna take a more conservative stance on things. And I think that balance between that push and pull is always super interesting because nobody's wrong. Like, you know, the product team want to innovate faster as they should. That's the whole point. But then the security team wanna keep people safe. So I've I've always found that, like, balance of that really kinda super interesting. Mhmm. Well, gentlemen, we are coming up right against time, here. I wanna thank you both for participating in this fantastic, discussion. Hopefully, Hopefully, we can move the state of the art forward a bit here when it comes to Linux distros, open source security, the role of distro. Really appreciate your insights here, and thank you for your time. Yeah. Thanks. Thanks for having me. All right. Thank you all for listening. We very much appreciate it. Have a great day. Thanks, everyone.