Rust Never Sleeps: Tonic with Lucio Franco
January 3rd, 2024
36 mins 24 secs
Your Host
About this Episode
Tonic is a native gRPC implementation in Rust that allows users to easily build gRPC servers and clients without extensive async experience. Tonic is part of the Tokio stack, which is a library that provides an asynchronous runtime for Rust and more tools to write async applications. Today, Lucio Franco (@lucio_d_franco) of Turso joins the podcast to discuss his unique experience maintaining Tonic and contributing to the asynchronous Rust ecosystem.
Contributor is looking for a community manager! If you want to know more, shoot us an email at eric@scalevp.com.
Subscribe to Contributor on Substack for email notifications!
In this episode we discuss:
The challenges of async Rust and ways the community has addressed them
Lucio’s plan on how to get a job in distributed databases
How the Tokio team avoided power dynamics
Problems around working on open-source in the corporate world
Why Lucio encouraged a collaborator to go on without him
Links:
People:
Carl Lerche (@carllerche)
Other episodes:
TRANSCRIPT
Lucio Franco:
The VP's making the choices about where to put money, they don't think about it. They take it for granted. And this is a huge problem in the corporate world with open source. It's like a fog of war, they don't see far enough down the software supply chain of what's powering everything.
Eric Anderson:
This is Contributor, a podcast telling the stories behind the best open source projects and the communities that make them. I'm Eric Anderson. Today we're on air with Lucio Franco, who is part of the team over at Turso. We talked to Glauber maybe a few episodes back. And we're going to talk about Rust and some emerging standards, and in particular a project called Tonic, part of the Tokio broader family. Lucio, tell us what Tonic is real quick. We'll see if we can encapsulate that in a elevator pitch and then we'll talk about the whole story.
Lucio Franco:
Tonic is a gRPC library at the base. It provides your client server, your basic code gen. The library is pretty simple, it just allows you to talk gRPC with any other language, because gRPC is a multi-language protocol, so it's pretty efficient. Tonic itself, some of its higher level goals have always been around being super ergonomic and efficient. So if you don't have any history around async Rust in the past, it's quite challenging to work with. It's improved a lot over the last five years, but when it first started it was quite hard to work with. So Tonic is essentially like I want to write a networking server, a client server, or talk to a server or client, but I want to do it really easily. I want to get started and not have to worry about the hard thing. So that's essentially Tonic's goal of a project is to enable those types of users that are somewhat new to the asynchronous world.
Eric Anderson:
And I don't know the right order to do this, Lucio, your background or Rusts background, or maybe you know a way to intertwine them, but in researching our conversation, I didn't realize there was a lot of history around async with Rust and there's not a standard way of doing this, and so the community has solved this in a few ways. Tokio, which Tonic is a part of, is an umbrella brand for a set of tools that all are going to work, interop well together.
Lucio Franco:
We can back up a little bit. Rust 1.0 in August 2015, July, something like that, so it's a very young language. And in effort to standardize or 1.0 the language to make it available to people, there was a choice to reduce the size of the standard library. The less surface area that you have to ship, the easier it is to ship, so that's the idea. Let's make package managers really easy compared to something like C++ or getting dependencies inside is very difficult. Rust is like, "Let's make that very easy and then let users innovate on their own and not have to deal with this whole RFC process to get stuff into the standard library that is, one, extremely slow and brain drain. It's exhausting." So Rust at the core has always started with a small standard library.
For example, a language like Go has a lot of stuff shipped with it, it has an HTTP implementation, it has green threading implementations that you can just take the standard library and just run with and actually built some pretty good stuff out of the box. Rust did not take that approach. Rust let you push that onto users. Tokio itself, there's two things here going on in the early stages of async Rust or the story of asynchronous networking in Rust. There's the design of futures, which would be the idea of how do we execute anything asynchronously, which was being driven by some compiler folks or library folks in Rust. And then on the side there was, okay, how do we leverage these interfaces and build something off of it that is actually async? And this is where Tokio came into play. Around 2016 ish, I think, was when they started developing Tokio a little bit. And it started off as a third party library.
If it works, why does it need to be a part of the standard library? Because putting stuff into the standard library is a lot of work. There's a lot of bureaucracy involved with it, which I completely agree we should have, because stuff that goes into the standard library is stamped and shipped to everyone in the world. So being able to do stuff outside of that enabled Tokio to innovate and push the boundaries of what was possible. And I think we're seeing a lot of the fruition of those effects in the modern day. Tokio is extremely popular now, it's being used at essentially almost every major tech company for something, and it's growing popular in the ecosystem as being a leader in this space.
Eric Anderson:
Maybe I interrupt you here because I think you've given us some early history into Rust and Tokio. And now what's your story? Because at some point you intercepted with Tokio.
Lucio Franco:
So actually my origin story with Rust actually started in college. Before I got into networking and all of this distributed systems stuff, I was like, "I want to write game engines," and I failed so many times trying to get OpenGL linked into my C++ project, banging my head against the desk reading Stack Overflow screenshots of Xcode settings and having no clue what's going on. And then I stumbled across Go and I didn't like the Go path thing, and then I stumbled across Rust and I was like, "Wow, I can just cargo add OpenGL and things just work." And I was like, "This is amazing. This is clearly the best." So I fell in love with it then and there, Rust as a core, everything just made sense. I read the book and I was like, "Why aren't other people doing this? This is so awesome."
And then obviously did my game stuff for a while, and as I was graduating I was like, "I need to do real stuff," because I didn't want to work in the game industry. And so I was like, "I love distributed systems and distributed databases, why not get involved?" Ironically now I work at a database company doing distributed database stuff, so if you think about my goals back then and where I've landed now, the path has worked out quite well for me. So I started learning Rust and then obviously I saw the future is in async networking stuff coming up, and I tried to play around with it for a while and I absolutely failed. It was not the easiest thing. The error messages coming from the compiler were quite confusing just using it in general, not having a clue what async networking was, I just knew it was a cool thing and it's high performance.
So if I was wanting to write a database, this is the stuff I want to use. So a couple of years went by, I worked in a different language and I started... Honestly, I don't know if you guys remember Gitter, it's a chat platform for GitHub, so I get a project to create a chat platform. And so me being curious, I just was poking around the Tokio stuff and I ended up going on Gitter and talking with Carl who's the author of Tokio. And we were talking back and forth a little bit. And that's basically how I got involved in Tokio was just honestly letting Carl talk to me about stuff. And I had no idea what was going on, but I was curious and wanted to learn. So I got myself involved and I think, instead of working, I was doing some open source stuff here and there, and it wasn't part of my full-time job.
That's how I got involved in the open source thing, but Tonic itself, the origin story is actually interesting because continuing on this path of seeing this projects and I started going to Carl, I was like, "Hey, where can I help?" Obviously, at the time I didn't really know what epoll was, I had some idea, but epoll's the Linux library that essentially Tokio wraps around, and that's the core API that powers everything asynchronous. And so I didn't even know the concepts of this, so I didn't really know what I was doing, but I wanted to learn. So I asked, I was like, "What are some space that I can help out with?" And it turned out that there was some experimental projects done with Tower, which is a middleware library. And it had been written, but it needed a lot of polish, as with everything. At this stage of Rust, everything was very new.
And so I started playing around with that a little bit. And then I saw this project that was called Tower gRPC, and I'd seen that Carl had actually written this. He was at the time working at a company called Buoyant writing a proxy that needed to do some gRPC stuff in Rust, and so he had written code. But as you are with any startup, when you write code, you're not writing code to release it to people. You're writing code into your product and, all right, it works, we're done, let's focus on the next priority, because it's a startup, so this thing was like this half hacked state, it worked, but it was the definition of unergonomic, not Carl's fault obviously, but at the state of where things were at, it was quite difficult.
So I decided if I wanted to work in databases and networking, that was the end goal, why don't I take this project and try to make it better? And roughly this is at the time when async functions were coming around. So we're going from handwriting state machines that were really complicated and extremely error-prone, extremely, extremely error-prone, to writing really nice generators that did a lot of the heavy lifting. So we were making this big jump, not just the Tokio people, the Tokio people actually this is all compiler stuff. So this is like the people in the compiler working on that singular futures interface, improving how people can write stuff from the compiler perspective. So for us it was a target of, "Hey, we have this new feature shipping in Rust that's going to really change the game. Really, really change the game. Let's ship something along with this."
So we were planning some releases of Tokio, and I was like, "You know what? Now is the time, now is my window to write a library that, one, can be really ergonomic and easy to use." So thinking back to me in college and I was to play around with some RPC libraries, I fell flat, I had no idea how to do this, how could I improve upon that experience? How could I make this easier for users? Not just discovery, but documentation examples, just how do we reduce the foot guns, how do we make it so you don't dig yourself into a hole that you didn't think was possible and then you couldn't get out of it? And so that was my main focus of time was how can we really change the game here? And so I spent a lot of wakeless nights, no, I slept, but a lot of nights after work coding, trying to get this done and shipped, and working on airplanes, and actually I found that airplanes was my best spot to focus on, because with my ADHD there was nothing else to do but code, so it was perfect.
And I got it shipped basically just in time for async/await to ship as well. And so technically Tonic was the first product, and I say product, I'm not really selling anything, but it was the first library on the market or in the Rust ecosystem that was taking advantage of this new async/await technology, was working, and basically production ready, and that was the goal is to lease this all at one time and then iterate upon it, and take lessons learned and stuff. Looking back at it now, in the last four years, the original design actually worked quite well and yet to hear people complain about stuff, which is either a scary or a very positive thing, but there's definitely some hidden things in there that I would love to fix in the future that I always put off to the end, but overall quite happy with the progress the library has made at this point.
Eric Anderson:
Maybe, Lucio, just to dig into a couple of things from that experience, is this your vision or is it Carl's vision?
Lucio Franco:
The real [inaudible 00:10:32] for Tonic was I wanted to work in distributed databases. So I was like, "What's the best way to get a job working in distributed databases?" It is to write the protocol that they're using. You go to an employer and you say, "Hey, you're literally using my code. You depend, your entire product, on me, hire me." So that was my vision, that was my motivation. Obviously, I think the goal that Carl was working on in general and the others in the Tokio group were really good, and I think we pushed for some similar fronts, but I think that there's, what I was pushing for, that ulterior thing for a job, but the goal of making things more ergonomic and easier was really important. And actually the third thing I think that was really critical to Tokio's current success that we really focused on was shipping.
A big problem that happens in the open source community, and we see this a lot with Rust. Most of my exposure has been to Rust, so I can't really speak to a lot of other languages and communities, but Rust, we have this decision paralysis. And the same thing with the standard library versus non-standard library discussions. If you want to put something into the compiler, it gets drilled down on you're going to have to answer a million questions, people coming left and trying to make decisions on, oh, what if we did it this way? We want to support this. What about that? And that, one, becomes very exhausting, but it doesn't allow you to ship. That's why things, for example, like the original async/await shipped four years ago, but we still don't have interfaces that are async/await. I think it's shipping now, but it's quite a time later.
This is just purely due to the politics that happens in open source. And Tokio took a different approach. Luckily we were all amicable to each other, so we all liked each other, there was no drama, but it was like, "Hey, why don't we not spend our time always discussing and going in circles, let's get something out there that people can use so people can build stuff with." Rust is not going to go anywhere if people are not able to actually build stuff, and people from any level. I'm sure someone who's very experienced could take some of the older stuff that was really difficult to work with and build something, but we have to support people who are mid-level and juniors, and we want to grow into that space like Go has taken really a strong advantage here of being really easy to work with.
So part of our Tokio's goal was to ship an easy to use ergonomic library but ship it. And that was the same goal with Tonic. That's why I was like, "Let's get it out in time for async/await, so everyone can, the day async/await lands, boom, what are you going to choose as your gRPC library? Tonic." So we were pushing on all fronts. I think we all had agreed that that was the right way to go. We didn't want to run in circles any longer. And so that's I think been a huge advantage of the Tokio group of us really not trying to foster drama or any power plays or anything like that, because open source tends to have a lot of power vacuums and people trying to take over.
Eric Anderson:
Maybe help me understand one aspect of this and that's that... I don't know if runtime is the right word, but this fundamental aspect of the program language that's not included in a standard library, async/await, when you write your gRPC, you write to a single approach to async/await, even though there may be others, and so Tonic can only work on Tokio's async/await, is that right?
Lucio Franco:
It's a little bit more complicated. So the async/await is just like syntactical sugar in the compiler to convert to some internal types. Think of it as like a transformer. In reality it is exactly a transformer. It transforms into a type that implements future. Future is some interface of something that can run asynchronously, and pause and resume. Those are all in the standard library. Finally, future made it into the standard library after a few years, but yes, that all made it to the standard library, Tokio itself just uses the future interface and a couple global variables to hook into your thing to notify you that you're ready, whether it be a timer or whether it be your epoll thing saying, "Hey, your file descriptor is not ready to read from because there's bites waiting in the TCP buffer," that stuff is tied to Tokio.
For example, if I use a TCP stream type from Tokio, it needs to run inside the Tokio runtime, and this is because the TCP stream type needs to know how to register itself with the API that Tokio provides. And there's no way around this. It's very hard to abstract that, or abstract it in an ergonomic way, because those APIs are so tightly ingrained with what it's doing in the scheduler that it doesn't really make much sense. So, yes, there's been this kind of issue in the Rust async ecosystem where you can't mix and match runtimes and swap out runtimes, and it makes it very challenging. It ruins your programmer mental model that you'd like to have where things are abstracted and I can just change one line and now I'm using a different runtime. In reality, the argument for that too is the use cases are not very high.
Yes, you have an embedded system, maybe you want a different runtime, completely agree, go use an embedded system runtime from the start. Tokio's net was never intended to work in that space. But spending all this time on abstracting things when, for example, we're using epoll in Tokio, but remember at the time that we were talking about all this abstract and runtime stuff, io_uring was coming around. If you don't know what io_uring is, it's essentially a new Linux API that allows you to dispatch assist calls asynchronously right in the kernel. So it's a really efficient way to do it, but it completely changes the model of how you do IO. From having to send the data into the kernel or just passing a pointer and being able to read through it, it makes it much more complicated. So then how can you abstract something that is still being innovated?
It's the same question about the standard library. If we haven't figured out the right abstractions yet for how to do epoll in Tokio, how the heck are we going to be able to put that into the standard library and convince people that this is going to be the implementation that we want for the next 10, 15, 20 years? It's not possible. And, in fact, for Tokio, we didn't really think about this. We didn't care about the abstraction because we wanted to get something that people could use. We weren't thinking about how could users swap it out in and out. It wasn't a play to be having monopoly whatsoever. Not at all. It was this isn't beneficial to achieving our current goals of getting this in the hands of users so they can build stuff.
That was primarily our number one goal about users not writing the most perfect system, not about writing the best thing ever, but about getting something that people can use and build real systems with. Because, in reality, it's how you bring a language forward. You can't just stay academic about everything. You need to be a little bit more pushy on getting things in people's hands. It's like doing a startup without the funding and all the stuff like that obviously.
Eric Anderson:
Well, actually that's a good segue. We will get into funding and economics related things in a moment. So this worked out for you, your plan of I'm going to work on the gRPC implementation and then I'll get hired by a database company.
Lucio Franco:
In fact, actually I had a friend, he was working at AWS at the time. He was like, "Hey, by the way, I just searched..." Because AWS, there's a way you can search every single code base that isn't security cleared or something, so anyone could see any implementation of EC2 or anything like that. And I guess he just searched Tonic and he found that Lambda was using it, and was like, "Okay, that's interesting." This is two months after I first released it. So maybe six months after I was like, "Hmm, maybe I should start working on this." That following March I actually reached out to him and I was like, "Hey, is Lambda still using it?" He's like, "Yeah." I was like, "All right, intro me to the manager." And boom, I ended up joining AWS Lambda and helping them build their new Rust service, and eventually transitioned into doing a lot more Rust stuff at AWS in general. But it worked that easily. I was impressed.
Eric Anderson:
Trojan horse almost, you write this thing, they have to use it, and then you have to work there so that they can maintain it.
Lucio Franco:
And what manager's going to tell you no, your entire networking stack, I can solve your problems.
Eric Anderson:
So you're working on Rust at AWS, but not necessarily databases, and then you find your way to Turso?
Lucio Franco:
Yeah, so I got caught in the layoffs. I was working on Rust tooling and open source, so I think I was one of the first people on the list to get laid off. I wasn't working on any moneymaking things. I was like, "The layers from the moneymaking was quite far away." And, again, it's probably a good segue into the open source funding thing, but when you work on open source, from the VPs making the choices about where to put money, you're so far away. They don't think about it. They take it for granted. And this is a huge problem in the corporate world with open source.
It's a fog of war, they don't see far enough down the software supply chain of what's powering everything, the bites operating on the CPU or whatever, they're not really thinking that far. They're just thinking about my customers directly. And so unfortunately I got caught in that crossfire. And good timing anyways, I was ready to leave, and I wanted to work on something. And this probably also segues into another talk about the mental health aspect of being an open source maintainer. And I'm, again, very happy with my choices and being at Turso now. And I'm finally actually working on a database, so that's the huge positive here.
Eric Anderson:
Before we go to that, how does the funding work for Tokio? So you got this somewhat of an organization of volunteers building libraries that they all are individually passionate about and it works, how does it all work?
Lucio Franco:
That's an extremely good question. For a while, a couple of us were being paid to work on Tokio. Carl works at AWS still and he has been working there maintaining Tokio and working with people inside the company to help them with Tokio. Sean has also worked at AWS and now is doing his own thing. He's the maintainer of hyper, which is the HTTP library, implements all the nice, crazy fun stuff around HTTP, but I think most of the other people are not paid full time to do this. A lot of it's either in our free time or work has given us some time to work on it. When I was on Lambda, I spent, and I'm going to put in quotes, 50% of my time working on Lambda, 50% of my time working on Tonic and open source. Definitely very hard to juggle.
But most people were in this situation where they were doing it either in their free time and stuff like that. And if you look at some of the projects, since 2019, things have stagnated a little bit, and obviously as things get popular, they're very hard to manage. For example, Tonic right now, I actually... Personally, I don't have any funding coming in for Tonic. Turso has given me time to work on it. Obviously it's hard to juggle startup priorities with open source library that is not as critical. Obviously I'm looking out for security bugs or anything that could be causing issues, and I'm going to respond to that as fast as possible. That's not a problem, but features, and I've wanted to re-implement certain things, I don't like how it's done, I just haven't had the time to do that. But there isn't much funding.
And in fact also we have some money that has been given to us through things like Open Collective, but we really haven't found a way to spend that money. It's been hard to find people to contract to because in reality the bar to contribute and to spend our money on somebody is quite high. To be able to get into that space, you need to show that you're able to do that. And so that barrier is already very difficult to overcome. And so finding people that are willing to do that and actually will stick with it is incredibly challenging. So in reality, besides people that are being paid to do it for work, there isn't much people really being paid to do stuff like that, which is unfortunate. And another thing for me at least is I haven't even set up GitHub sponsors for myself.
And another aspect of open source that's quite challenging is the guilt involved with it. I have the push rights to Tonic, I can publish things. And people will open up bugs that maybe it's blocking them from achieving something, and I don't have the time nor energy to always help them out, because sometimes it requires a lot of work. The quality of bar of contributors is all over the place, so sometimes you have to guide through users, sometimes users drop 2009 PR on you and are like, "Hey, can you review this?" I'm like, "Absolutely not. That's a lot." So one thing I've had to learn is how to say no, I've gotten very good at it now, but things like, "Hey, this is a lot. I don't have the time for this." But I feel guilty, so I actually don't accept money because if I accepted money, I would feel even worse.
I would feel like I would need to actually spend more time, more of my free time, more of my mental energy on it that I currently necessarily don't want to spend on. I've come to the point in my life where I want to set a certain amount of time a week to work on coding, and there's other priorities in life. So back when I was younger, I was able to come from work and code all night, but now I don't have that same interest or motivation, and so I have to have this boundary I put between the people there a little bit because I don't want to feel guilty and bad. I can say, "You're not paying me." Yes, if you want to get a release out, you do a lot of the heavy lifting and work, I will help get it through.
But if you're asking me to do something, I'm going to tell you, "Look, I just purely don't have time." And actually this happened literally today. Someone came in and was like, "Hey, is there any plans to implement this?" I'm like, "No, I currently don't have the time nor energy to do this." If you are interested in fixing it, come on, be a maintainer, come talk to me. And in fact, I've been looking for maintainers for a very long time and I've had some people come in, work on stuff. I've had one maintainer come in, but it turned out that actually pushing him to work on a different project, similar what I did with Tonic, I had him work on a HTTP library, like Django style library that allows you to... It's not HTTP, but a HTTP server express style library for Rust that we were really lacking, and I saw this really good opportunity for us.
And so I was like, "Hey, I know you're helping me a lot and I love having you help me and you're awesome, and I trust you, you have full rights to everything, but actually I think you're better doing this, championing this and running with it." He did that and now the project, which is Axom, is a wildly even more popular than Tonic, and I'm happy with that, but the downside is that I lost a collaborator because he doesn't have the time to spend time on both. So it's like I found somebody, but then the opportunity presented itself and I lost them. And in fact, I pushed them, I created the repo I was like, "Hey, go talk to Carl. Go implement this. Yes, go, go, go, go, go." So it wasn't like I was sad, sad, I'm still happy with the outcome, but since then I've not been able to find anyone to really stick with it and help me. Still been just me maintaining these things, and it's a lot of work.
Eric Anderson:
Lucio, you mentioned earlier there was some mental health aspect to open source that you feel like is fairly pervasive, it sounded like. Tell us what that is.
Lucio Franco:
When you create an open source project, let's say for me with Tonic, I started writing it because, one, as I said earlier, I was doing a job in distribute databases and stuff, but also I was just genuinely curious. I was enjoying it. I was having fun writing a library, figuring these things out, looking into other libraries and seeing how they did stuff, and trying to massage it into Rust and make it right, writing blog posts and spending time on the README and making it look all pretty, I loved that. The first pass I did that was amazing. But as the years went on, and Tonic grew and matured, and more and more people started using it, so the frequency of issues being opened or people asking me questions on Discord started to increase, the burden started to become quite hard.
I'm one person, I have a fixed amount of energy every week, and this thing is piling up on top of me and there's more people asking me stuff. And obviously as a project gets more complicated and the longer it exists, the harder it is maintaining it. There's this relationship where as you add more code, the burden of maintaining... Remember, the implicit dependencies of everything becomes very complicated. So the problems and the bugs to solve started to get much more complicated in a good and bad way, but it's the nature of software. But when you work at a company maintaining a piece of software, there's motivation. You have customers that are paying you and telling you, "Hey, we love this. This is great."
For example, let's say, a B2B company, maybe you have a customer success team, and they're giving you feedback from the customers. They're saying, "Hey, our customers really enjoyed this feature, good job," in the open source, what happens when I make a release? I spend all my energy and brain power cutting a release, writing the code, making sure everything's right, doing the change log, making sure everyone's happy, I click the button to publish, and then radio silence. If it's good, if the release has no bugs, radio silence, there's nothing. Did people download it? Did people try it?
Eric Anderson:
You're saying that most feedback is negative feedback.
Lucio Franco:
Right, all you hear is this negative feedback. You don't get the positive feedback loops that trigger the dopamine release in your brain that make you excited and motivated like you would have at a company like your boss saying, "Good job." That was really you make these releases, these bug fixes, you spend all this energy fixing these bugs and then it works, and no one tells you good job. It's quite a depressing experience and all you hear is these negative things. And then on top of that, because you are the maintainer, you are the only person to be able to review and push things through. And now someone opens a PR fixing a bug that a couple other people have, and then they comment on the PR saying, "Hey, is someone going to review this?" And I'm like, "Look, I am exhausted. I don't have the time for this. Maybe next week I can look at this," but I feel bad.
Inherently, I want to help these people, I have an internal want to be helpful, and so it really pains me to have to leave these people hanging. And it gets worse, we have some projects that issues are piled up like crazy, and the surface area is large that they need to work on, so there's a lot of stuff going on, and the overhead is really high. And you pair that with not being paid, say, a wage. The work that a lot of these open source people do is the work of senior engineers. We did this analysis when I was at AWS about the quality of the code these people are writing, what they have to think about.
And if you were to compare them to the leveling at AWS, for example, it's like you're a senior engineer, if you're able to do this stuff and you're able to make these choices and talk to people in a certain way and do designs, especially in an async collaborative world without someone being able to help you or another engineer helping you, that is truly the work of a senior engineer, but these people are being paid $0. It's quite crazy, the work they're doing is worth hundreds of thousands of dollars and they're being paid maybe they get a thousand dollars through sponsors or something. It's quite crazy this disparity. And that plus the guilt and plus other priorities in life turn you into burnout. They make you fall in this pit where you just don't know what to do. You're paralyzed.
Eric Anderson:
We talk about community, we use that word all the time, there's a big community around Tonic, and maybe it would help to understand who these people are actually, most of them are users of the library and they're fairly transactional. They show up, they download, they run, and they file bugs when they don't work. And that's what? 90% of the community?
Lucio Franco:
Yeah, there's maybe two or three people on Discord. We tend to use Discord for all of the Tokio projects, and so there's a Tonic channel that I monitor. And occasionally we have some other people that we have a tag for some people like a group where people that are helpful people, they're not contributors, they don't have push access, but they are just people that have been answering questions here and there. And this is also how I got involved by the way. I started going on Gitter back in the day and was helping people. And as a maintainer, I'm like, "Wow, I love that. I love when people come in and answer questions. If you don't know the answer, ping me, I'm happy, but please go and try to answer questions even if you're not confident. It's a good start." And that shows initiative.
So there's people that help out, but there's maybe three or four. This is just for Tonic. For Tokio, there's a larger group of people. So the Tokio umbrella and there's the Tokio project or the actual code library, that surface area is massive, so there's a lot more of a community there because also the user segment that uses Tokio is everything, whereas Tonic is one slice of that pie. So beyond that, there's not much of maintainers helping people answering questions. It's usually just other people maybe running into a bug and maybe they found a solution. Beyond that, it's transactional, people being like, "Hey, can I get some help for this?" That's it.
Eric Anderson:
And you can't just walk away because you have the keys to making updates. And as you mentioned, somebody's got a critical bug or a security issue, they want to push an update and you're blocking.
Lucio Franco:
So actually we have the permission set up in Tokio that if I were to get hit by a bus tomorrow, a large bus, people have access to rights to everything. It's part of an organization, there are escape patches, but people actively publishing new packages and stuff is just me mostly. Other people have rights, they know how to do it. For example, David, who works on Axom, the guy I mentioned earlier, he hasn't made a Tonic release in two years, so he's not really going to remember how to do all the little things here and there. So it's just me managing it all. I bet somewhat poorly probably, but tried my best.
Eric Anderson:
What's the end game? Eventually you hand this off to somebody? Do you just document how to do releases really well? I don't know. What have you seen in other open source projects? Is there a place?
Lucio Franco:
I think the real end game is waiting for Rust 2.0 to come out, so my library becomes incompatible and someone else has to take the reins of maintaining a library. No, there is no end game. And actually I thought the same thing, how do I get out of this? There's no real way to do it. And the hope is that some maintainer comes by that helps out and will be able to take charge. But the real solution actually is to get the library to a 1.0 state where it's just bug fixes. What this means for Tonic right now, it's probably I need to go through with an and start chopping off things. There's some things that I implemented in 2019 that made a lot of sense with the idea that we're the first library of the market, so I have to do a lot more heavy lifting to make up for the lack of other libraries.
So there's the gRPC stuff, but then there's a lot of load balancing and HTTP server client configuration stuff, I wrote a layer on top to make it easier, but now it's a lot easier to do it by hand, you could put the pieces together, but before it was not the case. So part of the end game is cutting surface area, this is a huge strategy in how to achieve 1.0, really just cut out all the surface area, maybe put it into a unstable crate that's not 1.0, and then just have a very solid 1.0. And that's really the end goal. The gRPC protocol is not going to change, the HTTP/2 protocol is not going to change. What we have now is pretty solid. It's very production ready, it's being used quite heavily all over the world, so I'm pretty confident in the code. It's more of just reducing surface area, but the problem with that is it's a lot of work to go and cut things out, because I have to write docs and update things. So it's a catch 22.
Eric Anderson:
From your experience at Amazon, what percent of big tech engineers, or an open source maintainer, have an open source side hustle that relies on them?
Lucio Franco:
From my perspective at Amazon, it's a very low percentage. Very, very low percentage. But I think that's a problem with a company like Amazon or Microsoft or Google in essence a little bit, but I think the Google personality varies from those two. Amazon has a very strong college pipeline, so you have a lot of people coming straight from college going into a pretty good, cushy tech job, working in some pretty bespoke technology. The build system is unique, the libraries they're using are unique. This is a big shift. Rust made a big shift at Amazon because a lot of the previous libraries they use, everything's in Java, so they already had handwritten every library you could ever want. You didn't have to go outside the little bubble, everything just worked. But for us, that was not the case.
That's why people started using Tonic, because they needed a library to do something now. You can't wait for a team to implement this in 1.0, but most people are not writing Rust, most people are writing Java or whatever at Amazon because it's a very large company. Those people are not really exposed to the outside world, so there's no point for them to do open source stuff. A lot of them don't. The job is already taxing enough, they're not going to really go out of their bubble. I think people at startups and companies where you do have to pull tools off the shelf, you do have to go on GitHub and see what other options there are. If I wanted to implement something for Turso, for example, and I didn't do my due diligence to go check for a Rust library that might do exactly what I want, that's really dumb. I should be finding the fastest way possible to achieve something. So that process also leads into people contributing back.
You'll see a lot of startups have GitHub organizations where they're like, "Hey, we open source this little library that we used to do this thing and it really helped us." And that can spark a little bit of that open source creativity, innovation and putting things out there that a large company doesn't really push. Obviously, there's going to be exceptions. There's going to be people that are curious and just want to do stuff. Like me, I was doing it all on my own, wasn't related to work, somehow found its way back into work, but there's a lot of people in this large company that are not really exposed to this type of stuff, and they're not exposed to the risks either. That's the other thing. I spent a lot of time at AWS thinking about what should we depend on. If I could choose a set of libraries that everyone should depend on, how do we make those decisions? And most people don't think about that. They just choose. It's good until things break and you have a zero day or something, but it's always a risky.
Eric Anderson:
Lucio, this has been super fascinating. Not only all the details on Tonic and Rust, but your personal experience navigating being an open source maintainer. Tell us, as we wrap up here, what folks can do if they're excited to learn more about Tonic or want to take the load off your shoulders.
Lucio Franco:
Well, if you're interesting in Tonic at all, come check out the project, try it out. There's a bunch of GitHub issues, there's Discord, getting involved is pretty easily. I'm available, you can find me on Discord, you'll see my green name show up. Play around, respond to some issues, review a PR. I will never stop anyone from providing their opinion on something. And actually I encourage people to act as if they're a maintainer without having the privilege of being a maintainer. Go ahead and review something as if you were the one trying to merge the PR. That generates a lot of confidence in us, I love seeing the initiative. And, again, I have an email, I have Twitter, X, I have Discord, reach out to me, come ask me questions. I'm always happy to talk and discuss these sort of things. I was a budding open source maintainer once upon a time not too long ago, so I understand very well what it's like to try to get into this space.
Eric Anderson:
Well, also, we appreciate your service, Lucio. You've given something back to the world.
Lucio Franco:
It's the least I can do.
Eric Anderson:
You can subscribe to the podcast and check out our community Slack and newsletter at contributor.fyi. If you like the show, please leave a rating and review on Apple Podcasts, Spotify, or wherever you get your podcasts. Until next time, I'm Eric Anderson, and this has been Contributor.