051: Rust and APIs with Steve Klabnik

Steve Klabnik @steveklabnik | Blog | GitHub

Show Notes:

• 02:56 - Getting Into Rust
• 05:51 - Working on Rust for Mozilla
• 07:01 - Writing Documentation and Preventing Burnout
• 13:24 - The Rust Programming Language
• 18:45 - Rewriting Firefox in Rust
• 21:20 - High-level Functions
• 25:23 - Typesystem and Concurrency
• 36:35 - Rust and Web Developers; Digging Into Rust on a Deeper Level
• 43:46 - The Rust Ecosystem and Using Rust on a Day-to-Day Basis
• 48:38 - The Rust Book

Resources:

• Rust For Rubyists
• Cargo
• Servo
• Application Binary Interface (ABI)
• MetaLanguage (ML)
• Tokio
• Systems Programming
• intermezzOS
• Steve Klabnik: Exploring Ruby Through Rust
• What’s new with “The Rust Programming Language”?
• rustbook

Transcript:

CHARLES: Hello everybody and welcome to The Frontside Podcast episode 51. I'm here, my name is Charles Lowell. I'll be hosting today. With me is Chris Freeman, also of The Frontside and with us is Steve Klabnik.

Now, most of you probably heard of Steve before. My first encounter with Steve was actually at the LoneStarRuby Conference back in... Gosh, I don't know. It was many, many years ago and he was giving a talk on Shoes, which I also had never heard of before. It was a wonderful story of a code archaeology project where he was kind of investigating, rehabilitating, and in carrying forward a project that the 'why the lucky stiff' had done. That was a wonderful introduction but it was certainly not the last time that I encountered him in his writings and in talks and stuff, mostly within the Ruby community. But it popped up again and again, talking about Rust APIs and always making a point to take a good knowledge that he'd learned and spread it around.

Personally, I've lost track of Steve or hadn't really heard much of what he was doing for a while. But then Chris came into the office and he was always talking about this language called Rust. While I've heard Rust, Chris was just all about it and wanted to have Steve come on the show because it turns out that Steve, you've been really, really, really into Rust these last few years and sounds like concentrating most of your work there.

STEVE: That is totally true and accurate. Also to go back a bit, that means that you are in attendance for my very first conference talk ever.

CHARLES: Really?

STEVE: That was literally the first one.

CHARLES: Wow, it was a great start. That was a great story. It was educational and also touching.

STEVE: Thank you. It's actually interesting because what happened was is that someone else who works on Shoes have encouraged me to submit to RubyConf and I was like, "Who would want to hear me talk at a conference?"

I submitted the talk and RubyConf accepted it and I was really excited. Then a bunch of other conferences noticed and two other conferences had asked me to give a talk before RubyConf happens and LoneStar was one of them and it was the first one chronologically. That moment was also very special to me as well.

CHARLES: Fantastic. What year was that?

STEVE: I want to say it was like 2012 or 2011. It’s really hard for me to pay attention to time and date. My history is so complicated that I often forget. I've literally told people that I'm 10 years old or younger than I am because I would like mess up to date on the things. It just happens.

CHARLES: Yeah, but it was a while ago and it's been quite a journey, in between now and then.

STEVE: Yeah, definitely and you're also definitely right. It is now literally my day job to work on Rust so it is definitely the focus of most of my efforts. Partly, why I made that happen was because it was the focus of all my hobby efforts before I made my job. It’s definitely been a couple of years that I've been a full-time on all the Rust stuff.

CHARLES: How was it that you actually got into Rust? How did you hear about it before everybody else and how did it capture your attention?

STEVE: I've always liked programming languages and learning different programming languages. Ruby was sort of where I became known professionally. But it wasn't the first language that I knew and I knew it was never going to be the last. As much as I always loved Ruby and I'm like literally have a tattoo on my body so I will be with Ruby forever. I always try to learn new stuff and I find it exciting.

I'm from middle of nowhere, Pennsylvania in the suburbs of Pittsburgh on a cattle farm and I was visiting my parents for Christmas one year. There’s not really a whole lot to do out of the very small town so I was just reading the internet, as usual and it turns out that that was the day that Rust 0.5 had been released. I saw this release announcement go by and I was like, "I vaguely heard of this programming language once or twice maybe. I don't have anything to do. Let's give it a try."

I downloaded and installed it. I looked at their tutorial and the tutorial has a problem that a lot of tutorials had, which is I read it, I said, "This all makes sense," I tried this down to write a program, and I had no idea how to actually write a program in it at all. I'm just completely confused. I couldn't actually apply the sort of syntax stuff that I learned.

At the same time, I was going to be working on this hypermedia book -- that was my plans for that trip -- as always, you just rewrite your tooling over and over again. You [inaudible] like, "Just don't write the thing. Write the tools that make the thing," so I wanted to try out a new way to take mark down and generate PDFs in HTML, involving pandoc. I sort of had that all set up and I said, "Well, let me give this a try run. What I'm going to do is I'm going to write down what I learned in Rust as I learned it," and sort of from a Ruby programmers perspective, I'll use that and working with my new tooling to see if it works to actually work on the real book and it will also help me understand Rust better because one of the reasons why I do all this sort of teaching and advocacy is because I think it helps me learn.

Just as much as I like helping other people learn stuff, I find that the repetition and being forced to explain something to someone else really make sure that I understand what I'm talking about. That's what the thing called Rust for Rubyist became boring. I'm a sucker for alliteration and that sort of became the first to tutorial for Rust from outside of the Rust projects proper.

From there, I went on to submit some pull requests because everything's open source so I wrote some documentation and funny enough, my first ever pull request to Rust was actually rejected based on procedural grounds. At the time, they didn't actually accept pull request to master, they accept this other weird branch and GitHub don't have the ability to re-target the branch of the pull request.

I also, always like this story because the thing that I now on the core team of, like my first attempt at getting involved was wrong and was turned down. But I'd fixed that pull request issue and got that in but it is kind of kept working on an open source capacity for a while and then decided to ask Mozilla if I can make it my job. Luckily they said yes.

CHARLES: Wow, so what? Your job at Mozilla, like you just kind of showed up and said, "I would like to have a pretty cool, awesome job, working on this brand new language," and they were like, "Sure, come on in?"

STEVE: To some degree, yes. That’s one way of putting it. There is always the devil in these details. The first thing is that that wouldn't have worked if I had wanted a different kind of job. But when someone comes to you and says, "I would like to write documentation for you all day," you go, "Oh, my gosh. This literally never happens."

If I had wanted to like work on the compiler, I'm pretty sure they would have said no. But because they knew documentation was important and they wanted documentation and because I had already been basically doing that job in an open source way, it's like I've had a year-long interview already.

Then finally, they actually didn't have headcount at the time so I actually moved on as a contractor initially and had to do some freelance work and then eventually, once we were able to hire a new person kind of got it in. They're like a cool kid story. It’s like, "Oh, yeah. I totally asked Mozilla for my perfect dream job and they just gave it to me," but like that's not really the way that it works.

CHARLES: Got you. That actually leads me into a question that I have wanted to ask you. You write a very good documentation as your day job and documentation is extremely hard. For me, it is extremely hard to get and stay motivated to document something that I've worked on. I think that is probably a common enough experience for programmers.

We don't recognize because we use documentation that it's extremely valuable and yet, it still this thing that is just a constant uphill battle. I'm curious, how do you manage to stay motivated to write documentation for an entire programming language over the span of years?

STEVE: As I'm often want to do, this has like three or four different components. I guess, there's a couple of different things involved. The first one is that I actually got accepted to go to English grad school, although I ended up not pursuing that. Like writing, it's something I have just always enjoyed.

I got a Bachelor in Computer Science but then I was going to go to grad school for English and due to university shenanigans, it didn't really work out. They told me I was going to get a free ride and then accepted me and then they were like, "Oh, wait sorry. You have to pay for this."

And I was like, "Wait, sorry. No, I'm not doing this anymore. That's ridiculous." That's kind of always a predilection for writing and I think that the reason why that is because I grew up basically like on Slashdot and eventually then on Dig and Reddit and all these other things. I've kind of been writing a couple paragraphs a day, basically every day in my life since I was a little kid. I think that's something that's sort of like underappreciated.

Documentation is hard but it's like a skill, like any other thing. Programmers will say, "I really want to learn TDD so I'm going to make myself do some TDD, I'm going to practice it, I'm going to focus on it and that's going to be a skill that I'm going to improve," and then they see documentation, and they kind of think it's this thing that you either have the skill or you don't. But writing is just another thing like anything else that you can practice at and get better.

I think maybe it's because it's a little bit farther away from the wheel house of what you do day to day, that people aren't as interested in it but it is something you're truly interested in, I think the best way to get better is just to do it and do it a lot. I say this is I'm kind of in the middle of a little bit of writer's block at the moment to be honest.

Then finally, I think the other reason that I'm motivated about docs is that I actually believe that documentation is an exercise in empathy. Like good documentation, the ideal as a programmer, the ideal thing that happens in documentation is I have a question about how to use something, I go to the documentation, and it says the exact sentence that answers my exact question.

As those varying degrees of vaguely gives you the right idea, versus literally tells you exactly what to do. I think that the way that you can accomplish that excellent documentation is by understanding what your users need and then preemptively figuring it and/or writing that down. I think that that requires being able to put yourself in their shoes to some degree. I'm not going to say that that's a thing that I am perfect at but I think that a valuable skill when trying to improve docs's like figure out what they actually need and then give it to them. It's doesn't always have to be in that order, like sometimes people will fail to find the thing they need, tell you what you need, and then you give it to them. That's a strategy I've used a lot and that's one reason why I hang out in the Rust IRC all the time, helping people is for a very long time, I would like sit in IRC, someone would ask a question, I would answer the question, I'd go look in the docs and see if they could have figured out themselves. If they couldn't, that would be might next doc PR.

It's just like even if it's just a couple sentences like add the question from IRC into the documentation and then just do that over and over and over again and then eventually, people start learning from the docs instead of actually ask questions because they already found what they needed.

CHARLES: Right. I have a question about that because once you develop those skill, I think you also still run the risk of like burning out. I know that one of the reasons I tend to always fall back to like, "I'm going to spend my time doing coding instead of documentation," Or, "I'm going to spend my time --" Even with TDD is a great example is like with TDD you get to experience those short term wins.

I think that kind of prevents you from burning out, where sometimes when I'm writing documentation, it feels like I'm screaming at the void. I might be screaming really loud and really, really well but I feel like a lot of times, I'm not experiencing those wins and I'm wondering if you have any tips for like experiencing those wins. Or getting that feedback to kind of keep you motivated and keep you doing the job. Also, trying to push the level of your own documentation skill and communications skill.

STEVE: Yeah, experiencing the wins is definitely a part of it. But one of the other things that is sort of part of it is that like I do the opposite. I do a lot of coding but that's my side projects. When I get fed up with writing documentation, I maintain the [inaudible] implementation that Cargo uses to resolve Rust packages, for example. If I'm feeling a little stuck on docs, I'll go write some software and then come back to the docs so that kind of help with burnout.

Another thing is that I think I'm just like perpetually in a state of just barely above burnout anyway so that also sort of factors in I guess. You know, it's like Bruce Banner. The secret is that I'm always angry so --

CHARLES: So you work on open source, is that what you're saying?

STEVE: Yeah, exactly. We're working on open source all the time. I've been lucky enough to make open source as my job for, basically almost my entire professional career. Although not totally. You know, at some point, you just kind of get used to it. But in terms of experience and the wins, this is also one of the reasons why I like to teach beginners specifically is that beginners allow you to remember what it's like to be a beginner, which is also part of building the empathy.

By interacting with beginners a lot, you also get a lot of those wins because beginners usually ask easy questions so it's easy to figure out the answer that stuff. Then you've got that positive feedback loop kind of going. To me it's maybe not IRC literally for every project but answering questions on Stack Overflow, or whatever message board forum you have, or Twitter, like actually interacting with other people. For me at least, that's how I get that kind of sense of not screaming into the void that you have to like go into the void and find the other people there, I guess, that I'm just like come to you necessarily.

CHARLES: Speaking of empathy for beginners, it just occurred to me that we didn't actually talk about what Rust is. We probably should do that. Why don't you tell us a little bit about the Rust, language, as well as, you've mentioned Cargo and [inaudible] ecosystem for us as well? Let’s talk about that.

STEVE: Yeah, totally. Basically, Rust is a new-ish. I should stop saying new because it's almost not really at this point. A kind of new-ish programming language, heavily sponsored by Mozilla in development. Its idea is to become a new low-level programming language. But I always hesitate when I say this because one of my old pitches for Rust used to be like, "Rust could be used anywhere. You can use C."

Then people go, "I would never write, C is so cool. Rust is not for me." I'm like not do that.

But the reason that people don't use C is a lot of the problems that we are also trying to fix. I guess the primary differentiator for Rust in terms of like programming languages theory is that it is safe and safety as they got specific meaning. But basically C is a very dangerous sharp tool and you can cut yourself and people who use those tools often do cut themselves, whereas Rust is like it's got a safety guard on it. It's a compiled language so its compiler actively prevents you from making some of the worst mistakes that you can make in a low-level programming language like C. It turns out that when you start building up these sort of safe abstractions on top of these really fundamentally low-level details, you actually end up with a relatively high-level programming language.

I talked to a lot of people, for example from JavaScript or Ruby world or Python world who come to Rust that are modulus, some libraries, and other things. This is actually high-level enough that I feel like I could do this instead of review JavaScript all day and I would be just as comfortable.

The other day, I did a little bit pair programming and we actually recreated a JavaScript library in Rust that had virtually the same interface because like you can actually build relatively high-level things so pass an enclosure to a function that does some stuff is totally normal and Rust world. That's also very familiar to people that come from the Ruby, JavaScript, Python background.

Also then, as part of that is we also culturally like Rust the projects, not Rust the programming language, really, really cares about helping people understand what systems programming and like lower-level programming means. A lot of people will not program and in C or C++ because they have no idea how to get help or to learn because many people in the low-level space have this RTFM attitude or like, "If you don't know what you're doing, then get out of here," whereas in Rust world, if you ask an extremely basic question, we're like, "Welcome. We would love to have you. I would be very happy to like walk you through," like explaining how that works on these kind of low-level details. Part of the culture of Rust is to bring this sort of low-level programming to people that have rejected it before for various reasons.

The reason that Mozilla cares and the reason Mozilla sponsored the project is that Firefox is written in C++, so like four million lines of C++ last I checked. Last time we did a security audit of a really pants-on-fire, terrible security bugs in Firefox, I go to this website and now they run arbitrary code on my machine kinds of terrifying bugs. Basically happened because C++ is dangerous and sharp. If you screw up, there's the kind of bad things that can happen. About 50% of those security issues in Firefox would be eliminated at compile time by the Rust compiler. That’s a really huge win in general so the idea is that we are slowly rewriting Firefox and Rust over time. That’s one angle of why Mozilla cares about Rust.

The second part is Servo, which is a rendering engine that's built in Rust from the ground up. If you think about Firefox proper, it's got Gecko as the rendering engine inside that actually determines where things go on the page and stuff. We’re also writing a new one of those from scratch called Servo in Rust. That was also to prove that the language was doing the kind of things that we need it to do. But also Servo is an impressive piece of technology in its own right so it might become its own thing and/or bits and pieces of it are already making their way into Firefox. It’s kind of also a way to improve our core products. That's why Mozilla cares.

CHRIS: I was curious with Servo and Servo is the layout engine. Do you know if there are any plans to write a JavaScript runtime in Rust?

STEVE: That question is complicated. Sort of what it boils down to is that a Git is inherently kind of unsafe by Rust definition of unsafety. It's actually controversial like when I talk to people that work on JavaScript engines, they're pretty much 50/50 split between, "Oh, yeah. Totally Let's absolutely rewrite the whole thing in Rust because we rewrite it every two or three years anyway from scratch so why not use Rust next time," to, "Since it's massively unsafe anyway, I don't see what benefit I would actually get so why not just stick with what we know." It's like very extreme ends. It’s definitely feasible but I don't know if it's going to happen and/or when exactly.

CHARLES: There were two questions that I had kind of to unpack some of the things that you said in there that were just really interesting to me. You said Mozilla plans to incrementally rewrite Firefox in Rust, where it's currently four million lines of C++. Now, how does that actually work where you're talking about swapping out large parts of the runtime with something that's written in a completely separate language? How does that communication happen between those language boundaries?

STEVE: There's this concept called an ABI, not API. It may sound very similar -- Application Binary Interface. What this really boils down to is assembly language does not have function calls. That’s not a concept, that's in assembly. People have come up with, "If I write a function and I map it to assembly code, what's the convention about how I do things like passing an argument and return values? How those all that stuff actually work?"

Because assembly is so low-level, there are multiple different ways that you can make that happen. There's a number of different specifications how to make that work so C, the programming language, has a very straightforward ABI so any programming language that knows how to call C functions, uses these convention at the assembly level to do the function call. What you can do with Rust is you can say, "Please make this Rust function follow the C calling convention," in that way, any sort of thing that knows how to call C functions can call Rust functions directly.

By doing that, you can sort of say like take a chunk of code, write it in Rust, expose a C interface, and then anything that knows how to talk to C, which is virtually everything, can talk to Rust equally as well. For example, one of the earliest production uses of Rust was actually inside of a Ruby gem because Ruby can be extended to C and Ruby knows how to have C extensions. It doesn't actually need to know that it's literally written in C. It just needs to know how to generate the assembly to call the correct functions. That’s actually like a thing. Basically, the process is like write a component in Rust, expose this language independent wrapper, and then call into it like you would in C code.

CHARLES: So it's really, just they're sharing memory and sharing is like right there in the process and there's no overhead for the intercommunication, it sounds like?

STEVE: Yeah, exactly. You could also do all the regular things with JSON-RPC over a socket or whatever if you wanted to. The most efficient way is to literally include it as your binary just like anything else.

CHARLES: Which kind of leads me into my next question, which is Rubyist and Pythonista people coming from JavaScript, one of the reasons we don't like to write in C is because, as you mentioned, they're so sharp so we have safety so that you don't have to worry about memory allocation for the most part, the garbage collector kind of has your back there. You access things by reference so you never have to worry about accessing memory. That's not there but kind of the conventional wisdom is that that all comes with a pretty big cost. It's like really, really expensive.

I know when I was getting into Ruby and I was explaining a lot of the pushback I got from people doing C and even Java, it was like, "It's going to be super slow because all those high-level features that you love so much, you're paying a lot. A lot for them." My understanding is that's not really true with Rust. Is that fair to say?

STEVE: Well, Rust does not have a garbage collector so, yes, it does not pay that cost because it doesn't exist. Now, that also raises a bunch of other interesting questions and basically what it boils down to is a compiler and especially one that has a typesystem, basically asks you to declare certain properties of your code like this function takes one argument only and it's always a string. That’s sort of what type safety means. It kind of like a fundamental level.

One of the ways that Rust uses type safety is to say, "This pointer to this memory always points to valid memory," and you have to be able to demonstrate that to me at compile time. From those couple of sentences, that sounds extremely complicated but it turns out that most programming code is written in a way that actually works this way.

For example, like I'd talk to Yehuda Katz a number of times because we're friends, he also works on the Rust project and he's also well-known in JavaScript and to you all, I would assume. It turns out that the style of Rust code I write is actually extremely similar to the style of JavaScript code that I write is just sometimes there are some tweaks. It is true that those features often do take up a lot of memory and/or rely on any sort of expensive, from a low-level perspective, way of doing things. But it turns out that's actually more of a function of the way that the programming language is made in semantics. You could design a programming language that feels very similar but as very different underlying characteristics.

For example, Closures in Rust, the compiler is smart enough to know that if you don't actually capture an environment. Say you're going to add one to every number in a list. You want to do like .map, pass in a closure that takes one argument X and adds one to every single X and then collect that up into like the map join kind of thing, to collect into a new array. That closure that you had passed a map, while it's a closure, it's taking that one argument X and doing X + 1, so it's not really capturing an environment at all.

There’s actually no reason to allocate a bunch of extra memory because it turns out, it's the same thing as a regular function. The compiler is able to optimize that call away completely to the same thing as if it was a normal function and not a closure, and therefore, you're paying no overhead. Even though, like syntactically, it looks kind of like a closure.

Then you're kind of think of that applied to almost everything in Rust. For example, Rust has methods but almost all of them are actually statically dispatched at compile time, as supposed to dynamically dispatched, where you need to look through some sort of object hierarchy because we don't really have inheritance. There's no way to say like this might result to a colon, this class or this class is super class, or this class is super class so I have to do this runtime look up to call functions that just doesn't actually really exist.

Part of it is through the fact that these coding patterns don't strictly require this stuff. It’s just the way those languages are built and part of it is because as we were building a language, we were extremely sensitive to not include features that would require this really heavy overhead. In a language, that's like a low-level of focus on details, it's extremely hard to talk about the details without code. There's a lot of details, it turns out.

CHRIS: One thing that I'm very curious about and one of the things that drew me to Rust actually is the fact that its typesystem is, I guess an ML typesystem. It is like much more [inaudible] to something that you would see in a functional programming language like Haskell, than you would like a regular C++ or Java.

CHARLES: Now, a Chris-acronym alert. What is an ML-style typesystem?

CHRIS: I'm sure Steve can answer this better than I can but it's a typesystem that uses the Hindley-Milner algorithm for type inference. It does a lot of the heavy lifting for you, in terms of correctness. Is that correct?

STEVE: Yeah, I would say more accurately, ML is a programming language. It's the name of the language so by saying like an ML-like typesystem, he means like a Java-type typesystem. It's like a similar statement but about a different language. I always forget what ML stands for specifically but like OCaml has got ML at the end so like OCaml is one of the languages that sort of the family of ML.

There’s like two branches of functional programming, which of course everything is wrong when you try to organize things this way. Like you could also argue Lisp as a third but there's kind of like the Haskell-style and the ML-style are these two big pillars of functional language stuff and Rust tends to be in the ML sort of family. There’s lots of common features between families of programming languages and all that kind of stuff.

I think the ultimate point that Chris is trying to make is when I say that Rust is a typesystem, I do not mean it's like Java. There is a wide variety of typesystems and they do all sorts of different things and actually Java has been getting increasingly better over the years as well. But it is much more canned to a functional language in the typesystem, which I think is what you were getting at and serves the actual question, right?

CHRIS: Yeah. Actually, I just looked it up and ML stands for MetaLanguage. It is actually is going to serve my question really well. ML was originally designed for theorem improving in math, which is part of why it works really well in functional programming languages. But it also makes sense if you use Rust, how the compiler work from the kinds of things that it catches, like a relatively low effort on your part because it is originally designed to completely prove out a theorem so the compiler is doing that to your program.

That leads to my question which is I recently heard someone else on the Rust core team talk about one of the things that Rust really seeks to improve upon is concurrency and parallelism, which is historically very hard. To do that, you could use things like mutexes or reference counting, which Rust has. But they also lean extremely heavily on the typesystem itself to sort of guarantee that your concurrent code is actually going to run safely.

On one hand, I'm interested in hearing you expound on that but I'm also really curious how the C, C++, Java programmers take to that sort of thing in Rust because as I understand it, that is a pretty novel approach to that kind of problem. I wonder if there's like pushback from the existing low-level systems community on that stuff.

STEVE: I'll do the second part first because it's a little simpler. One thing that I will say is we sort of didn't appreciate over time because we were creating Rust for ourselves, roughly the C++ programmers are working on Firefox, which we had to say for ourselves because I was not literally one of those people but you get the idea, is like assuming that C++ people would be the primary audience.

But it turns out that a lot of people that programming C or C++ are pretty happy with it and they like doing things that way. They're a lot smaller of a population than the number of programmers who do not program of those languages, which is true for any language, basically. The sum of all other people is bigger than your specific thing.

What that means, I think that in retrospect this seems obvious but at the time, it was like hard to figure out or I definitely did not understand this at that time, that most people would come to Rust from not C or C++ than they would from C and C++, just even by virtue of numbers alone.

A lot of the people who are not doing it are not doing it for reasons. They’ve already rejected it for some sort of purpose and the people who are still doing it often are like happy with what's going on. There’s definitely a little skeptical at times of the kinds of things that we can accomplish. Also, our success has been pushing C++ specifically to grow a lot of safety things so we hear a lot of people say like, "In five years, C++ is going to have this tooling that's going to make it also pretty safe, even if it's not as safe as Rust. I'll just wait for that instead." Surprise, low-level programmers are extremely conservative bunch in many instances.

The first part, which is the bigger and more interesting one, the typesystem is absolutely how concurrency works in Rust. This is extremely powerful for a number of different reasons. The first one, and I think the fundamental reason why it's done this way is that typesystems don't have any runtime overhead. When you're in a performance-heavy language, that's really the key.

Originally, a long ago in Rust, we actually had a garbage collector even, like a very long time ago in Rust. The primary goal was always safety and we thought the only way to accomplish that was with lots of runtime checking, heavy runtime, and all these things. Over time, as the typesystem grew, we realized we could use more and more of a typesystem to eliminate more and more of the runtime because types are checked to compile time so they have no overhead cost, which is awesome.

Like Rust references, doing this validation that they're always valid is completely a compile time construct that at runtime, they're literally the same thing as C pointers. That’s one reason why the typesystem is really heavily useful for concurrency because you want things to be safe. We also don't want to slow them down.

The whole point of concurrency in many instances is to get a speed up. If you introduce too many safety checks to make sure that your concurrency stuff works, you lose all the gains that you were trying to get from being concurrent in the first place. Having that like as low-cost as possible is extremely important.

The second one is that concurrent problems are extremely difficult to debug because you need to recreate the exact set of circumstances under which the bug happens. If you have a bug because you have two threads that have a particular access pattern on a particular variable and that's where the bug is introduced, good luck coercing your operating system scheduler into scheduling those two threads at exactly the same way as when the bug happens.

To some degree of the way that you fix a lot of concurrency bugs is by introducing an extreme amount of logging and then just kind of let it run and praying that you hit into the situation that causes the bug. That really brutal and doesn't really work. By using the typesystem and verifying it upfront, you just know it will work at runtime because you've already proved the concurrency property before your code even runs. It’s also just like a better debugging experience, I think in general.

The way that we accomplish this task is extremely novel. I guess I should also say extremely novel to working programmers, like almost all Rust is built off of existing research that has been known in academia for a relatively long time. That's actually one of the places where it gets the name from, it's like taking ten-year old ideas that have a little bit of rust on them, that have found usefulness and bringing them to [inaudible] research.

Anyway, the way we accomplish this basically is the typesystem in the standard library, the way that you spin up a new thread, it has a particular type signature and the type signature says, "Only allow the types to be sent to this new thread. There are safe to pass between threads," and/or like, "Only allow references between this thread and that thread of types that are safe to use across thread."

What that means is that when you try to spin up a thread and you passes a thing that doesn't work, you get a typesystem error. It turns out this is not concurrent safe collection so it does not have the prerequisite types so therefore, you cannot pass on this thread and you're done. That’s sort of like at a core level of how these things work.

Then for example, mutex is a type that does have that property so by sticking with non-concurrency thing into a mutex, now you can share it safely. That means we've guaranteed that the compile time that you'd safely done this transfer between threads and that kind of thing. It’s not just about mutexes but that's sort of the general approach.

The last thing I want to say briefly because I just said a whole bunch of things. I'm sure, I've raised a ton of questions here is that the other powerful thing about using the typesystem for concurrency guarantees is that other people can extend it. If you write a library in Rust, your library will be exactly as concurrency safe as the standard library and as the language itself. It's not like we provide the set of concurrent collections and then we vetted our own implementations and then you're kind of your own or building your own stuff. You can use those exact same types to help guarantee properties on your stuff. Also build alternate threading situations, as well that use the same things and the ecosystem all works together so everything is just concurrency safe by default because it's like a property of typesystems that are being built into the runtime or something.

CHRIS: I know that recently, there's been a lot of, I guess excitement about this library called Tokio. It's not like there's future that kind of like promises in JavaScript, then there have been abstractions just kind of consistently being built up but it seems like Tokio is the next step and it's building towards a whole stack of higher-level concurrency things. Is what you just said enables that kind of thing to happen?

STEVE: Yes. Tokio is using those exact same typesystem features in order to guarantee that when you have a chain of promises, to use the JavaScript terminology instead of future things, that you make sure that they're safe. This is not literally implemented yet but Tokio, for those who are not paid hyper attention to the Rust space because this is a cutting-edge, the library is gearing up for an initial release in the next week or two.

Soon after you hear this or maybe right before you hear this, it's just going to be released. It’s extremely cutting edge. But in some ways it follows sort of the node model of concurrency. There's event loops, you chained together, we call them futures, you call them promises together, you put that pile a future chain and do an event loop and watch the concurrency kind of go.

One example of how Rust can do cool things is you could -- this is not implemented yet but it will be in the future -- run, let's say, five event loops on five different threads. Then you just tell the framework, "Please run this future chain onto one event loop. I don't care which one," and then it will automatically load balance across the five threads and five event loops because you've guaranteed the compile time that everything is safe to pass between threads so we know that that's just trivial to do and therefore it's like not a big deal. We can add those heavy duty features without worrying about introducing very subtle bugs, which is really cool.

CHRIS: That kind of leads me to my next question, which is at The Frontside, we are pretty into web development, in case you didn't know. I am someone who follow Rust a lot and I find it very interesting. But for the most part, I don't have a need to do systems programming on a regular basis. I also wouldn't even really know where to start, if I wanted to do systems programming.

As I learned Rust, I tend to always gravitate towards wanting to do things that I would probably do in Ruby or Python, like write the back-end for some web app or something. That goes okay but Rust is very much still in the process of building those abstractions to the point that it's relatively digestible.

So I have a couple of questions. One is do you see Rust being a thing that would be used by web developers a lot more broadly and two, how would you recommend that people like me who aren't really familiar with systems programming start to really dig into Rust on a deeper level?

STEVE: I would like to think that web programmers will use Rust more often and to be honest, originally, I was extremely skeptical of this. But it's been changing rapidly as time has gone on. Part of that is because as we've gained more experience, actually in programming in Rust, the fact is Rust used to be a lot less ergonomic than it is and now it's fairly ergonomic and will only get more so in the future. That's something that web people or at least, I come from Ruby so Rubyist care a lot about ergonomics, maybe more than anything else frankly.

I'm not sure it's the first tool that you'll reach for but I do believe that sometimes, it makes a lot of sense. As one example that I will use, there's not a whole lot about this but basically, npm has started using Rust on the server side for powering the registry. They have three services in production now but they were basically like JavaScript as a language we all know what is the best language for doing this. We have a service that needs a little more oomph so maybe let's rewrite that in Rust instead and use it for those kind of things.

I think that there's a lot of situations for web developers where they don't realize they have the power to make things faster without just adding on more servers. I think that's kind of like a compelling sort of [inaudible]. Any sort of background job like any sort of job queue thing is like often better written in a faster language but you would not reach for that faster language first because traditionally, those faster languages have been terrible to use. I think we continue to win on the ergonomics and continue to win the libraries that web developers will reach for Rust like more often than not.

In terms of the learning rest on a deeper level, I think that one of the initial things and sounds like maybe you personally are a little past that but maybe not the people who listen this podcast is that I do think that sort of building the things that you would normally build in Ruby or JavaScript or Python is the good first step. For example, right now Advent of Code has been like a really fantastic way of having these little programming projects. If you haven't seen AdventOfCode.com, it's like every day in December up until Christmas, there's a new programming project that you can build the thing in.

I've been doing those in Rust and that's a lot of fun and it's a good way to practice and gain some basic literacy. But after that moving at a low-level stuff, my personal thing and I know something you've expressed interest in the past is my side project is building an operating system in Rust. More so, than just that the pitch is, "You've written JavaScript before. Let's write an operating system together. Here is this companion book and I'll show you how," and that's called intermezzOS. It’s like I'm basically trying to rebuild an operating systems curriculum but in Rust instead from nothing, like we start off with assembly code and move up into Rust code.

CHARLES: Now, you can't even use anything like all the things that we've been describing like threads, kernel level callbacks. You get none of that, right? You have to implement it all from scratch. You can't use POSIX or whatever. You know, 90% of your code ends up going through.

STEVE: It turns out that and it's sort of like for reasons that hopefully I'll be able to fix in the future, you need about like 200 lines of assembly code before you get into Rust and then you basically don't need to use assembly again, really. It’s not that big of a barrier in terms of [inaudible] things and its copy-paste stuff that I explained extremely heavily so it's like totally an accomplished real thing. Then you're in a real programming language and you can do more normal things on top of it.

But one thing about that because it is my side project, the kernel is actually farther along than the tutorial is and I actually need to find some time to write more of the freaking tutorial but this is kind of my personal long-term project over the next, let's say, decade and to have a completely free and open source tutorial for you to learn about operating system developments. That’s one of the things I've been doing.

Another one that I think that is really extremely useful is once you gain some amount of literacy on this, you can actually start to learn more about how your regular programming language works. I've been giving this conference talk recently. It's called 'Exploring Ruby Through Rust', and I'm like, "Once you know this low-level stuff and you gain this literacy, you can look at the source code of your language as interpreter and learn stuff about it and you can contribute to it maybe even."

Maybe that's not the most practical thing or whatever but now that I've spent a bunch of time with Rust, I understand Ruby on a far, deeper level than I ever did before because now I'm not afraid to go poke around in the internals and learn how it really works under the hood and I understand what those internals do far better.

Maybe five years ago, I could have told you like, "Ruby is garbage collector. It's extremely basic. But I don't really know what that means." And now I can be like, "Ruby has this mark and sweep generational garbage collector. But it's not compacting or concurrent yet but maybe in a year or two. Now, that's not just a bunch of buzzwords because I have this low-level literacy."

CHRIS: Yeah, that's definitely something. I forgot about but every time I go learn something in Rust and initially this happens a lot. Every time I do that and I go back to JavaScript or something else, I find that Rust inadvertently taught me something about the language that I actually work on every day. Especially, when it comes to things like references, values, and the difference between them and debugging weird prototype behavior in JavaScript became so much easier after I had spent some time working with Rust and had had to like actually deal with passing around references or dealing with life times or having the compiler yell at me for a lot of things that I thought were totally normal.

Then I'm going back to JavaScript, it's like, "Wait a second --" Suddenly a lot of these pieces are starting to fit together and before what was just as weird mystery, now I can totally see what is happening and start to think about how to fix it. Even though I don't even have the same tools that I do with Rust, it still is extremely useful from that perspective.

STEVE: That's awesome. I'm glad to hear it. That’s how I definitely felt with Ruby for sure.

CHARLES: You know, in terms of actually using it for day to day stuff, is there other plans, is the ecosystem already supporting things, say, a web framework? Like a low-level web framework like Sinatra or Express or even higher one like Rails.

STEVE: I guess, like you've already qualified it as web stuff. But I would say, in a broader sense, whether or not Rust is ready today for you, it depends entirely on the ecosystem. I feel like 80% is productive in Rust as I did ever in Ruby. But that's only if there's a library that I don't have to rewrite myself because it doesn't exist yet. That number is actually growing rapidly so I just look because it's like the end of the year and our package ecosystem is actually doubles. This is a request from earlier. I didn't expect Cargo so Rust basically has bundler or yarn/npm built into the language itself. We distribute it with Rust and we have all that great package ecosystem shenanigans.

Another great example of Rust over a language like C is the tooling. Basically, what happened was Yehuda and I kind of showed up in Rust world and we're like, "Why are you still using make files. We know a better way."

And they're like, "Okay." Then he builds the equivalent of bundler for Rust. Then everyone's like, "Oh, yeah. This is way better. We're not using make files anymore."

The tooling situation is very familiar to a dynamic programming language person because we literally had the same people write the tools. That also means you can share packages freely and briefly so operating system development thing is totally intense to be able to use your package manager to download packages to help you build an operating system.

For example, X86 has custom assembly instructions that you need to use when interacting with the hardware and someone has already built a package on [inaudible] that wraps the inline assembly up in a nice to use Rust functions. I can just include that package and use it when building my operating system, which is totally mind-blowing. The npm is sort of feel into OS development is just real intense and cool.

Back to the ecosystem thing, though. For web application specifically, it's good and also bad. There’s actually multiple different web frameworks already at different levels of comparison. For example, you have Nickle which is kind of like Sinatra and you have Pencil, which is kind of like Flask and Python, which is also kind of like Sinatra. Then you have Iron, which is kind of like expressed in JavaScript.

There’s also like I know of at least two. One of is has been worked on but it's not been actually released. But the code is at least open source yet. I know a second that is being developed fully in private that has not had any public release yet. Then when the Tokio stuff comes out, People are going to be building new frameworks on top of the new async shenanigans and/or porting the async stuff into the existing frameworks.

We kind of have a lot of options but there's also a lot of churn and activity and stuff going on in that space so that either terrifies you or makes you enthusiastic. They're basically is like that. We definitely don't have a Rails yet. I don't think that's because a Rails will never exist but because it's a much bigger project to build a Rails than to build a Sinatra.

CHARLES: Yeah, and you just need those foundational pieces there in place before you really want to attempt that.

STEVE: And I think Tokio is the real foundational piece and it's just taken us a long time to put it all together. The initial tests in Tokio, we could do a 'Hello, World' benchmark like the tech and power benchmark. Some of you are already familiar with those things, or not, they're like 'Hello, World' benchmark. We actually got faster than they are fat than all of them. It just edged out the fastest Java, which is currently the reigning benchmark on it. That’s like extremely compelling.

Even if after all this stuff is built on top of it but it's taken us a while to build those foundations and we're just getting that point like Tokio is going to have a release, hopefully before Christmas. I've been assured by the end of the year and then people are going to build stuff on top of it and it's just going to explode from there.

Here's another little interesting pitch. I'll give you for this, is that one of the things I like about Rust on early ecosystem is it means that if you want to be that person who built the library that does X that everyone uses, there's lots of opportunity in Rust world right now. Where there's a lot of foundational libraries that you could be the person who wrote that thing when everyone knows and loves and uses. Like JavaScript is still kind of there.

In Ruby, every library basically exists already so there's no more room to build a foundational thing. But if you're someone who likes working on open source and that story is compelling to you like getting involved in a younger ecosystem, it means that you can have a much larger impact. I maintained the [inaudible] library that things used. The only reason that's true is because I was around before we had one and then Yehuda wrote the initial version and now, I'm maintaining it. There’s tons of space out there so if writing a web framework is the thing that's interesting to you, Rust is a great place to explore and actually doing that at the moment.

CHARLES: Steve, one of the things that I know you do is you actually write the Rust Book. I heard that you're also in the process of rewriting it along with Carol Goulding, I believe. I was wondering if you could tell us a little bit about that.

STEVE: As part of this Steve getting the job right in the docs on Rust thing, I kind of working on lots of stuff so up to Rust 1.0, we knew we needed to have some long form explain all the things that Rust so that became what's called the Rust programming language which I named so because the C programming language and the C++ programming language, the names of the foundational books for those languages so I wanted to continue kind of in that tradition.

But there is some problems with that which is I'll say that I'm a little harder on my own work than I think other people are so I hear people tell me all the time that they love the Rust Book and that it's like one of the best programming books that have ever written. But I think it's not that great. The reason why is also because I just know that the way in which I wrote it.

You have to remember that Rust 1.0 happened in May of 2015. We were working on language for six or eight years before 1.0 happened so there was lots of changes, language is changing on a daily basis. Now, it's super stable like super, super, super stable. But what that also means is in some like deeper philosophical sense, nobody had had experience programming in what really was Rust yet because we were still like finishing building it's so like how do you write a book on a language that like the precursor language is what you're using and you're trying to see like what is it going to actually end up being like at 1.0. Because it's not like we can just say, "It's done. Now, go write a book, Steve and then we'll release it at that time."

The circumstances in which I wrote the original book were I had a very intense deadline of this has to be done by the 15th of May. While the language was coming together, it takes a couple months to put together a book so I had to make sure that the stuff I was starting I would need to go back and re-fix. That also means that I was like much more vague in some places where pieces were still falling into place and you're like, "This is definitely going to be the same. But this might change so I'm going to leave that part off," and then I just have to plow through because the deadline.

All those things coming together means that I kind of put together this book that while good and I'm proud of the work that I did, I can do much better. At this point in time, we now have a full year and a half after Rust 1.0 has come out. I know the struggles that people have when learning Rust. I know the ways in which they succeed or fail and I've talked to a lot of people so I'm sort of rewriting the book now, bringing that knowledge and understanding in as well as the fact that the language just been around for a minute so it's much easier.

As part of that, I brought on Carol. She goes by Carol Nichols or Goulding. She both has her maiden name and her married name. She's been one of my best friends for a very long time so I'm extremely happy that she's my co-author on this book. The two of us together and working on doing the rewrite, I think that it is possibly the best thing I've ever done or worked on as far as books go, like I'm extremely happy with it and you can read it online right now, if you want to and see if I'm right or wrong about that.

But I think it's a far better book than the original book was. It’s actually going to publish at No Starch as well. We’re donating all the proceeds to charities since we're being paid to actually write the book in the first place, like [inaudible]. It's going to be a much, much easier and better way to learn the language, I think as well.

CHARLES: If we want to check that out, where can we find the new version?

STEVE: I'll give you a link to put in show notes or whatever as well. But it's Rust-Lang.GitHub.io/book. There’s also just like a book repo in the Rust Lang organization on GitHub. All things in Rust is being developed fully in the open so you can read the drafts and see what's been done where. We're getting towards the end, slowly but surely so I'm hoping that's going to be done relatively soon.

CHRIS: Well, I'm looking forward to it.

CHARLES: Fantastic. Sounds like the documentation is there. It's excellent. The community is there. It's excellent and from what I'm hearing like the kind of the tower of the ecosystem is really being built up. It's not as high as a bunch of other places but it's definitely high enough to jump in and get your feet wet. If you're you know coming from almost any walk of programming.

STEVE: It's a lot of work but we seem to be doing good.

CHARLES: All right. Well, thanks for stopping in and talking about this with us, Steve.

STEVE: Thanks so much for having me. It's been a lot of fun.

CHARLES: Yeah, and now Chris, we do need to kind of figure out what is going to be our Rust project here at The Frontside.

CHRIS: I'm up for that challenge.

CHARLES: Yeah, that'll be some Christmas homework. All right-y. Take care everybody and thanks, as always, for listening. We’ll see you next week.