I realize it’s been a while since my last newsletter, but that’s shortly going to change. I’ve been working on a project that’s spun out some very interesting ideas and discussions and I plan to share them with you all fairly soon.

In the meantime, I created a Rust Editor Experience Survey1 to get some feedback on how Rust developers edit code and what parts of their editing experience they like and dislike. I’ll make a followup post with the results of this survey soon.

Consider taking the survey even if you have only been a Rust developer for a short time, I’m very interested in all perspectives on this. It tends to take about 4-5 minutes to complete.

Like any technology that has been around for decades, shells have a lot of features and responsibilities that were gained through gradual accretion. If a shell does almost everything you want except for this one thing I need right now, there’s a heavy temptation to add that one thing and not reinvent everything from scratch. Do that long enough and you end up with some things that a shell does well, and lots of things it does less well but, you know, well enough.

Some people have looked at the deficiencies of the shell1 and thought “How do we make this better?” Both fish shell and nushell are attempts to take what shells do and improve on it by making the shell more capable and its scripting language less crufty. In contrast, I think we should strip away responsibilities from the shell and hand them off to tools that are better suited.

Let’s take a look at what shells do, and see if we can tease apart what their strengths are and where other tools have come along and supplanted them. Then we can come back and see what’s left and see if there’s something better to be built. It’s usually a good idea to learn what something is doing before tearing it down.

Subscribe now

What capabilities do shells provide?

A way to spawn processes and pass startup options

In functional languages like Haskell, applying a function to its arguments is the most common and important operation you’ll do, so the syntax is very lightweight. No parentheses, no commas, just spaces between the function and its arguments. f a b is the syntax for calling a function, which is equivalent to f(a, b) in most imperative languages2. Removing that little bit of syntactic noise is worth it because function application is so central.

Likewise in a shell, spawning a process and passing it options is the central activity. Compare the way we invoke commands in a shell:

$ echo hello world

with how we call a process in Rust:

std::process::Command("echo").args(&["hello", "world"]).spawn();

Shells have reduced their overall syntactic flexibility in order to make spawning processes maximally convenient. Simple things like arithmetic and if/then conditions are syntactically very noisy in shell scripts, but spawning processes and passing it argv is about as easy as it gets.

A way to navigate the filesystem

In a shell, using cd and ls are about the most common things I do. Specifically, cd is so common we might not often pause to consider that usually it’s a shell builtin and not a separate executable on the path. If you write a naive cd program and invoke it in a shell, it’ll just change the directory of the subprocess and then exit, leaving the shell’s current directory unchanged.

I have a graphical file explorer on my system, but I don’t often use it. When I want to navigate somewhere on the filesystem I usually want to invoke some commands in that directory immediately afterward. It’s much easier for me to just cd some/directory than to click my way there in a GUI.

A default user interface for simple programs

Don’t want to think about your user interface? Just output text. Most languages3 make writing to stdout very easy, so it’s a very low friction way to allow your program to communicate with humans. For many programs that don’t need heavy interaction with the user, you accept arguments through the command line and output to stdout.

Imagine needing to construct a GUI text box in order to display output to a user. That would introduce a huge amount of overhead to creating small programs and testing things out. Instead the shell provides the default UI: text scrolls up as the program executes, which the user can scroll back through if they want. Then the shell returns to the prompt when the program exits.

There are downsides to this universal interface: it relies on the user having a gigantic library installed in their brain already: the entire human language your program is written in plus the ability to read the written form of that language. Most everyone can drag a slider or click a button, but a text interface is completely useless to someone who doesn’t know the language or how to read.

A way to connect the output of one program to the input of another

Besides spawning processes, this is the second thing the shell makes delightfully easy. The boundary between the pipe character and shell scripting language itself is a bit blurry, but it’s worth calling it out separately because all by itself it makes shells significantly more capable even without other programming language features like variables, loops and conditions.

It works really well because it turns out it’s the same thing as function composition. There are entire categories of math devoted to how much power you get just by composing functions. To make it explicit, here’s three things that are alternative syntax for the same thing:

cat /dev/urandom | head -c 200 | base64

base64(head(cat(“/dev/urandom”), 200))

cat(“/dev/urandom”).head(characters=200).base64()

Technically, processes return a second bit of information, the exit code, which makes this composition a little fiddly. But the basics are the same.

A resizable 2D grid that can display characters in color

This is getting into the border between the shell and the terminal emulator. Really it’s the terminal emulator that interprets various escape sequences and displays color and implements raw mode which enables writing to arbitrary character locations. But if we are willing to conflate shell and terminal emulator momentarily, command lines have the capability to create rudimentary graphical interfaces.

Sure we have to use some weird box-drawing unicode characters, but the limited graphical capabilities on par at least with xterm are expected by many common tools. I mean, heck, even systemd status uses color to indicate the status of services:

A simple programming environment

After piping, there are some basic programming features like variables, for loops, if statements etc. This is what tempts you to solve problems in a shell script rather than back up and realize you should switch over to solving the problem with a general purpose programming language.

The friction is so low though! Maybe by the time you’ve crafted half of the solution to your problem by piping shell commands together, there’s a little bit of sunk cost calculation going on. Do I really want to rewrite this entire thing over again in a completely different language, or should I just go look up the difference between [ and [[ one more time and bang this thing out?

Personally, I think this is the original sin of the shell. It gives you just enough rope to hang yourself. Once you’ve solved your problem in a shell script, doesn’t it make sense to just commit that as your code? And once you’re committing your shell scripts, doesn’t it make sense to add a linter like ShellCheck to your CI pipeline? This is the point at which the slope begins to slip.

A persistent global history buffer of previous commands

This might seem like a minor point, but I regularly use my shell command history as a very limited second brain. Especially with fancy history autocompletion, this becomes a really powerful way to keep arcane program invocations stored… somewhere. And, in contrast to curating and storing your useful commands in a wiki or something (probably copy/pasting it back and forth), searching the command history automatically puts the command exactly where it needs to be to execute it immediately.

A lowest common denominator execution environment

Because it’s always available on *nix systems, shell scripts offer a language to bootstrap other features from. MacOS won’t execute ELF binaries, but if you wrap your downloader in a shell script like rustup does, you can intelligently interrogate the host system and download the appropriate binary file for it. This makes distributing tools like rustup much easier.

Build systems commonly use this strategy to decide which tools are available to execute with. For example, autotools is essentially a gigantic shell script that checks the capabilities of a UNIX system extensively in order to determine the best way to compile a C program.

What are shells to a human?

So far I’ve talked about a bunch of technical capabilities and features of shells, but one layer above that is what roles shells play at a human level. By that I mean, what is a human trying to do when they reach for a shell? If there are two equivalent ways to do something, when do we choose to use a shell to do it?

A place to execute non-GUI programs

You could make the argument that the dividing line between an app and a program is whether you invoke it from the shell or not. Programs are available on your $PATH, not your desktop. They’re words you memorize and type out, not icons you click. They’re intended to be tools used in combination with other tools, not self-contained applications. The shell is how you access and configure programs like this.

A sense of location

When you cd into a directory, you are navigating there mentally. This is repurposing the brain’s well developed spatial awareness. From the computer’s point of view, absolute paths and relative paths are equivalent. From a human’s point of view, relative paths are a natural way of specifying directions that originate from a particular starting point.

You can see this effect in tools like git, which looks at the current directory to decide which project you’re working on. When non-technical users interact with computers, they usually don’t think of themselves being in a particular place in the filesystem. The location is either the desktop or an app. When you’re in the shell, you’re wherever your current directory is.

A place to tinker and figure things out

This is enabled by the REPL nature of the shell, and the interaction between the UNIX philosophies of everything is a file and the capability of piping data around. When Brett Victor talks about visual feedback and showing the data being more important than showing the code, it becomes obvious that the shell provides a crude form of that.

The arcane invocation you’re building up piece by piece may be difficult to understand by reading it, but that’s because you’re not reading the code, you’re running the code and seeing what it outputs. The process is to incrementally build the code to get the answer you’re looking for.

A stylized workflow in this mode is something like:

• Do a basic query, see what the output looks like

• Pipe the output into some program like awk or jq and start digging for the piece of the output you need

• Use Ctrl+R and ⬆️ to modify the previous version incrementally and re-execute

• Once the query portion is what you expect, simulate the write step by adding an echo that prints what command would be invoked

• When you’re comfortable with that, swap out the echo for the actual write operation

• Walk away. Don’t commit the result to source control or refactor, this is ephemeral write-only code for a one-off job

A place for hacker aesthetic

I’d be remiss if I didn’t mention that things running in the shell have a certain aesthetic that many developers find pleasing. This demo for the terminal UI kit tui-rs is undeniably cool:

Why is that? If someone made the same interface in a GUI toolkit, it wouldn’t look nearly as cool to me. I say this as someone who has used a shell daily for almost two decades, the novelty somehow hasn’t worn off.

If you want to build something with a terminal UI like this, your target audience is necessarily programmers and other technical users. Non-technical users are not going to put up with this stuff. The barrier to entry the shell puts up means programmers can have their dorky tools and make them the way they want, without anyone complaining that it looks ugly.

This also functions as a kind of costly signal: if a tool has an intricate terminal UI, it’s a good bet the creator of the tool cares about programmers. The code needed to write a terminal UI is just as complex as the code needed to output a basic html interface or use a standard OS GUI library. The difference is purely in the rendering layer, and the implicit message is “I’m rendering this for you, fellow hacker”

What replaces the shell?

This is a big topic and there are lots of answers, but I think we can point out some general replacements for aspects of what the shell does

Terminal UIs become browser interfaces

This trend is already happening. Terminals can be pressed into service to do graphical output, but human input in a terminal interface is abysmal. Browsers do input and output about equally well (that is to say: mediocre, but not badly).

Explore and tinker in a programming language’s REPL or code notebook

I mentioned this in the previous post: I’m a fan of the notebook concept over REPLs. But even a REPL in a good language is better than a REPL in a bad language (shell script).

Use IDEs or code notebooks to provide task-oriented location

IDEs also provide the metaphor of “project directory is where you are”, and code notebooks like jupyter do as well. You can change the current working directory in a notebook, but it rarely seems necessary because the implicit assumption is that the notebook file itself is in the directory you want to be in.

Minimize serialization logic and lean more heavily on library interfaces

This is a larger topic than I want to get into here, but the basic idea is that I think the UNIX toolbelt approach of making small, single responsibility tools that interact by piping plaintext to each other was a really good idea in the 70s, but we understand the problem domain better now and have myriad solutions for it.

Fundamentally, small tools communicating through plaintext has a few weaknesses:

1. It makes serialization, de-serialization and validation code a larger proportion of what your program does. It also means you lose much of the benefits of a language like Rust with strong typing. Serialization and de-serialization are the boundaries where the type checker can no longer help you, so you want to minimize those, not maximize them.

2. For a tool to really be usable in this way, you need to commit to backwards compatibility. This means tools have a limited ability to evolve before they become ossified in order to prevent breaking things.

awk, tr, and cut are all programs for parsing ad-hoc underspecified serialization formats. Let’s do away with the need for them by using the much richer interfaces provided by modern programming languages’ type systems and the tooling they have for expressive and flexible serde.

The alternative to small focused binaries communicating through plaintext is small focused libraries with richly typed interfaces.

What’s left for the shell?

I think this leaves the shell with:

• the default interface for simple programs

• a place to spawn processes and provide configuration arguments

• a cross-platform bootstrap environment

I hope that as WASM+WASI gains traction, it can take over the cross-platform bootstrapping capabilities. Or possibly something exotic like Actually Portable Executables will take root. But I don’t think shell scripts are a good long term solution for “write once, run anywhere”.

Unleashing Unsh

That leaves the default interface and process spawning responsibilities. To that end, I’ve written unsh (short for Un-shell). All it does is provide a loop where you can execute programs, provide them with an argv, and see what their output is:

It doesn’t have a programming language built in, and it doesn’t allow piping output:

Is unsh useful? Well, sure if you need those two things, and nothing else. It’s designed to hint strongly to the user that if they want something more complicated, they should probably use a real programming language.

If you want to be hardcore, try switching your login shell to unsh and see how it goes. I think people may be surprised at how far simple process launching gets you, even without the rest of a shell’s capabilities.

Leave a comment

Share

On tech publishing:

I had become really disillusioned with tech writing and with publishers pushing out content, which is like basically content marketing. So these database vendors, or like, let's say …

Read more

In acausal trade, two agents each benefit by predicting what the other wants and doing it, even though they might have no way of communicating or affecting each other, nor even any direct evidence that the other exists.1

In grad school I spent a lot of time writing C to try to maximize throughput on some algorithm benchmarks for my thesis, and I really came to enjoy the language a lot. There is just so much control over exactly how things execute and how things are laid out in memory. I also spent a lot of time in grad school coding in Haskell, because I love learning new type system features and Haskell basically has an endless supply of those.

This left me in the strange position of simultaneously loving a low level language where you could almost feel the bits directly and which has almost no abstraction capability at all, while at the same time loving an ultra-high level, lazy, garbage collected language with a never-ending tower of abstractions. I constantly found myself wanting some aspect of the other language:

• C is just missing some straightforward, obviously good language features like algebraic data types, closures, type inference, and generics

• Haskell has this garbage collector and runtime and yet you still regularly run across space leaks due to laziness!

Another grad student and I briefly investigated writing a non-garbage collected functional language but it looked to us like there wasn’t enough original research left to do. All of the hard parts had been thought up before, just nobody had combined them into a nice polished language. The closest thing was Cyclone, a safe dialect of C that pioneered the use of linear types to do safe memory management (which eventually inspired the Rust borrow checker). But Cyclone was also very obviously an academic research language, there was no big push behind it, and new code only got added to it when someone wanted to write a new paper.

So, for me, there was this puzzle piece that was obviously missing, and it looked like someone would create it any day now, all I had to do was wait. It turns out, Graydon Hoare had started work on Rust in 2006, but I wouldn’t have recognized it as that puzzle piece at the time. Rust still had garbage collection and an erlang-style green thread runtime back then. It wasn’t until after Patrick Walton’s 2013 blog post that Rust shed its garbage collector and most of its runtime and became the language as we think of it today.

I found Rust shortly after that in 2014 and, I guess the acausal trade was complete? A bunch of really smart and dedicated people had done all the hard work of creating and polishing a very nice language that satisfied all my requirements. In return I, uh… did nothing. Ok well maybe it was less of an acausal trade and more of an acausal gift2.

But, I recognize this is not a super typical route to discovering Rust. Although, since most people writing Rust are doing it in their free time rather than at their jobs (currently), there might not actually be a typical way to get into Rust. I recently spoke with Tim McNamara, author of Rust in Action, and he talked about his own route to the language:

My career path has been really, let's say non-linear. Quite weird.

I have an undergraduate degree in philosophy and German.​ A BA from a university that no one’s heard of at all because it’s from New Zealand. I also got a master's in public policy as well for various reasons. I was destined to be an official basically.

I spent several years as a data scientist doing some really quite neat things with text and search and so forth. I worked for Canonical, the company behind Ubuntu for a couple of years as well. But, I just really enjoyed playing with data, so now I work as a senior data scientist at a small (actually relatively large now) consultancy here in New Zealand.

I got into Rust sort of sideways. I've been a Python developer, I guess, for about a decade. Five-ish years ago now, it just sort of started kind of creeping up on me, this idea that I'm leaving a lot of performance on the table. I'm a relatively proficient Python developer, but, you know, people sort of always whisper that, “C extensions are where it’s at.” Like if you're using Python you're always missing out.

But I was always told that like… C is dangerous. You should never basically write C extensions because that's reserved for wizards and experts and you're not an expert. So you should not even try because you’ll create some sort of segfault, destroy your whole computer, or destroy the world (or expose your users to some pretty needless security concerns). So, I was always terrified of C.

My first playing around with Rust was doing natural language processing. Carving up a lot of text and converting it into big arrays of matrices of numbers, which was the input to a lot of these machine learning models which I can reuse in the Python world.

I got a 100x speed up, like, just by doing this Rust thing.​ And also, memory is like megabytes, instead of gigabytes. There was this massive mental shift that goes on, “Well actually, Rust is really good.”

At the same time, he had a bubbling interest in what was going on at the lowest levels of software

This is actually quite magical. My computer can talk to almost every other computer in the world and we just expect it to work and like… how does it work? It's so commonplace that it almost feels like we don't really consider the complexity that goes on when you click the enter key on a keyboard.

And yet, it actually initiates a massive cascade of events that enables pixels to be rendered on the screen or to do whatever needs to happen when that event occurs.

This led him to start writing Rust in Action, where he was able to explore some of these questions in depth. Sometimes one investigation led him to further topics and he just kept digging

The database chapter started out with the files chapter. I was like, how does a computer represent these data structures actually? How are they represented in a serial string on disk? In memory everything's hierarchical and there’s pointers everywhere and so forth. So we kind of need to flatten everything out.

How do databases do things like ensure that your data is safe, even if the hardware, like the underlying platform, might lose power or become corrupted? You know, if you can't trust anything, how do you ensure that your data is safe?

So the full version, which is kind of still sitting on my computer, it's like this full, really interesting B-Tree structure and like, it's quite elaborate. So we implement pretty cool semantics on top of a key value store.

Writing the book was the impetus behind exploring these topics, but the book format has limitations that he ran up against

I always have a little bit of a lament—kind of like a little bit of regret that I wasn't able to insert the full version of things. I can't explain everything there is to know about systems programming in a book about Rust. Like, I can wedge a lot in there, but… I can't write every book in one book.

Turns out that if you’re writing a chapter, readers are not really interested in, like, 90 page chapters. I don’t think they’re going to have the patience for me teaching them everything about databases and database internals.

You kind of need to whittle it down to only introduce one new Rust feature in every example. I can't introduce four new Rust concepts. I couldn't introduce like closures and Result at the same time. There are definitely people who have come to me and been like “Well this is a stretch.”.

I recognize that I'm not going to be able to write a single piece of tech writing that is going to be perfect to everyone. It’s just something that I've kind of had to acknowledge.​ I started out with this very grandiose idea that I could write the very best tech book, like, in the world.

I felt for myself that I would really produce something extremely high quality. And, I think one of the hardest things has been that I've constantly felt that it’s not quite good enough.

There was this case where I thought a section would be a couple pages, and then like I’m a hundred pages in now. Thankfully for my readers, my editors were very pragmatic and said “Tim, you gotta… this is crazy.”

We also talked a bit about the audience for the book, and the message he wants to convey through it

If you're anything like the person that I was a couple of years ago, like inquisitive about how computers work or really interested in understanding, like, “What is a pointer?” “What does it mean for a reference to be dereferenced?” things that are kind of like this tacit knowledge, which is assumed by people within the Rust community. That’s the kind of stuff you really get.

My intention is to put people at ease when they enter the Rust community. People talk about affine type systems and a whole bunch of jargon. This is kind of the resource that people can look to as giving a base level of understanding when someone says “We’ve got to use a syscall for that.”

If anyone is looking at Rust and thinking “Man, these people are really smart and I’m not smart”. That is the mentality I want to bridge and say “No, actually, you’re welcome. You’ve got a place at the table.” The Rust community is available to you, and Rust is a resource that you can use to keep your programs fast and your users safe.

Given he’s a data scientist, I asked him what he thought of doing data science in Rust

I think that Rust is going to struggle with the data science for sort of two reasons. I think primarily the ergonomics question is the problem. R and Python are always just going to feel a little bit more fluid.

I'm a very strange data scientist in that I have invested very heavily in understanding the software tools that I use and the world of software that I inhabit. But most data scientists do not take much time to think about software engineering as a… as a thing3. There's this new term called data engineer, which is probably where I sit.

Now, within Rust, I think things like TensorFlow should probably be written in Rust. I would love to see Kafka implemented in Rust and, and ZeroMQ and these messaging systems and log aggregation and kind of a lot of this boring stuff that enables data to be stored and transmitted and analyzed.

I think Andy Grove has done a really amazing job. He’s been shipping Rust inside the Apache Arrow project. So Apache Arrow was originally an attempt at creating a kind of peace between the Python and R worlds of data scientists by creating an in-memory format for being able to kind of send data structures between Python and R without costs.

Andy had this idea of reimplementing what we would have a couple of years ago said Hadoop, but more recently we would say implementing Spark with this project built on Arrow called Ballista.

So, these kind of big, heavy pieces that are infrastructure for data science, I think is where Rust is going to find itself very welcome. I think it will be very challenging for Rust to try to be Python. I think that Julia has a much better story as like a convenient, fast language for interactive use. You talked with the creator of the Rust kernel for Jupyter notebooks. And I think that is an enabler, but I don't think it's enough. Being able to interact and introspect variables and so forth is important too.

Apart from the book, where does he use Rust?

It was interesting that you interviewed Alexis because creative coding is one of the things that I also play around with. There’s a couple of YouTube clips where I go the tutorial for Nannou, which is the framework you were discussing before. So that’s kind of like side projects.

At work, we’re slowly going through the process of optimizing. For like utilities or stuff that is being used repeatedly, I will gently nudge work, “If we want something that's reliable and easy to deploy and can use like a micro instance rather than like some big VM on the cloud”, I’ll nudge them towards Rust. So, it's been a process of finding things that are very discrete and slowly kind of finding a spot for Rust. Either as a replacement or as like a supplement to things that are typically implemented first in Python.

Typically, it’s in CLI utilities or ETL which is the data transformation pipeline.

What is he excited about in either the language or Rust ecosystem?

I honestly feel that WebAssembly plus WASI is going to be the standard way that we ship software in some future state. That’s where I feel most excited for Rust.

One of the things that I'm delighted to see is the way in which the Rust language developers have embraced the async discussion.

I would love for that mental tax of async to kind of evaporate. I don't know if that's actually going to occur. I'm sure that it won't be perfect, but all the signs are there. The people taking the right approach like Niko and Ryan Livek and others are very focused on creating a structure that would enable Rust to become the most pleasant async programming environment that the world has.

On his development setup and tooling:

I have a surprisingly boring setup. I’ve got a Dell XPS 15, and VSCode with rust-analyzer.

Like, I kind of wish I was a wizard, but really I found the Rust tooling, just the default stuff, to be really good. I did pay for CLion, but one reason I don’t use it is that I do a lot of stuff on Twitch and YouTube. And I want to make sure that my content is really accessible. And if I use proprietary paid technology, which a couple of my earlier videos did, it kind of creates a barrier that I don't think should exist.

He also had some advice for newer Rust developers

It's okay not to use idiomatic Rust. If you're a Java developer or if you've done C# or whatever other programming paradigm you're from, you are going to naturally write Rust that looks a little bit like Java or C#, you know, that's just what happens.

You are going to write getters that to me look hideous, like you're going add, getThing. But that's my problem as someone who's an experienced Rust user. I am going to squirm a little bit when I see your code and that's fine actually, because you do not need to learn everything in one go. Don’t aim for perfection if you’re starting out.

Finally, to appease the Rust Evangelism Strikeforce, I have to include some effusive praise for the language:

Normally I expect a technology to have this two year honeymoon phase, and then things start to really irritate you. I just am waiting for this disillusionment to occur, but with Rust, it just feels like I'm just continually excited.

It can enable you to build cool software and safe software and that's all important, but actually being able to kind of grow people as humans, I think is one of the things that's a really strong distinguishing characteristic of the language and its ecosystem more generally.​

And because there must be balance in the universe:

I get in trouble sometimes because I'm not a very good language evangelist. Rust is really pedantic and it can feel very bureaucratic and stuffy. Sometimes you just want Rust to relax. But it’s never going to relax.

Where to find Tim

You can find his book Rust In Action on Manning’s site and you can find him on Twitter @timClicks and see his streams on YouTube at timClicks. He also has a blog about technical topics at tim.mcnamara.nz

Upcoming posts

Tim was a very fun person to talk to, and he gave me lots of (spicy?) opinions on things like Haskell, Emacs and data science that I am going to break into a separate post for subscribers.

If you’re interested in that kind of thing, feel free to subscribe. It does not have a lot to do with Rust, it was just a long conversation with lots of fun tangents and I wanted to collect the tidbits somewhere.

I also have a post coming up about doing constructive math and type theory in Rust, which Yay!, but also, sorry Tim.

Subscribe now

Share Always Bet On Rust

Things changed in a couple ways when Mozilla created asm.js (leading later to WebAssembly). Along with Emscripten, you could now compile C and C++ into a subset of JavaScript that executed very efficiently and provided the kind of linear memory model that C backends expect. Generated JavaScript can have pretty bad performance characteristics, so being able to take advantage of decades of research in optimizing compilers for C and C++ in the browser was a big step forward.

Rust uses LLVM, which Emscripten compiles into WebAssembly, so pretty early after the creation of WebAssembly, people were porting Rust to work in the browser. And very shortly after that, there were Rust frontend frameworks.

What’s the difference between Rust frontend code and say, Flash, which was a compiled binary being executed in the browser? Aside from the plugin not shipping with the browser, the main difference is that Rust on the web is designed to integrate with the other APIs in the browser. Specifically, the DOM. Previous generations of frameworks tried to paper over the web and pretend it was just like any other GUI application.

While frontend frameworks are taking web technologies seriously now, it’s also the case that WebAssembly is the result of the web2 taking languages other than JavaScript seriously. I’m personally a fan of TypeScript in the category of “JavaScript but with a little extra”, but if you’re already stuck with a compilation step, why be so concerned with whether the input language resembles JavaScript?3

Rust has a reputation of giving low level performance with high level language features. Does that translate into a big advantage on the web? Yeah, there are some performance benefits, but it’s important not to overstate them. The biggest costs on the frontend are less in “how many loop iterations can I do per second” and more like “how do I avoid reflows and batch my DOM updates”. This varies depending on what kind of application you’re writing, but just like on the backend where we don’t worry too much about language overhead if you’re IO bound, running a more efficient language on the frontend doesn’t buy you much if all your time is spent blocking on the DOM.

Frameworks mitigate much of this on the web. They take care of batching your DOM updates and calculating the minimal set of event listeners that need to be inserted, etc. I’ll write up a better post on these things later, but the bleeding edge of these techniques was pioneered by Svelte, which requires a compilation step. We now have tech similar to this in Rust as well with the dominator library, which MoonZoon utilizes in its frontend to get performance.

Interview with Martin Kavík

I spoke with Martin Kavík, the current maintainer of the Seed frontend framework and creator of a new full-stack framework called MoonZoon. We talk about how he chose Rust, becoming the maintainer of the Seed framework and some of the architecture and design decisions he made in MoonZoon.

The following interview was edited for length and clarity, and incorporates some followup questions I asked him.

Can you talk a bit about how you got into Rust development?

Yeah, I was working in some agencies in Prague in the Czech Republic and I was working with PHP and C/C++ and so on. And it was kind of dangerous. When our app got deployed to production, we’d very often find some problems that were very surprising bugs because of missing language features.

So I was trying to find a better language to eliminate these kinds of bugs, and I tried to learn Elm, Elixir, Go and so on. But I just didn’t like these languages. There were some very clever and nice things, but they were very dogmatic or too universal or designed for beginners (for example, Go). But then I found Rust, and that was it.

It’s the best language (I think), so I wanted to write in Rust and not use any other languages.

You’re the primary maintainer of Seed, how did you get involved with the project?

I was a frontend developer for some years, but there was only JavaScript there, so I moved to the backend. But after years of working in agencies and banks and similar companies, I wanted to move back to frontend because I was a bit tired of devops and microservices.

I was trying to find something like Elm, but in Rust. I tried a new framework, Yew4 and some similar ones, but they felt a little too hard to learn, a little too cumbersome, I didn’t like the API or they were missing documentation. Then I found Seed, and I didn’t really like the code, but the API was very good.

So, I wrote some pull requests to Seed, and the owner was pretty happy to merge them. That was my first Rust contribution to both GitHub and to open source in Rust. Then I saw the opportunity to rewrite and improve the codebase and API because the previous maintainer made me a provisional contributor. He was very good at designing the APIs because of their simplicity.

What led you to developing MoonZoon?

After maybe two years of development on Seed, I was able to write Rust pretty confidently. I had an idea that some of the problems with JavaScript architectures in Rust aren’t really solvable without many breaking changes in Seed’s users’ applications.

So, I had two options. The first one was to refactor and introduce many changes in Seed’s codebase and API. Or I could write another framework (or proof of concept at least) of something better, with a newer and more Rusty architecture without Elm’s influence. That was the motivation for Zoon, MoonZoon’s frontend part.

The second problem was I received many questions about frontend and backend integrations on Seed’s chat, and I wasn’t really able to answer them because there are too many frameworks and architectures on the backend side. Seed, or any other frontend frameworks, have to be flexible enough to support all of these frameworks.

So, I was thinking about how to resolve it once and for all, and the idea of one backend and one frontend was born. MoonZoon is solving the problem of integration, basically. The idea is the developer writes the application in MoonZoon, and it just works because it’s optimized for this small scope where you write the frontend and backend at once.

I was thinking about something like Apple’s technologies where they design the hardware and software, so they can optimize it much better than, for example, Windows where they have to deal with many kinds of hardware. That’s the basic idea.

Has using Rust caused you difficulties specifically?

Rust is a hard language to learn, and that has some advantages and disadvantages. The advantage is that it automatically filters, uh…, script kiddies, basically. So if you want to hire someone to write Rust for you, it’s pretty safe. They’ll already know how to program because it’s hard.

For particular Rust features, I was fighting for months with standard things like the borrow checker and lifetimes. And then you discover some feature with traits or some new features in the language and it’s kind of like relearning the language. So you have to learn constantly.

For example, when the async/.await syntax was new, we had to rewrite the Seed codebase, especially for the fetch API. We ran into some weird problems with the WASM file being too big because we added some dependencies like standard url. That dependency was too big, so we had to remove it and basically rewrite it from scratch. Problems or bugs, not particular to the language, but in dependencies and the ecosystem are the most painful things to resolve.

Most questions from Seed users or bugs in Seed apps are not due to the language itself; something like null or undefined. They’re just from semantic errors: they just can’t make it compile.

I think Rust has pretty different problems from JavaScript or PHP. So, you explain and they take this into account, that we will have to rely on the compiler. It’s new thinking for many many people from other languages and backgrounds.

You mentioned earlier that you just wanted to code in Rust from now on, do you mean literally, or do you think there are some places where you would still use another language over Rust?

Maybe there are some places where other languages will be a little bit better, but there are two problems for me. The first one is that my brain is comfortable with Rust. I don’t try to do the compiler’s work in my head. So, for example, when I was working for some months in a company where they were using Go. I would have everything running and working, and I would receive code review comments about something like missing references, or “This will maybe work, or maybe won’t work because of X”.

Rust eliminates these kinds of bugs with its features. In other languages there isn’t this safety net, so I had problems writing in these languages. Afterwards, I would like to write everything in Rust, because it’s basically a systems language for very low-level stuff, but we’re also able to write very nice abstractions with it. And you can compile it to native or anything else.

So, Rust everywhere for me.

What features are you personally most excited about adding to MoonZoon?

I’m writing the “Zoon” part of MoonZoon, so I ended up writing a JS benchmark example app for Zoon5. I’m already seeing the differences between Seed and Zoon, for example, displaying thousands of lines on the screen. When you click on one row, it changes basically immediately in the Zoon app, but in the Seed app you need to wait for it to render the entire page. There are some architectural features paying off that I’ve already seen, so that’s pretty exciting for me. That’s maybe a few months away.

The “Moon” part, the backend, that’s basically a new world for me because I was working on Seed for the last couple of years. I haven’t been able to find another Rust virtual actor framework6. There are some frameworks with virtual actors in Java and C#, but that’s something that’s missing in Rust. So it’ll be from scratch and I’ll be reading white papers and trying to implement it. The goal is to make it scalable and make it work without a database. So I’m excited because it will allow me to write web applications without Postgres and migrations and DB-ops and similar stuff.

Can you give a quick summary of how virtual actors work?

• Actor Model: An actor is an object that interacts with the world and other actors using asynchronous messages.

• Standard Actors: the actor has to be explicitly created and removed by a developer or by the actor's parent (aka supervisor) by predefined rules.

• Virtual Actors: A runtime system manages actors. It creates, removes or moves actors between servers. All actors are always active and cannot fail from the developer point of view. In the case of hardware failure, the lost actors are automatically recreated on another server and messages forwarded to them.

That’s interesting, so in other words it will pull the database into the framework itself so it’s not talking to an external service?

When you write a small application with a backend, you’ll be able to use virtual actors. But I know many apps will require some communication with some custom APIs (microservices or cloud APIs, something like that). So there will be some libraries to communicate with classic, standard APIs, but if you don’t need it you just use virtual actors and you’ll have scalability and basic stuff with authentication built in. It should make your development much faster and less complicated. Less setup for your local machine, and production should be less painful.

Are there any projects in the Rust ecosystem that you think are really interesting but maybe haven’t gotten enough attention?

• The first one is trunk. It’s like WebPack for Rust applications. The maintainer is a Seed user, so we know each other from the chat. I’ve already used trunk in a production app for my client, so it’s already working and usable with Seed and Yew (and probably other frameworks).

• futures-signals is super nice and is basically the core of Zoon now

• Another is whatlang, an application built in Seed that uses natural language detection.

  

• Finally, fst is pretty breathtaking. I was working on a frontend for a client in Rust, and the core was based on this library. I was able to search through maybe 20,000 movies by titles in a few milliseconds. When I was comparing it with some JavaScript libraries, they were taking maybe tens of milliseconds. So it’s a very nice example where Rust and WebAssembly is much faster and allows you to change your frontend architecture. You can move your search from the backend to the frontend side and save some money!

Share Always Bet On Rust

Subscribe now

Updates & Housekeeping

If you have a paid subscription and haven’t gotten your Always Bet on Rust sticker yet, fear not, I’m sending the rest out this week.

Also, if you know someone who works on an interesting Rust project you think would make a good post please let me know. I have a pipeline of new interviews, but the suggestions I’ve gotten so far have been great.

This post has some art in it, which you absolutely need to click through and go watch, because it’s meant to be seen in motion, and is pretty amazing. All images in this post are courtesy Alexis Andre.

Subscribe now

The technical side of making art

There is a stereotype of artists as strictly right-brained personalities. (Insert your favorite pop-culture image of a painter in a beret and a smock splotched with paint). This trope encourages an image of artists as people who are purely creative, unconcerned with practical matters.

But in reality, making art requires a significant time investment both terms of tools and learning the peculiarities of a particular medium. To state the obvious, it takes a monumental amount of work to gain mastery at a craft, even if the ultimate purpose of that mastery is creative in nature. When the medium is pencil, an artist gains a mastery of the HB grading scale, and blending on top of the prerequisite background of contrast, perspective and composition. When the medium is code, the artist gains a mastery of writing code and graphics drivers and building shaders.

Artists are in a pretty competitive market for attention1. They need to show the audience something interesting or beautiful or intriguing. Even if they make art only for themselves and there’s an audience of one. The easier a particular kind of art is to make, the more of it is made, and the novelty and originality wears off. Think of photo filters before Instagram: previously you had to have some PhotoShop skills, but once it’s an app, the filter style becomes a commodity. An artist needs to do something exciting or original2, rehashing the same thing is boring.

This treadmill is made even more explicit in the generative art world, where the artist writes code that creates the art. Once you’ve coded up something that looks interesting, you might naively expect trivially permuting the parameters will give you infinite art. Job’s done, you can go home: the art creates itself now. But, there’s inherent similarity between pieces of art using the same underlying code, and viewers pretty quickly catch onto that similarity and tune out.

Generative artists widely embrace this fact of diminishing returns. You have to keep moving, doing new things, writing new code. This is like exploring a latent space of visual experiences. Regions of the latent space that human brains find interesting are separated from each other by large stretches of things that are boring. It’s the artist’s job to travel between those gaps and bring back something cool. The tools they use can help them move more quickly through that space, or eliminate boring regions faster, or simply explore more dimensions. Speed of iteration is kind of a universally good currency here.

Rust brings a couple of potential advantages as a tool for generative art:

1. Faster execution means that not only are your graphics primitives executed by a well optimized library, but the top level art is as well. That means more elements can be added to a canvas, more advanced effects can be layered, and more for loop iterations can be executed.

2. The graphics library doesn’t have to represent an optimization boundary3. If the generative code is in Rust, the compiler can aggressively optimize the art from top to bottom

Additionally, it seems like a some combination of attributes of Rust are making it particularly fertile ground for cutting edge graphics projects like wgpu, vulkano and rust-gpu:

• cargo & crates.io make it easy to distribute packages versus the traditional model of creating a shared lib and convincing distro maintainers to distribute it (or trying to add the feature you want to an existing library that is already distributed)

• programmers who would never have chosen to code in C or C++ are now writing Rust and this naturally brings new ideas and energy into the systems software world (as any influx of new people would do)

• Sheer enthusiasm for Rust causing people to do ambitious things in it

Having so many high quality choices makes it pretty suitable as a medium for generative art.

Interview with MacTuiTui

I was excited to talk with Alexis André, a generative artist who creates his art primarily in Rust, using the Nannou framework. We talk about his history and how he got into making art with Rust, the iterative process of creating daily art and why Rust is a particularly exciting language for him.

You can find his dailies on Instagram as MacTuiTui and he posts a wide variety of updates on art on his Twitter (also MacTuiTui).

The transcript below has been lightly edited for length and clarity.

How did you get into creative, generative coding?

My background is in computer science. I come from the French system, where if you go through the elite course you get into one of the engineering schools and you end up as a general engineer. I decided to actually go to Japan (where I am right now) to get the master’s, and I did my master’s in machine learning. Then I switched halfway to get my PhD in computer graphics.

That’s where I started to learn not only about the techniques of rendering graphics, but also about perception, aesthetism, all the rules of color theory. From a technical perspective first, and also from an artistic perspective.

And that’s when I found Processing, because my supervisor was like, “You need to use Java! Java is going to be the future of graphics!” and I was like “I’m not sure… but why not.” That’s when I found out this new world of generative arts that I was not aware of. I just went through all of the big names at the time. So you have Casey Reas, Ben Fry, Jared Tarbell, Robert Hodgin. That was like 2004-5. Then I went through this habit of tinkering with stuff a lot until it became more and more like a hobby, then it became a job.

On learning new technologies

In my practice of doing graphics and generative art, I want to be on the technical front end of things, like, being aware of the latest technologies available because it’s a technical race. So I forced myself to learn new languages periodically. And I actually switched to Scala, and that’s when I started to do my dailies. I’ve been doing those for four and a half years, like 1500 days.

Then I was really intrigued by the latest GPU technologies where you had Vulkan, Metal and DirectX 12 that were coming out. The way Processing was was designed at the time, it was still stuck in the OpenGL way of doing graphics. That was not the way forward.

I was looking for a way to upgrade to Vulkan first, and that’s when I found Nannou. That was one of the few frameworks that were actually pushing the way forward, breaking things along the way if needed, but using the latest technologies. I went through a lot of version with them to maybe being the heaviest user of the framework.

Finally, I can hear my GPU working, where previously it was mostly CPU bound. It’s really interesting to get to the point where the computer is giving up. That’s really rewarding in a way where you can get like a GPU crash because the shader went into a weird loop.

Did you learn about GPU shaders in school or is that something you’ve taught yourself since then?

I was doing shaders at the very beginning of the technology, like I had been doing GPGPU stuff in 2005-6. In my lab, we were doing oil painting simulation and that was mostly shaders. But I’m not really a shader guy. If I look at what people are doing with shaders right now… I’m not there. I know the basics and I was there at the beginning, but it moves so fast that I’m still playing.

That’s why I’m interested in having access to those technologies in a nice framework so that I can manage to push forward.

I know Nannou does WebGPU rendering, is most of your work rendered as a video, or do you render it all on the browser side?

No, the whole compile to Web Assembly is not ready yet. That’s the end goal. Personally, I would really like to reach the point where I can just put the code for everyone to run in their browsers. That’s part of the appeal of using Nannou because eventually WebGPU will be available for everyone to use in the browser. Rust is the way forward for this.

In the next few months we’ll reach the point where I can just push everything that I do online and people can interact with it. Right now, it’s rendering as a native application on my computer. I export each frame and then I combine them as a video and push the video to social media, so it’s a two step process.

But it also allows me to do really intensive stuff where I don’t care about it being realtime as much because it’ll be compiled as a video afterwards. So I can cheat a little bit by making really intensive computation that looks crazy but isn’t real time.

Nannou has added a few features just based on your requests, can you talk about those?

The first one that I asked for, when I was doing my daily animations, I needed a way to save the current frame to a file so I can combine them. So for the first few months, I had a custom build that was just for me, where the screenshot feature was available until they managed to add it to the library.

The second one was using blending. Blending is how you mix colors on the screen. It’s really interesting to see how people have been doing it wrong in the past because of the limitations of OpenGL and the way it was done for ages before. So if you think about using colors as a 24-bit number, so we have 256 levels of color per channel, that’s not enough. We get banding. If we want to mix color together, we don’t have enough precision.

What Nannou has been able to do with the help of WebGPU, is that we have floats, real floats, for each color channel. So we can mix colors at a level that no other framework is able to do. We can reach truly bright colors and truly dark colors at the same time with the whole spectrum in between.

The team behind Nannou has been really helpful on a day-to-day basis. There were a couple of days where they really went down with me, documenting some features just to help me manage to complete the daily in time. So Mitch, Josh Batty, Tom thanks a lot for all the hard work!

One of the benefits Nannou mentions is that the language used by the framework is the same one the framework is built in. Have you found that that’s been a useful property?

The way I’m using the framework is pretty much still, from the Rust perspective, pretty much a beginner level. I’ve contributed a few examples to show what the framework is able to do, but I’m pretty sure any experienced Rust programmer will come and say “Oh, this is dumb.”4

But to be fair, creative coding is more about: “You take one circle, it’s fine. You take one thousand circles, then it’s becoming more interesting. Then you take ten thousand circles and make them move around.” It’s just a couple of forms and you have something that looks nice, and it’s a lot about the details about how you make them move and interact.

So, it’s not really about the language, because I’m not developing the framework, I’m just using the features that are available. I’d like to be able to contribute more, because I think I owe them that.

I still find Rust sometimes really challenging. I have not taken the time to get comfortable in terms of the advanced features of the language. Every time I see an Arc, I’m like “Oh, that’s not for me yet.” Like there are a few points of the language design I’m not comfortable using. So I just worked my way around and for me that’s enough. I should do more, but from the perspective of a heavy user, that’s where I am.

Can you talk a little bit about your experience with Rust in general? Do you regularly run into issues where you’re like “Okay, this is annoying because it’s in Rust?”

In the first six months, yes all the time. Especially the way Nannou has been designed. You have the model-view-controller pattern, so when you’re drawing something you cannot mutate the data. From the Processing way of thinking, you can do it whenever you want, wherever you want. Sometimes there is stuff that I’d like to do, where it makes sense to draw as you’re mutating the model.

Also, the thing that I’m not sure I’m doing right, even right now, is that when you have a list of things and you want to mutate them two elements at the same time. I just use split_at_mut.

I have a few tricks that I’m sure are wrong, but right now I think I’ve reached a point where I think before deciding what I’m going to do. So, I know that I will be able to do this with my skills in Rust. I don’t hit any compiler errors right now, because I’ve reached the point where I’m fluent enough to do what I want to do.

Are there any projects you’ve done that stick out in your mind as being really difficult from a technical perspective?

I think the hardest thing was when I tried to do real-time path tracing 2d simulation. There was a time of, maybe last summer where I had those really crazy effects where lights were bouncing around walls and shining.

That was the results of getting to know compute shaders in WebGPU. The whole stuff was like maybe 1000 lines of code just to prepare for the rendering. Then finally just inputting the data and making the stuff move was pretty easy. But it took me maybe a month of incrementally adding the next block to the puzzle so that I know that eventually I would be able to do that.

And when I was was able to do that I just abused it for two months. Like, I spent one month building the tech, I’m going to milk it for a long time!

That was really hard, but I learned so much about compute shaders, render pipelines, how to time the GPU to cache so that all the data is moving in the right way at the right time. So that, eventually, it works in almost real time.

That’s what I was talking about getting the GPU on their knees. That was the part where I managed to crust the system a lot. When you do compute shaders, you’re really free to do whatever you want. And when the GPU is not responding, the OS is like “I don’t know what is happening”, but it needs to just reset the driver, and you’re stuck in the middle of a loop where your computer is showing nothing… you’re hoping that you might come back…

Have you found compile times to be much of an issue?

Not that much. Right now, my compile time is like five seconds for the dailies. I use Vim, I have the LSP plugin, so the compiler will tell me any error before I try to compile. When I write the code, it’s almost too fast, the feedback I get from the computer.

From this perspective, I’m really happy with my workflow right now. When the compiler will not complain any more: I will get the results I want. Then it’s thinking about the algorithm itself, it’s not about the way to express it in code. So personally I’m super happy with the compile time and workflow.

I saw your GitHub Universe stream, and I was wondering whether that was a super polished presentation or whether that’s representative of your daily coding flow.

If you do daily times like I do, any bit of optimization is worth taking the time to master. It’s about finding the set of tools that will allow me to be efficient and not spend a lot of time in the tooling because I don’t have that time every day.

I also code a lot for work. So Vim is a skill that I don’t see a disadvantage of learning to use at a higher level. A few years ago, before LSP was the norm, I would say that maybe using Visual Studio might be a better choice, just to get the integration with the compiler. But right now with the new plugins and everything it makes absolutely no difference if you use VSCode or Vim.

I put Vim mode in all my other software. I’m so used to thinking in modes I do all my reports for work in Vim.

Have you used Rust for anything other than creative projects?

I’ve been using Rust and Nannou in my work as well. I’ve been building tools to design murals, to design tiles, to design walls. I’ve been doing a lot with small robots for my company and I’ve built a toolchain around them with Rust and Nannou. I’m actually using that hardware as an interface for my dailies, so that I can change all the parameters with them.

So yeah Rust right now is my default choice. I don’t think I can use any other language. I worked a lot in Java but that was 20 years ago. I don’t know any C or C++ to the point where it would make sense to consider them.

I also like cargo and the whole ecosystem behind Rust. It just runs. The whole process of making it run on a new system is really straightforward. I have yet to encounter an issue where I was not able to just take a computer, install Rust, install cargo, cargo run and then it runs. This has been so easy in terms of deployment.

Recently, I’ve done JavaScript and Rust. JavaScript when it has to run on the web. I’m hoping once WebAssembly is available in a few months that I can just switch everything to Rust. For me, there is no other choice right now.

Is there anything else you’d like to mention?

I think if anyone is interested in the concept of Rust and trying out something new, creative coding is something that is really satisfying. You write something, and you get a nice picture. In terms of not writing another Hello World tutorial, making a nice circle on the screen is a little more interesting in terms of getting to know the language. It could be a really good introduction.

Like, if you look at how I designed my presentation for GitHub Universe, I introduce a lot of concepts in an incremental manner. You just start with getting cargo to create a new binary to make it run. All the steps that you need to make your own projects, well, you can do them with Nannou.

And you get a nice picture as well which could be more rewarding.

Leave a comment

Subscribe now

For example, marine chronometers are a very complicated and hard to engineer way to keep time on the ocean1 used in the 18th and 19th centuries. In the 20th century, ships switched to using radio time signals. Radio signals are very easy to generate, and quartz clocks are a very accurate way to keep time. Both are much easier to build than a marine chronometer and they’re more accurate and robust to failure. We just… didn’t think of it first2.

Something like this happened with code notebooks3. REPLs (including shells like Bash) were invented and got popular with the rise of Unix and scripting languages. But the basic idea of a code notebook could have been invented any time. The core technology needed for a code notebook is the same as for a shell: you need a way to execute code in the language, maintain a context, and display the output.

As a consequence of inventing code notebooks embarrassingly late, it feels natural to describe them as “like a REPL, but with extra features like displaying images”. Instead we should be describing REPLs, including shells, as “kind of like a notebook, but worse and harder to use”. It’s hard to look at someone spamming ⬆️ and Ctrl+R in a shell and not think they would have benefited from using a code notebook where you can simply edit previous cells.

Notebooks are usually associated with data science work, but this is only because the shell doesn’t provide good visualization capabilities4. Developers on *nix systems learn to use the shell as an interface to all kinds of system tasks, so they tend to be comfortable with the command line. But data scientists generally don’t get this extensive shell indoctrination and Jupyter is readily available and more than capable. We can take from the fact that data scientists are not clamoring to do data science from the terminal as some evidence that, in fact, code notebooks are the more general solution.

And as long as we’re replacing the shell with a notebook, we might as well replace the language of the shell with Rust.

Quick overview of Evcxr

Evcxr is a Jupyter kernel for Rust. It makes iterative, one-off, and task-oriented coding much more enjoyable in Rust. It’s not that you can’t do iterative duct-tape-and-hairpin coding with cargo and an IDE, it’s just that it’s so much more ceremony.

With Evcxr, you can add dependencies from crates.io, and use them immediately:

This interweaving of what would normally go into Cargo.toml with the source code itself is exactly the kind of thing that’s great for iteratively solving a problem, but which most tooling avoids because it’s not very maintainable. That’s ok though: notebooks are not for maintaining. They’re a place to solve a particular problem, and if you’re feeling especially motivated and energetic, a place to document what you learned.

You can also do async and ? without any ceremony in a cell. It will wrap your code with the necessary invocations before running it:

Evcxr is really promising technology, but it has a few limitations. While async works, it looks like rustc made some changes to dynamic libraries that don’t interact well with Evcxr, so async is currently stuck on pre-1.0 tokio. The inline type checking was a little hit or miss for me as well. Sometimes it triggered, sometimes it didn’t. This may be a Jupyter issue. I would very much like to get autocomplete in cells.

Long term, because Evcxr is using rust-analyzer internally, it’s poised to benefit from the autocompletion and code fixes from that project. One of the reasons common shell commands have very short names (e.g. ls, mv, git) is because when you’re solving something, typing speed is really important. Good autocomplete mostly eliminates that advantage, and in return hands us back the ability to use more than three letters in our function names.

Overall, I’m very excited about what Evcxr unlocks.

Interview with David Lattimore

I chatted briefly with David, the creator of Evcxr who was kind enough to answer my questions. We talk about the internals of Evcxr, features he’d like to add, some of his work on structural search and replace in rust-analyzer, and what he’s excited about in the Rust ecosystem.

The following is a transcript of our conversation edited for length and clarity.

So I guess the first question I have is: how do you pronounce the project name?

I pronounce it “e-vix-er”. Whenever Evcxr is brought up on Reddit or something like that, someone will be like “It's a terrible name!” or like “I can never remember the name!”. It's quite searchable, which I guess is part of the motivation. You're not going to confuse it for something else or get results for something else.

But unfortunately, it's very difficult to remember and very difficult to work out how to pronounce.

Could you talk a little bit about what you worked on previous to Evcxr, and what kind of led you up to creating the project?

At the time I'd started it, I'd been playing with writing a linker,5 so I'd been playing in that space of loading and dealing with object files and shared objects and those kinds of things. And it just kind of occurred to me that it would be reasonably easy (or I thought it would be reasonably easy) to make a REPL like-thing just by loading shared objects and running the code.

It was reasonably quick to get something basic up and running. As always with these things the devil is always in the detail. It's a continuously evolving project to add the many subtle features and make different things work well.

So it started out as a REPL and you added the Jupyter kernel on afterwards?

Yes, although I added the Jupyter kernel before I released. I wrote the REPL and then I thought, “Well my wife is a data scientist, so she uses a Jupyter notebook quite a bit. That could be kind of fun to try and make a Jupyter kernel as well"

So I spent an extra couple of weeks and make a basic Jupyter kernel before I released.

Had you used Jupyter much previously yourself?

I'd never really used it myself. I'd just seen her using it and I thought it looked kinda cool.

It looks like you switched the project from using syn to using rust-analyzer. Can you talk about how that works?

Yeah sure, I switched to using rust-analyzer about six months ago.

Prior to that I would generate the code, and then ask the Rust compiler to compile it, and then look at what the errors were. I assumed that all the variables were of type String, and I would look at the errors were produced. The Rust compiler can happily format its output as JSON, so you get like a JSON representation of the errors and you just pull out the error message for a particular variable and see it says, you know "found this type, expected that type".

That worked for quite a few years, but then recently they changed the messages so that they no longer included the fully qualified type. So instead of saying, you know, std::collections::HashMap or something like that it would just say HashMap. And without the fully qualified type I couldn't use those types anymore, they just weren't sufficient.

So, I switched to using rust-analyzer. If rust-analyzer can't determine the type, Evcxr will still fall back to using the compiler messages. The only reason I've got that there is because rust-analyzer doesn't deal with fixed-sized array types. Like it doesn't tell you the size of the array in the type that rust-analyzer infers, so if you have like:

let array_of_ints = [0; 5];

rust-analyzer's type will just say:

[i32; _]

I can't use that, so for those particular cases I still fall back to using the compiler error messages to determine the types.

I would like to get rid of that code because it's... it's a bit ugly. It's a bit of a hack. But yeah for the most part I'm using rust-analyzer. I'm also not using syn any more, so I've switched entirely to using rust-analyzer to do the parsing.

How do you find using rust-analyzer as a library? I know it's broken into crates so it's theoretically usable as a library, but you're actually using it in anger, so what's that been like?

It's pretty good. Probably my biggest complaint would be that it's quite big and so it takes a while to compile. Certainly my continuous integration slowed down quite a lot when I pulled in rust-analyzer because now every time it runs it has to build all of rust-analyzer which takes a while.

An alternative way that I could have integrated that I didn't consider at the time was to actually pull in the rust-analyzer binary and talk to it using the language server protocol, instead of using it as a library. Obviously, that would have been better from a compilation time perspective, but I just wasn't sure whether I'd have access to everything I needed. So I didn't end up going that route.

What kind of features do you want to add to Evcxr that you feel are missing?

One feature that I started looking to try and add a while back, but I just haven’t had time recently, is allowing code that runs in a Jupyter notebook to interact with the notebook process. Allowing some way for JavaScript code that runs in the browser, to communicate with some Rust code that runs in a library.

Like maybe in a crate, where you pull in a crate that, you know, creates a graph or something like that, and that code then talks to the JavaScript code running in the browser. So you can have bidirectional communication going backwards and forwards allowing an interactive thing to be displayed in the notebook.

Like a slider widget, something like that?

Exactly, yep. So I started working on that I think I got some of the basic stuff working. I think I, I can't remember where I was up to with it but I remember I had issues with Mac. Like the changes I'd made when I pushed them, the tests failed on Mac.

It's probably because I didn't have access to a Mac, so the only way I can test these changes is like: put some print statements in, push to CI, and then wait for Travis to run Mac, which takes like half an hour or an hour. And then, you know, see what the values the print statements are. It can make a really frustrating and slow testing experience.

I should probably put a call out just to ask if anyone with a Mac wants to come and help. It would make things much easier!

It's currently pre 1.0, what are your criteria for making a 1.0 release?

You know, I haven't really given too much thought to 1.0. I guess, so, Evcxr itself is a library, which is then used by the Jupyter notebook and the REPL. And I make breaking changes to the API of that library moderately often. Just because it's convenient, and because the only two users of the library are in the same repository.

So I guess I'd need to think about how I want to treat it with regards to semver there. Maybe if I just kept the library pre-1.0, but made the notebook and the kernel be 1.0, then I could get away with it. Because I'd still like to be able to make breaking changes to the library.

The API of the library is very not stable.

It looks like after Papyrus and Rusti stopped being maintained, Evcxr seems to be the only Rust REPL that has any recent commits to it. How do you feel about being the sole Rust REPL at this point?

[It turns out I was wrong about this. After our conversation David pointed out IRust, which is a new Rust REPL that’s being actively developed]

At time I wrote Evcxr, there was only one other REPL, which was rusti. I think at the time it didn't say it wasn't maintained, but it was running on an old version of nightly from 2016. I think the issue there was that they were using some compiler internals, and those compiler internals got removed, so there was no way they could move forward to allow a new version.

I think it was using some compiler internals to get access to the LLVM IR or something like that. And then they'll be using LLVM to JIT it or something like that. It worked completely different to the way Evcxr works.

But there was no equivalent API provided, because the compiler just stopped... doing that.

Is this your main Rust project?

The original Rust project that I worked on and released was a thing called Rerast. Which was a structural search and replace tool. So it searched for Rust code, and replaced it based on a parse tree pattern. It used a lot of compiler internals and was nightly only. It was a bit of a pain to maintain and had a relatively low number of users.

But it broke very often, because the compiler internals were always changing. Because I depended on almost the entire surface area of the AST. Any time they changed the AST, my builds would break and I had to go fix it. So we had like fifty build fixes in one year, so round about one per week.

So eventually, once rust-analyzer got good enough, I helped out there. There was already some support for structural search and replace in rust-analyzer, and I built that up to close enough to feature parity to what I had in Rerast, and then deprecated Rerast.

So you're working on the structural search and replace in rust-analyzer itself?

Yeah I've done a lot of work on that. There's a lot more I'd like to do there as well, but yeah, I haven't really had time lately.

What I'd really like to see there would be some way for library authors to be able to make a change like they deprecate some API, and they put some annotations on the API that they're deprecating to provide a structural search and replace rule to allow you to automatically transition to the new API.

Then have the IDE automatically suggest that for you. Kind of how the compiler can make suggestions and the IDE can apply them. But this would be something driven by the library authors, not by the compiler team. So I'd like to see something like that, but I really haven't had time to drive that.

I'm curious if you have any places where you think Rust is particularly poorly applicable. Areas where Rust is maybe never going to be a good solution. Do you have any opinions on that?

I don't think it's a super hard language to learn. But I don't think it would be suitable as like a first language for kids, for example. For them you really want something that's simpler. I don't know whether Python is the right thing for kids, or scratch or something like that.

I'm not about to teach my kids Rust!

I guess in terms of industries, I think it probably could work pretty well in most cases. Certain kinds of problems can be a little annoying to try and do in Rust: graphs and things like that, that can be a bit annoying. Although there are ways to manage it, you've just got to structure your code a bit differently.

Often the ways you structure it end up being quite nice. So maybe it's not an issue. I found when I've done graphs in the past, just assigning integer ids to things, and things work out pretty well. Don't use pointers.

Nothing in particular comes to mind.

What are you most excited about in terms of Rust?

I've recently become quite interested in embedded programming in Rust. Mostly I'm excited to see Rust growing. And just seeing it continue to become bigger and better and more ergonomic and better tools and better libraries. Because that really does start to fill in the gaps where maybe it was a good language, but didn't have good libraries, but once you fill in the libraries then, there's really no reason not to use it.

I don’t know much about the embedded space. Is that growing a lot now in terms of usage?

I think it's a lot better than it was a few years ago. Like, I remember someone saying four or five years ago that there was pretty much one person working on embedded Rust. It probably wasn't true, but like there was one person really driving it. Whereas now, it feels like there's a lot of people you can point to where they're very clearly big in the embedded space. And you've got companies like Ferrous Systems doing great work in the embedded space.

It feels like there's quite a few more people working and although there's still bits and pieces missing it's a pretty nice experience and works reasonably well. At least depending on... It depends on what you're trying to use and what you're trying to do with it.

I don't know if you've seen Knurling. They've been doing a lot of Knurling sessions where they'll announce they're going to build, like, a CO2 sensor or something like that. They give you the part list of what you have to buy, and you order those online somewhere and then as the session goes on they release information about how to do different things with it. You then build this sensor and program it.

They don't provide all the details. I assume they provide code samples, but like the idea is that you write the code. So you're learning how to do it. There's other people who are learning as well, so if you're having troubles you can talk to other people to see what they've done.

They also make a bunch of tools, so there's a bunch of crates and utilities they've released to make it easier to do embedded Rust development. Things like probe-run where you can just type cargo run and it will compile your code and then run it on the device. And any log statements that you have in your code are on the device and just show up in your terminal. So it's like you're running it on the local machine but in actual fact it's running on a device you've got plugged in over USB.

If you Ctrl+C you get a stack dump of where it was on the device. It's quite a nice development experience in that regard. They've helped build out some of that stuff along with others in the community.

Thanks for taking the time to talk with me, is there anything you want to plug?

I need to figure out how to get Evcxr showing up in the first results when you search for “rust repl”. It’s at least on the first page, but that’s still not great given it’s been around for a few years now. I don’t know how to fix that kind of thing.

Leave a comment

Upcoming posts

I’ve got an interview with a generative artist that creates his works primarily in Rust, along with the primary developer of Seed, a frontend Rust web framework. Those should be out in the next couple of weeks.

Subscribe