Making Generative Art with Rust

Artists as technical experts and an interview with generative artist Alexis André

Apr 30, 2021

This post has some art in it, which you absolutely need to click through and go watch, because it’s meant to be seen in motion, and is pretty amazing. All images in this post are courtesy Alexis Andre.

The technical side of making art

There is a stereotype of artists as strictly right-brained personalities. (Insert your favorite pop-culture image of a painter in a beret and a smock splotched with paint). This trope encourages an image of artists as people who are purely creative, unconcerned with practical matters.

But in reality, making art requires a significant time investment both terms of tools and learning the peculiarities of a particular medium. To state the obvious, it takes a monumental amount of work to gain mastery at a craft, even if the ultimate purpose of that mastery is creative in nature. When the medium is pencil, an artist gains a mastery of the HB grading scale, and blending on top of the prerequisite background of contrast, perspective and composition. When the medium is code, the artist gains a mastery of writing code and graphics drivers and building shaders.

Artists are in a pretty competitive market for attention1. They need to show the audience something interesting or beautiful or intriguing. Even if they make art only for themselves and there’s an audience of one. The easier a particular kind of art is to make, the more of it is made, and the novelty and originality wears off. Think of photo filters before Instagram: previously you had to have some PhotoShop skills, but once it’s an app, the filter style becomes a commodity. An artist needs to do something exciting or original2, rehashing the same thing is boring.

This treadmill is made even more explicit in the generative art world, where the artist writes code that creates the art. Once you’ve coded up something that looks interesting, you might naively expect trivially permuting the parameters will give you infinite art. Job’s done, you can go home: the art creates itself now. But, there’s inherent similarity between pieces of art using the same underlying code, and viewers pretty quickly catch onto that similarity and tune out.

Generative artists widely embrace this fact of diminishing returns. You have to keep moving, doing new things, writing new code. This is like exploring a latent space of visual experiences. Regions of the latent space that human brains find interesting are separated from each other by large stretches of things that are boring. It’s the artist’s job to travel between those gaps and bring back something cool. The tools they use can help them move more quickly through that space, or eliminate boring regions faster, or simply explore more dimensions. Speed of iteration is kind of a universally good currency here.

Rust brings a couple of potential advantages as a tool for generative art:

Faster execution means that not only are your graphics primitives executed by a well optimized library, but the top level art is as well. That means more elements can be added to a canvas, more advanced effects can be layered, and more for loop iterations can be executed.
The graphics library doesn’t have to represent an optimization boundary3. If the generative code is in Rust, the compiler can aggressively optimize the art from top to bottom

Additionally, it seems like a some combination of attributes of Rust are making it particularly fertile ground for cutting edge graphics projects like wgpu, vulkano and rust-gpu:

cargo & crates.io make it easy to distribute packages versus the traditional model of creating a shared lib and convincing distro maintainers to distribute it (or trying to add the feature you want to an existing library that is already distributed)
programmers who would never have chosen to code in C or C++ are now writing Rust and this naturally brings new ideas and energy into the systems software world (as any influx of new people would do)
Sheer enthusiasm for Rust causing people to do ambitious things in it

Having so many high quality choices makes it pretty suitable as a medium for generative art.

Interview with MacTuiTui

I was excited to talk with Alexis André, a generative artist who creates his art primarily in Rust, using the Nannou framework. We talk about his history and how he got into making art with Rust, the iterative process of creating daily art and why Rust is a particularly exciting language for him.

You can find his dailies on Instagram as MacTuiTui and he posts a wide variety of updates on art on his Twitter (also MacTuiTui).

The transcript below has been lightly edited for length and clarity.

How did you get into creative, generative coding?

My background is in computer science. I come from the French system, where if you go through the elite course you get into one of the engineering schools and you end up as a general engineer. I decided to actually go to Japan (where I am right now) to get the master’s, and I did my master’s in machine learning. Then I switched halfway to get my PhD in computer graphics.

That’s where I started to learn not only about the techniques of rendering graphics, but also about perception, aesthetism, all the rules of color theory. From a technical perspective first, and also from an artistic perspective.

And that’s when I found Processing, because my supervisor was like, “You need to use Java! Java is going to be the future of graphics!” and I was like “I’m not sure… but why not.” That’s when I found out this new world of generative arts that I was not aware of. I just went through all of the big names at the time. So you have Casey Reas, Ben Fry, Jared Tarbell, Robert Hodgin. That was like 2004-5. Then I went through this habit of tinkering with stuff a lot until it became more and more like a hobby, then it became a job.

On learning new technologies

In my practice of doing graphics and generative art, I want to be on the technical front end of things, like, being aware of the latest technologies available because it’s a technical race. So I forced myself to learn new languages periodically. And I actually switched to Scala, and that’s when I started to do my dailies. I’ve been doing those for four and a half years, like 1500 days.

Then I was really intrigued by the latest GPU technologies where you had Vulkan, Metal and DirectX 12 that were coming out. The way Processing was was designed at the time, it was still stuck in the OpenGL way of doing graphics. That was not the way forward.

I was looking for a way to upgrade to Vulkan first, and that’s when I found Nannou. That was one of the few frameworks that were actually pushing the way forward, breaking things along the way if needed, but using the latest technologies. I went through a lot of version with them to maybe being the heaviest user of the framework.

Finally, I can hear my GPU working, where previously it was mostly CPU bound. It’s really interesting to get to the point where the computer is giving up. That’s really rewarding in a way where you can get like a GPU crash because the shader went into a weird loop.

Did you learn about GPU shaders in school or is that something you’ve taught yourself since then?

I was doing shaders at the very beginning of the technology, like I had been doing GPGPU stuff in 2005-6. In my lab, we were doing oil painting simulation and that was mostly shaders. But I’m not really a shader guy. If I look at what people are doing with shaders right now… I’m not there. I know the basics and I was there at the beginning, but it moves so fast that I’m still playing.

That’s why I’m interested in having access to those technologies in a nice framework so that I can manage to push forward.

I know Nannou does WebGPU rendering, is most of your work rendered as a video, or do you render it all on the browser side?

No, the whole compile to Web Assembly is not ready yet. That’s the end goal. Personally, I would really like to reach the point where I can just put the code for everyone to run in their browsers. That’s part of the appeal of using Nannou because eventually WebGPU will be available for everyone to use in the browser. Rust is the way forward for this.

In the next few months we’ll reach the point where I can just push everything that I do online and people can interact with it. Right now, it’s rendering as a native application on my computer. I export each frame and then I combine them as a video and push the video to social media, so it’s a two step process.

But it also allows me to do really intensive stuff where I don’t care about it being realtime as much because it’ll be compiled as a video afterwards. So I can cheat a little bit by making really intensive computation that looks crazy but isn’t real time.

Nannou has added a few features just based on your requests, can you talk about those?

The first one that I asked for, when I was doing my daily animations, I needed a way to save the current frame to a file so I can combine them. So for the first few months, I had a custom build that was just for me, where the screenshot feature was available until they managed to add it to the library.

The second one was using blending. Blending is how you mix colors on the screen. It’s really interesting to see how people have been doing it wrong in the past because of the limitations of OpenGL and the way it was done for ages before. So if you think about using colors as a 24-bit number, so we have 256 levels of color per channel, that’s not enough. We get banding. If we want to mix color together, we don’t have enough precision.

What Nannou has been able to do with the help of WebGPU, is that we have floats, real floats, for each color channel. So we can mix colors at a level that no other framework is able to do. We can reach truly bright colors and truly dark colors at the same time with the whole spectrum in between.

The team behind Nannou has been really helpful on a day-to-day basis. There were a couple of days where they really went down with me, documenting some features just to help me manage to complete the daily in time. So Mitch, Josh Batty, Tom thanks a lot for all the hard work!

One of the benefits Nannou mentions is that the language used by the framework is the same one the framework is built in. Have you found that that’s been a useful property?

The way I’m using the framework is pretty much still, from the Rust perspective, pretty much a beginner level. I’ve contributed a few examples to show what the framework is able to do, but I’m pretty sure any experienced Rust programmer will come and say “Oh, this is dumb.”4

But to be fair, creative coding is more about: “You take one circle, it’s fine. You take one thousand circles, then it’s becoming more interesting. Then you take ten thousand circles and make them move around.” It’s just a couple of forms and you have something that looks nice, and it’s a lot about the details about how you make them move and interact.

So, it’s not really about the language, because I’m not developing the framework, I’m just using the features that are available. I’d like to be able to contribute more, because I think I owe them that.

I still find Rust sometimes really challenging. I have not taken the time to get comfortable in terms of the advanced features of the language. Every time I see an Arc, I’m like “Oh, that’s not for me yet.” Like there are a few points of the language design I’m not comfortable using. So I just worked my way around and for me that’s enough. I should do more, but from the perspective of a heavy user, that’s where I am.

Can you talk a little bit about your experience with Rust in general? Do you regularly run into issues where you’re like “Okay, this is annoying because it’s in Rust?”

In the first six months, yes all the time. Especially the way Nannou has been designed. You have the model-view-controller pattern, so when you’re drawing something you cannot mutate the data. From the Processing way of thinking, you can do it whenever you want, wherever you want. Sometimes there is stuff that I’d like to do, where it makes sense to draw as you’re mutating the model.

Also, the thing that I’m not sure I’m doing right, even right now, is that when you have a list of things and you want to mutate them two elements at the same time. I just use split_at_mut.

I have a few tricks that I’m sure are wrong, but right now I think I’ve reached a point where I think before deciding what I’m going to do. So, I know that I will be able to do this with my skills in Rust. I don’t hit any compiler errors right now, because I’ve reached the point where I’m fluent enough to do what I want to do.

Are there any projects you’ve done that stick out in your mind as being really difficult from a technical perspective?

I think the hardest thing was when I tried to do real-time path tracing 2d simulation. There was a time of, maybe last summer where I had those really crazy effects where lights were bouncing around walls and shining.

That was the results of getting to know compute shaders in WebGPU. The whole stuff was like maybe 1000 lines of code just to prepare for the rendering. Then finally just inputting the data and making the stuff move was pretty easy. But it took me maybe a month of incrementally adding the next block to the puzzle so that I know that eventually I would be able to do that.

And when I was was able to do that I just abused it for two months. Like, I spent one month building the tech, I’m going to milk it for a long time!

That was really hard, but I learned so much about compute shaders, render pipelines, how to time the GPU to cache so that all the data is moving in the right way at the right time. So that, eventually, it works in almost real time.

That’s what I was talking about getting the GPU on their knees. That was the part where I managed to crust the system a lot. When you do compute shaders, you’re really free to do whatever you want. And when the GPU is not responding, the OS is like “I don’t know what is happening”, but it needs to just reset the driver, and you’re stuck in the middle of a loop where your computer is showing nothing… you’re hoping that you might come back…

Have you found compile times to be much of an issue?

Not that much. Right now, my compile time is like five seconds for the dailies. I use Vim, I have the LSP plugin, so the compiler will tell me any error before I try to compile. When I write the code, it’s almost too fast, the feedback I get from the computer.

From this perspective, I’m really happy with my workflow right now. When the compiler will not complain any more: I will get the results I want. Then it’s thinking about the algorithm itself, it’s not about the way to express it in code. So personally I’m super happy with the compile time and workflow.

I saw your GitHub Universe stream, and I was wondering whether that was a super polished presentation or whether that’s representative of your daily coding flow.

If you do daily times like I do, any bit of optimization is worth taking the time to master. It’s about finding the set of tools that will allow me to be efficient and not spend a lot of time in the tooling because I don’t have that time every day.

I also code a lot for work. So Vim is a skill that I don’t see a disadvantage of learning to use at a higher level. A few years ago, before LSP was the norm, I would say that maybe using Visual Studio might be a better choice, just to get the integration with the compiler. But right now with the new plugins and everything it makes absolutely no difference if you use VSCode or Vim.

I put Vim mode in all my other software. I’m so used to thinking in modes I do all my reports for work in Vim.

Have you used Rust for anything other than creative projects?

I’ve been using Rust and Nannou in my work as well. I’ve been building tools to design murals, to design tiles, to design walls. I’ve been doing a lot with small robots for my company and I’ve built a toolchain around them with Rust and Nannou. I’m actually using that hardware as an interface for my dailies, so that I can change all the parameters with them.

So yeah Rust right now is my default choice. I don’t think I can use any other language. I worked a lot in Java but that was 20 years ago. I don’t know any C or C++ to the point where it would make sense to consider them.

I also like cargo and the whole ecosystem behind Rust. It just runs. The whole process of making it run on a new system is really straightforward. I have yet to encounter an issue where I was not able to just take a computer, install Rust, install cargo, cargo run and then it runs. This has been so easy in terms of deployment.

Recently, I’ve done JavaScript and Rust. JavaScript when it has to run on the web. I’m hoping once WebAssembly is available in a few months that I can just switch everything to Rust. For me, there is no other choice right now.

Is there anything else you’d like to mention?

I think if anyone is interested in the concept of Rust and trying out something new, creative coding is something that is really satisfying. You write something, and you get a nice picture. In terms of not writing another Hello World tutorial, making a nice circle on the screen is a little more interesting in terms of getting to know the language. It could be a really good introduction.

Like, if you look at how I designed my presentation for GitHub Universe, I introduce a lot of concepts in an incremental manner. You just start with getting cargo to create a new binary to make it run. All the steps that you need to make your own projects, well, you can do them with Nannou.

And you get a nice picture as well which could be more rewarding.

Always Bet On Rust

Discussion about this post