Gamedev, graphics, open source. Shuffling bytes and swizzling vectors since 2004...
24 January 2022
It has been a while since I last posted an article. For the past 6 months I haven’t managed to get much coding done in my spare time. Partly due to work consuming most of my effort but also due to the relaxing of lockdown restrictions and making sure to take advantage of the opportunity to see my friends and family in real life. I had remaining holiday days to take from work, giving me 2 weeks off over the Christmas period and I was determined to get stuck into something new. Eventually I settled on building a new graphics engine in Rust. I already have a C++ engine, so this article is about why I am switching to Rust and why you should even build your own tech at all.
After finishing work for the Christmas break, on my first day off I wrote a new blog post which I never published. I started with an idea that I wanted to focus on modern graphics rendering techniques (mostly bindless, gpu-driven and ray tracing). My C++ engine on GitHub pmtech is very Direct3D11 focused and the design doesn’t incorporate these things. I could keep working on that engine, but I made the initial decision to start something fresh and to try and reduce the amount of code and dependencies required. Initially I was going to do this in C++/Win32/Direct3D12 and my ideas for reducing the amount of dependencies compared to pmtech was to use more modern C++ features (threads, filesystem and so on). pmtech actually started in the days of C++03 and in those days your base library would typically implement more platform abstractions - I have a fair amount of baggage to carry around, which would be nice to drop. I also wanted to ditch OpenGL and support fewer platforms and renderers as the baggage of maintaining CI for 5x platforms and 6x rendering backends (counting WebGL, GLES, OpenGL) is prohibitively time consuming at times.
I thought modern C++ would be a good starting point, I was going to go all in with C++20 and so I started to think about engine features I wanted, but almost immediately I got stuck in my tracks and wasn’t sure what to do. My first sticking point was with serialisation, JSON and more specifically reflection. In the engine I have developed at work we have this code generator tool called serj, lovingly named after the excellent Serde for Rust. It can generate automatic C++ to JSON serialisation and deserialisation code, as well as ImGui property browsers and other “Mad Scientist” features like swizzling SoA to AoS for serialising data-oriented component arrays. It’s a really useful tool, but It was developed for work so I felt like it wouldn’t be appropriate to use this code (and make it open source). If I were to implement something similar from scratch, it would end up being the same or similar to serj so I felt like this option was off the table. I looked into C++ Reflection which is still not widely available and other approaches to reflection require macros and other boilerplate code which I wasn’t keen on. I evaluated quite a lot of ideas and I spent a good few days thinking about it. The circle compiler has some crazily good looking features but is Linux only which is prohibitive for me as I want something cross platform.
I’ve spent the past 20 years building C++ game engines. So why start making another one? Well, to focus on modern graphics and have a clean slate was one reason, but learning another programming language in the process might broaden my horizons even further and give me opportunity to learn a lot more. So I decided since I so desperately wanted the serialisation support that serde provided in Rust, why not just use Rust instead of C++?
This is a polarising topic and I have already been shredded on HN before for making a similar data format to JSON5 called jsn with some tweaks and improvements that I find useful. People shoot you down with “Not invented here” syndrome and all that, so it seems pointless making anything new if you listen to them. If you try and compete with the likes of Unity and Unreal it can feel futile, @longbool discussed the perceived inevitability of Unreal.
There have however, been positive and encouraging articles on the subject of building your own tech. @TylerGlaiel wrote a nice article on how to build your own tech and why and @longbool again with some advice for people to find a way into graphics in the modern landscape. I would like to encourage people to just build something for the pure love of it… I am building my own tech and always will be. I choose to work on something for fun, to learn, just for sheer joy! I am ‘throwing’ away 5 years of work on a C++/Direct3D11 engine and starting again in Rust/Direct3D12 from nothing, but it’s the experience gained from previous work that will help make the next project better, so it’s not wasted; it’s part of the journey. Just like a buddhist monk wiping away complex sand mandala patterns after completion, not everything has to last, not everything has to be ground breaking or innovative, I am just trying to really absorb as much as I can and enjoy the process.
During my first industry job I worked at Juice Games which was owned by THQ, THQ were struggling and making redundancies and Juice was struggling to get projects green-lit. In my day job I was working on core tech and tools, and at home I was working on a deferred renderer (back in the days when this was cutting edge games tech). Some people at work had seen what I was working on and I got asked to investigate indoor lighting for Warhammer 40k KillTeam as the engine at the time only supported single directional outdoor lighting. I didn’t implement a deferred renderer in this instance, I was young and not confident enough to stick my neck out and pull it off on PS3 and Xbox 360. At this point there were no games I was aware of that had shipped deferred, but I did implement a forward renderer with per-object light selection, baked lightmaps and SH probes which gave us enough lights to play with. I wasn’t confident switching the deferred renderer but my development of lighting algorithms and understanding I gained in my work at home essentially kept me a job and got me my first credit on a game for still something I am very proud of.
With this new project, albeit for fun and for learning, there are still goals and reasons behind building something new from scratch. There is also an opportunity to learn from past mistakes. In pmtech, as any code base which is a few years old, there are inevitably things that could be improved upon. An example is the multi-threading model, pmtech has a single dedicated render thread which comes from a time of machines with fewer cores. So here is a rough outline of what I want to get out of this project:
I started the engine focusing on Direct3D12 and Win32 through windows-rs. This was really easy to get up and running, being familiar with Direct3D and Win32 it really is one of the things that made me start this project. I know there are many FFI bindings around but knowing Microsoft was actively developing it and supporting it gave me confidence to start.
It has been nice going back to basics and learning a new language, it has changed the pace at which I am used to working. I had previously written a few programs in Rust so I didn’t start off as a total newbie, but previously I had only written some small functional programs which were pretty straightforward, there was a lot of Rust I had not yet touched.
At first it was a little frustrating because I was trying to do things that come second nature to me in C++ and then I realised Rust doesn’t have the same features so it requires some research. But this has been refreshing and the change of pace has been good, a little humbling as you get stuck on simple things and make silly mistakes, struggle to get past the borrow checker and so on. I started to embrace idiomatic Rust and strive to do things the Rust way and I think it’s really great that the language provides so much for you to get you onboard.
After a week or so I was fully on board and was surprised at how much of my coding style had changed and how I naturally just started doing things I would not normally do. I began with a more granular TDD style approach because cargo test just makes adding tests so easy; it is already there and ready to use, you don’t have to make any decisions about which testing framework to use, the default one is just great. The same goes for cargo fmt and cargo doc. I will be honest and say that code documentation is not one of my favourite things… I have seen and worked with codebases with stale documentation, attempts to switch documentation generators and abandoned ideas and I’m not saying this is acceptable, but the barrier for entry with cargo doc is just amazing. Because your docs look the same as the official docs it just made me actually want to keep things up-to-date and to get good coverage. This is a really good thing in my opinion, Rust is making me want to do things the right way and to uphold good software development practises.
I’ll post some more articles about my progress on the engine and go into more detail about Rust and the gfx API I have been working on. The repository is already public on GitHub to take a look, or keep an eye on over time… Follow me on Twitter or GitHub if you are interested to see where this goes. I have been having so much fun so far and I have a good feeling about this.