A whole lot of animation, with a whole lack of compute.

Ok, 'lack' is probably a strong word. And this was a topic that, when I first began researching it for my complex games unit earlier this year, I had not heard of, despite its honest to goodness awesome nature!

I am, by no means, an experienced enough programmer to write a decisively fantastic resource on this topic, but, it would be cool to get words down about my experience using Vertex Animation Textures (VATs, from now on) and talk through the theory and implementation of them, assuming you're going from the ground up!

The idea to create and use VATs came about as a part of my capstone project at AIE and a core feature of the custom C++ Game Engine I was writing for it, and while at the time the project wasn't confirmed, it was a good first step in setting up theory and a solid framework to prove important concepts!

Chapter the First: What the heck IS a VAT?

Indulge me a little, for a piece of tech that I spent a good amount of time developing, I would like to infodump a lot. Feel free to skip forward to the next section if you're already pretty in the know about them!

But, first, in short, and in a fashion I would answer this question if you caught and asked me at my workstation, "This is":

I will assume no further questions from the audience.

But, in all seriousness, to understand what a VAT is it's best to first understand and revise the basics of what Skeletal Animation is, and why we may use them in its place. Skeletal Animation involves using a collection of matrices that represent the position, scale and rotation of a metaphorical 'bone' in a set of bones called a 'rig' (or 'joints' and 'armature,' for those of you more Blender inclined folk). These matrices operate on the positions of vertices, typically three-dimensional positions in space, and transform the base positions of a mesh to another pose. If you want a quick graphics programming focused rundown on the topic, this article on learnopengl.com, or these videos and articles by ogldev are both a good place to start! They're where I learnt from!

The perks of Skeletal Animation are honestly, pretty clear. With a robust enough system, and a cool enough animator, you can do mostly anything to any mesh! And it is the standard practise for animation in games, film and other mediums! The major con of Skeletal Animation is that, in a lot of ways, it is computationally expensive, given each entity needs to, every frame it would be animated, traverse its skeleton and update the transforms of each bone. That can be a lot, especially for a game like Mall or Nothing (my capstone at AIE, read more here!)where there were expected to be about 200-300 animated entities loaded at once! And given the time frame!

VATs address the computationally expensive part of the problem, with some admitted tradeoffs. The reason Skeletal Animation is a bit pricey but still used is its versatility. It can animate anything, and blend between mostly anything as well. It'll let you have complicated animated characters, which is where its strengths lie. It is why we used it for the animation of our 'leader' units in Mall or Nothing, they were bigger, more important units, and their animations had important information to communicate to the player. VATs let you have a lot of smaller animations, at the same cost of rendering a textured static mesh to the screen.

To properly answer what a VAT is, I'll grab a quick slice of the image I showed above and we can walk through it!

The two dimensions of the each represent the Vertex ID (on the Y-Axis/the height), and the Timestep (on the X-Axis/the width), and then by just walking horizontally along the texture over time, and diving down depending on what vertex you're looking at, you can quickly find the offset for any vertex at any stage of the baked animation!

To generate one we take the transformed positions of each of the vertices of a mesh, and then store the offset between them as a 3-dimensional vector, and then store than in RGB channels of an image. Before doing this, there is a pre-processing/planning step that needs to be done, where the 'maximum offset' a singular point can have needs to be defined. For Mall or Nothing, we assumed that 2 units (meters in our case) was a safe bet, but honestly that could have been clamped tighter. Then, the offset value is remapped from a range of negative to positive maximum offset, to the colour range of 0-255. That means that those grey splotches you see above are vertices that are roughly stationary at that steph of the animation!

Chapter the Second: Why the heck would you use a VAT?

The reason that VATs are so efficient for animating large swathes of things comes back to the idea of limited computation. A VAT removes the runtime cost of animating entities, instead replacing it with a fixed asset creation time cost. You only need to play any given animation once to generate the texture, and then load that texture into VRAM as it is needed. It removes any additional overhead of animating and reanimating and interpolating, because all of that is done for you.

As mentioned, Skeletal Animation does have its benefits, versatility is one. And VATs definitely sacrifice that in favour of pure performance above all else. Blending, Reactive Animation, Procedural Animations, all of these require runtime calculations to react to events and inputs from the game world and player. VATs are better when animations can be a bit more rigid, and a bit more simple, but you just need a lot of them. They're ideal for crowds, or in our case, lots and lots of really little guys!

You saw this lil friend on the landing page if you came that way, this is the Bat from my capstone project at AIE, Mall or Nothing! The bat itself has a remarkably simple rig, two bones only, one operating on each wing. The gif here is showing him animating with a VAT, but if I had shown you a version animating with a skeleton, it would look identical.

Chapter the Third: How did I implement them?

The initial implementation came from a modified version of my Skeletal Animation system I had written both for Vulkan, and for OpenGL. As I already had the basis, arguably the 'hard' part of writing your own tool was solved prior, but to walk through step one, get Skeletal Animation working in whatever renderer you are working in. That means loading the hierarchy of bones from a mesh, ensuring that animation plays back properly, and then seeing it work on screen. Trying to do this without this step, without having a semi-decent basis of an animator to work with, would be so much more painful.

The process to generate a texture was simple from there, as the method of generating the offset positions is very similar to how you animate a mesh on the GPU in a vertex shader. In a rough summary:
  - Work out what bones influence what vertices, and what 'weight' is attached to that (this is generally an asset import step, where each vertex can keep track of its relationship to the skeleton)
  - Iterate through the vertices for each time step you want (you can set this to whatever you please, our VATs assumed 60fps), and calculate their new positions by summing the matrices of each bone that influences that vertex
  - Calculate the delta vector to the offset position (some resources store the offset itself, this definitely reduces the scope of animation you can do)
  - Map each value in that vector to an RGB value, and store them in the texture.

And thats it! The method is pretty fast, handles any animation you could normally display with a skeleton. If you already have Skeletal Animation in and working, adding VATs isn't a huge time sink, but it definitely doesn't suit every project and use case!

Final Thoughts

I think that VATs are an awesome technique, and that can't be understated. They were a pretty fun task to implement, and something that I think lent Mall or Nothing a lot of character in a way it needed. The large amounts of minions all having simple little movements was a big part of the charm of the game, and if they didn't have that, I think a lot of the fun, wacky vibes we went for would have been lost.

I am excited to try and do more with them going forward, there was a lot of stuff I didn't get the chance to do for Mall or Nothing that I think I still want to. Things like normal generation, varying maximum offsets for differing animation resolutions, and even creating blending textures are all things I would love to explore in a VAT toolset.

Keep an eye here! When I finally publish my VAT Generator Tool I'll put a link here! Just got some clean up to do on it!