Latest development projects


#1

Over the past week, I’ve broken ground on a bunch of new stuff, none of which has actually gone into a build yet. Here’s what I’ve been working on:

  • Binary saved games

Saved games are currently saved in a text format, which makes them easy to debug and easy for players to edit to give themselves more money (or anything else they want). Sadly, these text files take a lot of time to read.

I now have a proof of concept for saving/restoring these files from a binary format. For the largest saved game I’ve received from a tester (“Infinity Worlds”), saving using the binary format drops the file size from 129 megabytes to 118 megabytes (still far, far too big), but reduces the time it takes to load the file from 15 seconds to 9 seconds. Or to put it differently, it reduces the time it takes to figure out the contents of the file from 6 seconds to a fraction of a second. (the rest of the time is spent creating the objects specified by the file)

Save files still grow way too big. Even the binary-format file can be shrunk by a factor of ten just by zipping it, so I’m probably going to have to do that for saved games. (We already have zlib in the project, so I’ll probably just use that, I guess?)

Update: With a further optimisation, I now have that loading time down to 3 seconds, from 9. Which is pretty startling, but also awesome! Definitely need to make this all robust enough to put into a real build for people.

  • Scenarios

“Scenarios” are miniature plots/events which occur while you’re playing the game. Currently, I’m roughing out a scenario which involves an Internet personality streaming your game. In effect, it adds the concept of a “VIP” to the game; that person’s happiness with your game influences your game’s popularity with everyone else. You get alerts to let you know when they’re going to connect and when they’re streaming, and you can interfere or not; up to you.

I’m currently on the second big rewrite of the Scenario system; I think this version is going to work for almost anything we want to do. Crossing my fingers!

  • Pathfinding

I’ve now broken ground on the new pathfinding system, which will replace the current one. The new system should be vastly faster than the current one, when complicated walls and other curving obstructions are present in the world. This is still early-days, but it’s in progress.

  • Lots more art stuff.

I’m working with a new artist, now. You’ve already seen a few new models enter the game, and there’ll be more over the coming builds. This is taking a lot of my time right now, as we tweak shaders and discuss the general look of the game.


#2

A little update on the binary-format file saving:

I’ve reduced the size of binary files to about 60% of the text format files by reducing the space used for a few fields in the save file format. And I can probably reduce it further. And I’ll probably still zip it at the end of the process.

The massive 130 megabyte save file that I’m using for testing is more than 10x bigger than any other save game I’ve ever seen, which makes it a great test case. Right now, in binary format it takes up 88 megabytes, and zipping the file can drop that as low as 11 megabytes.

Right now, saving in binary format is slower than it should be; it currently takes about twice as long to save a file in binary format as to save in text format, for a couple of reasons. (writing that huge save file in binary format currently takes about 8 seconds, where it only takes about 4 seconds in text format). I’m working on those issues; once sorted out, binary saving should be really fast, like binary loading. :slight_smile:


#3

Oh hey, I’ve discovered that I actually have another big saved game. This one is about 111 megabytes in size. (This one is named “Blades of Legend”; the one I was testing earlier was “Infinity Worlds”. Both came from testers, and I’m embarrassed to admit that I don’t remember off the top of my head who each of those testers were!)

With the binary save, the Blades of Legend saved game drops from 111 megabytes to 75 megabytes. And when compressed, just 8.8 megabytes.

Saving that game now takes about 3.5 seconds as text, 3.3 seconds as binary, and 4.4 seconds when compressed. Loading the save now takes 9 seconds as text, or 3 seconds as binary. (I haven’t implemented loading compressed saves yet, so I’m not sure how much time that will take)

All of the above numbers were taken while loading and saving data to a solid state drive. If you’re using a traditional hard disk, these numbers will presumably all be larger, and the savings from reading/writing less data to the disk (in the compressed case) would presumably be even bigger.

Also note that when I say “loading the save”, I’m just talking about how long it takes to load the data from disk and create the in-game game objects from the saved game. The time it takes to build the world geometry and do other setup tasks isn’t included.


#4

Update: …and now loading of compressed save files works. 3.2 seconds, for loading from an 8.8 megabyte compressed save game, compared against 3.0 seconds to load the same data from a 75 megabyte uncompressed save game.

So, with decompression taking only an extra 0.2 seconds, and with the save data size shrinking by 90%, it’s hard to argue against compressing our save data…

I also need to ask whether it’s worth compressing (and converting into binary) all our other data files. I’m not too concerned about their size on disk, but the load speed improvement is pretty tempting. On the other hand, it makes it harder for people to peek into those files and mod them. I guess I’d want to provide some sort of tool which could unpack and repack the files.

Maybe I’ll defer that. For right now, I feel like I’ve proved out the concept and benefits of binary-format files, and saving/loading in compressed format.

There’s still a lot of work to be done before I can ship this functionality; right now, the game is hardcoded to only be able to load one type of saved game (text, binary, or binary-compressed). And the saved-game selector can only read save metadata out of text-format games, regardless of which type of data the rest of the game wants to read.

I need a way to load any of the three save types, so that folks can keep loading their old games, and save them into the new format. And the saved-game-selector needs to be able to read metadata out of all the different save formats. Maybe this just means sticking a string at the front of the file, declaring its file format.

Some other comments. 3 seconds to load a saved game is completely acceptable. However, 3 seconds to save the game is problematic, since I’m currently having the game autosave periodically. Sudden 3 second freezes in the middle of gameplay just because the game has decided to do a periodic save simply isn’t acceptable. I guess I’ll need to figure out where all this time is going… or else tie autosaves to particular user actions, instead of merely doing them periodically, so that they don’t feel so arbitrary.


#5

And just as one more quick update on the “how long it takes to save” question:

I now have it down to 0.7 seconds (on my rather fast computer), to save that huge game, down from about 7 seconds. Similarly, loading is down to about 2 seconds, from about 15 seconds (this doesn’t include loading art assets and creating the world mesh; only loading the game state data).

I must confess that I’m cheating a little bit, though; the 0.7 seconds isn’t actually how long it takes to save the game. Rather, that’s how long it takes for me to gather a copy of all the data that we’re going to save.

I then pass that data into a background thread, and let the gameplay continue while we compress the data and save it to disk at our leisure. This was suggested to me by a friend this afternoon, and it was such a neat idea I had to immediately drop everything else I was doing to try it out. Works a treat! This also means that saves should appear to take about the same amount of time, regardless of whether you’re saving to an SSD or a hard disk or even slower media; we only need to pause the game while we’re saving game data into a separate buffer in RAM. Once the game state snapshot is there, we can take as long as we like to write it out to disk, while the game itself continues!

I still have some adjustments I need to make to bring the GUI up to date with the new file saving regime, and ensuring that we retain support for loading from the old text-format files, but the bulk of this work is now complete! Looking forward to getting it all into a build for folks to test. :slight_smile:


#6

And the save-game feature is now wending its way through the build server; expect to see those changes in the Steam build in a few hours. Next task: updating pathfinding to no longer require raycasts! (And maybe simplifying the collision geometry for walls). Those two things together should vastly improve the performance of pathfinding, particularly in games containing a lot of subscribers and a lot of walls.


#7

And you’ll want to flag that you are “saving” so that the user can’t exit before the save actually hits the media, yeah?


#8

Yup, already done! There’s now a little “Saving” animation in the bottom right corner of the screen, which appears when a save is happening. (and possibly for a little before and after)

Additionally, the game won’t let you exit without saving; it always saves the game before exiting. And any save which was in progress when you close the game will delay the game shutdown until the save completes.

(Of course, all bets are off if you do a force-quit via Task Manager or Activity Monitor or kill -9. We still do our best to avoid winding up with any half-written invalid save files even in that situation, but no promises!)


#9

Aw shucks! You done thunk of everything! :slight_smile: :+1:


#10

I’ve been working on replacing our current pathfinding (using dynamic node graphs) with a new system (using nav meshes). Here’s a thing I’ve noticed:

Screenshot from 2017-09-01 13-33-42

What you’re looking at here is a nav mesh for a small portion of a region. It draws walkable areas in white, and you can see the ground through unwalkable areas. Black lines surround individual triangles in the nav mesh.

So some things to note. There are a LOT of long, thin triangles, and in particular, a lot of vertices along walls, even along their straight sections, but particularly around curves. On the top and bottom corners of this little enclosure in particular, there are so many thin triangles that they appear to just be big wide black lines. It’s all these extra vertices which turn out to be the root cause of how badly walls are affecting pathfinding speed in current builds of the game. As it turns out, they’re getting so many vertices because I accidentally told the geometry library I’m using (ClipperLib) to output rounded edges, instead of straight ones. If I change that, we instead get this:

Screenshot from 2017-09-01 13-31-40

LOTS fewer vertices around that wall!

As it turns out, just this change by itself vastly improves pathfinding performance, using the old system. It improves it by enough that I’m not sure whether it’s even worth the effort to convert us to using nav meshes at all.

In fact, it’s even better if I do some basic mesh simplification on the obstruction mesh; we can bring it down to this:

Screenshot from 2017-09-01 14-23-39

The obstruction doesn’t follow the subtly curving back wall quite as well here, but that won’t really cause any problems,and look how many fewer triangles there are!

So… at this point, I’m not sure how important it actually is to continue with this. Simplifying the obstruction geometry has vastly sped up the old pathfinding system. So… maybe that’s good enough?

On the other hand, look at this nav mesh inside a little town:

Screenshot from 2017-09-01 16-38-53

Is it not a thing of beauty? I almost want to finish the nav mesh system just so that I can have little subscribers pathfinding over it.

But really, I should probably get back onto the actual game mechanics, instead; leave this all for later, if pathfinding becomes a real performance problem again in the future.


#11

Here’s a screenshot where I’ve turned on shadows for players, NPCs, and monsters.

This has highlighted a little gotcha that I’d never noticed before. See how ‘Torywa’ is floating above the ground? That’s because he’s on a complex bit of terrain. There’s a crease crossing the geometry diagonally. When we place an object on the terrain, we just look at the four points around them and kind of blend between them based upon which corner is closest; we’re not aware of any hard crease that might exist. And that can lead to the character floating above the surface, when the terrain crease bends downward.

I guess our “how high is the terrain right here” code is going to have to be more clever! Once shadows are enabled, these slightly-floating characters are going to be extremely obvious!


#12

Took some doing, but here are players now walking along the ground in that same spot, now correctly following the actual ground mesh.

Shadows!

Still requires some more tweaking to keep shadows happy on small characters such as these, while also being happy on terrain viewed from a long distance away. But this is progress!