Hello, My name is REDACTED. But I mainly go online as “Little Cat”. If you want to reach out to me, I’m mainly active on my Twitter.

More info about me on my About page.

  • A Challenging World of CI/CDs

    Jul 8, 2020

    a.k.a. “If you want something done right, do it yourself!”

    When you’re developing a huge project (in this case, Toontown Offline), there’s no doubt that having a CI/CD pipeline is an absolute must have, and it has became standard practice in software development today, but the problem is that if you’re working on a closed-source project, you’d have to either pay for a CI/CD service, self-host your own instance, or just plain build one yourself from scratch. For Offline, we host our own at first, but later on we went with the building one ourselves. Here’s how it all went down.

    In case you don’t know, CI (Continuous Intergration) is an atomated service which helps maintains the source code of a project. You can make it build the code for you, run the tests for you, or both. While CD (Continous Deployment) is the same thing, but for deploying your project.

    Back when Toontown Offline used to use Nirai to compile the game, we would just run a script on a server and it would pull the latest code, run through the Gamedata building process (both the game engine executable and the game logic were built seperetly back then), and deploy the thing off, just like that. But with Pypperoni however, there are no such thing as Gamedata files. All of the Python logic for the game gets generated into C code and compiles right into the executable and we have to do this for all three of our target platforms, Windows, macOS and Linux, and we do NOT want to run a seperate series of scripts to build all three platforms at once. So I thought: “Hey, we should try using a CI service to build the game for us!”. And I knew what the exact service to try out on. Past me didn’t realize that we would be making a huge mistake.

    That server in question was AppVeyor. It supports our self-hosting Git repository, has the ability to run Windows pipelines and they have the option to self-host your own instance instead of paying for one of their plans. Cool! So Gustavo (our co-lead developer) went ahead and purchased a few servers for us. One for the web instance, one which runs on Windows Server 2016 to do Windows builds on, and one Linux server (which never went used). We haven’t really figured out how to handle macOS builds at the time, so we’ve let that aside. Once everything has been hooked up and ready to use, I quickly got to work. Few days, all seem to work well, except for one problem.

    Because Pypperoni generates thousands and thousands of C files to compile for our project, the Windows builds are slow. painfully slow. Like, it would take about one hour to build the game. And guess what, by default the build will automatically fail when the build time reaches above one hour for some reason (either because of the windows server suddenly became slow and needs a reboot, or the compile process was just slow overall). It doesn’t help with the fact that the Windows server only has 4 threads to work with. But it does exactly what we needed it to do. Automatically build the game for us. Meanwhile, we’ve modified the build script on our other server to act as a “deploy” script. Which fetches the Windows build artifact from AppVeyor and does its usual deployment process.

    For throughout the entire v1.0.0.x version history, we’ve stuck that way. But the nagging issues from AppVeyor have finally paid the price on us. We grew sick and tired of having to wait a full hour to build and deploy the game even if its just a simple batch of bugfixes. Not to mention that the servers are just too costly, and the fact that it sometimes kept dying on us. Also, we can only do Windows builds through there. Yes, we have a Linux server ready to go, but my love for AppVeyor was already over at that point to even try getting Linux working there. And then another worry has hit us, what if we release a huge update (for example, Seek Out Scrooge), and it adds to the build time considerably? We absolutely do NOT want to go through that possibility. So we went ahead and do a totally different and custom made solution, but it’s just enough to fit our needs.

    Gustavo ordered out a full-fledged custom made dedicated server (which contains a 32 Core CPU with 12 TB of storage) which we’ve named “Finland”. And under my wanted specifications, he has also developed a custom Discord bot which resides in our Discord server under the name “Robo-Patrick” (from Battle for Bikini Bottom) which acts as our own CI/CD solution to pull and build the game from within Finland. And it would complete builds within more than 7 minutes! Which is a massive improvement than our Windows server for AppVeyor.

    From our v1.0.1.11 build (the first with macOS implemented):

    You might have noticed that we have all three platform builds setup instead of just one. That is because we now have the ability to cross-compile our builds for other platforms and it’s all thanks to Docker containers! We have two Docker images built within Finland. One for Windows builds (which is based on MSVCDocker, making use of Wine to compile the game) and another for Linux builds (which is just a stock Ubuntu 16.04 image). For macOS, we’re using the OSXCross toolchain but instead of containing within a docker image, it is installed system-wide, which was probably the reason behind the fast build time comparing to Linux’s.

    And the best part about all this is that because we have a 32 Core CPU installed in Finland, we have access to a whopping 64 threads to use. More than enough to build all three platforms simultaneously!

    The first 48 out of 64 threads being used during the build process:

    That way, we can use the first 16 threads for Windows builds, the next 16 for macOS, and the next 16 after that for Linux. Leaving us the last 16 to spare whenever we need it.

    Not only Robo-Patrick does game builds, it can also deploy the game for us too!

    I know, it’s nothing at all fancy compared to what you see at AppVeyor, Travis CI, etc. But for us, it was perfect. We don’t need all that other fancy schmancy stuff to build and deploy our game; if we need anything else, Gustavo could code it right in for us. He is also considering open-sourcing the code to Robo-Patrick on GitLab at some point in the future. If he does decide on doing so, I’ll edit the post to include the link to it.

    But yeah, the moral of the story is that sometimes, it’s best to do things our own way instead of someone elses. In this case, who needs AppVeyor and their slow Windows build servers while we can slip up our own, because we can? Right now, I love both Finland and Robo-Patrick and I don’t see that changing in the far future. Glory to Finland and our robot overlord Robo-Patrick!

    To close the blog post off, I am going to leave you off with a few words from the Creative Lead of Toontown Offline and fellow friend of mine, Benjamin:

    The deployment process we have now has been really, really motivating for myself and probably most of the other developers on the team. At the very least I speak for myself when I say this- but whenever I’m working on prod, non-SOS updates, I have this strong sense of control in the back of my mind. I think to myself that the only thing limiting me from updating the game now are my own two hands. In the past, there were so many barriers that came in between adding new content, and then that content becoming playable by other people. Before v1.0, sometimes it would be months before we had any certain date for when we could get a feature out.

    Building the game used to be seen as a special event to look forward to. But with the new system Gustavo and Little Cat have created, I now feel that that the only thing in between me and releasing a feature to the public is myself- my own ability- my own motivation. And to me, that is very motivating in and of itself. If I want a feature to be out by Tuesday night, it will be out by Tuesday night if I have anything to say about it. That’s how I feel at least.

  • Starting a blog!

    May 21, 2020

    So here we are. I’m finally starting a blog, with not a clue where to start from, huh? Doesn’t all blogs start out that way? Seems to be the norm.

    So hey folks, if you don’t know who I am, I’m Little Cat. I basically work on all these cool fan projects based on the now-closed MMO “Toontown Online”, since 2013 (Man, how time flies). I plan to write a series of posts about my Toontown career, so if that sparks your interest, look out for the first post!

    This blog may look a little barebones right now, but I’m just learning how to use Jekyll. I might change themes at some point, but right now, I’m happy with the plain and default one. I’ll be sure to tweet on Twitter whenever I make a new post. Or if you prefer, there’s also an RSS feed in this site if you’d like to add it to your reader.

    Alright, I’mma head out. Thanks for visiting!