Let me just take a look at this issue reeeeeal quick...

Today, a Github user filed an issue on the Tillerinobot issue tracker. It's a small issue, but it kept me busy for a whole while. I wanted to tell the story behind this because every developer has had a day like this, and I thought that maybe somebody might get a kick out of it.

The issue

lualex403 describes that the pp query breaks for a map. Something about a corrupt file. And interestingly, this also happens when they copy the beatmap, but only when they query that new beatmap a second time. Very interesting. I comment "Weird, lemme take a look," and get to work.

Diagnosing the issue

I remember that a long time ago, some beatmaps could be corrupted when downloaded directly. The MD5 sum of the beatmap file would then not match the MD5 sum presented in the beatmap info. Well this doesn't necessarily have to be the same problem, but it might be related. So it's probably a good idea to compare the MD5 sum of the beatmap with the MD5 sum on record. I hop over into the Swagger UI of Tillerinobot where it is easiest to query the API when you don't know all the URLs by heart. I fill in the query, click Execute and... "TypeError: NetworkError when attempting to fetch resource." Oops.

Diagnosing the other issue

Uh, that's weird. Well, I guess I'll check out what's going on, then. So, I open the Firefox Developer Tools and check out the console tab. "Cross-Origin Request Blocked." What? That can't be. This should work just fine, there's a bit in the code that handles this. "Reason: CORS preflight response did not succeed." Oh, huh. Weird. Let's look at what the request does! So, I hop over to the network tab, click Execute again, and sure enough: "500 Internal Server Error." Oh boy, that's not good.

So, let's dig out the logs. "Could not send response error 500: java.lang.RuntimeException: A MultiException has 4 exceptions." That doesn't sound good. Blahblahblah ... "javax.ws.rs.ProcessingException: Error creating a JAXBContext" blahblahblah. Oh, JAXB. I seem to remember that JAXB was once a part of the standard Java distribution but was removed during the Java 9 massacre. Or it's in some non-standard module now. Should be easy enough to fix. I briefly wonder: what does it even need JAXB for? But I don't really care right now. I'm trying to fix something else, afterall.

So, the next step is to reproduce the error in an automated test. Since this issue is probably coupled to the specific version of Java that we're running on, and might even be related to command line options, I should make sure that I'm as close to the running thing as possible. The good thing is that the bot runs in a Docker container and I already have everything in place to automate tests against that container. Cool. So then let's write a test.

Testing the other issue

All I need to do is have a unit test fire an OPTIONS HTTP request against any part of the API. Seems simple enough. My IDE is complaining about compilation errors, though. Oh that's just because I was messing around with stuff. I'll just do a mvn install on the master branch of Tillerinobot and I'll be set... "Failed to execute goal com.github.spotbugs:spotbugs-maven-plugin:3.1.11:spotbugs." Uh, what now? Now let's see here... Blahblah "Could not initialize class org.codehaus.groovy.vmplugin.v7.Java7". Java 7? That came out in 2011. I'm running... [java -version] 14! Wait, really? Cool, I must have upgraded recently.

Whatever, I'm not even going to try to figure out what's going on here. Probably an upgrade of the Spotbugs plugin will fix this. Let's bump this to... 4.0.4. Looks like this one was overdue anyway. Pleeeease don't find any new bugs, please, please... "build success". Excellent. Let's push this straight to master. Doesn't hurt.

Sidenote

Pushing a change to master in the Tillerinobot repository will deploy straight to "production". That's right. Live what you preach. In addition to the Travis CI, there is another CI set up on Gitlab.com which will grab changes from Github.com (yes, Gitlab.com and Gihub.com working together. The most ambitious crossover...), sprinkle some stuff on top, build Docker images, test these images, pushes them to a Docker registry, and deploys the new images on a Kubernetes cluster.

I spent quite some time setting all this up in a way that isn't flaky as heck. Not necessarily because it is required for this use case, but also as a great learning opportunity. We'll learn some more about this deeper into our voyage.

Back in the saddle

Great, now that that's out of the way, I can get back to... what was I doing? Right, back to figuring out the OPTIONS issue. All my compilation errors have vanished, and the test is quickly written. However, when executing it, there appear to be problems with Docker. Oh well, of course there are. Docker isn't even running. D'uh.

I like working on my old Desktop. I don't use it that often, mostly I work on a Macbook. But sometimes this Desktop is just what I need. 144Hz Monitor, great Keyboard and Mouse. It's a little under-powered, but it still runs just fine. How old is it? Well, it's running Windows 7, and I'm too lazy to upgrade.

You might recall that Windows 7 doesn't support Docker (at least the last time I checked). What I'm not too lazy for is having a Virtual Machine run Docker on Linux with an exposed Docker socket, so that I can have Docker on Windows 7. Take that! So, all I need to do is start the Virtual Machine and get going. However, after starting the Virtual Machine, Docker isn't reachable.

Diagnosing dockerd

Alright, let's see what going on here. So, I log in to the Virtual Machine. It's running Ubuntu 20.04, so we're in the world of systemctl, journalctl, and... Snap. That word alone makes me shiver. So I need to google a bit to figure out all the commands, how to even find out the name of the Docker service ("snap.docker.dockerd.service" smh).

Aaanyway, the logs say something about "overlayfs: missing 'lowerdir'." Some googling reveals the following: apparently, this can be an issue after an update. There's something about a full hard drive, and for somebody a graphics card driver was the issue. Yeah, you know what? I'm not even going to bother. I'm just going to set up the machine from scratch. I remember that finding the config files in Snap's weird folder structure was a bit annoying when I set this up a while back, but other than that, this shouldn't take long.

Setting up Ubuntu... again

Now here's some luck: I still have the Ubuntu 20.04 image in my Downloads folder. Looks like the machine broke after just a couple of months ... from an automated update. Good job, Canonical. Alright, let's get going. I've done this a million times. OpenSSH? Yes, please. You can install Docker from the OS installer? Neat. Yes, please. This is going great.

The setup finishes, the machine reboots. I log in and type docker ps. The machine tells me that Docker is not installed and what options I have if I want to install it. There's an apt option, and a Snap *shiver* option. At this point, in a fit of rage, I delete the Virtual Machine. And just to make a point, I delete the old one, too (the one that broke with an update). Screw you, Ubuntu. You were once cool, but since you've started with all that "cloud" stuff, you've become kind of a douche. And I hate that Snap kid. You can both just heck off.

Finding new friends

A short google search reveals that there are more options, if you really only want to run Docker. I settle on Photon OS. The ISO is tiny, the installation takes 33 seconds. Wow, that's pretty cool. I set up dockerd to also work through a TCP port. Photon OS is completely firewalled in by default, so I once again get to duel an old friend: iptables. It doesn't stand a chance. My complete ignorance of security on my home network is too strong.

Before too long (wait, when did I start with this?) Docker is working on my dusty Windows 7 machine, once again. I make sure to take a snapshot (haha, get it? Snap) of the Virtual Machine so that I can revert any breaking updates in the future. Cool, let's get back to the thing that were doing. Uh...

Things escalate

Sometime in the last however-long-I-took, I got an email informing me that the automated deployment of the changes that I pushed earlier had failed. The changes were only an upgrade of a plugin that does not affect the software directly, so there couldn't have been any issues with the actual software. However, when you have tests that require timeouts, sometimes there are glitches, and the tests fail. So probably nothing to worry about. Let's check out the logs of the deployment job... "Error: Unauthorized". That's not good.

The error happens during a Helm command which is supposed to get Kubernetes to deploy the new version of the Docker image that was just built. I figure something must be wrong with the authentication of the deploy script. I built this over a year ago, so I have no idea how it even works.

A little digging reveals the following: The deploy script authenticates with a certificate. Somehow Kubernetes recognizes this certificate, but at this point, I'm not sure how. The certificate (and also the private key) are Base64-encoded in the ~/.kube/config file of the Gitlab pipeline. A quick look at the certificate reveals that it... expired. It was good for one year.

Making certificates

While there is now an explanation as to why the deployment doesn't work, I have no idea how to fix it, since I set up all of that over a year ago. So... how does all that work again? I remember being confused but that there was a sudden clarity when I first worked on it. An uncoordinated google search of several vague phrases that clearly show that I have no idea what's going on (I reviewed these later but am too embarrassed to share them) slowly leads me on the right path. Apparently, you just have to have a certificate signed by the Kubernetes cluster (or a trusted CA) that is made out to the common name equal to the username. So basically, the way that HTTPS works, but for clients instead of servers. That's pretty cool. All I need to figure out is how to sign such a certificate.

I find this guide in the official documentation. Since I don't want to be in the same position in a year, I make sure that everything I do is reproducible and do everything in a small script that I plan to commit to the git repository where I keep all the Kubernetes configuration. Everything slowly falls into place, I create the new configuration, and...  "Error: Unauthorized"

What's wrong? Date is fine. Common name is fine. Why did my old certificate have an organization? Whatever, let's replicate that in the new certificate. Try again:  "Error: Unauthorized". Everything looks identical except the signature, and the date. Why? Whyyyy? At this point, I pull up both certificates next to each other and literally cross my eyes at the screen (diff is for suckers), and there it is: "X509v3 Extended Key Usage: TLS Web Server Authentication". Web Server? The old one said, "Web Client". That's right: the guide that I followed signs a certificate for a Server. Oof.

Don't worry. This tiny detail only cost me half an hour to find *panting*. No biggie *twitch*. Maybe I can just change that one word in the code that I don't understand from the guide that I skimmed, and it will work? Oh look: It works. Well, cya next year!

Fixing the other issue

What did I even want to do in the first place? Right, somehow fetching OPTIONS doesn't work. Somewhere deeper in the stack trace, we find "javax.xml.bind.JAXBException: Provider com.sun.xml.internal.bind.v2.ContextFactory not found". Just as suspected. Easy fix: just need to add the default JAXB runtime implementation. Works? Yup. Let's push it. Should be deployed automatically. It better...

Fixing the actual issue?

The automated deployment works as expected and so does the fix. I download the beatmap that is stored by Tillerinobot. I check out the cached beatmap data. The checksum matches. Wait, then what's the problem? I query the API to get some pp info on the map and it... works just fine. Wait, what's even the issue here. I search the logs of the last day for "corrupt" and there was no such message sent out. What is this issue even about? I guess I'll have to comment once more and have to find out. Github says my original comment is from "8 hours ago". I started writing this towards the end of the journey which took up around half of that time. So, four hours for nothing?

What even happened?

I feel like there is a lesson here. In a professional setting, this would have been a catastrophe. Specification unclear, outdated tooling, unmaintained infrastructure. Since I don't do this for money, though, we can let that slide. I did learn a bunch and did fix some issues.

There are two things that are visualized nicely in this story, though. For one thing: estimations are garbage. I thought that this was a quick adventure. In and out. Nope. There is this rule that suggests that you multiply however long you think it takes to do something by Pi. When I first read this, the article had some nice pictures to explain this. I can't find that article anymore, so let me just draw this with words:

What you think you need to do is this (draws a straight line). What you actually need to do is that (extends the line to twice the length). And here is how you get there (draws a half-circle from the start to the end of the extended line). The length of that half-circle is Pi times the length of the first line.

Even that estimation probably would have been too low in my case. I don't really know, since I didn't fix the original issue. The point is: things pop up that you just couldn't have thought of. While I should have had an automated test for CORS headers, and probably should just just upgrade from Windows 7, there was just no way that I could have foreseen the Kubernetes certificate having expired.

The other thing: this is what programmers do a lot of the time. We don't sit there thinking about algorithms to solve the problem assigned to us or frantically hacking away at the keyboard. Large amounts of time just go into fixing weird stuff by googling our butts off.

Finishing thoughts

The world of software is just too large to know - let alone foresee - everything. Look at what this journey touched on: CORS, JAXB, Java, Groovy, Spotbugs, Docker, Ubuntu, Snap *shiver*, Photon OS, iptables, Virtual Machines, Gitlab (CI), Kubernetes, X509, not even mentioning all the little tools around the edges (git, bash, IDEs, Maven, openssl, Firefox Developer Tools, ...). I'll admit that some of these things are just unnecessary in this context. I chose some of those technologies because this is where I get to learn. And there are a couple of things in that list that I would confidently say I have mastered. But still: nobody actually knows everything about all the technologies they're using. They couldn't and that's fine.

However, this can be a big hurdle for more junior developers. Ask somebody who just started out coding how to go about programming this-and-that project. Probably they'll just tell you to use the programming language that they've been learning and that'll be it. Why? Because it enables them to do things and they like it. Ask a slightly established developer the same question and they'll throw a shopping list of technologies at you. Why? Basically the same reason: Each of those technologies solves a specific problem that they've had in the past. It's just that they've come across a range of problems that you're probably not thinking about and don't need to be thinking about.

What sucks is when you find yourself in a situation where you have to use the giant list of fancy technologies despite not seeing the need, e.g. in a professional environment. A friend of mine quit his job as a developer partly because he was frustrated by the endless stream of new stuff he had to familiarize himself with just to accomplish basic things in the projects that he worked on professionally. What adds to that frustration is that there's always somebody around who appears to already know everything. Don't worry. They don't. Nobody actually knows everything about all the technologies they're using. They couldn't and that's fine.

I'll stop trying to wrap this text up in a neat way. Honestly, I just thought that some programmers might get a laugh out of this story since we've all experienced this. I kind of want to write more and will dip my toes in the water when an opportunity arises.