## June 24, 2017

The most surreal experience I ever had on discord was when someone PMed me to complain that my anti-spam bot wasn't working against a 200+ bot raid. I pointed out that it was never designed for large-scale attacks, and that discord's own rate-limiting would likely make it useless. He revealed he was selling spambot accounts at a rate of about $1 for 100 unique accounts and that he was being attacked by a rival spammer. My anti-spam bot had been dragged into a turf war between two spambot networks. We discussed possible mitigation strategies for worst-case scenarios, but agreed that most of them would involve false-positives and that discord showed no interest in fixing how exploitable their API was. I hoped that I would never have to implement such extreme measures into my bot. Yesterday, our server was attacked by over 40 spambots, and after discord's astonishingly useless "customer service" response, I was forced to do exactly that. ## A Brief History of Discord Bots Discord is built on a REST API, which was reverse engineered by late 2015 and used to make unofficial bots. To test out their bots, they would hunt for servers to "raid", invite their bots to the server, then spam so many messages it would softlock the client, because discord still didn't have any rate limiting. Naturally, as the designated punching bags of the internet, furries/bronies/Twilight fans/slash fiction writers/etc. were among the first targets. The attack on our server was so severe it took us almost 5 minutes of wrestling with an unresponsive client to ban them. Ironically, a few of the more popular bots today, such as "BooBot", are banned as a result of that attack, because the first thing the bot creator did was use it to raid our server. I immediately went to work building an anti-spam bot that muted anyone sending more than 4 messages per second. Building a program in a hostile environment like this is much different from writing a desktop app or a game, because the bot had to be bulletproof - it had to rate-limit itself and could not be allowed to crash, ever. Any bug that allowed a user to crash the bot was treated as P0, because it could be used by an attacker to cripple the server. Despite using a very simplistic spam detection algorithm, this turned out to be highly effective. Of course, back then, discord didn't have rate limiting, or verification, or role hierarchies, or searching chat logs, or even a way to look up where your last ping was, so most spammers were probably not accustomed to having to deal with any kind of anti-spam system. I added raid detection, autosilence, an isolation channel, and join alerts, but eventually we were targeted by a group from 4chan's /pol/ board. Because this was a sustained attack, they began crafting spam attacks timed just below the anti-spam threshold. This forced me to implement a much more sophisticated anti-spam system, using a heat algorithm with a linear decay rate, which is still in use today. This improved anti-spam system eventually made the /pol/ group give up entirely. I'm honestly amazed the simplistic "X messages in Y seconds" approach worked as long as it did. Of course, none of this can defend against a large scale attack. As I learned by my chance encounter with an actual spammer, it was getting easier and easier to amass an army of spambots to assault a channel instead of just using one or two. ## Anatomy Of A Modern Spambot Attack At peak times (usually during summer break), our server gets raided 1-2 times per day. These minor raids are often just 2-3 tweens who either attempt to troll the chat, or use a basic user script to spam an offensive message. Roughly 60-70% of these raids are either painfully obvious or immediately trigger the anti-spam bot. About 20% of the raids involve slightly intelligent attempts to troll the chat by being annoying without breaking the rules, which usually take about 5-10 minutes to be "exposed". About 5-10% of the raids are large, involving 8 or more people, but they are also very obvious and can be easily confined to an isolation channel. Problems arise, however, with large spambot raids. Below is a timeline of the recent spambot attack on our server: messages 19:41:25 19:41:45 19:42:05 19:42:25 19:42:45 This was a botched raid, but the bots that actually worked started spamming within 5 seconds of joining, giving the moderators a very narrow window to respond. The real problem, however, is that so many of them joined, the bot's API calls to add a role to silence them were rate-limited. They also sent messages once every 0.9 seconds, which is designed to get around Discord's rate limiting. This amounted to 33 messages sent every second, but it was difficult for the anti-spam to detect. Had the spambots reduced their spam cadence to 3 seconds or more, this attack could have bypassed the anti-spam detection entirely. My bot now instigates a lockdown by raising the verification level when a raid is detected, but it simply can't silence users fast enough to deal with hundreds of spambots, so at some point the moderators must use a mass ban function. Of course, banning is restricted by the global rate limit, because Discord has no mass ban API endpoint, but luckily the global rate limit is something like 50 requests per second, so if you're only banning people, you're probably okay. However, a hostile attacker could sneak bots in one-by-one every 10 minutes or so, avoiding setting off the raid alarm, and then activate them all at once. 500 bots sending randomized messages chosen from an English dictionary once every 5 seconds after sneaking them in over a 48 hour period is the ultimate attack, and one that is almost impossible to defend against, because this also bypasses the 10-minute verification level. As a weapon of last resort, I added a command that immediately bans all users that sent their first message within the past two minutes, but, again, banning is subject to the global rate limit! In fact, the rate limits can change at any time, and while message deletion has a higher rate limit for bots, bans don't. The only other option is to disable the @everyone role from being able to speak on any channel, but you have to do this on a per channel basis, because Discord ignores you if you attempt to globally disable sending message permissions for @everyone. Even then, creating an "approved" role doesn't work because any automated assignment could be defeated by adding bots one by one. The only defense a small Discord server has is to require moderator approval for every single new user, which isn't a solution - you've just given up having a public Discord server. It's only a matter of time until any angry 13-year-old can buy a sophisticated attack with a week's allowance. What will happen to public Discord servers then? Do we simply throw up our hands and admit that humanity is so awful we can't even have public communities anymore? ## The Discord API Hates You The rate-limits imposed on Discord API endpoints are exacerbated by temporary failures, and that's excluding network issues. Thus, if I attempt to set a silence role on a spammer that just joined, the API will repeatedly claim they do not exist. In fact, 3 separate API endpoints consistently fail to operate properly during a raid: A "member joined" event won't show up for several seconds, but if I fall back to calling GetMember(), this also claims the member doesn't exist, which means adding the role also fails! So I have to attempt to silence the user with every message they send until Discord actually adds the role, even though the API failures are also counted against the rate limit! This gets completely absurd once someone assaults your server with 1000 spambots, because this triggers all sorts of bottlenecks that normally aren't a problem. The alert telling you a user has joined? Rate limited. It'll take your bot 5-10 minutes to get through just telling you such a gigantic spambot army joined, unless you include code specifically designed to detect these situations and reduce the number of alerts. Because of this, a single user can trigger something like 5-6 API requests, all of which are counted against your global rate limit and can severely cripple a bot. The general advice that is usually given here is "just ban them", which is terrible advice because Discord's own awful message handling makes it incredibly easy to trigger a false positive. If a message fails to send, the client simply sends a completely new message, with it's own ID, and will continue re-sending the message until an Ack is received, at which point the user has probably send 3 or 4 copies of the same message, each of which have the same content, but completely unique IDs and timestamps, which looks completely identical to a spam attack. Technically speaking, this is done because Discord assigns snowflake IDs server-side, so each message attempt sent by the client must have a unique snowflake assigned after it is sent. However, it can also be trivially fixed by adding an optional "client ID" field to the message, with a client-generated ID that stays the same if the message is resent due to a network failure. That way, the server (or the other clients) can simply drop any duplicate messages with identical client IDs while still ensuring all messages have unique IDs across their distributed cluster. This would single-handedly fix all duplicate messages across the entire platform, and eliminate almost every single false-positive I've seen in my anti-spam bot. ## Discord Doesn't Care Sadly, Discord doesn't seem to care. The general advice in response to "how do I defend against a large scale spam attack" is "just report them to us", so we did exactly that, and then got what has to be one of the dumbest customer service e-mails I've ever seen in my life: Excuse me, WHAT?! Sorry about somebody spamming your service with horrifying gore images, but please don't delete them! What happens if the spammers just delete the messages themselves? What happens if they send child porn? "Sorry guys, please ignore the images that are literally illegal to even look at, but we can't delete them because Discord is fucking stupid." Does Discord understand the concept of marking messages for deletion so they are viewable for a short time as evidence for law enforcement?! My anti-spam bot's database currently has more information than Discord's own servers! If this had involved child porn, the FBI would have had to ask me for my records because Discord would have deleted them all! Obviously, we're not going to leave 500+ gore messages sitting in the chatroom while Discord's ass-backwards abuse team analyzes them. I just have to hope my own nuclear option can ban them quickly enough, or simply give up the entire concept of having a public Discord server. The problem is that the armies of spambots that were once reserved for the big servers are now so easy and so trivial to make that they're beginning to target smaller servers, servers that don't have the resources or the means to deal with that kind of large scale DDoS attack. So instead, I have to fight the growing swarm alone, armed with only a crippled, rate-limited bot of my own, and hope the dragons flying overhead don't notice. What the fuck, Discord. ## June 1, 2017 ### Programmers Should Take Linguistics The older I get, the more I realize that 90% of all disagreements or social drama results from a miscommunication of some kind. Every time I wind up having to resolve a dispute, I'll try to get both sides of the story, only to realize that they're the same story, and both parties were actually either in agreement or fighting over a perceived insult that never actually existed. Unsurprisingly, a disproportionate amount of this miscommunication often involves programmers being far too strict with their interpretations of what certain words mean. A linguistics class teaches you about what a language actually is - a bunch of sounds that we mutually agree mean certain things. Language is, intrinsically, a social construct. The correct definition of a word is whatever the majority of your primary social circle thinks it is. However, this also means that if you interact with a secondary social circle, and they all think the word means something else, then whenever you interact with them, it really does mean something else. Language is inherently contextual, and the possible meanings of a word can change based on who is saying it and to whom they're saying it to. If everyone else on Earth has decided that 'literally' can also mean 'figuratively', then it does, even if the dictionary says otherwise. It also means most people don't actually care if you say jif or gif, they'll just say whatever pronunciation gets you to shut up about it. It's important to realize that a word's meaning is not defined by a dictionary, but rather by how people use it. The dictionary is simply a reflection of it's usage, and is generally a few years out of date. Just as the pronunciation of a word can vary by dialect, so can the potential meanings of a word. Meanings can be invented by regional subdialects and spread outward from there, which is the origin of many slang terms. Sometimes we invent entirely new words, like "dubstep", but young words may have fuzzy definitions. In some dialects of electronic music listeners, "dubstep" is not actually a a specific genre, but instead refers to all electronic music. Using dubstep to refer to any electronic song is currently incorrect if used in general parlance, because most people think it is referring to a very specific kind of music. However, if this usage of the word continues to be popularized, eventually the meaning of the word will change into a synonym for electronica, and the dictionaries will be updated to reflect this. The fluid nature of language is why prescriptive grammar is almost always unnecessary, unless you are deliberately conforming to a grammar standard for a specific medium, such as writing a story. In almost any other context, so long as everyone in your social group understands your 'dialect' of English, then it is valid grammar. However, if you attempt to use this dialect outside of your social circle with people who are not familiar with it, you will once again be in the wrong, as they will have no idea what you're talking about. This, however, does not mean there are no mandatory grammar rules, it's just that most of the rules that are actually necessary to speak the language properly are usually so ingrained that you don't even think about them. A fantastic example of this is a little known rule in English where all adjectives must come in a very specific order: opinion-size-age-shape-color-origin-material-purpose Noun. So you can have a lovely little old rectangular green French silver whittling knife, but if you switch the order of any of those adjectives you'll sound like a maniac. Conversely, you can never have a green great dragon. Despite the fact that this grammar rule is basically never talked about in any prescriptive grammar book, it is mandatory because if you don't follow it you won't be speaking proper English and people will have difficulty understanding you. True grammar rules are ones that, if not followed, result in nonsensical sentences that are difficult or impossible to parse correctly. However, this does not mean all sentences that are difficult to understand have incorrect grammar. In fact, even some words are completely ambiguous by default. If I say I'm "dusting an object", the meaning of the phrase is completely dependent on what the object is. If it's a cake, I'm probably dusting it with something. If it's a shelf, I'm probably dusting it to get rid of the dust. Programmers tend to be very literal minded people, and often like to think that language is a set of strict rules defined by their English class. In reality, language is a fluid, dynamic, ambiguous, constantly changing enigma that exists entirely because we all agree on what a bunch of sounds mean. We need to recognize this, and when we communicate to other people, we need to be on the lookout for potential misinterpretations of what we say, so we can provide clarifications when possible. If someone says something that seems ridiculous, ask them to clarify. I'm tired of resolving disagreements that exist only because nobody stopped to ask the other side to clarify what they meant. Stop demanding that everyone explain things in a way you'll understand. That's impossible, because everyone understands language slightly differently. Instead, ask for clarification if someone seems to be saying something unusual or before you debate a point they made. Maybe then we can keep the debates to actual disagreements, instead of arguing over communication failures. ## May 4, 2017 ### Why Bother Making An App? I have an idea for an app. According to startup literature, I'm supposed to get initial fundraising from small-time investors, or "angel" investors, possibly with help from an incubator. Then, after using this money to build an MVP and push the product on the marketplace, I do a Series A round with actual venture capitalists. Now, the venture capitalists probably won't give me any money unless I can give them a proper financial outlook, user growth metrics, and a solid plan for expansion, along with a market cap estimation. Alternatively, I can just use enough meaningless buzzwords and complete bullshit to convince them to give me$120 million for a worthless piece of junk.

Either way, venture capitalists usually want a sizable 10-30% stake in your company (depending on if it's Series A, Series B, or Series C), given how much money they're pouring into a company that might fail. That's okay though, because my app does reasonably well and sells lots of copies on the app store and journalists write about it. Unfortunately, soon sales start tapering off, and ad revenue declines because customers either purchase the pro version or block the ads entirely. While the company is financially stable and making a modest profit, this isn't enough for the investors. They want growth, they need user engagement, they need ever increasing profits. Simply building a stable company isn't enough for them.

So the investors start pushing for you to be bought out. You get lucky, and your app would make a great accessory to Google Assistant, or Cortana, and you get huge buyout offers from Microsoft, Google, and Amazon, because they have more money than most small countries. Investors immediately push for you to take the most lucrative offer from whoever is willing to give you the most cash for fucking over all of your customers. You can push back, but your power is limited, because those investors hold a significant chunk of your company. At best, you can pick the offer that is least likely to completely destroy your product.

If you get lucky, your cross-platform app that worked on everything gets discontinued and re-integrated into one device that people have to buy due to vendor lock-in. If you aren't lucky, your app gets discontinued and completely forgotten about, until someone else comes up with the same idea and the process repeats. Maybe this time they'll get bought out and actually integrated into something.

Either way, your customers lose. Every time. They are punished for believing that a new app, by some new company, could actually survive long enough to be useful to them without being consumed by the corporate monstrosities that run the world. If the company founders are nice, maybe some of the employees walk away rich, but most of them will probably just end up trapped inside a corporate behemoth until they can't take it anymore and finally quit. In your efforts to make the world a better place, you've managed to screw over your company, your customers, and even your employees, because investors don't care about your product, they care about milking you for all you're worth.

But hey, at least you're rich, right?

## March 23, 2017

### Companies Can't Be Apolitical

One of the most common things I hear from people is that companies should be "apolitical". The most formal way this concept is expressed is that a company should make decisions based on what maximizes profits and not political opinions. Unfortunately, the statement "companies should only care about maximizing profits" is, itself, a political statement (and one I happen to disagree with). Thus, it is fundamentally impossible for a company to be truly apolitical, for the very act of attempting to be apolitical is a political statement.

How much a company can avoid politics generally depends on both the type and size of the company. Once your company becomes large enough, it will influence politics simply by virtue of its enormous size, and eventually becomes an integral part of political debates whether or wants to or not. Large corporations must take into account the political climate when making business decisions, because simply attempting to blindly maximize profit may turn the public against them and destroy their revenue sources—thus, politics themselves become part of the profit equation, and cannot be ignored. Certain types of businesses embody political statements simply by existing. Grindr, for example, is a dating app for gay men. It's entire business model is dependent on enabling an activity that certain fundamentalists consider inherently immoral.

You could, theoretically, try to solve part of this quandary by saying that companies should also be amoral, insofar that the free market should decide moral values. The fundamentalists would then protest the companies existence by not using it (but then, they never would have used it in the first place). However, the problem is that, once again, this very statement is itself political in nature. Thus, by either trying to be amoral or moral, a company is making a political statement.

The issue at play here is that literally everything is political. When most everyone agrees on basic moral principles, it's easier to pretend that politics is really just about economic policy and lawyers, but our current political divisions have demonstrated that this is a fantasy. Politics are the fundamental morals that society has decided on. It's just a lot easier to argue about minor differences in economic policy instead of fundamental differences in basic morality.

Of course, how companies participate in politics is also important to consider. Right now, a lot of companies participate in politics by spending exorbitant amounts of money on lobbyists. This is a symptom of money in general, and should be solved not by removing corporate money from politics, but removing all money, because treating spending money as a form of speech gives more speech to the rich, which inherently discriminates against the poor and violates the constitutional assertion that all men are created equal (but no one really seems to be paying attention to that line anyway).

Instead of using money, corporations should do things that uphold whatever political values they believe in. As the saying goes, actions speak louder than words (or money, in this case). You could support civil rights activism by being more inclusive with your hiring and promoting a diverse work environment. Or, if you live in the Philippines, you could create an app that helps death squads hunt down drug users so they can be brutally executed. What's interesting is that most people consider the latter to be a moral issue as opposed to a political one, which seems to derive from the fact that once you agree on most fundamental morals, we humans simply make up a bunch of pointless rules to satisfy our insatiable desire to tell other humans they're wrong.

We've lived in a civilized world for so long, we've forgotten the true roots of politics: a clash between our fundamental moral beliefs, not about how much parking fines should be. Your company will make a political statement whether you like it or not, so you'd better make sure it's the one you want.

## March 7, 2017

### I Can't Hear Anything Below 80 Hz*

* at a comfortable listening volume.
EDIT: I have confirmed all the results presented here by taking the low frequency test with someone standing physically next to me. They heard a tone beginning at 30 Hz, and by the time I could hear a very faint tone around 70 Hz, they described the tone as "conversation volume level", which is about 60 dB. I did not reach this perceived volume level until about 120 Hz, which strongly correlates with the experiment. More specific results would require a professional hearing test.

For almost 10 years, I've suspected that something was wrong with my ability to hear bass tones. Unfortunately, while everyone is used to people having difficulty hearing high tones, nobody takes you seriously if you tell them you have difficulty hearing low tones, because most audio equipment has shitty bass response, and human hearing isn't very precise at those frequencies in the first place. People generally say "oh you're just supposed to feel the bass, don't worry about it." This was extremely frustrating, because one of my hobbies is writing music, and I have struggled for years and years to do proper bass mixing, which is basically the only activity on the entire planet that actually requires hearing subtle changes in bass frequencies. This is aggravated by the fact that most hearing tests are designed to detect issues with high frequencies, not low frequencies, so all the basic hearing tests I took at school gave test results back that said "perfectly normal". Since I now have professional studio monitor speakers, I'm going to use science to prove that I have an abnormal frequency sensitivity curve that severely hampers my ability to differentiate bass tones. Unfortunately, at the moment I live alone and nowhere near anyone else, so I will have to prove that my equipment is not malfunctioning without being able to actually hear it.

Before performing the experiment, I did this simple test as a sanity check. At a normal volume level, I start to hear a very faint tone in that example at about 70 Hz. When I sent it to several other people, they all reported hearing a tone around 20-40 Hz, even when using consumer-grade hardware. This is clear evidence that something is very, very wrong, but I have to prove that my hardware is not malfunctioning before I can definitively state that I have a problem with my hearing.

For this experiment, I will be using two JBL Professional LSR305 studio monitors plugged into a Focusrite Scarlett 2i2. Since these are studio monitors, they should have a roughly linear response all the way down to 20 Hz. I'm going to use a free sound pressure app on my android phone to prove that they have a relatively linear response time. The app isn't suitable for measuring very quiet or very loud sounds, but we won't be measuring anything past 75 dB in this experiment because I don't want to piss off my neighbors.

The studio monitor manages to put out relatively stable noise levels until it appears to fall off at 50 Hz. However, when I played a tone of 30 Hz at a volume loud enough for me to feel, the sound monitor still reported no pressure, which means the microphone can't detect anything lower than 50 Hz (I was later able to prove that the studio monitor is working properly when someone came to visit). Of course, I can't hear anything below 50 Hz anyway, no matter how loud it is, so this won't be a problem for our tests. To compensate for the variance in the frequency response volume, I use the sound pressure app to measure the actual sound intensity being emitted by the speakers.

The first part of the experiment will detect the softest volume at which I can detect a tone at any frequency, starting from D3 (293 Hz) and working down note by note. The loudness of the tone is measured using the sound pressure app. For frequencies above 200 Hz, I can detect tones at volumes only slightly above the background noise in my apartment (15 dB). By the time we reach 50 Hz I was unwilling to go any louder (and the microphone would have stopped working anyway), but this is already enough for us to establish 50 Hz as the absolute limit of my hearing ability under normal circumstances.

To get a better idea of my frequency response at more reasonable volumes, I began with a D4 (293 Hz) tone playing at a volume that corresponded to 43 dB SPL on my app, and then recorded the sound pressure level of each note once it's volume seemed to match with the other notes. This gives me a rough approximation of the 40 phon equal loudness curve, and allows me to overlay that curve on to the ISO 226:2003 standard:

These curves make it painfully obvious that my hearing is severely compromised below 120 Hz, and becomes nonexistent past 50 Hz. Because I can still technically hear bass at extremely loud volumes, I can pass a hearing test trying to determine if I can hear low tones, but the instant the tones are not presented in isolation, they are drowned out by higher frequencies due to my impaired sensitivity. Because all instruments that aren't pure sine waves produce harmonics above the fundamental frequency, this means the only thing I'm hearing when a sub-bass is playing are the high frequency harmonics. Even then, I can still feel bass if it's loud enough, so the bass experience isn't completely ruined for me, but it makes mixing almost impossible because of how bass frequencies interact with the waveform. Bass frequencies take up lots of headroom, which is why in a trance track, you can tell where the kicks are just by looking at the waveform itself:

When mixing, you must carefully balance the bass with the rest of the track. If you have too much bass, it will overwhelm the rest of the frequencies. Because of this, when I send my tracks to friends to get help on mixing, I can tell that the track sounds better, but I can't tell why. The reason is because they are adjusting bass frequencies I literally cannot hear. All I can hear is the end result, which has less frequency crowding, which makes the higher frequencies sound better, even though I can't hear any other difference in the track, so it seemes like black magic.

It's even worse because I am almost completely incapable of differentiating tones below 120 Hz. You can play any note below B2 and I either won't be able to hear it or it'll sound the same as all the other notes. I can only consistently differentiate semitones above 400 Hz. Between 120-400 Hz, I can sometimes tell them apart, but only when playing them in total isolation. When they're embedded in a song, it's hopeless. This is why, in AP Music Theory, I was able to perfectly transcribe all the notes in the 4-part writing, except the bass, yet no other students seemed to have this problem. My impaired sensitivity to low frequencies mean they get drowned out by higher frequencies, making it more and more difficult to differentiate bass notes. In fact, in most rock songs, I can't hear the bass guitar at all. The only way for me to hear the bass guitar is for it to be played by itself.

Incidentally, this is probably why I hate dubstep.

For testing purposes, I've used the results of my sensitivity testing to create an EQ filter that mimics my hearing problems as best I can. I can't tell if the filter is on or off. For those of you that use FL Studio, the preset can be downloaded here.

Below is a song I wrote some time ago that was mastered by a friend who can actually hear bass, so hopefully the bass frequencies in this are relatively normal. I actually have a bass synth in this song I can only barely hear, and had to rely almost entirely on the sequencer to know which notes were which.

This is the same song with the filter applied:

By inverting this filter, I can attempt to "correct" for my bass hearing, although this is only effective down to about 70 Hz, which unfortunately means the entire sub-bass spectrum is simply inaudible to me. To accomplish this, I combine the inverted filter with a mastering plugin that completely removes all frequencies below 60 Hz (because I can't hear them) and then lowers the volume by about 8 dB so the amplified bass doesn't blow the waveform up. This doesn't seem to produce any audible effect on songs without significant bass, but when I tried it on a professionally mastered trance song, I was able to hear a small difference in the bass kick. I also tried it on Brothers In Arms and, for the first time, noticed a very faint bass cello going on that I had never heard before. If you are interested, the FL studio mixer state track that applies the corrective filter is available here, but for normal human beings the resulting bass is probably offensively loud. For that same reason, it is unfortunately impractical for me to use, because listening to bass frequencies at near 70 dB levels is bad for your hearing, and for that matter it doesn't fix my impaired fidelity anyway, but at least I now know why bass mixing has been so difficult for me over the years.

I guess if I'm going to continue trying to write music, I need to team up with one of my friends that can actually hear bass.

## February 13, 2017

### Windows Won't Let My Program Crash

It's been known for a while that windows has a bad habit of eating your exceptions if you're inside a WinProc callback function. This behavior can cause all sorts of mayhem, like your program just vanishing into thin air without any error messages due to a stack overflow that terminated the program without actually throwing an exception. What I didn't realize is that it also eats assert(), which makes debugging hell, because the assertion would throw, the entire user callback would immediately terminate without any stack unwinding, and then windows would just... keep going, even though the program is now in a laughably corrupt state, because only half the function executed.

While trying to find a way to fix this, I discovered that there are no less than 4 different ways windows can choose to eat exceptions from your program. I had already told the kernel to stop eating my exceptions using the following code:
HMODULE kernel32 = LoadLibraryA("kernel32.dll");
assert(kernel32 != 0);
if(pGetPolicy && pSetPolicy && pGetPolicy(&dwFlags))
pSetPolicy(dwFlags & ~EXCEPTION_SWALLOWING); // Turn off the filter

However, despite this, COM itself was wrapping an entire try {} catch {} statement around my program, so I had to figure out how to turn that off, too. Apparently some genius at Microsoft decided the default behavior should be to just swallow exceptions whenever they were making COM, and now they can't change this default behavior because it'd break all the applications that now depend on COM eating their exceptions to run properly! So, I turned that off with this code:
CoInitialize(NULL); // do this first
if(SUCCEEDED(CoInitializeSecurity(NULL, -1, NULL, NULL, RPC_C_AUTHN_LEVEL_PKT_PRIVACY,
RPC_C_IMP_LEVEL_IMPERSONATE, NULL, EOAC_DYNAMIC_CLOAKING, NULL)))
{
IGlobalOptions *pGlobalOptions;
hr = CoCreateInstance(CLSID_GlobalOptions, NULL, CLSCTX_INPROC_SERVER, IID_PPV_ARGS(&pGlobalOptions));
if(SUCCEEDED(hr))
{
hr = pGlobalOptions->Set(COMGLB_EXCEPTION_HANDLING, COMGLB_EXCEPTION_DONOT_HANDLE);
pGlobalOptions->Release();
}
}

There are two additional functions that could be swallowing exceptions in your program: _CrtSetReportHook2 and SetUnhandledExceptionFilter, but both of these are for SEH or C++ exceptions, and I was throwing an assertion, not an exception. I was actually able to verify, by replacing the assertion #define with my own version, that throwing an actual C++ exception did crash the program... but an assertion didn't. Specifically, an assertion calls abort(), which raises SIGABRT, which crashes any normal program. However, it turns out that Windows was eating the abort signal, along with every other signal I attempted to raise, which is a problem, because half the library is written in C, and C obviously can't raise C++ exceptions. The assertion failure even showed up in the output... but didn't crash the program!
Assertion failed!

Program: ...udio 2015\Projects\feathergui\bin\fgDirect2D_d.dll
File: fgEffectBase.cpp
Line: 20

Expression: sizeof(_constants) == sizeof(float)*(4*4 + 2)

No matter what I do, Windows refuses to let the assertion failure crash the program, or even trigger a breakpoint in the debugger. In fact, calling the __debugbreak() intrinsic, which outputs an int 3 CPU instruction, was completely ignored, as if it simply didn't exist. The only reliable way to actually crash the program without using C++ exceptions was to do something like divide by 0, or attempt to write to a null pointer, which triggers a segfault.

Any good developer should be using assertions to verify their assumptions, so having assertions silently fail and then corrupt the program is even worse than ignoring they exist! Now you could have an assertion in your code that's firing, terminating that callback, leaving your program in a broken state, and then the next message that's processed blows up for strange and bizarre reasons that make no sense because they're impossible.

I have a hard enough time getting my programs to work, I didn't think it'd be this hard to make them crash.

## February 12, 2017

### Owlboy And The Tragedy of Human Nature

Owlboy is a game developed by D-Pad studios over a protracted 9 year development cycle. Every aspect of the game is a work of art, meticulously pieced together with delicate care. While it has mostly gotten well-deserved positive reviews from critics, some people have voiced disappointment at the story arc and how the game was eventually resolved. Note: I'm about to talk about how the game ends, so an obligatory warning that there are massive spoilers ahead. If you haven't already played it, go buy it, right now.

On the pirate mothership, it is revealed that the titular "owlboy" is not referring to Otus, but is actually referring to the mysterious cloaked figure, who is actually Solus. Otus merely serves as a distraction so that Solus could steal the relics from the pirates. Once the heroes follow Solus up to Mesos and beat him into submission, it is revealed that Solus masterminded the events of the entire game, promising the pirates power in exchange for retrieving the relics, then using Otus as a distraction to steal the relics and use them to power the Anti-Hex to save the world.

Unfortunately, as the heroes immediately point out, this resulted in the destruction of Advent and the deaths of countless innocent people. Had Solus just asked for help, all of this could have been avoided, and the world could have been saved without incident. Solus admits that he felt he had no choice, as he simply had nobody he could trust. Our heroes offer to help finish the ritual, at which point Molstrom shows up and ruins everyone's day. Solus' methods have finally backfired on him, and he is now too injured to complete the ritual—but Otus isn't. In an act of desperation, he imbues Otus with the power of the artifacts, and Otus is able to complete the anti-hex in his stead, obliterating Molstrom in the process and saving the world.

A lot of people take issue with this ending for two reasons: One, it means almost everything you fought for in the game technically meant nothing, because you are actually working against Solus the entire time. Two, the entire thing could have been avoided if Solus had just trusted someone, anyone, instead of engineering a ridiculously convoluted plot to get what he needed through deceit and betrayal. Otus and Solus here represent a hero and an anti-hero. They are both fundamentally good people with flawed goals for opposite reasons. Solus is doing the right thing for the wrong reasons, whereas Otus is doing the wrong thing for the right reasons. Solus is doing the right thing, which is saving the world, but he achieves it by sacrificing thousands of innocent lives and betraying everyone. Otus is unknowingly dooming the world to destruction, but only because he and everyone else is operating on faulty information. He always chooses to do the right thing, and to trust others.

This is important, because the way the game ends is crucial to the narrative of the story and the underlying moral. Solus may have nearly saved the world, despite his questionable methods, but he would have failed at the end. Molstrom would have found him and easily defeated him, taking all the relics and then destroying whatever was left of the world as it rose into space. Only after Solus explained everything to Otus was Otus able to finally do the right thing for the right reasons, thanks to the friends he made on his journey holding Molstrom back. By doing this, Solus allows Otus to atone for failing to secure any of the relics, and Otus absolves Solus of the evil he had committed in the name of saving the world by annihilating Molstrom with the anti-hex.

The whole point of this story is that it doesn't matter how many times you fail, so long as you eventually succeed. Otus may have failed to save the world over and over and over, but at the very end, as the world is coming apart at the seams, he is finally able to succeed, and that's what matters. This moral hit me particularly hard because I instantly recognized what it represented - failing to ship a game over and over and over. Owlboy's story is an allegory for it's own development, and on a broader scale, any large creative project that has missed deadlines and is falling behind. It doesn't matter how many times you've failed, because as long as you keep trying, eventually you'll succeed, and that's what really matters. The game acknowledges that humans are deeply flawed creatures, but if we work together, we can counteract each other's flaws and build something greater than ourselves.

This is why I find it depressing that so many people object to Solus' behavior. Surely, no one would actually do that? However, at the beginning of the game, Solus' defining character moment is being bullied and abused by the other owls. His response to this abuse is to become withdrawn, trusting no one, determined to do whatever is necessary without relying on anyone else. This is exactly what happens to people who have been abused or betrayed and develop trust issues. These people will refuse help that they are offered, pushing other people away, often forcing an intervention to try and break through the emotional wall they have built to try and keep themselves from being hurt again. That's why you have to fight Solus first, because he's spent his whole life building a mental block to keep other people out, and Otus has to break through it to force him to finally accept help.

Far from being unbelievable, Owlboy's plot is entirely too real, and like any good work of art, it forces us to confront truths that make us uncomfortable. Owlboy forces us to confront the tragedy of human nature, and deal with the consequences. At the same time, it shows us that, if we persevere, we can overcome our flaws, and build a better world, together.

## February 1, 2017

### DirectX Is Terrifying

About two months ago, I got a new laptop and proceeded to load all my projects on it. Despite compiling everything fine, my graphics engine that used DirectX mysteriously crashed upon running. I immediately suspected either a configuration issue or a driver issue, but this seemed weird because my laptop had a newer graphics card than my desktop. Why was it crashing on newer hardware? Things got even more bizarre once I narrowed down the issue - it was in my shader assignment code, which hadn't been touched in almost 2 years. While I initially suspected a shader compilation issue, there was no such error in the logs. All the shaders compiled fine, and then... didn't work.

Now, if this error had also been happening on my desktop, I would have immediately started digging through my constant assignments, followed by the vertex buffers assigned to the shader, but again, all of this ran perfectly fine on my desktop. I was completely baffled as to why things weren't working properly. I had eliminated all possible errors I could think of that would have resulted from moving the project from my desktop to my laptop: none of the media files were missing, all the shaders compiled, all the relative paths were correct, I was using the exact same compiler as before with all the appropriate updates. I even updated drivers on both computers, but it stubbornly refused to work on the laptop while running fine on the desktop.

Then I found something that nearly made me shit my pants.
if(profile <= VERTEX_SHADER_5_0 && _lastVS != shader) {
//...
//...
//...
}

For almost 2 years, I had been feeding DirectX total bullshit, and had even tested it on multiple other computers, and it had never given me a single warning, error, crash, or any indication whatsoever that my code was completely fucking broken, in either debug mode or release mode. Somehow, deep in the depths of nVidia's insane DirectX driver, it had managed to detect that I had just tried to assign a vertex shader to a pixel shader, and either ignored it completely, or silently fixed my catastrophic fuckup. However, my laptop had the mobile drivers, which for some reason did not include this failsafe, and so it actually crashed like it was supposed to.

While this was an incredibly stupid bug that I must have written while sleep deprived or drunk (which is impressive, because I don't actually drink), it was simply impossible for me to catch because it produced zero errors or warnings. As a result, this bug has the honor of being both the dumbest and the longest-living bug of mine, ever. I've checked every location I know of for any indication that anything was wrong, including hooking into the debug log of directX and dumping all it's messages. Nothing. Nada. Zilch. Zero.

I've heard stories about the insane bullshit nVidia's drivers do, but this is fucking terrifying.

Alas, there is more. I had been experimenting with direct2D as an alternative because, well, it's a 2D engine, right? After getting text rendering working, a change in string caching suddenly broke the entire program. It broke in a particularly bizarre way, because it seemed to just stop rendering halfway through the scene. It took almost an hour of debugging for me to finally confirm that the moment I was rendering a particular text string, the direct2D driver just stopped. No errors were thrown. No warnings could be found. Direct2D's failure state was apparently to simply make every single function call silently fail with no indication that it was failing in the first place. It didn't even tell me that the device was missing or that I needed to recreate it. The text render call was made and then every single subsequent call was ignored and the backbuffer was forever frozen to that half-drawn frame.

The error itself didn't seem to make any more sense, either. I was passing a perfectly valid string to Direct2D, but because that string originated in a different DLL, it apparently made Direct2D completely shit itself. Copying the string onto the stack, however, worked (which itself could only work if the original string was valid).

The cherry on top of all this is when I discovered that Direct2D's matrix rotation constructor takes degrees, not radians, like every single other mathematical function in the standard library. Even DirectX takes radians!

WHO WRITES THESE APIs?!

## January 17, 2017

### Our Software Is a Beacon of Hope

As I draw closer to releasing the first early alpha of a library I've been working on for several years, I've noticed that it's release time coincides with some rather depressing politics that I wish I could ignore. Unfortunately, it's difficult to ignore politicians who seek to destroy the lives of your closest friends. A common theme I hear from my friends is a feeling of helplessness, as though we all know terrible things are happening, but no one really knows what to do about it.

But we are not helpless. It is so easy to lose ourselves in pessimism and cynicism, to think that everything will continue to be shit simply because it's been shit for so many years. A common refrain I heard while still working at a large corporation was, we know our code is shit, but it's okay, because everyone else's code is shit. This kind of mentality really bothers me, not because it isn't true, but because it seems to reflect a resigned, cynical view of the world, and never strives to do better. Yes, everything might end up being shit in the end, but if we don't even attempt to build something better, we never will. If we hadn't been trying to write better code we never would have invented structured programming, or object oriented programming, or functional programming. Each innovation builds on the last, and each innovation has it's faults, but each brings us one step closer to not being shit.

What disturbs me is that the software industry's inability to strive for better code now mirrors a feeling of political resignation, a malaise that seems to be settling on our polarized political discourse. If democracy isn't working, what else can we do? If the entire system is corrupt, what hope do we have of fixing it? As wealth inequality rises to ever more absurd levels, our society is tumbling towards a breaking point. Just like a badly maintained software stack, our society is threatening to topple over under economic stresses it was never built to handle. How much longer can we keep it running with duct-tape and terrible hacks? It is our duty to push for real change, to build a better foundation for our future using whatever skills or resources we have.

As I prepare my library for its initial release, I'm not just zipping up a bunch of files to upload to GitHub. I'm lighting a fire, a beacon of hope that I will cast out into the endless night, praying that it might lead us to a brighter future. I am joined by millions of other flickering flames, many of which will fade with time. A few, however, will grow brighter and brighter, until they bring forth a new dawn and shepherd us into a bright new future, even if only for a moment, before sinking below the horizon. Our entire society is built on these brief moments of clarity. Some are longer than others, but all are important. Even when the night seems to stretch on without end, the rays of hope from a thousand stars can still point us in the right direction.

We all wish for a brighter future, a future free from disease and poverty and war and hunger. We seek the future that we envision for ourselves in our stories and movies. Sometimes, it seems like society is going backwards, when everything is going wrong and nothing seems to go right for anyone. But it is in these moments that we must not forget the most important thing: we cannot wait for the future to come to us, we must build it ourselves. 2016 was not a very good year for many people. 2017 shows signs of being even worse, but I don't care.

I'm going to make 2017 be a better year. I'm going to make the future I want to live in. I'm going to build the tools I wish I had. We are not pawns in some cosmic game, we are human beings. We are free agents capable of making our own decisions, and I'm making a decision to make the future happen, whether it wants to or not.

I'm not letting this year take me down without a fight.