Lost in async-hole.

The past week and a half or so I have been living up the reputation of the typical basement-dwelling geek. I like to call it “my cave” regardless of where it is or whether or not it has windows to the outside. It’s a tunnel-vision state of mind where I spend a significant amount of time in a sort of a trance; I recently drove myself crazy while trying to solve two particularly sneaky problems with the part of WordPress that shows you your notifications.

The criminal in one of the problems was caused by an asynchronous worker – a piece of code that was running in the background without making itself well-known. Imagine, for instance, that you are a grocery store manager and have instructed your staff to keep the storefront tidy and clean. One day, you decide to make a display of your fresh breads in the window to entice all who might walk by to come in for some pastries or pretzels. Strangely, every time you return to the front of the store you notice the missing bread and replace it, confused as to where it’s going. Meanwhile, unbeknown to you, one of your employees routinely makes his rounds and sees the bread, thinking it was misplaced, and promptly brings it to the bakery section, placing it on the shelf. This is what was happening to me in the code.

Despite the fact that the solution was simple – instruct the clerk to leave the bread alone – you spend hours trying to figure out who might be swiping it away or how it keeps disappearing. Programming and engineering large systems is often like this: trying to see the whole picture where multiple actors are at work and where different components from different teams all interact.

Remember the Mars Climate Orbiter? Thankfully my mistakes were much more forgiving than the wrong unit conversions were for that space probe. Still, I learned a few lessons through this experience.

First, it was a good reminder to take real breaks from my work when I get stuck in a rut. We can spin our gears in a certain direction because we think the answer lies at its end, but if we keep on turning them and get nowhere, we (at least I) need a chance to let my mind reorient itself. “Thinking outside of the box” might even be a little too cliché and overkill to describe this because we really just need to get out of the tedious cycle we put ourselves in. Try something different. In this case, I solved the problem this morning after a full night’s rest and after pushing aside all of the work I have spent on it over the past few days. In the end there was an “aha moment” and I had the solution within a few minutes.

Second, I learned to remember the things that aren’t visible. Maybe it’s a piece of code that someone else wrote that you never saw that runs in the background or maybe its a whole repressed segment of society, but there’s always more to be seen than meets the eye and we’re not capable enough or thoughtful enough to always keep that in mind.

In conclusion, lots of my experience with engineering is like this: you spend heaps of time trying to visualize the end but when it’s all said and done, the solution is basic – like an honest-to-goodness Jackson Pollock.