Introducing Digbuild

I’d like to introduce Digbuild, an open-source game engine inspired by the excellent game Minecraft (and Infiniminer before it — that’s right, Minecraft is itself a clone). I’ve been working on it on and off in my spare time for a few months now, and today I decided that it’s ready to show to the world. For the last couple of months I was debating when it would be time to publish it. I didn’t want to release it in such an early stage that it was unusable, and in particular I didn’t want to release it in a state where it was nearly impossible to build. This weekend, though, my good friend Blake Miller took it upon himself to build Digbuild (say that 5 times fast), and as it turns out, the build system is relatively workable. So, have at it!

What Digbuild Is

Right now Digbuild provides a randomized, voxel-based world for the player to explore. In this regard, it’s very similar to Minecraft. You can create and destroy blocks, and thus you can build castles and any other structures that spring to mind. Digbuild has several improvements over Minecraft:

  • Infinite world height. You can build structures as tall as you like.
  • Colored lighting. Different blocks emit different colors of light, and colored glass blocks filter the light that flows through them.
  • Translucent materials. Want to build a castle out of six different colors of stained glass? Go for it.
  • Bump– and specular-mapped textures: Glass is shiny and rocks are rough.
  • Open source. Want to improve something that’s not changeable through an existing API? Hack the source to your heart’s content.

What Digbuild Isn’t

Although Digbuild is heavily inspired by Minecraft, it does not strive to be just like it. If you want to play Minecraft, go play Minecraft! The ultimate goal is for Digbuild to go in several directions. We’re planning a Python-based scripting engine to make building plugins easy, and it can always be forked. There’s a lot of things that Digbuild lacks at the moment:

  • It’s unfinished. If you want to play a game, don’t choose Digbuild. It’s still early in development, and right now is targeted towards hackers.
  • There’s no multiplayer support. It’s planned, but is still a ways off.
  • There’s no crafting. The crafting system will eventually be fully Python-based, but there’s no support for this yet.

How to Contribute

We’d be thrilled if you wanted to help make Digbuild better. It’s got a long way to go before it’s really a video game, but building it is (at least) half the fun, right? If you’re interested in working on it, just fork it on Github and go crazy. Add something cool? Issue a pull request and see it get merged into the main game.

There’s plenty of work to do aside from coding, as well. We need to create textures for new materials, come up with ideas for gameplay, and eventually add sound effects.

Finally, we’re under no pretense that Digbuild is perfect. It’s still a work in progress, and any kind of feedback at this stage could be helpful. So don’t hold back your criticisms or ideas!

Learning More

I plan to write a series of articles on what I consider a few of the more interesting bits of the Digbuild implementation. Right now the topics I expect to write about include the random terrain generation, graphics optimizations, and efficient collision detection algorithms. If there’s anything else interesting about how Digbuild works, let me know and I’ll consider writing about that too!


Boot a Kernel over Serial with U-Boot and Kermit

I’m doing a little bit of work that involves frequently rebuilding the Linux kernel and installing it on a headless ARM board. The particular ARM board I’m working with has some vendor support for flashing kernels, but it’s slow and clunky, and I have to run it inside a Windows XP VM. The ARM board uses the U-Boot bootloader, though, so it’s possible to boot the kernel in a couple of different ways. One way would be to load the kernel via TFTP, but I haven’t gotten that working yet on my board. The other option is to load it via serial, which isn’t very fast but requires very little setup.

U-Boot’s loadm command allows a kernel to be loaded, via serial, into a memory location. The bootm command may then be used to boot the kernel directly, which saves time compared to writing the kernel to the flash memory and loading it from there. The trouble is that loadm expects the kernel to be sent via the Kermit protocol. I found a few examples of how to deal with Kermit, but none of them directly applied to loading a kernel with U-Boot.

I came up with the following Kermit script to solve my problem. This script automatically waits for the board to reset, sends the loadm command, pushes down the kernel, and runs it via the bootm command. After it boots the kernel, it turns into an interactive console. This script relies on C-Kermit, which I installed under Ubuntu as follows:

bash$ sudo aptitude install ckermit

The script I’m using is as follows. There are a lot of settings hard-coded into the script, so read the comments carefully to determine what parts you might need to change to suit your setup. To use this script, simply copy it into a file named, for example, boot-kernel, give it executable permissions, and run it.

#!/usr/bin/kermit

# Serial port setup.  These settings will likely need to be
# changed to match the configuration of your workstation
# and the ARM board you're working with.
set line /dev/ttyUSB0
set speed 115200
set serial 8n1

# General C-Kermit settings.  These probably don't need to change.
set flow-control none
set file type bin
set carrier-watch off
set prefixing all
set modem none

echo "Prepared to boot new kernel.  Reset the board now."

# This is the string that my board outputs to allow the user to
# gain access to the U-Boot console.  Change this to suit your
# setup.
input 60 "Hit SPACE to stop autoboot"
# If your board wants you to press a different key to get to
# U-Boot, edit this line.
output " "
input 5 "u-boot>"
# Here, 0x800000 is the memory address into which the kernel
# should be loaded.
lineout "loadb 0x800000"
# This should be the absolute path to your kernel uImage file.
send /path/to/uImage
input 5 "u-boot>"
lineout "bootm 0x800000"

# This command drops you into a console where you can interact
# with the kernel.
connect

Once the script has given you console control, you need to use the Kermit escape key to exit. By default, this is set to Ctrl+\ (that’s a backslash). To see a list of commands, type Ctrl+\ and then ?. The command to immediately exit the console is q.

One last thing to note: this script doesn’t do any error checking. Each of the input commands can fail, if it does not see the text it’s looking for in the specified time. The script could be extended to check for errors using Kermit’s IF command.


Telecommuting Has Benefits, Too

Recently I’ve run across a few articles (on Hacker News and elsewhere) about the drawbacks of telecommuting. I agree that there are drawbacks, but I believe that they can be counterbalanced by the benefits under the right circumstances.

The Right Circumstances

Not every person is cut out to telecommute, and not every job is suitable to be performed remotely. Furthermore, there are many tools available to make telecommuting much more effective.

The single most important traits for a telecommuter to have are strong writing and comprehension skills. There are no two ways about it; a telecommuter is going to engage in a lot of written communication. You can’t yell over the cubicle wall to ask for a quick clarification. Since they are not physically present, any communication with them requires a small amount of overhead. Thus it’s important that each bit of communication with the telecommuter be clear and concise.

The ever-present communication overhead implies that jobs which require more frequent communication are less suitable for telecommuters. The best jobs are those in which a lot of “heads down” work needs to be done. These are the kinds of jobs where even if the employee were physically present, they’d want an office with a door that shuts tight. Many nuts-and-bolts, back-end software engineering jobs fall into this category. For instance, writing a device driver requires large chunks of up-front communication, but after that it requires deep concentration and few interruptions — perfect for a telecommuter. Other jobs, such as project management, require constant communication and incur a much greater telecommuting overhead.

Finally, tools are instrumental in making telecommuting work. In a software shop, a good Wiki system allows for collaborative documentation. A bug/feature tracking system helps keep everyone in sync on priorities. File sharing, phone conferencing, source control, desktop sharing, VPN systems — all of these are absolutely critical to enable a telecommuter to do their job.

The benefits of telecommuting only apply fully when the above circumstances are met. It’s easy to see how telecommuting could leave a bad taste in someone’s mouth if it was attempted with the wrong person, job, or tools.

The Benefits

Better documentation. One of the major drawbacks of working with someone far away is that you can’t walk up to their desk and pick their brain. Sure, you can call them, but once you’ve resigned yourself to the overhead of a phone call, more likely than not you’ll just send an email or instant message. But there’s a hidden benefit to this: more knowledge ends up written down. Informally, you end up with more knowledge in your email or IM history. More formally, you have more opportunities to write documentation. A good telecommuter knows when an email thread has become overgrown and needs to be dumped into a Wiki article.

Higher throughput. For software jobs that require extended periods of deep concentration, telecommuting can often provide the best work environment. This can require some effort on the remote employee’s part (e.g. establishing a no-interruption rule with the kids), but when it’s pulled off successfully it can be orders of magnitude better than being cramped up in a cubicle next to a salesperson who’s constantly on the phone.

More hours. The lack of a commute and the ease of making a quick lunch at home save a lot of time for a telecommuter. When a doctor’s appointment comes up in the middle of the day, it’s easier to justify working late to make up for it, instead of taking personal time off.

More flexible pay. The market value for a talented engineer differs between, say, the Bay Area and Wisconsin. The cost of living and market demand vary drastically between different geographical areas. A business in an expensive metropolis can save tons of money by hiring a telecommuter from an area where it’s cheaper to live. This can benefit the telecommuter as well, if the business, for instance, splits the difference between the local and remote market salaries with the employee.

Conclusions

In no way am I trying to prescribe telecommuting as a panacea or some kind of magical efficiency booster. But, as a telecommuter myself, I have seen it work out really well firsthand, and I feel the need to point out the fact that it does have a few tangible benefits. Like any other business decision, though, it shouldn’t be chosen without careful thought and planning.


Sloptimize

hide-first-letter

slop·ti·mize | slop·ti·mized | slop·ti·miz·ing

verb \ˈsläp-tə-ˌmīz\

  • To make a program run faster by decreasing the accuracy of its output.

Examples of sloptimize

  • We changed the calculations to use 32-bit floats instead of 64-bit doubles, and got a 20% speedup.

Please Don’t Request User Input in the Middle of a Lengthy Task

Just Say No

It’s happened to everyone. You kick off a software installer, answer a few questions about how you’d like things set up, click next and you’re presented with a long progress bar. “No problem,” you think to yourself, “this is a good excuse to grab a cuppa joe.” You leave the computer to its business and hit the kitchen, maybe catching a glance at the paper. After some time has passed, it occurs to you that the installer’s probably been finished for a while, so you head back to your computer to start using your fresh new software. And then BAM! You get slapped in the face with just one last question that the installer needs you to answer. It turns out that it’s only partially complete, and when you click next again, you’re presented with another long progress bar. Now you’re faced with a decision: do you switch tasks again, or do you babysit the installer, in case it has another question?

This behavior drives me absolutely mad. I’m impatient with installers to begin with; they’re a hiccup (albeit a necessary one) between me and the software I want to use. Of course, it’s not just installers that suffer from this problem. Any piece of software that has to perform some kind of long-running task can be subject to this annoyance, simply by requiring user input anywhere except at the very beginning or end of a lengthy task.

The amount of frustration this bug causes is directly proportional to how long the task will take. I recall a recent mishap where I was installing an older Debian distro on an extremely slow ARM machine. I thought I had answered all of its questions, and left my office to do errands for several hours. I was confident that when I returned, the machine would be ready to go. Of course, you know how this story ends: upon my return, I found the installer waiting for input, and it took several more hours for the installation to complete. My work for the day was set back, and my schedule was thrown off.

Thankfully, the solution to this problem is extremely obvious: batch up and prompt for all of the necessary user input before starting a long running task. Never, ever interrupt the task to prompt for more input unless it is 100% unavoidable. If truly unforeseen circumstances do require user action, try to continue any work that can still be performed. If all of the work is dependent on the user feedback, consider continuing the work in the background by guessing the most likely user response. If the user shows up and enters a different response than the one guessed, back out the guess work and do the right thing. If the user is not present to see the prompt, at least there’s a chance that the long-running task will continue down the right path uninterrupted.


Introducing cppsh: A bash-Like Shell with C++ Syntax

cppsh

It’s been a long time in the making, but I am proud to announce the first beta release of cppsh, the bash-like shell specifically designed for those engineers who find themselves most comfortable at the reins of a C++ compiler. The best features from both bash and the C++ language come together in cppsh to make you a more productive shell user. Some of the most important features of cppsh include:

File Iterators

File iterators allow you to traverse the files in your working directory using the convenient C++ STL iterator syntax:

cppsh>
for ( cppsh::file::const_iterator file_iterator =
        cppsh::list_cwd().begin();
      file_iterator != cppsh::list_cwd().end();
      ++file_iterator )
{
  typedef cppsh::command_line_entry<std::string> entry;
  std::vector<entry> command_line;
  command_line.push_back( entry( "mv" ) );
  command_line.push_back( entry( "-f" ) );
  command_line.push_back( entry( *file_iterator ) );
  command_line.push_back( entry( *file_iterator + ".bak" ) );
  cppsh::execute_command( command_line, std::cout );
}

Command Pipelines

Following the UNIX tradition, cppsh makes it easy to feed the output of one command into another command:

cppsh>
typedef cppsh::command_line_entry<std::string> entry;
typedef std::vector<entry> command;

command cat_command;
cat_command.push_back( entry( "cat" ) );
cat_command.push_back( entry( "datafile" ) );

command sort_command;
sort_command.push_back( entry( "sort" ) );
sort_command.push_back( entry( "--unique" ) );

command wc_command;
wc_command.push_back( entry( "wc" ) );
wc_command.push_back( entry( "--lines" ) );

cppsh::command_pipeline<command> pipeline;
pipeline.push_back( cat_command );
pipeline.push_back( sort_command );
pipeline.push_back( wc_command );

pipeline.execute( std::cout );

Configurability

Like many UNIX programs, cppsh can be configured by editing the .cppshrc file in your home directory. Unlike most UNIX programs, however, the .cppshrc file is a full-fledged C++ header file. The .cppshrc file is responsible for defining the cppsh_shell type. This is done by creating a user-specific traits class and passing it as a parameter basic_cppsh_shell template:

#ifndef DOT_CPPSHRC
#define DOT_CPPSHRC

#include <cppsh/basic_cppsh_shell.hpp>

namespace cppsh {

struct user_cppsh_traits
{
    typedef vi_editing_mode editing_mode_t;

    static const int command_history = 1000;

    static std::string prompt()
    {
        return "cppsh>";
    }
};

typedef basic_cppsh_shell<user_cppsh_traits> cppsh_shell;

} // namespace cppsh

#endif // DOT_CPPSHRC

Extensibility

If you’re a demanding user, you might find that the .cppshrc file does not offer the power you need to customize cppsh to fit your needs. You’re still in luck! All of the features described above (and more) are packed into only 412,011 lines of C++ code, so you can easily hack cppsh to fit your own needs. Internally, cppsh makes extensive use of template metaprogramming, so the code is terse and easy to understand.

What are you waiting for?

Get started with cppsh today — visit the project page for downloads and documentation. You’ll be happy you did.

The cppsh team


New Song — Codename Mystic

Fractastic

Here’s another song. I haven’t bothered naming it yet, so I’ll just release it under the codename I’ve been using. The song again features Ableton’s Collision instrument for the bells in the beginning. It was partly inspired by the upbeat and airy sound of Aphex Twin’s Flim, which is one of my favorite tracks of all time. The audio cutting techniques that I used were probably a result of my deep love for Machine Drum’s music, which has some of the sweetest audio slicing that I’ve ever heard. Machine Drum is to audio as a teppanyaki chef is to an onion tower.

Evan Mezeske — Codename Mystic


The BigBoxoCo Disco Party: Why Segmentation is Good

As the freshly brewed coffee enters my mouth, I experience my first glimpse of consciousness for the day. “Where am I?” I mutter, in broken English. The gray walls around me slowly come into focus, lit by the flickering of a long-in-the-tooth fluorescent bulb. The top half of a man’s face appears over the top of my cubicle wall.

How’s the wonderful world of iNetConjoinApp?”

The caffeine must have made it past my blood-brain barrier, as I recognize at once that I’m at EnergyModCo, where I am one of a handful of employees. The half-head belongs to Freyr, EnergyModCo’s COO, lead customer service rep, and deployment technician.

Umm, it’s, well, I just started working on the –”

Great, that sounds good. You remember BigBoxoCo?”

You mean, as in our biggest cust–”

There’s a problem at one of their warehouses. Something to do with our lighting controller.”

Oh?”

I just got off the phone with the warehouse manager. All the lights went out for a few minutes, but they’re back on now.”

Uh, that’s bad. Thank goodness they have skylights.”

Nope. This is their first two-story warehouse. The only light the first-floor customers had was from the emergency floodlights.”

My throat tightens. “Well, I’m on it. We can’t let that happen again.”

The weight of the situation slams into me like an over-packed palette of giant mayonnaise jars. After being awake for only 25 seconds, I’m not ready to douse this kind of blaze. I don’t have a choice though, so I lean back in my chair and gaze at the craquelure on the ceiling tiles. How could this have happened? I recall that at one time we did have problems with the smart-breakers that switched the lights. They would sometimes mysteriously ignore the commands sent to them by EnergyModCo’s software. But I fixed that by adding a watchdog that would retry the switch commands if they did not take effect. After a brief palpitation subsides, I admit to myself that the lighting control watchdog must contain a nasty bug.

After some brief email digging, face palming, and silent cursing, I manage to get a VPN connection set up so that I can SSH into EnergyModCo’s on-site lighting controller. I bump up the logging verbosity, which requires that I restart the system, and start looking for clues. After a few minutes, I see the periscope that is Freyr’s forehead rise above my cubicle wall.

The lights are off again! I’ve got the BigBoxoCo manager on hold, and he’s about to lose it!”

My lip quivers as I struggle to suppress my fight-or-flight instinct. It can’t be a coincidence that the lights went off right when I restarted the software. What have I done? In a panic, I force our software to turn the lights back on, and thankfully it works. At this point, I am paralyzed with fear. I want to disable our software entirely, but what if stopping it is what made the lights go off just now? I pull my hands away from the keyboard, fearing that anything I do might cause the BigBoxoCo manager to enter sudden cardiac arrest, or worse.

Without being able to touch the on-site software, I dive into the source code, hoping to track down the bug analytically. I pore through the entire stack, following the data flow and logic for the relatively simple lighting control subsystem. The scheduling code makes sense. So does the the timer code. The trickiest code, for the watchdog system, looks entirely correct. I rack my brain; what am I missing? I am startled by the crack of thunder, but there doesn’t seem to be a storm outside. As my nerves resonate with the imagined sound, Freyr’s forehead crests my cubicle wall.

“The BigBoxoCo manager is flipping out. He says, and I quote, that ‘There’s a God damned disco party going on’ in his warehouse. They are going to have to stop accepting customers.”

Content with having surpassed even my worst expectations, Freyr jogs back to his office. I follow him briskly.

Freyr, can’t the manager hit the manual lighting override? I think it might take me a while to figure out the problem.”

What are you doing away from your desk? No! Only BigBoxoCo’s maintenance engineer has a key to the enclosure, and he’s AWOL. Go!”

I save three seconds by running back to my desk. At this point, I’m bouncing ideas off our other programmer, Nate. No good; he has never worked on this system, and can only offer limited advice. I go back to staring at the code. I may have been unconscious an hour ago, but now the fire of my mind is burning with the focused intensity of a TIG welder. The coffee is gone. I begin questioning all of my assumptions. Compiler bug? Memory corruption? Cosmic rays? Nate complains about the sound of my forehead slamming against the desk. In my heightened state of awareness, I perceive the ghostly sound of footsteps come to a stop outside my cube. Moments pass before the shrunken form of Freyr emerges from the hallway. I am calmed by his lack of speed as well as the fact that he is not using his periscope.

I’m sorry,” he says.

Uh, hey Freyr, what’s up…?”

I fixed the lights. It was my fault.”

Until this point, it had not crossed my mind that the problems may have been caused by the lighting controller being configured incorrectly. “Wha — what the hell happened?”

I had configured the lighting controller at a different site with the IP address of the smart-breaker at the disco warehouse. The other site was in a different timezone, and its schedule said that the lights should be off.”

The problem was too simple. Why didn’t I think of this? One controller thought the lights should be on, and the other thought they should be off. Thus, the lighting watchdogs at each site were fighting over control of the lights. Neither controller knew about the other one; they just thought that the smart-breakers were disobeying them and retried their commands. Over and over. I subdue my first instinct to tackle Freyr on the spot, and murmur, “Okay. Thanks for letting me know.”

I sit still for a few minutes, allowing the turbulence of my rage to subside. My initial response is to be angry at Freyr for wasting my time and terrifying me. However, as I calm down and regain clarity, I realize that he did nothing wrong. Who hasn’t mistyped an IP address before? I know I certainly have, many times. Freyr made a simple and understandable mistake. The problem was that the BigBoxoCo network was set up in such a way as to allow a simple mistake to wreak utter chaos.


As it turned out, BigBoxoCo had all of their hundreds of warehouses on the same virtual network. Not only could BigBoxoCo’s corporate headquarters reach machines at every single warehouse, but so could any individual warehouse. A PC at a BigBoxoCo in New York could ping a PC at a BigBoxoCo in Oregon with no problem. Even ignoring the security repercussions of such a setup, there are good reasons to avoid it. If the network was set up with a star topology, with only the corporate headquarters having access to every single warehouse, the disco party fiasco could have been easily avoided. In other words, segmentation is good.


New Song — Spelunkatronis

Spelunkatronis

I don’t have a whole lot to say about this track, other than that it is one of my first songs that actually feels somewhat finished. I had a lot of fun making this one. In particular, I fell in love with Ableton Live’s Collision instrument and the infinite variety of bell-like sounds that it can create. And, of course, the song features some pads courtesy of my Alesis Andromeda A6. I think I’m going to name it Padosaur.

Evan Mezeske — Spelunkatronis


C++ Streams & Typedefs: Be Charful

The C++ typedef keyword is indispensable in many situations, especially for writing portable low-level code. However, in some circumstances it can cause trouble, particularly when it comes to function overloading. Consider the following C++ template class:

template <typename T>
struct foobar
{
    foobar( const T foo ) : foo_( foo ) {}
    T foo_;
};

One might want to write a simple stream output operator to format the template class’ member values, e.g. for debugging purposes:

template <typename T>
ostream& operator<<( ostream& s, const foobar<T>& fb )
{
    return s << "foo: " << fb.foo_;
}

This seems reasonable. Now, let’s assume that this template is going to be used in a context where T will be one of several fixed-width integer types. These are usually typedefs from a header like stdint.h (for those that don’t mind including a C header) or boost/cstdint.hpp (to be a C++ purist). They are commonly named int64_t, int32_t, int16_t, and int8_t, where the X in intX_t specifies the number of bits used to represent the integer. There are also unsigned variants, but we’ll ignore those for this discussion.

Let’s now explore what happens when we initialize a foobar<intX_t> instance with its foo_ member set to a small integer and print it to standard output via our custom stream output operator:

cout << foobar<int64_t>( 42 ) << endl;
cout << foobar<int32_t>( 42 ) << endl;
cout << foobar<int16_t>( 42 ) << endl;

Each of these statements prints “foo: 42″, as expected. Great, everything works! But wait, there was one type that we didn’t test:

cout << foobar<int8_t>( 42 ) << endl;

This prints “foo: *” instead of “foo: 42″. This is probably not the expected result of printing the value of an int8_t. After all, it looks and feels just like all of the other intX_t types! What causes it to be printed differently from the other types? Let’s look at how the integer types might be defined for an x86 machine:

typedef long int int64_t;
typedef int int32_t;
typedef short int16_t;
typedef char int8_t;

The problem is that the only way to represent an integer with exactly 8 bits (and no more) is with a char (at least on the x86 architecture). While a char is an integer, it is also a… character. So, this trouble is caused by the fact that the char type is trying to be two things at once.

A simple (but incorrect) approach to work around this is to overload1 the stream output operator for the int8_t type, and force it to be printed as a number:

// This is incorrect:
ostream& operator<<( ostream& s, const int8_t i )
{
    return s << static_cast<int>( i );
}

The problem with this approach is that the int8_t typedef does not represent a unique type. The typedef keyword is named poorly; it does not introduce new types. Rather, it creates aliases for existing types. By overloading the stream output operator for the int8_t type, the char type’s operator is being overloaded as well. Since the standard library already defines a stream output operator for the char type, the above definition would violate the One Definition Rule and result in a compiler error. Even if it did compile, the results of redefining the way characters are printed would probably not be desirable.

An alternative (working) solution to the problem is to overload the output stream operator for the foobar<int8_t> type:

ostream& operator<<( ostream& s, const foobar<int8_t>& fb )
{
    return s << "foo: " << static_cast<int>( fb.foo_ );
}

This definition does not clash with any existing overloads from the standard library, and it effectively causes the int8_t to be printed as an integer. The downside is that it will cause unexpected behavior when a foobar<char> is printed, if the programmer intends char to represent a character. The only way to avoid this would be to define int8_t as a class instead of making it a typedef, and providing a well-behaved stream output operator for that class. The class’ arithmetic operators could be overloaded to make it look almost exactly like a POD integer, and it wouldn’t necessarily take up any extra memory. However, this solution is still not ideal, because classes behave differently than POD types in subtle ways (e.g. POD types are not initialized by default, but classes are).

If there’s anything to take away from this, it’s that the C++ char type is an odd beast to watch out for. Also, the name of the typedef operator could use some improvement…

  1. If you are curious as to why I suggest overloading instead of template specialization, see this article. []