Saturday, May 4, 2013

New Blog

Since it's been 3 years since I've written, you can probably imagine that things have changed.  I just started a new blog, because I've switched gears and started doing something new.  My new direction (C++/OpenGL/PC/Mobile) might not be interesting to those following this blog, and Gabob is officially over, making the URL obsolete (but Tim and I are still working together).

Friday, May 28, 2010

Credit Card "Security"

I've run into a bit of a problem with, and I thought I'd share what has happened in case someone can help me understand it better.

About a month ago, I placed an order with them. That's nothing new. I've been an customer for more than 10 years. About a week later, my stuff arrived, and everything was peachy...or so I thought.

Last Sunday, I got an email from them saying that another customer complained that the charges for my order ended up on their credit card. By the time I got that email, I had 3 other ones saying that my order had been refunded (my order ended up in 3 shipments, each charged separately). Those refund emails included the last 4 digits of the card I used to pay, which were wrong. Instead of AABC, I typed CCAB.

But being a responsible credit-card-accepting vendor, requires more than just a card number. They also require a billing address and the 3-digit security code on the back of the card. I entered my address and the 3-digit code from my card when I placed the order. Since I messed up the card number, the billing address and security code did not match the card number I entered, but the charge went through anyway! The card number happened to match someone else's card and whatever computer system verified the payment information didn't check the address or security code.

The email I first got from them said that they could charge a card stored in my account already and that I could authorize payment if I told them the last 4 digits of one of them. I'm not sure what cards I still have stored in my account, so I thought I'd log in to see, but my account has been disabled, so I can't check. I emailed them back saying they could charge my card AABC if it's in there, but if not, they could call me, and I gave them my phone number. I also asked how this could have happened.

I got an email back a couple days later blaming Visa for the mistake. They said that they just pass all the card info the customer enters to Visa, and Visa accepts or rejects the charge.

They also told me to call customer support and tell them I need to pay for this order over the phone. I called support, gave them the order number, and told them I need to pay for it, but the guy told me he can't access my order because my account is locked. He told me to wait 24-48 hours and try logging in again to change the payment for that order. Based on previous experience with entering incorrect payment info on an Amazon order [1], I didn't think I could change the payment info on a completed order in their web interface, and I suspected he didn't understand that the order was already completed. I told him that, but he just repeated what he told me, so I told him I'd try it. It's been 48 hours now, and my account is still locked.

Getting the run-around with support techs who don't know what's going on and don't seem to communicate with each other is frustrating, but nothing new.

What's new for me is that I bought almost $300 of stuff from using another person's credit card. I would have thought that in 2010, using a stranger's credit card would have been a lot harder than that.


[1] If you buy mp3's from Amazon, you authorize payment and then can download the songs immediately. However, Amazon waits to charge your card, probably to let you buy multiple songs and group them all into one transaction. So if you accidentally use an expired debit card to authorize the payment, you get an email an hour or so later saying that your card was rejected, long after you've already downloaded the songs. Because the order was completed with items delivered, you can't change the payment info for the order. I emailed support about that, and they had no way for me to pay for it over the phone either. I had to go find all the songs and buy them all again with a new card.

Wednesday, February 10, 2010

Sad Games

Every once in a while, I download several game demos or take advantage of a Steam weekend sale, and try a bunch of games. This gives me a chance to keep up on what others are doing and learn from their example, both things I should and shouldn't do. There probably isn't any game developer that would say that good game design is not important, but actions speak louder than words, and many of the games I play have some glaring design problems.

Of course, much of the field of game design is simply opinion, so just because I think a game has a "glaring design problem" doesn't mean it really has a problem or that it won't sell. Even if I'm pickier than the average potential customer, there are players out there like me, and incorporating the following guidelines would probably help.

1. Only make interactive those parts of the story that should be interactive.

Use short cut scenes for the rest. Cut scenes do serve an important purpose. They define the setting, flesh out characters, explain story elements that would be difficult to convey with gameplay, they provide rest after intense play sequences, etc. Those are good things, as long as they're kept short, because the player isn't playing a game to watch cut scenes.

That said, it is possible to go too far the other direction: making some parts of the game interactive when they shouldn't be. The original Assassin's Creed is the best example of this flaw I can think of. In between sessions on the table, you actually have to control your character to get off the table, walk into his bedroom, and press the "use" key on his bed. The game seriously won't continue until you do that. If you take too long, say, because you think there's a reason you're actually controlling the character, so you start to explore the lab, the man just keeps saying stuff like, "Go to your room to rest."

After going to bed, the screen turns black, and then you're waking up. Your guy stands up by the bed and then waits for you to take over. The guy in the lab starts telling you to get on the table. Yep, that's right. You have to walk all the way in there and press the "use" key to get on the table.

There is one part I remember where you can talk to the guy's female assistant and get some info, but all that boring walking for that little story element? Not worth it. The player should only be doing interesting things. Less interesting things should not even be in the game, but if they really are necessary for some reason, they should only be briefly covered in a cut scene.

This is the reason I never even tried WoW. Whenever I saw my brothers playing, they were doing one of two things: running or waiting. They were running from one part of the world to the next, just running, running, running, and more running. I was playing a different game on my computer, and whenever I looked at my brother's screen, he was still running. If they weren't running, then they were waiting for people to join their group for a raid. That was years ago, and I know the game has changed a lot, but for me, grinding in general is a problem. More on this below.

2. The coolest and/or most critical parts of the game should not be in the cut scenes.

I first learned this design principle from Warcraft III. There's a character who turns evil and becomes a Death Knight or something. At one point in the cut scene, he charges out through a crowd of soldiers, and he 1-hit kills a bunch of guys in his way—BAM, BAM, BAM, they fall with every stroke of his arm, and he road away untouched. It was awesome. I haven't touched the game in several years, and that's all I remember about the cut scenes. Then in the game, you have him as your Death Knight hero. Can you do that with him? Nope. He stands there and hacks at the same enemy soldier over and over again like an impotent 2 year-old. Disappointing. But it sure was an awesome cut scene!

Assassin's Creed made this error too. I played about three missions into the game, and each one went as follows. You work your way through town, and you finally get to the target's lair. The only way in is through the front door (when do assassins use the front door?). As soon as you enter the doorway, you're ripped out the game experience and thrown into a cut scene. You proceed to watch your guy call out his target (when do assassins do that?), talk smack, the target tries to recruit you to his team, and then talks smack back when you refuse. And then after stirring up a beehive of angry bees and plopping you in the middle of it, the game gives you control of your guy again. Every time that happened, I could hear the designers laughing at the ridiculous situation they got me into during that cut scene. If I could have controlled my guy, I certainly would have done things differently.[1]

Let the player do the cool stuff and make the critical decisions.

3. Read Raph Koster's Theory of Fun, and apply it.

In that book, I think Raph hits the nail on the head. Games are fun because they tap into our natural desire to learn and master new skills. Games can tap into other natural desires, like curiosity, when we play to find out what happens in the story, find out what gadget X does, or to find out what's on the other side of that mountain. Movies and books also tap into that curiosity, but only games can do the former.[2]

My #1 reason for uninstalling a game I didn't finish (which is most of them) is that I just get bored. I've never been hooked by a game's story, and I'm rarely intrigued enough by the gadgets and abilities to play just to find out what they do (not because I'm smarter or more refined and crap—it's just me, my preference). The main reason I play is to do something new. Games which aren't continually requiring the player to improve their skills, learn new skills, and apply their skills in new ways miss out on this most compelling reason to play a game at all. Again, there are other reasons players might play a game, but the skill angle is unique to games.

There are many popular games that have this problem. Those other reasons players play games might still be sufficient for them to have fun, finish the game, and recommend it to their friends. But I submit that those players would have loved the game even more if it kept up the skill learning.

Requiring the player to continually learn and improve his skills doesn't mean the game needs to be really hard. It just means that the gameplay needs to keep evolving. Games are hard when the incremental steps that are required of the player are far apart. Games are boring when those steps disappear or even almost disappear.

FPS games frequently have this problem. You encounter harder, stronger enemies, and you get bigger, stronger guns to kill them with, but the core gameplay of "run, aim, fire, and keep firing until everyone's dead" stays the same. Usually a shooter will incorporate some new mechanics which set it apart from other games in the genre, but then the game still needs to evolve and stay fresh throughout. Usually, if the new features of the game are compelling enough, I'll play for an hour or so, but by then I've gotten a handle on them, and nothing new happens, so I'm done. Or I'll play for an hour and never get past the same old stuff and conclude the new stuff isn't worth my time to even try.

The FPS genre is one of my favorites to play, but I have only finished 2 FPS single player games: Far Cry (the original) and The Chronicles of Riddick: Assault on Dark Athena. I finished the former because the game kept giving me new situations to solve and requiring me to push my skills to new levels. I played Dark Athena a few weeks ago, and while I had other issues with it (the melee combat and ridiculously gross dialogue by Jaylor, to name two), it let me do some awesome stuff, like drive a mech and control the drones (the Alpha Drone was particularly cool). Even while not controlling those things, the weapons had different uses and required different play-styles, and that kept the game fresh (I'm a sci-fi fan too, and that never hurts).

The defining core mechanics of other mainstream genres, e.g. RTS and RPG, are also done to death, and I don't play through those single-player games either.[3] In the case of the latter, grinding isn't fun because it's the same thing over and over again. I've never played an RPG that didn't include grinding as part of the game. It might be part of the genre's definition though—the slot machine effect: you're dumping minute after minute of your life into the game, hoping the next monster will drop something good. I have no use for that as a player, and I wouldn't enjoy making a game like that either.

It's hard to make a game that continually stays fresh. I don't know if we've succeeded at it, because by the time I was done with Now Boarding, I couldn't bear loading another airplane. I can't objectively judge the final product. As employees take over the mundane tasks of grouping passengers and loading, docking, and undocking airplanes, the gameplay does evolve. The different maps offer different routing problems to solve. But it's only $15, so it doesn't need to last for weeks before players grok it.[4]

An RPG that is a long quest would be a ton of work to make it feel epic and stay interesting from a skill perspective. Maybe development costs would be prohibitive, but anything developers can do to require new and different skills whenever they can would improve the game.

4. Give the player clear short-term goals.

Clear means that the player understands what he is supposed to do and can reasonably figure out how to do it from what he's learned about the game world to that point.

Short-term means that the goal won't take too many steps to accomplish. Finishing missions is a natural hook for players, so let them do it—and keep doing it. Don't go too far and make the goals too short and small (unless you're developing a game like Achievement Unlocked). I like to finish what I'm doing before I quit playing, and I don't want to be locked in for hours. It's nice when the game can be enjoyed in small bites. I can always eat more bites in one sitting if I want to.

Not knowing what to do or how to do it is probably my #2 reason for uninstalling games. Most games are pretty good about this, but occasionally I run into a situation where I'm stuck. I go over past dialogue (if it's available), I retrace my steps and look for things I missed, but once I exhaust my resources in the game and/or my patience, I'm done. Alt-F4. And if Alt-F4 doesn't instantly close the game, then I instantly uninstall it when I finally get out. If Alt-F4 works and the game was otherwise compelling, I might google for a walk-through or something.

The most extreme example of not giving the player enough information is Silent Hunter 3. It's so bad I had to find walkthroughs just to explain the main menu. The reason is it's hardly even a game, but rather just a simulation tool. Usually a single player game has some kind of guided experience, i.e. you click on "Play" or "Career" or "Campaign" or something like that, and then you play through the game, and it gets increasingly difficult. In SH3, the "Career" option doesn't let you play anything (I never learned what it was for. I created a new profile with my name, but I couldn't select it or anything). The single player game is actually just a list of scenarios, each with several load-out options. All are available from the start, so there's no way to be sure you're playing the "right one" when you first play. After reading through a walkthrough to find out how the missions work, I played through the missions, which were quite fun (well, excpept the Gibraltar mission which was ridiculously long and tedious...I don't think I have the patience to be a real U-boat captain).

Aquaria also lost me for this reason. Where do I go? What do I do? How do I unlock that door? I have no idea. It's a beautiful game visually, but I hardly got to see any of it. Other artsy games like The Path and Blueberry Garden don't give the player clear goals either (well, The Path seemed to: "Stay on the path." I did, and it was short and boring, and I got 0's for all my scores. It lied to me. :) ). It's great that other people like those kinds of games. I just can't get into them. Exploration and experimentation are not reasons I play video games. I do a lot of that in the real world. But again, that's just me. If you make a game that encourages and rewards exploration and experimentation, why not add some goals in there too? You don't need to give the goals time pressure or anything that would hamper the explorer's experience, but then people like me would like your game too.


These are just four major things I've picked up over the years. All games have more specific issues that could have been done better and which are useful to talk about, but I wanted this post to be generally useful. Most games could be improved in these areas, and if thought is given to these from the beginning of a project, they probably wouldn't require any more work.

I think it's sad when a game really stinks. A game represents a lot of time and creative effort, the expenditure of real human life value. Usually, the difference in work between a crappy game and a good game is a relatively few small changes.[5] But when those changes are left undone, all that effort is wasted.


[1] About the same time I played Assassin's Creed, I also played Hitman: Blood Money. That is an assassin game done right, and I played every level until I got the "Silent Assassin" rating. Assassin's Creed should have been called Pompous Brawler's Creed.

[2] There are other reasons people play games too. MMORPG's tap into our social natures, and games can also stimulate our OCD tendencies, e.g. when I have to finish a collection or find all the hidden packages or whatever. Some players also play games to escape reality, and that's sad.

[3] Multiplayer games have the inherent ability to push players and keep their skills evolving, but they are difficult and expensive to make. Multiplayer is the reason I like FPS games so much. I've played a lot of Counter-Strike, Team Fortress Classic, Team Fortress 2, Combat Arms, and Left 4 Dead. The most successful games have been multiplayer ones, but they're more of a "cash dragon" than a "cash cow"—if you can manage to slay the dragon, you'll make a lot of money.

[4] Grok is the term Raph Koster uses to describe when a player has mastered the skills of a game. This is the point at which I usually quit playing.

[5] The key term is "relatively". If you think about a game project, there's a lot of art and a lot of code that went into it. As an example, it wouldn't have been much more work to make the "Career" mode in SH3 take the player through each scenario with the recommended load-out. In the final stages of Now Boarding development and now Clockwords development, little changes that require very little work end up making a big difference in the overall player experience.

Friday, February 5, 2010

Flash, Flash, Flash....

...Why you gotta be like that?

I was tempted to call this post my "Flash 'Dear John' Letter" in reference to Danc's Flash Love Letters, but since we're not actually dumping Flash (at least not yet), it wouldn't be accurate, just catchy.

So what is this post about then? My gripes with Flash. Many people have love-hate relationships with tools and products they use. We're finishing up our second premium Flash game right now, and while I love the good things about Flash more than ever, I'm increasingly frustrated by it. I hope that by joining with other Flash game developers in airing our grievances, Adobe might take notice and make some changes for us. If they don't, we won't be the only developers switching to Unity.

The Good

We use Flash for two reasons:
  1. Wide Reach – Almost all internet-connected computers can run our games.
  2. Rapid Development – Games come together quickly.

We like these things about Flash development:
  • Tight integration between assets and code.
  • AS3. I have to admit, I enjoy writing AS3 code. I have several years of C/C++ experience and a few more of C# experience, but I'm having a lot of fun with AS3. Yeah, it's missing some language features, and it can be quirky, but it's fun.
The Bad

These are things that are really hampering large-scale Flash game projects for us.

1. Two necessary tools that don't work together very well.

Flash CS4 doesn't cut it for software development—it's terrible. It's great for creating visual assets, laying out interfaces, and exporting compressed sound. Those things are necessary for game development, and Flex Builder can't do them, so Flash CS4 is a necessary tool (if there are alternatives that really work, I haven't found them).

Flex Builder is vastly superior to Flash CS4 for compiling and debugging code. Flex Builder 3 is the only Flash profiler, so it's necessary for game development, IMO. I know not all Flash game developers use it, but most don't create large programs that need it. Garbage-collected languages need memory profiling at the very least. That's about all I use the profiler for. The performance profiler just tells me what I already know: all the execution time is spent rendering the game.

In theory, CS4 could export a .swc that FB imports, and I could load the symbols and do my thing with them. But there are bugs.[1] The only way I've found for objects exported from CS4 to always work as expected when imported into FB is to export a .swf from CS4 and load it dynamically in my FB AS project.[2]
This integration is clunky and tedious to setup and maintain.

Solution: Fix the bugs, remove the hoops we have jump through.  Make FB and the Flash IDE work together seamlessly.

2. All asset data is contained in the .fla's.

We have a couple hundred MB of bitmaps and wave files in our game. All that data is imported into .fla's. The visual things are laid out and composed into symbols which I can use in code. The sounds and music are compressed and processed by Flash, which does a great job.

This leads to two problems:
  1. Tim and I (and our other contractors) all work at different locations, and our SVN repository is on the internet. We already have hundreds of megabytes to check in and update, and then the .fla's double it. While most check-ins and updates don't touch nearly that much of the project, sometimes they do, so I postpone my check-in until I'm done for the day so I can start it and walk away. If Tim needs something sooner, then I have to do it during the day and interrupt my work.
  2. Whenever a file on the disk changes, we have to open the .fla it's in and update it manually. This is stupid. That's the nicest word that accurately describes this "feature" of the Flash IDE.
Solution: Store references to files, not the file data itself in the .fla. Unity does this.

3. Flex Builder caches source files.

Ironically, FB has a similar problem as CS4. I can't stand the FB code editor or the Flash IDE's code editor, so I use an external one (gVim FTW!). Other people probably like it just fine—that's not the problem. Code editor preferences are personal, to each their own. The problem is that when I change the files on the disk, FB doesn't notice, so it doesn't recompile them when I build the project. This is true of the .swf's that I export and embed in the project and the .as files themselves.

There are 2 work-arounds for this problem:
  1.  Open the .as file in FB, because then it will say "Hey, the file changed, do you want to reload it?"
  2. Project -> Clean -> OK. Then it recompiles everything, which takes a lot longer. I have to clean the project whenever I re-export from CS4. If I just change code files, I have to estimate which work-around will be faster.
Solution: Don't copy all the source files, just use the ones I'm editing. It's trivial to detect when files change.

4. Both tools are bloated and slow.

It doesn't take a very complex scene for CS4 to start lagging, and I have a nice machine. While our projects might be big by Flash standards, they are small by the rest of the software world's standards. 8 years ago, working at Acclaim, I could build our much larger project (a AAA-sized title) on my totally garbage Win95 computer just as fast as my core i7 machine builds my much smaller Flash project today.

What gets me though is how the CPU on my computer is hardly doing anything while CS4 is exporting or FB is compiling. What the heck is the program doing? CS4 has just 1 file to read (as already covered), and it's even slower than FB. I have all my big music wave files in 1 .fla, and even exporting it doesn't use any CPU to process all that data. Yet it takes minutes to export, and during that whole time, CS4 is completely unresponsive. FB doesn't go unresponsive when compiling, but it still takes a long time and doesn't utilize the CPU at all. No CPU utilization means the process is spending most of its time waiting.  That is very poor software design.  All other compilers I've ever used are much faster.

Another result of this issue is that our game assets need to be split up into many .fla's to keep compile times reasonable. That adds additional headache for managing what is where, which compounds with issues #1 and #2 above, since more .swf's have to be dynamically loaded, and then assets need to be loaded from the right .swf's, and when changes are made, the right .fla needs to be manually updated.

5. The Flash Player is slow.

It uses a software vector renderer, which is never going to be fast compared with the hardware bitmap renderers that most games use. For small, simple games and ad banners, it's great. For large, complex, fullscreen games, it stinks. AIR does perform better fullscreen than the Flash Player does, but it's still nothing compared to hardware acceleration.

In our latest game, Clockwords, we've made almost everything bitmap-based to bypass the slow vector renderer, and that keeps our game working on slower machines. But this comes with another source of headaches: we need low-res versions of our art for the web game, and high-res versions for the AIR game. That means we need another set of .fla's and another set of art. If Adobe hadn't introduced conditional compilation, then this would have been way too retarded to accomplish. With conditional compilation, it's just retarded, but still possible.

Solution: Make real hardware acceleration a higher priority. There are opensource and proprietary products that can run .swf's in a hardware accelerated environment, so this is not an unsolvable problem. It just hasn't been a priority for Adobe. Don't worry about models, animations, shaders, or any of that stuff. Stick with 2D, it's OK. Just write code to transform vector shapes into triangle strips, and you're just about there. ;)


Flash has historically been targeted at small project sizes, like ad banners and short form games. Adobe is spending lots of time and resources on Flex, which is targeted at RIAs. Long form Flash games require the software dev tools of the Flex world and the visual tools of the Flash world (with improved support for large projects). We are stuck in the middle without good tool support on either side.

While we're reaping the benefits of wide reach, the poor tool support is greatly increasing our development burden. We're sore from hitting our heads against the ceiling Adobe has placed over the Flash platform.


It would be nice if Adobe addressed these issues, because then we could keep doing what we're doing. We've invested a lot of time and money into Flash development, and the libraries and community support are great. Companies like MochiMedia, FGL, and the countless Flash portals make game development and distribution much easier than other platforms.

I think Unity is positioned to grab some significant market share. All it needs to take off is content, and indie developers are already flocking to it now that it's free for indies (or at least poor ones). When there is sufficient content for the Unity browser plugin, users will install it, and portals will support it. We're not really that interested in making the jump to 3D, but we're definitely interested in hardware acceleration...and a sane art pipeline...and just 1 slick tool. As Unity gets more developer support, it will get wider reach, and so the #1 reason to use Flash will be reduced.

We're kicking around ideas for our next project after Clockwords, and we're pretty sure we want to do a bigger one. Unless the Flash scene changes significantly, taking on Flash's limitations and headaches to get its reach probably won't be worth it.

[1] The primary bug was that timeline script didn't always execute. Some animations need some script, and so I need the script to work.

[2] You can see how I do it in our Global Game Jam 2010 submission. The source is in the .zip file you can download from there. The file contains a list of all assets exported from the GameAssets.fla. That way, I can write my code against the file to let the compiler make sure all my references are spelled right.

Monday, February 1, 2010

Global Game Jam 2010

As many of you probably know, this past weekend was the Global Game Jam! We decided to do it, and we hooked up with the group at Newport, Wales, so we could start and end earlier than our own time zone. We partnered up with a great artist named Scott Thompson in Wales, and we made a game in 2 days.


Making a game in 2 days requires some different skills and thinking than making a game at a "normal" pace does. I had to set aside some of my OCD tendencies and make some last-minute, dirty hacks to get a couple things working.

Perhaps the most useful thing was the necessity to limit the scope of the project. It's really easy to say, "Oh! Let's add this!" We say that a lot when we're working normally. When prototyping, it's important to really stick with what's essential for the core gameplay. The reason is because until you have the core done, you don't even know if it's going to be fun. While we've always prototyped our ideas before making full games out of them, we tend to let our prototypes bloat a bit, and we've thrown some out, so we could have saved the time if we stuck with the core features.

With a 2-day limit, it's also a good idea to stick with what you know, just for practical purposes. Some might disagree with that because it can limit creativity, but it just depends on how complete you want your game to be after 2 days.

Game Overview

The theme for this year was "deception", and the GMT constraint was "a sink, a wink, a rink" (in no particular order). We made a game where you play as a mouse looking for cheese in a big kitchen. You have to sneak around, avoiding being seen by the humans in the kitchen. Cheese can be hidden in one of a dozen places, and you have to find it and get it back to the drop zone near where you start. Each time you collect one piece of cheese, another piece and a mouse trap spawns on the map. Running into a trap kills you instantly, game over.

To make things more interesting though, you can only see traps, cheese, and obstacles a couple feet in front of yourself. Also, some of the kitchen appliances can be used to distract the humans and make them look at them. Click on objects that glow green to start them up. Some take longer to activate than others (e.g. filling a sink to over-flowing takes longer than knocking over a garbage can). Gray squares can only be traversed when you don't have the cheese, so you have fewer sneaking options when you're carrying it.

You can play the game here:

This is the GGJ page for our game where you can download the source and stuff:


I'm not sure (at least at this point) if we'll do it again, because of the sleep deprivation and time away from families, but it was fun. Maybe in a year, we're be ready. In the meantime, we'll definitely keep our prototypes more focused.

Monday, November 30, 2009

Choosing a Business Partner

Over the course of about 3 weeks from the end of October to about half-way through November, we passed a nasty cold through our family.  During that time, I didn't shave at all, so when I was feeling better I thought I'd cut off most of the beard I grew and leave a goatee.  When one of my sons saw me the next morning, he said, "Whoa, Dad,  you and Tim both like games, and you both have goatees.  You guys are practically the same!"

I thought that was funny.  We also have noses and two eyes, but other than facial hair configuration (I've since shaved it all off), common human features, a desire to make fun games, and a common vision for our company, our similarities don't extend much past those things.

That's what I want to write about today.  The success of a start-up business depends heavily on one's choice of business partner(s).  Gabob is my second attempt at a serious business venture with partners.  The first never got off the ground because of my choice of partners, and this one is working quite well because I chose wisely.  However, it wasn't until after Tim and I started working together that I realized that I chose wisely and why.

There are two critical factors to consider, IMO.  First is determination.  How determined is your prospective partner to "make it happen"?  Is he/she as determined as you are?  Will he/she see the project through to the end?  If not, then it doesn't matter how smart or talented he/she is: that person is not a good choice.  That was part of the problem with my first business attempt.  We'd talk a lot and plan a lot, but when the meeting was adjourned, I was the only one who got working.  I was the only one who had anything to show at the next meeting.  After a couple meetings like that, I just didn't schedule the next one.  None of them did either....[1]

Without determination, no start-up will succeed.  The second factor, of which my son inadvertently reminded me, is to pick someone different than you.  The primary reason is this: there are a lot of things that need to be done in a start-up, and if you both like doing and are good at doing the same things, then deciding who will do the rest will be drudgery at best and a source of conflict at worst.

Because Tim and I are so different, we don't compete for tasks.  We do game design and make business decisions together, but when it's time to work (which is most of the time), I do my thing, and he does his.  I write code and keep the books, and he does most everything else.  I'm good at focusing on one thing, and he's good at juggling several things at once.  I'm good at planning projects and organizing data and assets, and he's good at managing our contractors and negotiating with portals.  We each see things the other doesn't.  I mean, my skillset is so specialized (I write code) that if I partnered with someone like me, we'd be doomed. [2]

The ironic thing is Tim and I met at a company where we both worked as programmers.  The difference is that I was doing what I was born to do, and Tim kinda just fell into it before he found what he was born to do.  He has now found it [3], and our business is benefiting from having two very different, complementary co-founders.


[1]  There were many other reasons the project was doomed to fail, but I didn't see the other reasons then.  I bailed on the project because I was the only one working on it.  The project was way too big for the team we had.  That's another thing we've done better with Gabob.  We also had a hard time agreeing on what product to make.  There were strong opinions on certain issues that were at odds with each other.

[2]  Seeing things differently and having different tastes has the potential to cause conflict.  If a person is too stubborn to listen to another viewpoint or change his/her own thinking, then that person will not reap the benefits of partnering with someone who is different and will likely cause the relationship to end.  I'm more interested in doing the right thing for our business than doing the thing that was my idea, and so is Tim.  In the past 2.5 years we've worked together, we've only had a couple discussions that caused one or both of us to take some deep breaths (at least that's all I know about...maybe he has to exercise more patience with me than I am aware of :) ).  But each time, one or both of us let go of one or more of our own ideas in order to accept one or more ideas from the other.

Usually our final decision is a combination of our ideas and is better than either of us came up with on our own.  I can think of multiple times when he has persuaded me and when I have persuaded him.  There have also been a couple of times when he accepted something I proposed without questioning it, and that made me a little nervous.  I count on him to see the holes in my thinking, and when he doesn't see any holes, I know it's not because there aren't any—it's because he doesn't see them either.  In those cases I can't help but wonder what kind of "surprise" will come up later.

[3]  Tim is a hustler.  By that, I mean that he is really good at getting other people excited about what we're doing.  Whether it's contractors, publishers, portals, or fans, he gets people on board.  While we wouldn't have a product if I didn't code it, we wouldn't make any money if he didn't sell it.  Both are necessary for a business to work, and I'm glad he can do and enjoy what I can't, because that frees me up to do what I enjoy.

Monday, November 9, 2009

AS3 Memory Management

After developing in C++ for several years, I must admit I found the idea of garbage collection in higher-level languages like Java, C#, and ActionScript to be alluring.  Making sure every "new" was matched by a "delete" was tedious but absolutely necessary.  Like any serious C++ programmer, I developed patterns and habits for making sure the code I wrote was clean as I wrote it, because tracking down memory leaks after they've crept in is even more tedious than thinking about "delete" whenever I typed "new".

My first experience with a "garbage collected" language was C# (.NET).  After getting deep into development of a new product, I quickly discovered that the garbage collector was really not so magical, and not even really that smart (at least not for a highly memory-intensive application).  I learned that I had to think just as much about making sure memory was collected as expected and when needed as I did when writing C++.

When I first started with AS3, I figured I probably could trust the garbage collector this time (different language, different application, smaller scale), but as Now Boarding (our first AS3 project) neared completion, it became apparent that memory was again not so magically taken care of.  Really, I shouldn't have been so surprised, but I was somewhat disappointed, because my understanding of the garbage collector was that I shouldn't have had one of the problems I ended up facing in NB.

I'm writing this post as a practical guide to AS3 memory management.  You can read the technical specs and algorithms which the garbage collector uses elsewhere.  This guide is based on my many hours of experience with the memory profiler in Flex Builder 3 and is about what actually happens and what you can do to fix/avoid memory problems in AS3.

Now that the background is over, let's set the stage:

Why does memory management matter in AS3?

There are probably many smaller-scale games and applications out there in which the author thought basically nothing about memory management.  Sometimes that works.  For the larger-scale games we've made and are making, poor memory management manifests itself in a critically bad way: poor performance.  As the pool of allocated objects grows and grows, it takes the garbage collector longer and longer to process the references, and it has to run more frequently.  Each collection pass is a noticeable pause in the game which happens every few seconds.  The game also generally runs more and more slowly.  Eventually, the game becomes unplayable.

How does that happen?

The short answer is that the game is maintaining references to the objects that are no longer being used, and those references make the garbage collector (GC for short) think the objects are still being used.  The GC only frees memory allocated to an object if the object has nothing referencing it (or a group of objects that reference each other have nothing else referencing any of them).  If the game would only null out all references to an object, the GC would reclaim the memory, and the pool of objects would not keep growing.

Unfortunately, it's not quite that simple.  I tried to treat it like that, and it didn't work as I expected.  In NB, each game mode is an object that contain everything for that game mode.  The game mode object itself is the root of the whole object hierarchy.  I figured that if I just set the 1 reference to the game mode object to null, then the GC would recognize that the whole hierarchy was orphaned.  Nope!  Every game mode was staying in memory after the next one was loaded, so going in and out of the game made the memory go up every time until the game was unplayable.

In practice (maybe this is in the spec, I don't know, but I missed it if it was), there seems to be a limit on just how big an orphaned group of objects can be before the GC just gives up and assumes the whole group is still being used.  That brings us to my first tip:

TIP #1:  Explicitly null out your object references.  If you have an object hierarchy that you want to be collected, null out the references each object has to the other objects in that hierarchy explicitly.

Don't asume that the GC will recognize the whole thing is orphaned.

Along with nulling references, event listeners are also extremely "sticky" when it comes to making sure objects "stick" around.  In my experience, using weak references doesn't matter.  If one object (Object A) subscribes one of its methods to an event on another object (Object B), then Object A and Object B will never be collected, and any objects they reference will also never be collected.

TIP #2: For every call to addEventListener, make sure you call removeEventListener.

To fix the memory leak between game modes in NB, I had to make sure every single event listener was removed.  Each one that "leaked" caused the objects involved to stay in memory.  I give every object that references another object a destroy method that I make sure is called.  In that destroy method, I set all references to null (after calling destroy on all children that have a "destroy" method, of course), and I make sure all event listeners are removed.  Yes, that's obssessive-compulsive, but it entirely solves all memory leaks (well, unless you really are keeping references around...).

This is where the memory profiler in Flex Builder becomes indispensable.  Particularly useful, IMO, is the "Live Objects" view.  Running under the profiler, I can load a new game mode and see the object associated with the old game mode disappear (or not disappear, and then I can drill down and see what is still referencing that old game mode object).

TIP #3: Use the Flex Builder memory profiler.  It's only in the $699 version, but it has a 61-day trial, which is plenty of time to use it to fix your game.  Then just reinstall Windows when it comes time to work on your next game. ;)  If you're a serious AS3 developer, then the $699 price tag shouldn't deter you.  Get the tools you need!

Object Pooling

When an object isn't collected when the game is no longer using it, that's a memory leak, and if you have enough, then the game's memory usage will grow and grow until the game doesn't work any more.  That's a memory management issue AS3 developers need to be aware of.

Another potential problem that can frequently come up in various types of games is creating lots of objects while the game is running.  If the objects are leaking (i.e. not being collected), then the game's memory usage will grow as the player plays the game, leading to slow down and eventual crash (follow tips #1 and #2 to fix that).  If the objects aren't leaking (i.e. the memory is being reclaimed), but if the number or size of objects being created is large enough, then another problem happens: frequent pauses.

When the GC runs, it causes a noticeable pause in the game, as it collects all those properly orphaned objects.  If you watch the memory usage graph in the profiler, it will look like a saw blade.  Memory rises as those objects are created, then it drops back down suddenly every time the GC runs.  The more the GC has to collect when it runs, the longer it takes, and the greater the pause in the game (and the more jagged the saw blade).

If you have saw blade action in your memory usage graph, then you would most likely benefit from object pooling.  That's a fancy term that essentially means reusing your objects so that they never have to be collected.  Then your memory usage graph stays nice and flat, and the GC can basically just go to sleep while your game runs, never interrupting your code or the player's fun.

Once I saw the saw blade thing going on in NB, I tried to retrofit some object pooling into the game.  While I was able to drastically reduce the jaggedness of the graph, I couldn't pool everything (ran into some weird issues with one MovieClip), and so NB retains some saw blade action to this day.  I wrote some classes to facilitate object pooling which I used from the start in Clockwords, so CW doesn't have any saw blade action going on.  Its memory graph is a perfectly horizontal line, even though letters, explosions, hit animations, and bugs are continually being created and destroyed.  Then when the game is over, the memory usage drops back down as all objects in the old game mode are collected properly.

TIP #4: Pool objects that need to be created and destroyed frequently.

Object pooling is really simple.  I posted the class I wrote (along with a few helper classes which it uses, such as my linked list class) here.  There's no license or copyright or anything in the code, and you can do whatever you want to it or with it.

To use the gaObjectPool object, just create one and pass in the Class type you want to pool and a block size, which is just how many instances of Class the pool will instantiate if a caller wants an instance of Class, but there are no more left in the pool.  You can use 1 if you want (you can even make that a default value, since you have the code ;) ).

So once you create a gaObjectPool instance for the type you want to pool, stash the reference to that object wherever you need it.  Then when you want an instance of that type, call getObj, and cast the return value to your type.  Use the object as you please.

When you're done with the object, call the destroy method on it (you have one right?).  I find it useful as a design pattern to give all objects I pool a destroy method which resets all member variables to a known, default state, as if the object were just constructed.  That way when subsequent callers of getObj get the same instance back, they don't have to worry about what might be lingering in the reused object.  After calling destroy on the object you want to reuse, pass it in to the gaObjectPool.returnObj method.  That'll put the object back in its internal linked list so that the instance can be returned by a later call to getObj.

Last word of warning: don't return the same object instance to the pool more than once at a time.  That'll cause the object pool to hand it out more than once to different callers who then both try to use it for different things.

A quick note on gaList: it's a multi-purpose linked list class, which you probably noticed pools its node objects.  Linked lists are preferrable to Arrays when the contents of the list changes frequently.  In practice, most of my collections change frequently, so I use gaList for most of them.  For example, a queue (first in, first out) is a type of list that is perfectly suited to a linked list implementation.  To make a queue with gaList, use "add" to add objects to the end of the queue, then "removeHead" to remove and return the first item in the queue.  If you don't want to remove the head of the queue, then just use "getFirst" to peek at it.


If you're making a decent-sized game, you will probably run into some memory issues during the course of development (or...gasp...after release).  The issues might not be serious enough to refactor your code to retrofit these tips, but you should understand the potential pitfalls of AS3 memory management and know how to prevent them or at least fix them if they end up causing problems for you.