So we’re at the last mile (it’s a long mile..) of development on Tilt To Live. Adam and I are starting to get some pre-release buzz built up around the game as we put our finishing touches into place. With the holidays around the corner, I will be out of commission until mid January. Marketing is a whole other beast we’re trying to deal with and learn as we go.  With a market as vicious and saturated as the App Store, it’s been one challenge after another!

As many have stated, to succeed in the app store these days requires a a good bit of marketing muscle. The ‘gold rush’ is pretty much over and the app store is now similar to any overcrowded market. I’ve been reading up on marketing in general and also seeking out articles on the app store specifically. App Gamer had a very interesting piece for newcomers to the app store regarding PR and marketing.

Have a great holiday (to those that are celebrating) and see you guys in the new year! We’ll be starting off that year with a great game and hopefully many more to come!

Tilt To LiveIt’s been a while since my last post. As things started entering crunch mode and the whole holiday rush was happening I unfortunately let my blogging schedule slip as a result. It’s still going to be irregular for the next month or so as I’m trying to finish up Tilt To Live. The official Tilt To Live site has launched with a few screenies and more info though! It’s a basic info site at the moment, but I’ll be sprucing it up as time goes on. Starting to ramp up some pre-release hype around the game but it’s proving to be challenging :].

A second take on Multi-threaded asset loading

FrustrationAfter much reading, experimenting, more reading, and just plain frustration I’ve toned down the multithreading goodness on asset loading for tilt to live. The main cause for this decision was speed. I seem to have miscalculated the amount of time it takes to load images. It was somewhat of a compound problem. Firstly, loading textures with libpng was on average 4 times slower than the standard UIImage/CGContextDrawImage! I’m not entirely sure if my implementation of libpng was inefficient in some way, but it was taken from several samples and documentations on loading pngs. Secondly, loading textures on a background reduces the speed by another 2 fold!

I’ve finally implemented texture atlases, so I got a bit of a speed boost on rendering. Although, I now have a 1024X1024 textures. Using libpng on a background thread took ~8 seconds to load a single 1024×1024 png image. I put the image loading on a main thread and it reduced it to 4 seconds on average. A great improvement, but with 2-3 large textures this was still unreasonable. After doing some profiling with the basic GLSprite sample on Apple’s iPhone developer site, I decided to revert back to the UIImage/CGContextDrawImage approach and it allowed me to load 1024×1024 textures in a little less than a second. Alright, progress! I’ve reduced a 20 second load time down to about 4-5.

As a result, I removed all background texture loading from the game. To further reduce the initial launch time I see that I can defer one of the 1024×1024 atlas textures til later as it’s only needed when the main game is up. That’ll be another second shaved. This is definitely one of those times where a supposed ‘good’ thing that was implemented has come back to bite me in the ass….

unlock_lightning

Lessons Painfully Learned

The slickness of having background loading while the game is giving the user some feedback is nice, but ultimately not worth a larger load time. The discrepancy here is that on a 3G iPhone the load time is huge with background threads, but on a 3Gs it’s very fluid and barely noticeable. So if I was making a 3Gs-specific app then this is definitely something to look further into and optimize a bit more. But with the goal of trying to reach as many different gen devices as possible, I needed to compromise and give up some of the ‘flare’ for speed. Right now the target release date (or at least submitting to Apple for approval) is in January 2010.

In the meantime, follow us on Twitter and Facebook to get the latest news on the game!

…looks like I’ll be hitting a bit of crunch time in the coming weeks. Blog entries will probably be dwindling over the next month or so as I try to put the finishing touches on Tilt To Live. On that note:

No matter how much energy you feel like you have and no matter how motivated you are at seeing a project to the end, you still need to take breaks from it. Going non-stop at it for days on end is a one way street to burnout and reduced quality.

I hit my limit on Sunday where I felt if I worked on any more of the game it’d have more bugs than features by the end of the session. Was nice to take the day off and just relax to some good music and video games :). Weird how many times I’ve heard the above repeated and yet still fell into the same trap. It kind of sneaked up on me I guess. Oh well…

Tilt To Live

Almost there!

App Store Games

As I’m zeroing in on finishing Tilt To Live some of the bigger assets are coming through from Adam and the game size has grown a lot faster than it has in past few months. One of the unstated goals of mine was to keep the game under 10 MB. Why? You are not allowed to download apps, podcasts, or videos that are larger than 10 MB over 3G. You are required to connect to wi-fi in order to do it on your iPhone. Alternatively, you can use a PC/Mac and just sync the said data to your phone/iTouch, but who bothers to do that?

Having a 10MB limit on an “unlimited” data plan sucks, but I’m not here to discuss that. This limit possibly poses a constraint on iPhone developers who want their games to be widely available. I just started digging into this, but haven’t really found anything conclusive on downloading habits of iPhone users beyond a few small-sample surveys and articles.

In either case, the question I wanted to answer was:

Do you stand to lose a significant amount of potential sales/downloads for apps over 10 MB?

I decided to do some unscientific data gathering and some manual data mining with good ‘ole excel :].

The Process

I fired up iTunes and after fumbling about in the iTunes app store I was able to get to the Games top 100 free and paid section. This was actually my first time digging around in the itunes desktop app store, as I download and buy 100% of my apps and podcasts directly from my phone. Connecting my iPhone to my computer is a rare event indeed…

Not having released a game yet on the app store I didn’t have any data on hand that’d prove useful for this. I took a rather low-tech approach and just consulted the all mighty top 100 lists of the app store. I took the top 100 games in the paid list and the free list and compiled them separately to try to find two things:

  • How many 10MB+ games exist in the top 100?
  • How are those 10MB+ games distributed in the top 100?

So the caveats to using the top 100:

  • From my understanding, I’m only seeing the US top 100?
  • It’s not representative of the HUGE library of other apps that are still somewhat successful but not the over-night hit sensation that a lot of the upper rung apps are
  • I don’t have a good idea of the size distribution of games that aren’t in the top 100, so I may be missing the big picture. So any experiments seeking to find a distribution has a very limited view “window” of what is actually going on.
  • This data includes iTouch devices, but since they lack 3G this could possibly skew the data.
The Results

First, I’ll explain what I thought the results would be. Before doing any of the number crunching I hypothesized I would find that the most successful apps are under 10 MB and any app that happens to be over 10 MB would be weighted towards the lower rungs of the list.

Let’s first take a look at the free games in the app store. Below is a table showing the number of apps above 10 MB split into top N lists.

Top N Free Games

[table id=2 /]

We find in total that there are only 18 out of the top 100 free games that go over the 10 MB limit. Not exactly conclusive, but it shows that majority of top 100 free games are available over the air (OTA).  As you decrease the size of the list the number of 10MB+ games starts to dwindle naturally, but when you look at the percentages the amount of those 18 stays relatively the same until you hit the coveted top 10. The only 10MB+ free game in the top ten is Real Racing GTI, a promotional game for Volkswagen that weighs in at a rather hefty 59.3 MB!

Now how does the distribution of these “heavy” games look in the top 100 free list?

Distribution of 10MB+ Free Games

[table id=3 /]

Hmmm…The distribution of 10MB+ games is pretty even all the way up to the top, with a slight dip in the mid section.

Let’s look at paid games.

Top N Paid Games

[table id=4 /]

One thing I didn’t really account for in my original hypothesis is a difference between paid and free games. 53 paid games are over 10MB! While by a slight margin, the majority of paid games seem to ignore the 10MB rule and go for broke! Egad! This above table completely refutes my original hypothesis that most successful apps are under 10MB (Wow those last few sentences sounded super nerdy…). This is assuming you equate that success = number of apps sold * price of app = more $$$, where as in the free model this kind of breaks down. Anyway, moving on to the distribution of paid game juggernauts…

Distribution of 10MB+ Paid Games

[table id=5 /]

Interesting. While the paid app list has a majority of 10MB+ games the bulk of them live in the top 20-40 section, again, only by a slight margin. The stand out number here is the top 20 section. It appears games that are under 10MB here still win out the majority, so maybe there still is some relevance to my original hypothesis but to a much lesser extent than I imagined.

Probably not as useful, but in case you’re curious here’s a simple area graph plotting top 100 free and paid game sizes, there are some monster sized games in the top 100 so that is encouraging! Click it for full resolution.

App Store Game Sizes

The Truth Revealed?

Looking at the free games market the results decidedly point to a more “logical” trend, in my opinion, on how games are being designed. Yet, when you look at the paid games market you find that:

  • Developers don’t give a crap and make what suits them (Hurray freedom!)
  • Consumers don’t seem to give a crap either on how big it is when they are paying for a game. Maybe they even irrationally prefer bigger (in physical size, not content) games for their dollar?

While the results point to opposite trends on app store game sizes, it doesn’t really uncover if developers are consciously shooting for sub 10MB games or simply letting the chips fall where they may in either case. Could free games just not warrant enough budget to make enough assets that grow bigger than 10MB? Are free games specifically targeting “viral” game status by shooting for sub-10MB downloads to allow it to spread more easily? I also find that a good bit of the free games are also “Lite” versions (62 of the top 100 in fact). So it shows that people like to try before they buy at least. With Apple’s announcement that they’ll allow free apps with in-app purchases, hopefully things will clean up a bit. But that’s a different issue for another day.

If you’re interested in getting my scrawny little excel spreadsheet with the free/paid top 100 lists to run any fancy shmancy analytics you’re welcome to it.

significantly

I’ve made some serious progress as of late with Tilt To Live. The age old saying “I’m 90% done…” is rearing its head and  yet it feels like there’s 90% more to go! This is a bit regurgitated, but saw an interesting bit in a post on burnout and motivation at Zen Habits:

1. Achieve in increments. When you only focus on a big goal someday, it’s easy to get burned out by the daily grind. It’s like driving toward a mountain in the distance. You can drive for hours, but the mountain doesn’t seem to get any closer. And spinning your wheels gets real tiring real fast.

The solution is to give yourself a way to measure and record every little step forward you take. Here’s how:

  • Get a journal, notebook, or calendar. Writing things down is important.

While I’ve been doing the above for a while now, I’m finding the advice becoming more critical as I polish up the game and there are only smaller, less visible changes  to the gameplay or code. Along with all the milestone tracking, to-do lists, and time logs I tend to write down where I stand on the project at a high level almost on a daily basis. It gives me a good “whole picture” look at the game to see what else is coming up on the horizon or if I’m finding myself focusing too much on any single feature. Anyway, back to work… :]

Tilt To Live

I finally got rid of the stupid…

"ld: warning in <foo>.so, file is not of required architecture."

…warning when using iSimulate. If you’re not using iSimulate or a similar technology you’re losing valuable time even if you don’t use the accelerometer or GPS functions. Then again, I might not be the best spokesperson on time since I spent the weekend re-inventing the wheel.

isimulate

I have this need to make sure my projects compile w/o warnings. I let it sit for a while because I’m not an XCode expert so I knew I wanted to conditionally link it in but I couldn’t find the UI for it. The process in which they suggest you integrate iSimulate into your XCode project truly is easy, but I felt icky when it came up with warnings when you compile against the actual device.  There are better ways of doing that. Namely, conditional build settings. Of course, it requires a few more steps and a bit more knowledge of the XCode environment so I suspect that wouldn’t help their marketing. Regardless, having the iSimulate linker flags and library search paths only be added when you’re compiling for a simulator is relatively easy to setup.

It’s getting around that time when I’m spending more time working on the game as I realize how many little things need to be fixed, added, tweaked. I’ve recently given up most of my weekends now in an effort to get more hours in. I’m currently working 6 out of the 7 days a week on the game in some way. It’s mostly game programming, sometimes graphics, web development, emailing, phone calls, or getting business forms and other legal stuff squared away.

This week was a rather productive week on several fronts:

  • Got a lot of new art assets from Adam and implemented them
  • Feedback from testers started trickling in and the reaction has been positive along with tons of great suggestions/ideas
  • Fixed a lot of old bugs that were sitting in the queue
  • Improved some memory usage of textures

The week wasn’t without it’s headaches though. This completion date graph illustrates just how badly I estimated this week’s tasks:

Bad Estimates Ahoy!

The amount of tasks was unusually large for the past week, but the amount of tasks wasn’t as big an issue as some tasks taking 10 times longer than I wanted.

The biggest one was getting multi-threaded texture loading on the iPhone working. The hours I logged into getting it working is my own personal testament to how a seemingly simple task can blow up to ridiculous portions when you mention the word “multi-threading”. Had I known it’d take this long I would’ve opted for a more basic solution and leave it to a side project. The bright side is now I can load audio and graphics in parallel along with the main thread being able to continue any animations and such. It took about 20 hours to get it working and debugged, which is somewhat embarrassing haha!

Hair-pulling Threading Issues

It started with me wanting to off load the texture loading to another thread. Simple enough. I’ve done this before on other projects. It took a couple of hours to learn how to get OpenGL setup correctly to allow this. Then I discovered OpenGL wasn’t being the meanie, but a function call from core graphics wasn’t allowing me to call it from a background thread (sometimes…ugh). I researched it for a couple of days while finishing up other things, but wasn’t able to find anything conclusive. I then decided to take the leap and implement my own PNG loader using libpng as a base.  I mean really…how hard could it be? Hah…hah…ha…

Saying it was frustrating would be the understatement of the year, and last year. I usually try to avoid copy/pasting code as I like to read and understand the API I’m using so when it DOES break I’m not utterly clueless. Libpng’s interface is rather low-level and not exactly the cleanest. After reading documentation on it for what seemed forever I decided to start with an example program and modify it to fit my needs.

Let’s not forget that I’m an XCode/Unix newbie so getting libpng/zlib working as a static library in XCode was a nightmare for me. I ended up building from the source against the correct config’s and SDK’s for the iPhone to avoid the “invalid architecture” warnings. Next came actually implementing the loading. It looked pretty straight forward in all the examples I saw, except I wanted one feature I didn’t see in any of the sample code: be able to load non-power-of-2 textures onto the iPhone. All of the game’s assets are of various shapes and sizes, and several are non-power of 2. I haven’t taken the time to do proper texture atlasing and I wanted the ability to quickly drop any assets Adam sent me into the game’s folder and load it without fiddling with size alignments and the other mess.

This is where things started to get hairy because I was seeing issues crop up that weren’t necessarily from my code so I was lead astray a lot. The caveat is the iPhone has a rather special way of “compressing” PNGs. It took me a while to understand the implications of this because I could load the PNGs fine but holy shit were they rendering wrong in OpenGL! It finally dawned on me to change the blend function OpenGL was using. Much to my surprise I couldn’t find one that would give me the equivalent alpha blending that I got with premultiplied alpha’s. And I wasn’t about to go back and re-design all the assets. By this time I was pretty comfortable working with libpng and I decided to just write a custom transform function to premultiply the alpha when loading it to mimic the previous technique of loading:

Finally, you can write your own transformation function if none of the existing ones meets your needs. This is done by setting a callback with

png_set_read_user_transform_fn(png_ptr,
       read_transform_fn);

You must supply the function

void read_transform_fn(png_ptr ptr, row_info_ptr
       row_info, png_bytep data)

See pngtest.c for a working example. Your function will be called after all of the other transformations have been processed.

Alrighty, so now I can load things ~20 hours later. Talk about a huge step backwards. Doing the pre-multiplication at runtime is a trade off of convenience over speed. When it comes time to release the game I might consider doing this prior to compiling so the assets are “cooked”. But doing it at runtime adds about 1-1.5 seconds from my unscientific tests.

What’s amusing was once this was working it took 15 minutes to get multi-threading working. Now I decided to be adventurous and load the audio and graphics on separate threads from the main thread, for a total of 3. Speed tests show it’s on average no faster or slower than having both asset types be loaded in order on a single thread *Shrug*. Oh well, if the iPhone ever gets another core then I’ll be set ;). Ultimately, I now have the freedom to toss asset loading on a background thread while the game continues playing with minimal hit to the frames per second.

This  kind of thing is probably a bit advanced and overkill to do for such a simple and small game as Tilt To Live. If I have time (unlikely) over the next several weeks I’ll try to put together a complete code listing for those looking to have more control of their asset loading with libpng on the iPhone.

canabalt-378x253

Every now and then I come across a game that is rather inspiring. Strangely, a lot of them come from the flash community. I really like it when someone can take a game mechanic that is so utterly constrained and create something so enjoyable out of it. The epitome of these design challenges might be the ‘One-Button’ games. I just recently came across Canabalt, and it had me playing for a while despite the need for sleep.

The sense of urgency and the whole atmosphere implies a bigger and more ominous ‘universe’ than what the gameplay focuses on, and is a treat. I’ve died several times trying to figure out the fleeting images in the background only to slam into a wall, haha!  I’ll be checking it out on my iPhone later this week. Hats off to them for getting it onto the App Store and good luck!

I’ve had a rather productive weekend! Implemented a handful of new art assets from Adam, finished some more gameplay code, optimized a bit, and even released a new build to testers tonight. It wasn’t a smooth process, but when is it ever? I was having an issue with loading the assets of the game at app launch and it caused the game to crash if it ever took more than 20 seconds to load the assets.

Admittedly, the asset loading code could use some optimization. Namely, I need to start using texture atlases for the art assets to minimize the state changes in OpenGLES. The reason I haven’t done this yet is art assets are still in flux and changing almost daily. I haven’t found any decent tools to pack textures into single images yet (haven’t looked too hard yet) and I haven’t taken the time to re-write the resource management in the game to be aware of multiple resources in the same texture.

So I’m putting it off for now and in the meantime wanted to redo the overall asset loading the ‘right way’ with a proper load screen and what not. I was faced with:

  • The Easy Way: Just draw up a loading screen and then load the assets right after drawing. The game essentially is blocked and there’s no user feedback that the game is running until the assets are all loaded
  • The Difficult Way: Whip out some multi-threading API’s and have the assets load on another thread and have some animation play on the main thread.

Naturally, I chose the ‘difficult way’. After doing some reading up on multi-threading assets on the iPhone I figured it wouldn’t be too difficult to get running and I could implement it for tonight’s build. Wow was I wrong. I took the 20 minutes to implement the correct EAGLShareGroups and EAGLContexts and the supporting code to play the load screen and have the background thread load the assets.

loader

It all worked beautifully and having the circles spin while the “loading” words doing some idle animation worked surprisingly well with little stuttering. I tested it on the Simulator and it worked fine, but to get accurate time results I began testing on the device itself. With everything in order I kept testing the loading screen and nothing was amiss. I was minutes away from calling it done and about to start making final test builds when I tried it one more time unhooked from my mac mini… I was looking at the circles spin when suddenly they locked up. The screen turned black, and the next second I was staring at my home screen.

After debugging the build I was still at a loss as to what was going on. Each time the application crashed it was an EXC_BAD_ACCESS error in the CGContextDrawImage function. I’m using this function to load a texture into OpenGL and it’s worked just fine in a single-threaded environment. Unfortunately, I haven’t found any definitive info on whether it’s possible to call this function from a 2nd thread or not. This code listing has a very similar process inside the ‘textureWithName’ method (single threaded) in case you wish to see how it works.

Unless I come across something that shows how to use CGContextDrawImage correctly in a multi-threaded application I’m probably going to go the route of libpng. The portions of code I added have worked in regards to openGL so I believe I have that part solved, it’s just the data conversion from a file to raw bytes before openGL gets it’s hands on it is the problem.

Some Data You May Find Useful

My decision to use multiple threads for asset loading wasn’t based on any conceptions of a speed boost. If it loaded faster due to multi-threading that’d be great but it was secondary. I was doing it from a polish point of view and to make the whole experience on the user more seamless. So here’s a list of some data I’m working with:

  • 68 png images totaling 609kb. There is huge performance gain potential here when I implement texture atlases.
  • 22 audio (caf) files totaling 10.9Mb. Audio is certainly the majority of the game’s data, and I hope to reduce this by tweaking the compression on these files later.

Some load time data, don’t have the log files off hand at the moment so these are approximates:

[table id=1 /]

So the single threaded time is pretty straightforward. In the 2nd column it was my attempt to do both audio+graphics loading in a single background thread. After realizing I wouldn’t be able to fix this in time for the build I opted for a different multi-thread approach, which is shown in the 3rd column.

I load all the graphics assets on the main thread but just before I do that I spin off a background thread to load the audio assets. The result? Pretty much no performance gain! The graphics load in pretty much the same speed but audio loading takes twice as long while doing it at the same time as graphics loading. The net time is still around 7-8 seconds, which is no different than a single-threaded approach.

I’m sure I’m probably doing something dumb in the loading of both assets since both have to hit the file system at the same time and there may be some thread contention there, causing it to lock the audio thread. I’ll need to look into it further because if I could do both simultaneously with minimal blocking then this approach would be the way to go.

The visual result? The load screen animation freezes for 3-4 seconds at the beginning while loading the graphics, but once only the audio is left it starts to animate again. Not ideal, but it doesn’t crash. I’ll be working on getting it all on the background thread in the coming weeks.