Parade of Rain

Tilt to Live has been unleashed on the unsuspecting public! So far the general vibe has been extremely positive! Adam and I are both pretty excited about getting our first app store game out the door. What’s interesting is the workload went from “lull” to “overdrive” in a matter of days as we ramped up for release and still are trying to coordinate things for a bigger media push in the coming weeks.

You can download Tilt to Live in the app store here. One of the cool things about AGON Online integration is how it’s easy to check leaderboards outside the game. Their community page for Tilt to Live is rather snazzy. We’ll be looking into integrating some of those widgets into our own landing page on onemanleft.com. But that’ll have to wait for now.

We’ve got plenty of ideas in store for Tilt to Live for future updates. So tell your friends, your mom, your dog, your twitter followers! The higher the rating the better! Things are rather hectic at the moment as you can probably imagine, but hopefully I’ll have some breathing room and be able to look back on all of this and write up some (hopefully) useful posts.

So I’m using OpenAL to do the audio in Tilt to Live. Over the course of development audio became the bottleneck of my game, both in size and in performance. The following could help you if you’re experience audio performance issues and are somewhat new to OpenAL. Let me preface this with: Don’t blindly optimize. Measure, Measure, MEASURE! Know what your bottleneck is before trying to tune code!

1. Don’t make more than 32 sources objects in OpenAL. Of course, this number may vary from device to device and what generation the device is. Making any more than the limit the device support makes openAL fail silently, and at this point you’re wasting cycles.
2. Load your audio buffers and create your source objects ahead of time and re-use them. This is a big one. Don’t generate and delete source objects in the middle of your game update. Instead, it’s much faster to just grab a ‘free’ source that is not playing a sound any longer and attach it to another buffer. I was pre-loading my audio-buffers, but creating/deleting sources on the fly in Tilt to Live. Then I started ‘pooling” and I got a decent gain out of re-using source objects.
3. Keep OpenAL calls to a minimum. Especially in tight update loops, don’t repeatedly call openAL functions that don’t change frame-to-frame. I found that a good portion of my time was spent doing something as simple as ‘alSourcei()’ on each source ID per frame was causing a significant slow down.
4. Don’t query openAL state if you don’t have to. In my case, I wrapped openAL Sources in objects with properties to get volume,pitch, etc. Initially those properties simply called the openAL equivalent and returning it instantly. This was hurting my frames due to some some innocent looking “backgroundMusic.volume += someVal” happening each frame along with other audio sources doing the same thing. Save any state you can in instance variables, and as a last resort hit openAL when you need to.
5. As for size, you should consider compressing your sound FX and music to a reasonable size. If you’re like me, you hate giving up quality; especially if you listen to the full quality one and then the compressed one. It can seem like night and day. But in reality, when your users won’t have a full quality audio file to compare it to, they will not notice the difference.

As a sidenote, you can look at my first dev tip for batch compressing audio files.

We're extremely close to submitting our first title "Tilt to Live". We plan on submitting it this upcoming Friday. After a long and grueling (but awesome fun!) development cycle we've learned a ton. I figured an interesting way to distill our new found knowledge it would be in very short "dev tips" for developing an iPhone game. Today I start out with audio:

Compressing Audio

By the end of development our game weighed in at 16 MB. We wanted to go below the 10MB limit so users could download our game over-the-air. Quickest solution? Compress the audio. Now, we were already using some compression but we have over 60 audio files! The main crux of the problem is we want to be able to quickly compress all our audio to a certain format, build it, look at final build size, and test for quality. For this I wrote a quick script:

Now for a little explanation on my setup:

• I have a single folder (let's call it 'finalAudio') with all the original, uncompressed wave files used in our game.
• The script sits inside this 'finalAudio' folder as well and I run 'chmod +x build_audio' where 'build_audio' is the name of my script so I can just click on it in Finder when I'm wanting to re-export my audio.

What the script does:

1. It removes any folder/file named 'caff' in the current directory
2. it makes a new directory inside 'finalAudio' called 'caff'. This is where the compressed audio will go
3. It looks through all wav files inside 'finalAudio' and runs the 'afconvert' utility on each file and puts the resulting compressed file in 'caff'.

I'm not going to go into the details of all the options you have with afconvert and what audio formats the iPhone supports. There's plenty of documentation online covering that already.

Just an interesting sidenote: we took our 13-16MB game and shrunk it to 6.5 MB using ima4@22000 with 1 channel.

So we're at the last mile (it's a long mile..) of development on Tilt To Live. Adam and I are starting to get some pre-release buzz built up around the game as we put our finishing touches into place. With the holidays around the corner, I will be out of commission until mid January. Marketing is a whole other beast we're trying to deal with and learn as we go.  With a market as vicious and saturated as the App Store, it's been one challenge after another!

As many have stated, to succeed in the app store these days requires a a good bit of marketing muscle. The 'gold rush' is pretty much over and the app store is now similar to any overcrowded market. I've been reading up on marketing in general and also seeking out articles on the app store specifically. App Gamer had a very interesting piece for newcomers to the app store regarding PR and marketing.

Have a great holiday (to those that are celebrating) and see you guys in the new year! We'll be starting off that year with a great game and hopefully many more to come!

It's been a while since my last post. As things started entering crunch mode and the whole holiday rush was happening I unfortunately let my blogging schedule slip as a result. It's still going to be irregular for the next month or so as I'm trying to finish up Tilt To Live. The official Tilt To Live site has launched with a few screenies and more info though! It's a basic info site at the moment, but I'll be sprucing it up as time goes on. Starting to ramp up some pre-release hype around the game but it's proving to be challenging :].

A second take on Multi-threaded asset loading

After much reading, experimenting, more reading, and just plain frustration I've toned down the multithreading goodness on asset loading for tilt to live. The main cause for this decision was speed. I seem to have miscalculated the amount of time it takes to load images. It was somewhat of a compound problem. Firstly, loading textures with libpng was on average 4 times slower than the standard UIImage/CGContextDrawImage! I'm not entirely sure if my implementation of libpng was inefficient in some way, but it was taken from several samples and documentations on loading pngs. Secondly, loading textures on a background reduces the speed by another 2 fold!

I've finally implemented texture atlases, so I got a bit of a speed boost on rendering. Although, I now have a 1024X1024 textures. Using libpng on a background thread took ~8 seconds to load a single 1024x1024 png image. I put the image loading on a main thread and it reduced it to 4 seconds on average. A great improvement, but with 2-3 large textures this was still unreasonable. After doing some profiling with the basic GLSprite sample on Apple's iPhone developer site, I decided to revert back to the UIImage/CGContextDrawImage approach and it allowed me to load 1024x1024 textures in a little less than a second. Alright, progress! I've reduced a 20 second load time down to about 4-5.

As a result, I removed all background texture loading from the game. To further reduce the initial launch time I see that I can defer one of the 1024x1024 atlas textures til later as it's only needed when the main game is up. That'll be another second shaved. This is definitely one of those times where a supposed 'good' thing that was implemented has come back to bite me in the ass....

Lessons Painfully Learned

The slickness of having background loading while the game is giving the user some feedback is nice, but ultimately not worth a larger load time. The discrepancy here is that on a 3G iPhone the load time is huge with background threads, but on a 3Gs it's very fluid and barely noticeable. So if I was making a 3Gs-specific app then this is definitely something to look further into and optimize a bit more. But with the goal of trying to reach as many different gen devices as possible, I needed to compromise and give up some of the 'flare' for speed. Right now the target release date (or at least submitting to Apple for approval) is in January 2010.

In the meantime, follow us on Twitter and Facebook to get the latest news on the game!

...looks like I'll be hitting a bit of crunch time in the coming weeks. Blog entries will probably be dwindling over the next month or so as I try to put the finishing touches on Tilt To Live. On that note:

No matter how much energy you feel like you have and no matter how motivated you are at seeing a project to the end, you still need to take breaks from it. Going non-stop at it for days on end is a one way street to burnout and reduced quality.

I hit my limit on Sunday where I felt if I worked on any more of the game it'd have more bugs than features by the end of the session. Was nice to take the day off and just relax to some good music and video games . Weird how many times I've heard the above repeated and yet still fell into the same trap. It kind of sneaked up on me I guess. Oh well...

Almost there!

As I'm zeroing in on finishing Tilt To Live some of the bigger assets are coming through from Adam and the game size has grown a lot faster than it has in past few months. One of the unstated goals of mine was to keep the game under 10 MB. Why? You are not allowed to download apps, podcasts, or videos that are larger than 10 MB over 3G. You are required to connect to wi-fi in order to do it on your iPhone. Alternatively, you can use a PC/Mac and just sync the said data to your phone/iTouch, but who bothers to do that?

Having a 10MB limit on an "unlimited" data plan sucks, but I'm not here to discuss that. This limit possibly poses a constraint on iPhone developers who want their games to be widely available. I just started digging into this, but haven't really found anything conclusive on downloading habits of iPhone users beyond a few small-sample surveys and articles.

In either case, the question I wanted to answer was:

Do you stand to lose a significant amount of potential sales/downloads for apps over 10 MB?

I decided to do some unscientific data gathering and some manual data mining with good 'ole excel :].

The Process

I fired up iTunes and after fumbling about in the iTunes app store I was able to get to the Games top 100 free and paid section. This was actually my first time digging around in the itunes desktop app store, as I download and buy 100% of my apps and podcasts directly from my phone. Connecting my iPhone to my computer is a rare event indeed...

Not having released a game yet on the app store I didn't have any data on hand that'd prove useful for this. I took a rather low-tech approach and just consulted the all mighty top 100 lists of the app store. I took the top 100 games in the paid list and the free list and compiled them separately to try to find two things:

• How many 10MB+ games exist in the top 100?
• How are those 10MB+ games distributed in the top 100?
  

So the caveats to using the top 100:

• From my understanding, I'm only seeing the US top 100?
• It's not representative of the HUGE library of other apps that are still somewhat successful but not the over-night hit sensation that a lot of the upper rung apps are
• I don't have a good idea of the size distribution of games that aren't in the top 100, so I may be missing the big picture. So any experiments seeking to find a distribution has a very limited view "window" of what is actually going on.
• This data includes iTouch devices, but since they lack 3G this could possibly skew the data.

The Results

First, I'll explain what I thought the results would be. Before doing any of the number crunching I hypothesized I would find that the most successful apps are under 10 MB and any app that happens to be over 10 MB would be weighted towards the lower rungs of the list.

Let's first take a look at the free games in the app store. Below is a table showing the number of apps above 10 MB split into top N lists.

We find in total that there are only 18 out of the top 100 free games that go over the 10 MB limit. Not exactly conclusive, but it shows that majority of top 100 free games are available over the air (OTA).  As you decrease the size of the list the number of 10MB+ games starts to dwindle naturally, but when you look at the percentages the amount of those 18 stays relatively the same until you hit the coveted top 10. The only 10MB+ free game in the top ten is Real Racing GTI, a promotional game for Volkswagen that weighs in at a rather hefty 59.3 MB!

Now how does the distribution of these "heavy" games look in the top 100 free list?

Hmmm...The distribution of 10MB+ games is pretty even all the way up to the top, with a slight dip in the mid section.

Let's look at paid games.

One thing I didn't really account for in my original hypothesis is a difference between paid and free games. 53 paid games are over 10MB! While by a slight margin, the majority of paid games seem to ignore the 10MB rule and go for broke! Egad! This above table completely refutes my original hypothesis that most successful apps are under 10MB (Wow those last few sentences sounded super nerdy...). This is assuming you equate that success = number of apps sold * price of app = more $$$, where as in the free model this kind of breaks down. Anyway, moving on to the distribution of paid game juggernauts...

Interesting. While the paid app list has a majority of 10MB+ games the bulk of them live in the top 20-40 section, again, only by a slight margin. The stand out number here is the top 20 section. It appears games that are under 10MB here still win out the majority, so maybe there still is some relevance to my original hypothesis but to a much lesser extent than I imagined.

Probably not as useful, but in case you're curious here's a simple area graph plotting top 100 free and paid game sizes, there are some monster sized games in the top 100 so that is encouraging! Click it for full resolution.

The Truth Revealed?

Looking at the free games market the results decidedly point to a more "logical" trend, in my opinion, on how games are being designed. Yet, when you look at the paid games market you find that:

• Developers don't give a crap and make what suits them (Hurray freedom!)
• Consumers don't seem to give a crap either on how big it is when they are paying for a game. Maybe they even irrationally prefer bigger (in physical size, not content) games for their dollar?

While the results point to opposite trends on app store game sizes, it doesn't really uncover if developers are consciously shooting for sub 10MB games or simply letting the chips fall where they may in either case. Could free games just not warrant enough budget to make enough assets that grow bigger than 10MB? Are free games specifically targeting "viral" game status by shooting for sub-10MB downloads to allow it to spread more easily? I also find that a good bit of the free games are also "Lite" versions (62 of the top 100 in fact). So it shows that people like to try before they buy at least. With Apple's announcement that they'll allow free apps with in-app purchases, hopefully things will clean up a bit. But that's a different issue for another day.

If you're interested in getting my scrawny little excel spreadsheet with the free/paid top 100 lists to run any fancy shmancy analytics you're welcome to it.

I've made some serious progress as of late with Tilt To Live. The age old saying "I'm 90% done..." is rearing its head and  yet it feels like there's 90% more to go! This is a bit regurgitated, but saw an interesting bit in a post on burnout and motivation at Zen Habits:

1. Achieve in increments. When you only focus on a big goal someday, it’s easy to get burned out by the daily grind. It’s like driving toward a mountain in the distance. You can drive for hours, but the mountain doesn’t seem to get any closer. And spinning your wheels gets real tiring real fast.

The solution is to give yourself a way to measure and record every little step forward you take. Here’s how:

• Get a journal, notebook, or calendar. Writing things down is important.

While I've been doing the above for a while now, I'm finding the advice becoming more critical as I polish up the game and there are only smaller, less visible changes  to the gameplay or code. Along with all the milestone tracking, to-do lists, and time logs I tend to write down where I stand on the project at a high level almost on a daily basis. It gives me a good "whole picture" look at the game to see what else is coming up on the horizon or if I'm finding myself focusing too much on any single feature. Anyway, back to work... :]

I finally got rid of the stupid...

"ld: warning in <foo>.so, file is not of required architecture."

...warning when using iSimulate. If you're not using iSimulate or a similar technology you're losing valuable time even if you don't use the accelerometer or GPS functions. Then again, I might not be the best spokesperson on time since I spent the weekend re-inventing the wheel.

I have this need to make sure my projects compile w/o warnings. I let it sit for a while because I'm not an XCode expert so I knew I wanted to conditionally link it in but I couldn't find the UI for it. The process in which they suggest you integrate iSimulate into your XCode project truly is easy, but I felt icky when it came up with warnings when you compile against the actual device.  There are better ways of doing that. Namely, conditional build settings. Of course, it requires a few more steps and a bit more knowledge of the XCode environment so I suspect that wouldn't help their marketing. Regardless, having the iSimulate linker flags and library search paths only be added when you're compiling for a simulator is relatively easy to setup.

It's getting around that time when I'm spending more time working on the game as I realize how many little things need to be fixed, added, tweaked. I've recently given up most of my weekends now in an effort to get more hours in. I'm currently working 6 out of the 7 days a week on the game in some way. It's mostly game programming, sometimes graphics, web development, emailing, phone calls, or getting business forms and other legal stuff squared away.

This week was a rather productive week on several fronts:

• Got a lot of new art assets from Adam and implemented them
• Feedback from testers started trickling in and the reaction has been positive along with tons of great suggestions/ideas
• Fixed a lot of old bugs that were sitting in the queue
• Improved some memory usage of textures

The week wasn't without it's headaches though. This completion date graph illustrates just how badly I estimated this week's tasks:

The amount of tasks was unusually large for the past week, but the amount of tasks wasn't as big an issue as some tasks taking 10 times longer than I wanted.

The biggest one was getting multi-threaded texture loading on the iPhone working. The hours I logged into getting it working is my own personal testament to how a seemingly simple task can blow up to ridiculous portions when you mention the word "multi-threading". Had I known it'd take this long I would've opted for a more basic solution and leave it to a side project. The bright side is now I can load audio and graphics in parallel along with the main thread being able to continue any animations and such. It took about 20 hours to get it working and debugged, which is somewhat embarrassing haha!

Hair-pulling Threading Issues

It started with me wanting to off load the texture loading to another thread. Simple enough. I've done this before on other projects. It took a couple of hours to learn how to get OpenGL setup correctly to allow this. Then I discovered OpenGL wasn't being the meanie, but a function call from core graphics wasn't allowing me to call it from a background thread (sometimes...ugh). I researched it for a couple of days while finishing up other things, but wasn't able to find anything conclusive. I then decided to take the leap and implement my own PNG loader using libpng as a base.  I mean really...how hard could it be? Hah...hah...ha...

Saying it was frustrating would be the understatement of the year, and last year. I usually try to avoid copy/pasting code as I like to read and understand the API I'm using so when it DOES break I'm not utterly clueless. Libpng's interface is rather low-level and not exactly the cleanest. After reading documentation on it for what seemed forever I decided to start with an example program and modify it to fit my needs.

Let's not forget that I'm an XCode/Unix newbie so getting libpng/zlib working as a static library in XCode was a nightmare for me. I ended up building from the source against the correct config's and SDK's for the iPhone to avoid the "invalid architecture" warnings. Next came actually implementing the loading. It looked pretty straight forward in all the examples I saw, except I wanted one feature I didn't see in any of the sample code: be able to load non-power-of-2 textures onto the iPhone. All of the game's assets are of various shapes and sizes, and several are non-power of 2. I haven't taken the time to do proper texture atlasing and I wanted the ability to quickly drop any assets Adam sent me into the game's folder and load it without fiddling with size alignments and the other mess.

This is where things started to get hairy because I was seeing issues crop up that weren't necessarily from my code so I was lead astray a lot. The caveat is the iPhone has a rather special way of "compressing" PNGs. It took me a while to understand the implications of this because I could load the PNGs fine but holy shit were they rendering wrong in OpenGL! It finally dawned on me to change the blend function OpenGL was using. Much to my surprise I couldn't find one that would give me the equivalent alpha blending that I got with premultiplied alpha's. And I wasn't about to go back and re-design all the assets. By this time I was pretty comfortable working with libpng and I decided to just write a custom transform function to premultiply the alpha when loading it to mimic the previous technique of loading:

Finally, you can write your own transformation function if none of the existing ones meets your needs. This is done by setting a callback with

png_set_read_user_transform_fn(png_ptr,
       read_transform_fn);

You must supply the function

void read_transform_fn(png_ptr ptr, row_info_ptr
       row_info, png_bytep data)

See pngtest.c for a working example. Your function will be called after all of the other transformations have been processed.

Alrighty, so now I can load things ~20 hours later. Talk about a huge step backwards. Doing the pre-multiplication at runtime is a trade off of convenience over speed. When it comes time to release the game I might consider doing this prior to compiling so the assets are "cooked". But doing it at runtime adds about 1-1.5 seconds from my unscientific tests.

What's amusing was once this was working it took 15 minutes to get multi-threading working. Now I decided to be adventurous and load the audio and graphics on separate threads from the main thread, for a total of 3. Speed tests show it's on average no faster or slower than having both asset types be loaded in order on a single thread *Shrug*. Oh well, if the iPhone ever gets another core then I'll be set . Ultimately, I now have the freedom to toss asset loading on a background thread while the game continues playing with minimal hit to the frames per second.

This  kind of thing is probably a bit advanced and overkill to do for such a simple and small game as Tilt To Live. If I have time (unlikely) over the next several weeks I'll try to put together a complete code listing for those looking to have more control of their asset loading with libpng on the iPhone.