But every once in a while even the most polished and refined software exhibits that nastiest of malfunctions: the crashing bug. In the middle of a possibly important user task, the application comes screeching to a halt and disappears, leaving the user with nothing more than a cryptic message from the operating system, along with an offer to send the crash log off to Apple.

Needless to say, crashes suck. A lot. For users, especially. If there’s one behavior of your application that you should focus on eliminating, it’s the behavior of crashing. Above all other “nuisances,” this is the one that is absolutely unacceptable.

Which is why crash logs are so important. Occasionally, when a user is sophisticated enough to understand the value of the crash log and, instead of choosing to send it to Apple, copies and pastes it into an email to you, you should be overjoyed! Finally! A crash has been captured in the wild and delivered to you for easy inspection. This artifact not only proves that a crashing bug exists, but usually includes highly detailed information about where and how the crash has happened.

Which is why it boggles the mind to think that some developers see these crash logs as relatively useless. In my own experience of reporting bugs to other developers, I have found frequently that my crash log is considered more as a statistic than as a specimen. While I expect a response like “great, I see what the bug is, and I’ll get this fixed,” I often get responses more like “sigh, I’ve seen this a few times but have no idea what’s going on!”

Sometimes I find that the developer in question simply needs to be educated about the tools at his or her disposal. I imagine the hours wasted speculating about what the crash log means, and can’t help but think if they knew how to analyze it, they could have the bug solved (in many cases) in moments. Oh – and I’m sounding high and mighty right now, so before I get too carried away, let me just say that I too need to be educated on a great many things, but making sense of crash logs is not particularly high on that list. I leave it to other bloggers to fill in the gaps of my education.

Crash Logs: The Crappy Way

So what is this big thing that developers are missing? Well, a crash log essentially shows the “path” that the little hamsters in your computer were running on when they fell off the cliff. It paints a picture of where the program got misguided and went off doing something it never should have done. Problem is, the picture is a little blurry. Below is an actual crash log submitted by an actual MarsEdit user (abbreviated and condensed to fit the blog):

Thread 0 Crashed:
Foundation 0x93cc2a2e _NSAPAddPage + 94
Foundation 0x93cc3b27 __NSAutoreleaseObject + 263
Foundation 0x93cc3962 -[NSObject(NSObject) autorelease] + 18
Foundation 0x93cceaca +[NSString stringWithFormat:] + 106
marsedit   0x000637fb 0x1000 + 403451
marsedit   0x00053883 0x1000 + 338051
marsedit   0x000539ea 0x1000 + 338410
marsedit   0x00053d7b 0x1000 + 339323
marsedit   0x00053e45 0x1000 + 339525
marsedit   0x0005238b 0x1000 + 332683

For any non-programmers who are still reading (bravi!), you may have a little trouble understanding what the heck this means. This is an ordered description of the path the computer followed while trying to do something. Problem is it ended up crashing. The path is shown in reverse-order, so the crash is at the top of the list, inside Foundation (an Apple system library). Trying to make human sense of this goes something like this:

1. Something in MarsEdit at address 0x0005238b…
2. … called something in MarsEdit at address 0x00053e45
3. … called something in MarsEdit at address 0x00053d7b
4. … called something in MarsEdit at address 0x000539ea
5. … called something in MarsEdit at address 0×00053883
6. … called something in MarsEdit at address 0x000637fb
7. … which called a standard string library method “stringWithFormat”
8. which eventually crashed in the system.

You can sort of see how some developers are forced to throw up their arms in despair, right? But what if you could turn this muddled picture into something a bit sharper? The fact is, you can. The reason the picture is so blurry is because MarsEdit shipped without debugging symbols. The reasons for this are mostly to keep the shipping application as small as possible, but there are also some good reasons to hide the symbols if they reveal too much about your business strategy.

Crash Logs: The Happy Way

Many developers seem to think that gathering useful crash logs requires shipping a symbol-laden application, but aren’t willing to do so. The fact is, you can have the best of both worlds by shipping a symbol-stripped version of your application, but keeping a symbol-laden version on-hand for the developer’s convenience.

What am I getting at? Apple offers a tool called “atos”, which makes it relatively easy to map an address from a symbol-stripped application to a name in a symbol-laden version of that same application. As long as the two applications were built with the same sources and using the same build options, the addresses are bound to match up. It’s just the symbols that are “stripped” from the shipping application.

Let’s take a look at what happens when I apply my atos powers to the crash log I pasted above. From the Terminal, I invoke atos with the literal memory addresses of all the mysterious points in the “path” from the crash log. You can list the addresses in whatever order you want, but I’m listing them so that the result will be in order from top to bottom, the same way I reasoned out the crash in plain “crappy” English above:

iBook> cd MarsEdit.app/Contents/MacOS
iBook> atos -arch i386 -o MarsEdit \
	0x0005238b 0x00053e45 0x00053d7b 0x000539ea 0x00053883 0x000637fb

Note also that I specified the architecture explicitly. This is because I am debugging this on a PowerPC Mac, but the crash log I received showed that the customer is running on an Intel machine. This yields a magical response with, behold, real meaningful symbols!

-[RSXMLRPCCall invoke:] (in MarsEdit) (RSXMLRPCCall.m:397)
-[RSXMLRPCRequest requestText] (in MarsEdit) (RSXMLRPCRequest.m:239)
-[RSXMLRPCRequest serializeParams:] (in MarsEdit) (RSXMLRPCRequest.m:216)
-[RSXMLRPCRequest serializeDictionary:toString:] (in MarsEdit) (RSXMLRPCRequest.m:153)
-[RSXMLRPCRequest serializeData:toString:] (in MarsEdit) (RSXMLRPCRequest.m:123)
-[NSData(extras) base64StringWithLineLength:] (in MarsEdit) (NSData+extras.m:215)

Now I can apply my human sense again, but with the benefit of symbolic names instead of raw memory addresses:

1. MarsEdit invoked an RPC network request
2. … which while building the actual request cargo text
3. … was trying to serialize the parameters
4. … when it tried to serialize a dictionary object
5. … whose data
6. … upon being converted to a Base64 representation
7. … caused a “stringWithFormat:” call that went blammo!

Not only does the plot to this story make a lot more sense now, but I’ve even got handy pointers to the line numbers in the source code. Let’s see, NSData+extras.m, around line 215:

NSString *charString = [NSString stringWithFormat:@"%c", base64EncodingTable [outbuf [i]]];

Survey says? That’s the line I’m going to zero in on now as I try to figure out why this crash is occurring. Sure, I’ll have to now speculate as to why, and under what circumstances the observed crash might occur. So the magic of “atos” doesn’t take all the work out of debugging a crash log, but it brings most crashes out of the “frustratingly impossible” into the realm of “all in a day’s work.” I can use my knowledge of the situation to ask intelligent questions of the user, and hopefully get to the bottom of it, so that no other user ever has to see the nasty crash again.

I can do more than just tell the user “thanks for the report, I have no clue what’s happening here. Sorry.”

Doing It For Yourself

I hope I’ve intrigued you by the power of what I was able to achieve with “atos”, but of course to do the same with your own project, you’ll need to make sure you can come up with a “symbol-laden version” of your release app, so you can apply the same magic. The best way I know to do this is to simply add another build configuration in xcode, that is just like your “Release” configuration, but doesn’t strip out debugging symbols. You can read more about this approach and others in Apple’s fine documentation.

If you find yourself dealing with crash logs on a regular basis, it might be a good idea to make building a “symbol-laden” version of your app part of the standard build and distribute process. For myself, I’ve found it happens rarely enough that I’m happy with a process whereby I re-checkout a tagged version of the sources from Subversion, and then rebuild a copy using the “Release With Symbols” build configuration.

Nota Bene: If you’re on Leopard and trying to replicate a symbol-laden version of an app you shipped with Tiger, you’ll probably have to tweak Xcode to use the old linker, in order to get a build whose memory addresses match up with your old Tiger-built application. Might be easier to just go back to Tiger and rebuild it there.

I know this was probably common knowledge for some of you, but I also know from experience of chatting with my colleagues that this is completely new, forehead-slapping “why didn’t I know about that?!” type material for many others. I hope it saves you hours of work. Possibly even enough hours to continue reading long articles such as this one!

I am going to need something to develop my iPhone/iPod applications on pretty soon.

I couldn’t be more thrilled, because not only does the list include everybody I suggested as good candidates, it also includes me!

I’m particularly happy to share the honor with so many people I have so much respect for. I would try to itemize them but there are just so many that I am still in awe of actually being listed among them. Many thanks to those who voted for me. It feels great to be recognized, and I’m glad if it means I’ve had a positive influence on you.

I would also like to point out that everybody who has programmed for MarsEdit is on the MacTech 25 list this year :) This project has the golden touch!

My entry received an honorable mention as the “Best Non-Screensaver.” I thought everybody would do a screensaver, and I should try to be a little different. Viva is an application that lets you tile multiple screensavers on your desktop. Fun times. But run the app before you run the .saver – there’s a bug.

Big thanks to Jon for sacrificing many hours and making the difficult decisions as judge.

Start your engines lady and gentleman coders, get ye to the Iron Coder starting line this Friday evening!.

Sooo… I’m encouraging anybody who hasn’t voted yet to do so, and to vote for people who influence you directly. I voted for three people from among the following names which I present merely as memory toggles for you. The problem here of course is I’m sure I’ll forget somebody. But at the very least I feel comfortable proposing any of these names as worthy recipients for an honor such as this. These names all strike me as being particularly appropriate this year. All of these people go out of their way to develop a personal relationship with the community on top of a professional relationship. Any three of them are deserving of your vote:

John Gruber. I probably don’t have to convince most of you that John is an influential part of our lives. Whether you read his content or not, his opinions push developers, users, and the press to reconsider their views on all things Mac. This past year was an especially influential year for Gruber, having dropped some serious food for thought in his HIG Is Dead speech at the C4 Conference.

Aaron Hillegass. Every year, Aaron’s Big Nerd Ranch “vacation” boot-camp for aspiring Cocoa developers churns out a new batch of programmers freshly-versed in the “right atittudes” to get started programming on the Mac. My belief is that more developers on the Mac will ultimately make all developers on the Mac more successful and happy. We’ve got a great community, and as new members are successfully indoctrinated into the group, our shared resources become larger and larger. I expect that as the years go by, more and more of the Mac developers you meet socially will have in their pedigree some kind of training from Aaron and the rest of the staff at Big Nerd Ranch.

Paul Kafasis. As CEO of Rogue Amoeba, Paul has had his hands full. But he still found time over the past year to write several thought-provoking commentaries on the Rogue Amoeba blog, prompting debate about terms like “Delicious Generation.” He also serves up repeated doses of “good indie software sense” in blog series such as Should I Exhibit At MacWorld. On a personal note, Paul’s advice to me has been invaluable as I’ve gotten to know him and tried to learn from his experiences in successfuly building Rogue Amoeba into a robust young business.

Leo Laporte.I grew up listening to Leo on SF Bay Area radio. At the time I was more into Amigas and Sun Workstations than Macs, but I still found Leo’s show fascinating. The fact that all these years later I’m still listening to him, and he’s still pushing the Mac, is pretty awesome. Leo’s This Week In Tech podcast is the most popular tech show online, and his empire continues to grow as he adds other podcasts, videocasts, and satellite radio to the collection. Most of Laporte’s shows are not Mac-focused, but he always has Mac in his mental repertoire, which no doubt leads many of his listeners to consider giving the Mac a shot. More Mac users equals more Mac customers. And that’s good for the developer world.

Merlin Mann. Merlin Mann tirelessly (OK, I’m sure he gets tired!) promotes the Mac as tool numero uno in the battle against productivity failures. Through his world-famous 43 Folders blog, he uses his playful writing style to remind the world that there are tricks and techniques to an organized life, and that the best ones require a Mac. Merlin’s influence on productivity software can be witnessed by the growing trend for “Getting Things Done” applications. As advisor to The Omni Group during the development of their “OmniFocus” GTD application, his opinions will no-doubt help establish some of the standards for all such software in the future. Merlin Mann is also an integral part of the Leo Laporte podcast MacBreak Weekly, the one show in Laporte’s empire which is dedicated to Mac and Mac alone.

Jonathan “Wolf” Rentzsch. Wolf has been personally influential to me for a few years now, as his work within the developer community on projects like mach_inject have made him well-known as a highly skilled Mac OS X developer and consultant. But this past year, Wolf turned the corner and became incredibly influential by launching a new Mac developer conference called C4. Even in its maiden year, the conference left attendees unanimously satisfied and eager to return for more fun and englightenment in the coming years. Wolf has been working hard on planning the 2nd C4, and I’m sure his influence will continue to grow as the conference becomes even more renowned.

Brent Simmons. What can I say, the man had a huge impact on me this year :) As the person who was instrumental in putting me on NewsGator’s radar as a potential buyer for MarsEdit, I am very grateful to him. He also deserves your vote because of the ways in which he helps to define the Macintosh user experience through his development of NetNewsWire, and through his participation in the developer/user communities.

Scott Stevenson. Scott Stevenson has long been a valuable resource in the Cocoa community in particular, serving up delicious tutorials on his Cocoa Dev Central site. He upped the ante recently by adding an editorialized take on the world of Cocoa blogging, in his Cocoa Blogs digest. Scott’s writing style combined with his passion for the Mac and an ability to spot quality are turning his content into the first-stop for quality Mac development information.

For some dumb reason, I’ve been living without such conveniences for several years now. I know there are many 3rd party products that add an epic amount of functionality to the standard copy and paste of the system, but I’m not really interested in running a separate application for this purpose.

It occurred to me that this would be a perfect use of Xcode’s Menu Scripts. Xcode not only supports the addition of custom scripts to the menu bar, but supports a unique format by which the scripts can advertise keyboard shortcuts, and integrate seamlessly with Xcode’s text editor. For instance, you can use a script to process the selected text in Xcode and spit back out something else in its place.

Append To Pasteboard is a simple python script that takes advantage of the “pbcopy” and “pbpaste” commands to tack whatever you’ve selected in Xcode on to the end of the existing pasteboard. You stick this within your (horribly located) user scripts folder, where you must name the folders and files with numeric prefixes to inform Xcode where they belong. For example, the script lives in my home directory:

/Users/daniel/Library/Application Support/
	Apple/Developer Tools/Scripts/
	10-User Scripts/1-DCJ/

Unfortunately Xcode doesn’t handle the situation where some scripts come from your home directory, and some come from the /Library installation directory. So I put everything in my home directory. The “10-User Scripts” is how I tell it to name the menu bar item “User Scripts.” The “1-DCJ” folder is the folder with all my custom scripts in it, and since it is “1,” it comes before all of Apple’s standard scripts that I copied in (10-Open, 20-Search, etc). The process of providing scripts to Xcode is ugly but when you finally get it all put together, this is what you end up with:

Now it’s a snap to cruise through a .m and accumulate method declarations, for pasting into a corresponding .h. Note that the script always appends a newline to the existing pasteboard, which could be annoying depending on what you want to use it for. Since I envision myself almost always double-clicking a function or method declaration and appending it to a list, I find this behavior to be most convenient.

Thanks to Jon Wight for helping with Python.

Update: Any productivity gained by using this macro was certainly lost as I tried desperately to debug Xcode’s latest propensity to “beachball” on launch. Finally, I attached with gdb and found a telling backtrace. Hmm? Yep, when I “zipped” the script for upload to the blog, I left a zipped copy in the scripts folder. Next time I launched Xcode, it choked big time on trying to treat that zip file as a script.

Unless. Ughh!!!!! Angst, angst, angst! It’s got crap in the index. You jump to a definition, make a load of changes, build, run, and … nothing’s changed. What on earth? You thought for sure you fixed it. Finally you stumble upon the nuanced fact that the source file you’re editing isn’t located in your project directory. In fact, looking at the full path you see fragments like “.bak” or “old” or “copy” in the hierarchy. This is some bogus copy of your source base that you moved aside, but somehow Xcode caught up with it.

I’m not exactly sure how this happens, but I suppose the recommended procedure for fixing it is to click the “Rebuild Code Sense Index” button from the Project Inspector window. Heck, for all I know this might work. But it gives no feedback – no sense of whether the reindexing has started, or when it will be done. And especially no feedback about what it chose to index. Well, truth be told you can see some vague feedback about the indexing progress if you open the “Activity Monitor” window. But it’s still not completely reassuring.

I stumbled on the “xcodeindex” command-line tool, which is a real gift. With this you can not only explicitly remove the index, but you can recreate it with a variety of debugging log levels set, to see exactly what’s going on.

But Apple really made my day by including a “dump” option. For instance:

% xcodeindex -project FlexTime.xcodeproj dump > MyIndex

Yields a tab-delimited file containing all the indexed terms that Xcode knows about for my project. Nifty!

Anatomy Of An Index Dump

The index file contains thousands of rows divided into 5 tab-delimited columns detailing the symbol name, description, type, language, and file. Let’s take a look at a sample line from my FlexTime index, column by column:

adaptPlaybackToDisappearingItem:

This is the symbolic name, what I would have to double-click on to get Xcode to wake up and notice it was an indexed term.

-[RoutineDocument(Running) adaptPlaybackToDisappearingItem:]

This is the description, which seems to be what Xcode displays for instance in the popup menu when there are more than possible choices. Clearly this description will vary based on the language the symbol happens to belong to.

Instance Method

The type is probably the most interesting column, after name. In this case I’m told it’s an instance method. Other rows in the output include types like “Instance Variable,” “Function,” and “Type.”

Objective-C

Language. Well, that’s pretty obvious.

/Users/daniel/Sources/FlexTime/RoutineDocument+Running.m

Source file. Everything has a source file, including items from the System APIs, which point into the root volume or the appropriate SDK.

Great Expectations

What’s really exciting about this index dump file is that I can read it. You can read it. We can write scripts to do nifty analysis tricks with it. Xcode’s standard project indexes are kept in some (presumably) proprietary binary format that we dno’t have easy access to, but the dump file opens things up to the masses.

For instance, sometimes I’d really like to know the answer to questions like “which of my custom classes is depended upon by the fewest of its peer classes?” I’m not sure how I would figure that out (no doubt some of you will comment with another clever solution), but I’m pretty sure I can whip up a Python script to do the trick with this dump file.

Update: Well, call off the party balloons. The index dump file is cool, but not quite as cool as I was imagining. I just realized that because the “Instance Variable” types don’t indicate the type of the variable, they’re not very useful for tracking down dependencies.

The workaround is, of course to simply restart Xcode when it gets flakey. But often this is an ordeal because you then have to go through your recent projects and manually reopen the one (or ones) that you were working on. I found myself frequently cmd-clicking the project window’s title to get a Finder reference to the project, so I could quickly reopen it after quitting.

Restart Xcode does all of this for you. Save the script to your ~/Scripts/Applications/Xcode folder, and keep it at easy reach from FastScripts or the Apple Script Menu. When you run it, it asks Xcode to quit, but not before accumulating a list of all the project documents you’ve got open. It waits for the quit to finish, then promptly reopens Xcode and all the documents.

Until Apple improves the uptime of Xcode, this script is going to save me a lot of frustration.

I recently added Keychain Support to an application where the sensitive password information needs to be fetched at launch-time, every time it launches. That’s fine, but one of the important security features of Keychain Services is its ability to detect changed copies of a previously permitted application. This is to ward off abuses in case a hacker has replaced some trusted application with one that is programmed to siphon off your keychain passwords or something:

Usually this dialog appears only once in a while, after updating an application to a newer version, or installing an Apple software update. But when you consider the product development cycle: develop -> build -> test, you can imagine how frequently an application under development will trigger this warning.

Until provoked by this nuisance, I didn’t know about a feature in Keychain Access that allows users to open up access to a particular key. In general this would be a pretty insecure choice, but it makes a lot of sense for development purposes:

You’ll find the option in the “Access Control” tab after double-clicking any keychain item. So for the purposes of development, test against a key that doesn’t actually have any sensitive information in it, and set the access to unrestricted. Now you can rebuild and test as often as you like without being pestered by Keychain Services.