1. Navigate to a .app file in the Finder, e.g. /Applications/Mail.app
2. Control-click the app icon and select “Show Package Contents”
3. Navigate to Contents/Info.plist
4. Open with a plist editor or your favorite text editor.

These files advertise a number of interesting attributes including the application’s official version and copyright strings. They also include important clues to the system about what kinds of files the application knows how to read and write, and what its minimum system requirements are.

Minimum System Requirements

Declaring the right Info.plist values not only allows a developer to control which operating systems a product should run on, but which of the application’s architectures should be considered for a particular operating system. For example, Black Ink uses a number of keyed values describing its requirements:

	<key>LSArchitecturePriority</key>
	<array>
		<string>x86_64</string>
		<string>i386</string>
		<string>ppc</string>
	</array>
	<key>LSMinimumSystemVersion</key>
	<string>10.4.0</string>
	<key>LSMinimumSystemVersionByArchitecture</key>
	<dict>
		<key>i386</key>
		<string>10.4.0</string>
		<key>x86_64</key>
		<string>10.6.0</string>
		<key>ppc</key>
		<string>10.4.0</string>
	</dict>

Notice the fidelity of some of these options. Black Ink is an application that is compiled with binary code for three separate architectures embedded into it: 64-bit Intel, 32-bit Intel, and 32-bit PowerPC. In order to give the user the best possible experience after they double-click the application icon, the system consults the Info.plist values to determine which binary is most appropriate to load on the current system.

The options for Black Ink start by indicating that all things being equal, it prefers to run as 64-bit Intel, but will settle for 32-bit Intel or PowerPC in a pinch. Next, it conveys that it should not be run on any operating system earlier than Mac OS X 10.4. Finally, it adds an additional restriction by architecture, stating that in the case of 64-bit Intel, the system should consider Mac OS X 10.6 the minimum required OS.

Why so picky? For example, Mac OS X 10.5 will run on computers that are new enough to support 64-bit Intel code, but the 10.5 operating environment itself doesn’t contain enough compatible libraries to make it very useful to a full-featured Cocoa application. So we need to convince the 10.5 system to ignore our 64-bit Intel code. And this is how it’s done.

Embedding Info.plist

This is all well and good for applications, but what about other code that may be run on a variety of hardware and operating systems? In particular, command-line tools do not come in the same bundled-folder format that applications do. Fortunately, Apple thought of this when they implemented support in their developer tools for embedding the contents of an Info.plist directly into the binary itself. You just have to pass the Info.plist path to Apple’s linker tool when it’s constructing your binary file. (Thanks to Paul Kim for reminding me of this functionality). What this means in practice is adding arguments like this to your Xcode target’s “Other Linker Flags” build setting:

-sectcreate __TEXT __info_plist $(INFOPLIST_FILE)

This instructs the linker to crete a new text segment named “__info_plist” in the resulting binary, and to fill it with the contents of the referenced Info.plist file. To make sure the INFOPLIST_FILE expands correctly, just make sure you set it in the separate build setting for identifying the Info.plist (the one you’d normally use in an app, and that is just ignored by default for a command-line tool target).

I brushed up on all this information because I wanted to help ensure that some recent work I did to create a standalone MultiMarkdown tool would be backward compatible with older systems. This required adding an embedded plist much as I’m describing here.

(Note: I have gotten some feedback to the effect that the __info_plist segment’s minimum system version requirements may not actually be consulted properly on Mac OS X 10.5. This would be a bummer, and in fact sort of reject the whole point of embedding the Info.plist in this case. I’m going to see if I can confirm whether it doesn’t in fact work.)

You could technically embed all kinds of stuff in the binary, with different section names. But the contract in this case is that when Mac OS X is determining information about an executable, it will consult the bundle’s Info.plist file, if it happens to be an executable in a bundle, or the binary’s __info_plist segment, if it happens to have one.

Another common reason for needing to embed Info.plist information into a standalone binary is if you are using Apple’s code signing mechanism to sign an individual binary tool. Code signing uses information from the Info.plist of a product, and in the case of standalone binaries, the only product-cohesive place to put this is in the binary itself.

Examining Binary Info.plists

After you’ve performed the magic above, you’ll probably be keen to inspect your command-line tool’s Info.plist to see if it worked. To do this, you can use the otool command from the Finder. If you run otool with the “-l” option to show all load commands, it will include the load command that identifies the __info_plist segment. Use grep to narrow it down to the part you’re interested in:

otool -l ./yourTool | grep info_plist -B1 -A10

Any results at will confirm that there is Info.plist information in the binary, but how does it look? The “-s” option to otool lets you examine the contents of a specific segment in a binary, but it dumps it out as hex code. Fortunately, another system-bundled tool called “xxd” is capable of easily converting between hex-dump and text formats. As it turns out, some of Apple’s bundled developer tools (at least on my Mac) contain these embedded __info_plist sections, so if you don’t have a command line tool of your own to try this on, you can peek at what the “atos” tool has to say for itself:

otool -s __TEXT __info_plist /usr/bin/atos | xxd -r

In Summary

Info.plist values are important for conveying information about your product, from the crudest, most obvious stuff like version number, to the most nuanced details such as which architectures are supported. Knowing how to specify an application or tool’s requirements as precisely as possible will ensure that the product performs optimally on whatever supported system your user happens to be using. I’ve barely scratched the surface here with a few common deployment keys that control execution behavior on Mac OS X. Be sure to skim Apple’s Information Property List Key Reference to see if there are other useful bits of information you can share for your application or for your command-line tool.

For the longest time, I assumed it was simple mistakes: I had neglected to ever approve “remember this password,” or I had somehow invalidated the password in my keychain. But recently I have come to realize that Safari is no longer auto-filling passwords for me on any site. Ever. This sucks. Usually when something like this happens, the first thing I do is “Repair Keychain” from the Keychain Access application. But in this case, the keychain is reportedly not damaged. No help.

To research this problem, I started the way I usually do: Googling it. While there are many promising search results, most of them lead to “solutions” that are not applicable to my scenario. Most commonly it seems the user has somehow ended up with the appropriate “User names and passwords” checkbox being turned off in Safari’s AutoFill preferences.

So how does a programmer like myself approach a problem like this? In recent years, Apple’s Instruments application has become an incredible asset for tracking down weird behavior not only in my own apps, but in other apps and in the system itself. I opened it up and started a new “Time Profiler” instrumentation session, targeting Safari. I know that Safari used to autofill my password whenever I typed in an account name that it recognized, so I start Instruments “sampling” Safari while I type my PayPal account name into the PayPal login page, and stop it after I’m done.

I want to figure out where Safari is failing to get my password, so I take a wild leap and assume the corresponding function call in Safari contains the word “password.” Typing this into the Instruments search box yields promising results:

Now I know that Safari is at least making an effort to fetch the corresponding password for me. I see that it’s calling -[NSURLCredential password], which in turn calls a lower-level Keychain Services function, SecItemCopyMatching(). But why isn’t it filling in my password?

At this point I switch to gdb, where i can attach to the running instance of Safari and dig a little deeper into the issue.

% gdb
(gdb) attach Safari
(gdb) b -[NSURLCredential password]
Breakpoint 1 at 0x7fff8ddff4af
(gdb) c
Continuing.

I know from Instruments that Safari will call this method when it tries to autofill my password, so I go back to Safari and enter my PayPal ID again. Sure enough, Safari “freezes” indicating that gdb has interrupted it where my breakpoint was set. Now, I realize this isn’t for the faint of heart, but you can use gdb to effectively debug an application, even if you don’t have the source code. You just have to know a bit about how Apple’s compiler translates source code into corresponding assembly language instructions, and where the arguments are placed when calling methods.

Breakpoint 1, 0x00007fff8ddff4af in
-[NSURLCredential password] ()
(gdb) d
Delete all breakpoints? (y or n) y
(gdb) po $rdi
<NSURLCredential: 0x7ffd34d62920>: paypal@jalkut.com
(gdb) po [$rdi password]
2: x/i $pc  0x7fff9192760e <gdb_class_getClass+4>:
	mov    0x20(%rdi),%rax
Can't print the description of a NIL object.
(gdb)
(gdb) p (int) [$rdi hasPassword]
$11 = 1

What I’ve done here is first delete the breakpoint, so I don’t get interrupted while poking around from gdb itself. Then, I display the NSURLCredential object’s description, confirming it corresponds to the account I’m trying to log into. Finally, I call the password method on this NSURLCredential object myself, so I can see quickly and easily what the response will be. Gdb’s objection to trying to print a NIL object confirms that the password is not being returned. I am further able to confirm with NSURLCredential’s “hasPassword” method that the keychain does have a password for this account, but it isn’t being returned.

Here we have a situation where apparently Safari is trying to get my password, but the system is vexing it. Let’s confirm by setting a breakpoint a little deeper, at the Keychain Services level that we also noticed in Instruments.

(gdb) b SecItemCopyMatching
Breakpoint 3 at 0x7fff85919582
(gdb) c
Continuing.

Breakpoint 3, 0x00007fff85919582 in SecItemCopyMatching ()
(gdb) po $rdi
{
    acct = "paypal@jalkut.com";
    atyp = form;
    cdat = "2007-06-02 16:17:01 +0000";
    class = inet;
    desc = "Web form password";
    icmt = default;
    labl = "www.paypal.com (paypal@jalkut.com)";
    "m_Limit" = "m_LimitOne";
    mdat = "2011-07-29 15:30:15 +0000";
    port = 0;
    ptcl = htps;
    "r_Data" = 1;
    srvr = "www.paypal.com";
}

So far, so good. It looks like NSURLCredential is properly passing a set of criteria to the keychain, describing the kind of password item it’s looking for. I take a peek at the second argument to SecItemCopyMatching, which is a pointer to where the result will be stored, and then I ask gdb to continue executing until the function is complete.

(gdb) x/x $rsi
0x7fff62eb3590:	0x0000000000000000
(gdb) fin
Run till exit from #0  0x00007fff85919582 in SecItemCopyMatching ()
0x00007fff86417c29 in URLCredentialInternetPassword::copyPassword ()

When gdb returns control to me, I know the function has finished calling, so I examine the result area in memory.

(gdb) x/x 0x7fff62eb3590
0x7fff62eb3590:	0x0000000000000000

Indeed, it’s NULL! What is going on, here? I decided to dust off my Usable Keychain Scripting tool, which makes it easy to use AppleScript to search and inspect the keychain. Is the inability to access the password a system-wide thing, or just something that is vexing Safari?

tell application "Usable Keychain Scripting"
	tell current keychain
		password of keychain item "www.paypal.com (paypal@jalkut.com)"
	end tell
end tell

The script works perfectly, first causing the system to ask permission to authorize “Usable Keychain Scripting” to access the keychain item, and then returning the expected password string. This got me thinking, is it possible I’ve somehow got Keychain Accessing refuse access only to Safari?

When I look at the “Access Controls” for the affected keychain item, it lists Safari as one of the permitted apps. What the hell, I think. I’ll remove it from the list, and then re-add it. To try to “dislodge” something. Confoundingly, everytime I try to add it, it just silently fails to appear in the list. I am able to add any other app without an issue.

As a quick check, I change the setting to “Allow all applications to access this item,” and save the changes. When I return to Safari and type in the login name, it instantly fills in the password for me.

My system has a functional keychain but Safari seems unable to be authorized to access it, either on a one-time basis or as a “always allow” permission. I finally got the clever idea to look at /var/log/secure.log, which frankly, I should have done about 5 hours earlier. What’s this?

com.apple.SecurityServer[33]: suppressing keychain prompt
for invalidly signed client /Applications/Safari.app(2938)

Aha! So that would explain it. Funny, I don’t remember tweaking anything inside the Safari application bundle, but this is certainly a very real example of how a broken code signature on Mac OS X can cause extremely subtle, hard to track-down “bugs.” I re-signed the applicaton on my own:

codesign -f -s - /Applications/Safari.app

When I reopened the app, and went to PayPal’s site, voila! It prompted me for permission, I approved, and now everything is back to normal.

I examined the Safari.app bundle contents to try to remember what I might have tweaked. Then my memory was jogged. I had opened and edited Safari’s Info.plist file to play around with some settings. Don’t ask, it’s an even longer story than this.

I take full responsibility for getting myself into this mess. I’ll certainly be more careful in the future to consider the implications of breaking code signatures on apps. However, I do think that the failure behavior could have been more informative. That NSLog I found in the console about refusing to prompt the user because of a broken code signature, could have been presented as a dialog that would instantly inform the user something is fishy. If my copy of Safari actually had been compromised by a nefarious individual, I wouldn’t have thought twice about continuing to enter my passwords and trying my damnedest to get Keychain to share my passwords with the compromised application.

Major Memory Meltdown

I have always been pretty proud of my ability to manage memory usage in my code. In addition to using good programming habits, I periodically run Apple’s memory leak-detection tools, and the clang static analyzer. These tools do a great job of finding accidental memory management errors. But as I discovered, they are not enough.

For months I have received anecdotal feedback that MarsEdit is using more RAM than it should. But I couldn’t find any evidence of wrong-doing on my part, so I committed a major mistake: I just assumed it was normal. MarsEdit uses a lot of Apple’s frameworks, including WebKit, which is so big and expansive, I assumed there was something about the degree of WebKit I use in MarsEdit that explained the large amount of memory consumption.

Wrong! I was inspired by a recent one-two punch of feedback, combined with a recent blog post I had read, to dig deeper.

Twitter user @Fletcher_Chris was exceedingly polite when he asked if MarsEdit’s 200+ MB memory usage was “normal.” Ugh! Normal? I don’t know? I think so? Not knowing about why your software does what it does is a major alarm bell.

Then my friend John Gruber wrote to let me know that not only is he noticing major RAM consumption, but that it’s something that has bothered him for some time now. A recurring, nagging issue, noticed by somebody who is technically competent enough to know a real issue from a false one. I have a problem on my hands.

It was lucky timing that my friend Bill Bumgarner recently published an article on the very subject of hard-to diagnose memory leaks. I had actually read this article, enjoyed it, and blithely assumed it didn’t apply to any of my current software challenges.

It turns out that MarsEdit was in fact suffering from exactly this kind of leak: one that the brilliant leak-targeting tools from Apple are not able to detect. For a variety of reasons mainly involving retain cycles, I was neglecting to clean up after a major part of the MarsEdit post editor’s user interface, which means that every newly created and closed document would leak hundreds of KB of RAM.

Mea fricken culpa. Fixed in 3.1.4.

Tumblr Photo Posts

Tumblr has been a major challenge for MarsEdit because of the degree to which it doesn’t follow the usual paradigm for media management. This means in a nutshell that things you can do with almost every other blog system are not possible in Tumblr, making the user-interface a little clunkier to handle. In particular:

1. Tumblr does not allow direct file uploads. Even though their web-based UI allows you to upload an image for direct insertion into a post, this is not something they have exposed for remote clients such as MarsEdit.
2. Tumblr supports the notion of a “Photo” type of post, but it takes at-most one single photo. Usually MarsEdit is happy to add as many photos as you like to a post, but with Tumblr it has to be limited to one, and requires that the post type be switched to photo.

In MarsEdit 3.1.4, I take baby steps towards improving the situation when users do want to publish photos to a Tumblr blog. First, the per-blog “image constraints” settings are now applied to Photo posts on a Tumblr blog. So if you’ve got it set to limit posts to no more the 400×400 pixels, it will apply to any photo you drag in to Tumblr’s photo well.

I also streamlined the experience when choosing a photo from the Media Manager for a Tumblr-style post. Not only will the image constraints selected in the Media Manager be applied to the photo, but for example selecting and inserting a photo will automatically set the photo for the open Tumblr post document, or create a new post if one is not already open. This should make the photo-blogging experience on Tumblr a lot smoother.

A Curious Lack of Crash Reports

Since MarsEdit 3.0 was released, I have included a crash reporter mechanism that will automatically notice when MarsEdit quits for any unexpected reason, and offer to let the user send me a crash log plus any additional information they choose. This has been a great tool for measuring the number of harder-to-find bugs that remain in MarsEdit, and for zeroing in on and fixing some of them.

Luckily, the flow of crash reports is a relative trickle, compared to the number of users running MarsEdit on a daily basis. But after MarsEdit 3.1.3, the crash reports all but dried up! Was I really so brilliant as to accidentally fix any remaining bugs in the application? 3.1.3 forever!

No, what actually happened was I was brilliant enough to break the crash reporter in 3.1.3, so that it would fail to present the dialog to users offering to send in the report. Crashes were still happening, I’m sure, but I wasn’t hearing about them. Now fixed in 3.1.4, and I’m anxious to get back to monitoring the pulse of MarsEdit stability with this great feature.

This functionality is based largely on the great work of the iMedia project, which was spearheaded by Karelia to replicate Apple’s own ubiquitous media browsing interface, some variant of which is available in Pages, iWeb, Mail, etc.

iMedia has been around for a while, but in the past year or more it has been undergoing a significant overhaul as its primary developers gear up to release the official 2.0 milestone. I have been involved in the 2.0 project for a long time now as an “early adopter,” using MarsEdit as a testing ground during the beta phase, and taking (mostly!) only the refined and polished aspects of the product for the final MarsEdit 3.0 release.

We are still ironing out some of the kinks, especially in parts of the project that were not tested in MarsEdit. One of the cool new features is support for the Flickr photo-sharing service. Like MarsEdit’s own Flickr-browser, it shows thumbnails of images on Flickr, and lets you treat them more-or-less like ordinary image files that you’d find on your computer.

Things get a little bit complicated when a user selects and drags one of these Flickr image thumbnails to another app, such as the Finder, or Photoshop. Since the image lives in the internet, and not on the local hard disk, it is not something that can be popped over to the application in an instant. Instead, a progress dialog comes up while the image is downloaded to the disk, and then the local copy is passed to the receiving application.

I try to contribute back when possible in the form of bug fixes and architectural feedback. Some of the work I did for iMedia 2.0 involved updating its drag-and-drop functionality to take advantage of new features in Mac OS X 10.6. In the process, I became a sort of de facto expert on handling dragging, which is a shame, because we could sure use a real expert, instead!

The other day, an issue came up that relates to the dragging of Flickr files. If the download is taking a while, and the user tries to cancel, mouse clicks are apparently ignored and they are stuck with the progress dialog above. I agreed to look into the issue, but there was just one problem: my network is too fast! Whenever I try to drag a Flickr image to another app, it’s done within a couple seconds, and I don’t have time to play around with clicking the cancel button.

If only I could get a really slow connection to Flickr, then I could easily test this. In the old days of Mac OS 9 or earlier, I may have resorted to plugging in a slow modem to simulate the experience of the less bandwidth-fortunate. In Mac OS X however, I can take advantage of the advanced firewall software that comes bundled with the operating system, and which allows me to configure “traffic shaping” policies on the traffic coming in and out of my computer. I found a great hint from Macworld that got me on the right track.

My goal is to cause Flickr downloads to be artificially slow. In order to achieve this goal I need to find the specific web address that the downloads come from. To do this, I use the standard tcpdump terminal utility, which also comes bundled with Mac OS X:

sudo tcpdump -Atqp -s 0 -ien1

The only part you may want to change is the bit at the end that says “en1″. Depending on the network interface you’re using, you might want en0 or en2 instead. With this running in the terminal, I go back to iMedia and start downloading Flickr files. This lets me see what the addresses of those files look like, and I discover the pertinent host name is “farm5.static.flickr.com”. Using the information I gleaned from the Macworld article, I set up the following firewall traffic shaping rules:

sudo ipfw pipe 1 config bw 15KByte/s
sudo ipfw add 1 pipe 1 src-port 80 src-ip farm5.static.flickr.com

Translation?

1. Create a pipe (you might prefer to call it a “tube”). A pipe is an artificial pass-through which can be configured to slow down, block, or otherwise alter the network traffic that goes through it.
2. Configure the pipe to limit bandwidth to a crawling 15KByte/s.
3. Configure all traffic coming from port 80 (the standard HTTP port), and originating from a specific hostname at Flickr, to pass through that pipe.

As soon as these commands are executed, the pipe is in place, and Flickr download speeds are brought to screeching halt. I can now debug the issue with the download dialog and the cancel button patiently and curiously, instead of racing to try clicking the button while I still have the chance.

When I’m done debugging, I definitely want to delete the pipe, so that download speeds go back to normal:

sudo ipfw delete 1

Mucking around with the built-in firewall on Mac OS X is not for the extremely faint-of-heart. This is why all the commands listed in this entry require “sudo” superuser privileges. Yes, you could seriously screw up your computer’s network connection if you do something wrong. But if you are the type of person whose work requires you to test and debug network related tasks, this traffic shaping functionality can be a real asset when it comes to mimicking the network environments of other people.

Addendum: I should have known I might be doing it the hard way, as usual. Clint Ecker reported on Twitter, and several folks in the comments below, that speedlimit, a Mac OS X preference pane, gets the job done in a much friendlier manner.

Unfortunately, this seemingly minor update presented a major failure in my debug builds, which I’m adventurously compiling using the LLVM clang compiler. They built fine with Xcode 3.2.2, but starting in 3.2.3, I got a linkage failure, tracing back to the popular open source RegExKitLite:

Undefined symbols:
  "___gxx_personality_v0", referenced from:
      _rkl_debugCacheSpinLock in RegexKitLite.o
      ...
  "__ZSt9terminatev", referenced from:
      +[NSString(RegexKitLiteAdditions) clearStringCache]
      in RegexKitLite.o

What? Those symbols smack of C++, but there isn’t any C++ code in my project. At least, I don’t think there is. RegExKitLite is so famously advanced in its designed, I couldn’t easily tell you whether there was stuff in there intended as C++ or not. On a whim, I tried to switch the file type to C++ to see if that made clang any happier. No, it screamed bloody murder. It agrees with me, RegExKitLite is not C++ code.

So why the link errors? I examined the compile line carefully for RegExKitLite and confirmed that it’s using clang in a manner that should only be generating plain C linkage:

/Developer/usr/bin/clang -x objective-c -arch x86_64 ...

But when I look at the resulting RegExKitLite.o with the “nm” command-line tool, it shows the culprits symbols have been listed as references in the binary object file:

% nm RegexKitLite.o | grep -e terminate\\\|personality
                 U __ZSt9terminatev
                 U ___gxx_personality_v0

Normally when I’m getting some kind of issue like this, I just type in the problematic symbols in to Google and usually find somebody else has found and solved the problem. No such luck this time. There are lots of false positives for the typical reason somebody runs into this link error: they are compiling legitimate C++ code but neglecting to link with the stdc++ library. But in our case, we are not compiling C++ code, but we’re ending up with C++ dependencies nonetheless.

I hopped on to the #clang IRC channel on irc.oftc.net, where a couple extremely helpful clang engineers (thanks to dgregor and rjmccall) helped talk me through the problem, and determine that it does seem to be a clang 1.5 bug. They confirmed that with their latest and greatest clang the bug seems to be fixed, but I was curious to confirm it myself, so I checked out and built the latest clang, too. Sure enough, all is well for future generations!

But what do we do now, stuck with the clang 1.5 that ships with Xcode? There is a crude workaround: simply link to stdc++. But yuck! All that C++ linkage dirtying up my pristine Objective-C project? Since I only use clang (for now) to build debug builds, I don’t have too much of a problem doing this. But if I was shipping these binaries I would be very hesitant to make that concession.

Radar #8230225: clang 1.5 forces linkage to C++ for non-C++ source file
(Open Radar Link)

The best we can do is report the issue to Apple and let them decide whether it’s worth their time to backport whatever fix in later clang is missing from clang 1.5. If you have other suggestions for how to work around the problem in the mean time, please do chime in with a comment below.

Update: The author of RegExKitLite kindly chimed in below with a pointer to the revision in the clang source repository that that fixes the issue. He also suggests a simple, and seemingly safe workaround, involving a slight tweak to the RegExKitLite source code.

“Every time I press cmd-N to create a new document and start typing, instead of accepting the text, it just beeps at me.”

This seems impossible to me. Creating a new document and typing into it is fundamental to my product. Surely if it was broken I would know from my own daily testing of the application. I try diligently to reproduce it, but of course, I can’t. Create a new document. Start typing. Everything works, everybody’s happy. Except the person who reported the bug.

At some point in my tinkering I stumble upon the bug. Beep. I can’t type. There it is! What a drag. This application sucks!

Now that I’ve seen and heard the bug with my own eyes, I’m more committed than ever to fixing it. But how on earth did I do it? I try several more times and am unable to reproduce the problem.

I have only spotted the issue once in my dozens of trials, yet the customer claims it affects him every single time. What explains this? Maybe it simply doesn’t happen as often on my Mac as it does on his. My thoughts turn to the brute force of AppleScript. If this happens only once in a blue moon on my Mac, surely I can reproduce it by running a script that iterates the test conditions hundreds or thousands of times. I open AppleScript Editor and enter the following:

tell application "System Events"
	repeat 100 times
		keystroke "n" using command down
		keystroke "Testing typing"
	end repeat
end tell

Nothing! Even after running it 1000 times, nothing. How can this be? How could something so common for the customer, something that I saw myself once, be so difficult to reproduce on my Mac?

After playing around a bit more, I started to reproduce it more readily. What was happening? I couldn’t figure it out, but something “felt” a little different when I reproduced it. I thought carefully about the parameters of the bug, as exemplified (I thought) by my script:

1. The command key is pressed.
2. The N key is pressed.
3. The N key is released.
4. The command key is released.
5. Typing into the new document is attempted.

I looked again at the script. Perhaps this “keystroke” command is not literal enough. For all I know it’s circumventing the normal key events that get generated when you type on the keyboard. On a lark, I modified the script to be more explicit:

tell application "System Events"
	repeat 100 times
		key down {command}
		key down  "n"
		key up "n"
		key up {command}
		key down "x"
		key up "x"
	end repeat
end tell

Here we have a more literal match with my analysis of what happens. Hold down the command key, press the N key, release, then type something. In this case I just type an x character because I figure any typing is enough to trigger the bug.

I ran this 100 times as well and no luck. The bug is simply not reproducible through scripting. I guess I’ll have to figure out what conditions are making it more likely for the customer to run into the bug. Maybe he has some 3rd party software that is interfering. Or … wait a minute.

tell application "System Events"
	repeat 100 times
		key down {command}
		key down  "n"
		key up {command}
		key up "n"
		key down "x"
		key up "x"
	end repeat
end tell

I ran the modified script and the bug immediately exhibited itself. So what changed? Look carefully and you’ll see that my previous script assumed that when a keyboard shortcut is pressed, there is a sort of nested symmetry to the order of pressing and releasing the keys. When I press keyboard shortcuts, I hold the modifier keys down throughout the stroke, and release only after the letter key has been released. But this customer has muscle memory that inclines him to instead release the keys in the order they were pressed. When he releases the N key, the command key has “long” since been released.

This subtle difference turns out to trigger a bug in my event monitoring code that, to make a long story short, robs first responder status from the default view in the document. So when he creates a new document, there is no first responder, and his typing just causes a bunch of beeps. When I create a new document, there’s always a first responder, so I don’t see the bug.

As difficult as this bug was to track down, it would have been near impossible without the theory testing, disproving, and ultimately proving tool of using AppleScript and System Events to zero on in the behavior. Sometimes when the steps a customer provides to reproduce a bug seem exactly the same as what you’re doing, you’re just not trying hard enough to find the difference.

But after Apple released 10.4.11, I started receiving crash reports at an accelerated pace. The log is the same in every instance, and culminates in what appears to be a threaded NSURLConnection caching operation:

-[NSURLConnection(NSURLConnectionInternal)
	_didFinishReturnCachedResponse:]
-[NSHTTPURLProtocol startLoading]
-[NSURLConnection(NSURLConnectionInternal)
	_performOriginLoad]

Now it so happens that WebKit is the one crashing in many of these cases, but I suspect the problem is more widespread than that. In any case, the problem is definitely more widespread than just MarsEdit. Searching Google reveals a number of similar results, from a variety of applications, including Apple’s own.

If you’re a developer and your application uses WebKit or NSURLConnection, chances are you’ve noticed an increase in crash reports, too. Always on 10.4.11, always involving NSURLConnection. If you’re a user running 10.4.11, chances are you’ve noticed that network-enabled applications seem to be a bit more flakey and crash-prone.

The fact that the bug seems to be fixed in Leopard makes me think that this is a bug whose source was identified and fixed. Now the question is, will Apple ever ship a 10.4.12 containing a fix? Or will those users be stuck in crash-ville forever?

Apple needs to get this fixed. For these poor 10.4.11 users who got stuck with a buggy URL loader, and for us poor developers who have to gently explain to them that it’s out of our control. I have to assume there’s already a bug reported against this, but better safe than sorry, I wrote up another one: Radar #5704773.

So why I am I writing this blog entry? Because I want other developers who read my blog or who find it by Google search to know that they’re not crazy. And I want users who run into this crash to please remember to send the crash report to Apple. They need to know that this is a serious issue that is impacting you and the apps you use.

What does it mean for a project to be alive? I’m adopting a definition of life that is roughly “as freaking close to original functionality as possible.” That means the application needs to be launchable. It needs to perform its advertised purpose. Ideally, a project in “life support” should be less than a week away from “shippable” at any time.

The bad news? To maintain this standard, you might put up a ton of scaffolding code today to keep the application alive, only to tear it down in a few days when the complementary structures in another part of the application have been comparably revised. Some would argue that it’s better to bid farewell to the life of the application, and commit to a dark period of zero functionality while everything is redesigned and/or ported to the desired specification. This would certainly make for less “throwaway” code, but an unfamiliar, non-functioning application scares the living daylights out of me.

Clean Code In A Strange Land

Back to the specific project at hand. You’ll recall that during triage, I discovered some cool Objective-C hacks that allowed me to fix the heinous crashing bugs on Intel, while leaving essentially the entire Java code base unchanged. This trick relied on a basic truism of the Cocoa-Java runtime: calling methods on Java objects from Objective-C is relatively easy, while calling back to Objective-C from Java is not.

To call a Java class method from Objective-C, you look up the class by name, and send it a message that will, by convention, map to an associated Java class method. When the Java class you’re messaging is in fact a subclass of a Cocoa class, the messaging can feel extremely natural. For instance, assuming you’ve got a Java-based Preferences Window controller, this is how you might message it to show its window, from an Objective-C based class:

Class theClass = NSClassFromString(@"PrefsController");
NSWindowController* myPrefs = [theClass sharedInstance];
[myPrefs showWindow:self];

Going the other direction, from Java to Objective-C, requires compiled-in bridge code. Much of the Cocoa frameworks from around the 10.2 era is represented in the bridge code provided by Apple. Everything else, including any custom Objective-C classes you’ve written, would require new bridge code to be generated and linked in to your application. This is apparently still possible, using Apple’s unsupported and poorly documented “bridget” tool. Personally I didn’t want to touch it. The idea here is to eventually get rid of the Java code completely, so I don’t want to waste a lot of time making it easier for it to stick around.

So here’s my approach: pragmatically move stuff from Java to Objective-C, keeping in mind that everything I move “up” must not be depended on by any other Java code. For example, if I move the Preferences controller from Java to Objective-C, then no longer can any Java code say “PrefsController.sharedInstance().showWindow(this)”. So deciding how to proceed will require at least some sense of the dependencies between the various Java classes in my project.

I was hoping there might be a nifty way of automatically gathering this information from Apple’s class diagram tool in Xcode, but it turns out not to be as entirely useful as I’d hoped. I used a combination of its member variable information, combined with some brute-force global text searches, to roughly plot out a list of all classes, and the Java-based classes that they depended upon. For instance my Java-based application delegate depended upon the Java-based document class, which depended on several Java-based model classes. So I can’t really port the document or model classes to Objective-C, but I can port the delegate, because no other Java code links directly to it.

The idea is to look for “childless nodes” in the class hierarchy, and decide whether it is pragmatic to port them no or later. As classes are migrated up from Java to Objective-C, it reduces the amount of Java code in the project, while keeping the application functionally identical to its former self.

The Objective-Java Chimera

Obviously things are never quite so simple. In a real project, dependencies between classes never fall into a perfect tree structure where one can simply pluck classes one-by-one from the top of the tree. In some cases, two classes might mutually depend on one another, in which case it would make sense to port both of them to Objective-C in one fell swoop. Or maybe the vast majority of a class is not dependent on Java, but one small component is. It might make sense to port the class with the caveat that things will not be completely functional, for a short period. The application isn’t dead! It might be wheezing, but it still basically works.

I came up with a technique which I found pretty useful, especially for some of the largest classes in the project. What if a class is half dependent on Java, and half not? It would be great to be able to port just the half of the class that can make it to Objective-C, while postponing the rest for later. Using a “Chimera” object that is conceptually part Java and part Objective-C, I was able to achieve this monstrous goal.

I applied the technique first to a custom NSView subclass, because I observed a slowness in drawing that I guessed might be sped up by minimizing the amount of cross-bridge messaging and structure conversion that going on every time the view was drawn. In any case, I knew I was not confident about applying Shark measurements to a Java class, and would have a much easier time fine-tuing the performance if all drawing was done in Objective-C. The custom view in this case was a major piece of the application, though. It had its hooks into the model code, and other classes had their hooks into it. And it was relatively huge. It made a great candidate for the Chimera technique.

The technique depends on Objective-C’s ability to forward messages a class doesn’t implement to another class. So for most practical purposes, an Objective-C class can “wrap” another class such that clients of the class think all the work is being done by the wrapper. In this case, we want to sneak a new CustomView class in to the runtime, where the nib file will unarchive and automagically contain an instance of our view instead of the old Java one. Then we’ll fulfill the contract of the old view class, both by implementing our own methods and by dispatching to it when necessary. Simplifying the situation somewhat, converting a Java class to a Chimera class involves the following steps:

1. Rename the Java class to something new. For instance, “CustomView” becomes “JavaCustomView.”
2. Create a new Objective-C class, named after the original. In this case, “CustomView.m” is our new source file.
3. Implement message-forwarding methods. These are where the magic happens.

That’s the skeletal Chimera. Now you can add whatever methods you like in ObjC and they will automatically take precedence over the Java equivalents. You can even rename the Java equivalents if appropriate, and partly implement methods in ObjC. For instance, you could implement “awakeFromNib” such that it first called the Java class’s “javaAwakeFromNib” and then proceeded to do whatever Objective-C based setup was appropriate.

So which are the “message-forwarding methods” that you need to implement? In my experience, it’s important to implement respondsToSelector:, methodSignatureForSelector:, and forwardInvocation:. In each case, you’ll want to dispatch to the underlying Java object, but only when appropriate. To have an instnce of the Java object around, you’ll need to add code that, at init time and awakeFromNib time, respectively, instantiates and configures the underlying class. You may also need to do some patching up of IBAction connections or whatnot. The fine-tuning will all be very application-specific.

The idea for the “responds to” and “method signature” methods is you want to return the union of your Objective-C class functionality and the Java class functionality. So first ask your superclass to handle the method, and then only if the result is NO or nil, pass-through to the Java class for further inquiry. The forwardInvocation method will only be reached if nothing in your class could handle the method, so you can just blindly pass these through to the Java class, if it will take them.

To make this work, you have to manage the dependencies between the two halves of the Chimera. For instance, it might make sense to maintain redundant copies of the IBAction outlet variables, so that both sides of the world have access to “outside the class” dependencies. A pitfall is you might end up storing data in one half of the Chimera and not realizing that the other half is still trying to depend on it. I never said this wasn’t going to be messy, but it gets the job done.

Applying this technique essentially opens the doors wide to staging the port. You can carefully port parts to Objective-C while keeping the application alive. You’ve got a full-fledged Objective-C class to pass around in “clean land,” while Java objects continue to operate as necessary on the Java subset of the class.

A Healthy Outlook

In the next and final installment of this series, I’ll talk about the grunt-work of actually going the final mile, and what we gain by getting there. Porting all those Java classes to Objective-C is fairly easy work, but it’s tedious. I came up with some tricks that might affect your approach in attacking a similar problem.

But while I’m debugging the code that actually does the manipulation, I find myself stuck doing stupid stuff like:

(gdb) p/x ((char*)[imageData bytes])[14]
$1 = 0x28

Ugh! That’s no way to win the software wars. It occurred to me today that what I really want is something like Hex Fiend built right into the debugger. Why shouldn’t I be able to hex-dump my NSData objects directly from the console? It turns out I wasn’t alone in this wish, because Dan Wood recently published a category on NSData that allows just that.

By simply compiling the NSData category into a given application, you give NSData’s debugging description superpowers. For instance, here’s me at the gdb console, examining the contents of the favicon.ico response I just received from NSURLConnection:

(gdb) po imageData
NSData 350 bytes:
00 00 01 00 01 00 10 10 10 00 00 00 00 00 28 01 | ..............(.
00 00 16 00 00 00 28 00 00 00 10 00 00 00 20 00 | ......(....... .
00 00 01 00 04 00 00 00 00 00 C0 00 00 00 00 00 | ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 FF FF | ................
FF 00 00 00 FF 00 00 00 EF 00 00 00 DE 00 00 00 | ................
CE 00 00 00 BD 00 00 00 AD 00 00 00 9C 00 00 00 | ................
8C 00 00 00 7B 00 00 00 52 00 00 00 42 00 00 00 | ....{...R...B...
29 00 00 00 10 00 00 00 00 00 00 00 00 00 11 11 | )...............
11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 | ................
11 11 11 11 11 11 13 31 11 11 12 55 21 11 1B B1 | .......1...U!...
13 D8 7F FF FB 31 1B B1 6F D4 FB 65 9F D2 1B DB | ....1..o..e....
FB 21 84 11 15 F5 1B FF FC 41 12 78 8D D1 1B B1 | .!.......A.x....
3C C1 2D FF FB 51 1B B1 1A D1 5F 71 11 11 1B C5 | < .-..Q...._q....
BF 81 1C C7 58 61 1A FF C6 11 12 BF FF 81 12 54 | ....Xa.........T
11 11 11 13 52 11 11 11 11 11 11 11 11 11 11 11 | ....R...........
11 11 11 11 11 11 11 11 11 11 11 11 11 11 00 00 | ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00       | ..............
Current language:  auto; currently objective-c
(gdb)

Dan’s code is pretty much perfect as-is, but I couldn’t resist tacking on some additional features. His category hardcodes the dumped data to 1024 bytes, but in my case there are times when I’m interested in seeing more. I added two category methods to the mix:

// startOffset may be negative, indicating offset from end of data
- (NSString *)descriptionFromOffset:(int)startOffset;
- (NSString *)descriptionFromOffset:(int)startOffset
				limitingToByteCount:(unsigned int)maxBytes;

Now I can peek at a specific subrange, or expand the default limit of 1024 bytes. I can even peek at the end of a chunk of data, by specifying a negative offset:

(gdb) po [mDocumentData descriptionFromOffset:-11]
NSData 109 bytes (showing bytes 98 through 109):
02 43 41 54 41 54 4F 4E 49 43 00                | .CATATONIC.
(gdb)

Wow! This is a pretty huge addition to my debugging and development toolbelt. Many thanks to Dan Wood for providing the ready-made code to speed up my implementation.

NSData+RSHexDump is freely available to you, under an MIT license.

Unfortunately, Apple has all but abandoned the technology. The implications for developers are that Cocoa-Java applications cannot easily adopt new APIs, are mysteriously buggy, and are not easily maintained with the latest versions of Xcode. In short, they are a mine-field of opportunities for disappointment and despair.

Nonetheless, I decided to buy a Cocoa-Java application, after looking through the existing sources and making best and worst-case scenario predictions. The previous owner had already disclosed some nuanced issues with running on Intel Macs, so I knew there would be some work before I could ship it in good faith. The amount of work would range from “a few hacks to get it working on Intel as Cocoa-Java” to “a complete rewrite in Objective C.” I planned to port to Objective C eventually, but I hoped to be able to put the product on the market soon with just a few tweaks to make it robust on Intel. Then I could port to Objective C at my leisure while counting all the money.

Now that I’m 95% done with the process of getting the application into shape, I’m ready to start writing about it. I hope that by explaining the process of evaluating and overcoming obstacles in the process of doing this port, it will establish a reference for others who are stuck doing the same kind of work. I expect there will be many Cocoa-Java parts in the months to come, as support for the legacy technology continues to diminish. Hopefully some of the wisdom I have earned will come in handy for others.

Evaluate The Source Base

The first thing you’ll want to do in approaching any port is to look closely at the existing source files. You don’t need to completely understand what everything does, you just need to see it and take away an impression. At this stage the style and organization of the content should pose as much interest to you as the raw technical functionality. For example, is the object hierarchy conducive to a typical Cocoa design? Is the code well-commented? Does the code exploit language specific features being that will be difficult to replicate in the target language?

In my case the code is well-written, well-commented, and since it’s Cocoa-Java, it already conforms more or less to a conventional Cocoa way of doing things. Lucky me! Nice purchase, Daniel. The only part of the project that doesn’t directly translate to Objective-C is the notion in Java of a class that “extends Thread.” There is no Objective-C class representation of a “thread,” so this would obviously take some modest reorganization. But the use of threads was not extensive, so I did not get scard off.

It doesn’t hurt that Java and C are such close siblings, any amount of porting that does need to be done will retain much of its syntactic appearance. In particular, when porting Cocoa-Java to Cocoa-ObjC, probably 80% of the work will fall into two broad categories of concern:

1. Convert syntax().like().this() to [[syntax like] this].
2. Pay attention to memory management.

The syntax point is a simplification, but really a lot of it could be ported by trained monkeys, or better yet, regular expressions. Anyway, I’ll talk more about this in a later segment, but suffice to say that the work of straight porting from Java classes to Objective-C classes can be tedious, but is eminently doable.

Build And Run

Realistically, this step was probably the first thing you did in evaluating the sources. You’ve got the Xcode project staring right back at you, how can you resist at least trying? The great news is that my product built and run flawlessly, from the minute I unpacked the source archives and installed them on my Mac. The bad news goes back to those aforementioned “subtle Intel problems.” Specifically? Crash on print. Crash on window zoom. Ouch.

The crashes in either case point to the Cocoa-Java bridge. An infuriatingly cryptic backtrace essentially lets me know that Cocoa-Java was trying to print some log message (perhaps explaining why it was about to crash?) when it bit the dust:

#0  0x90a53387 in objc_msgSend ()
#1  0xbfffe250 in ?? ()
#2  0x908103a2 in _CFStringAppendFormatAndArgumentsAux ()
#3  0x9080ec8c in _CFStringCreateWithFormatAndArgumentsAux ()
#4  0x925e2a5d in -[NSPlaceholderString initWithFormat:locale:arguments:] ()
#5  0x92604670 in -[NSString initWithFormat:arguments:] ()
#6  0x9672d308 in _BRIDGELog ()
#7  0x9673305d in _BRIDGEMethodImpStructReturn ()
#8  0x93519c6e in -[NSWindow _standardFrame] ()
#9  0x93732d7c in -[NSWindow zoom:] ()
#10 0x9335cd88 in -[NSApplication sendAction:to:from:] ()
#11 0x9335cce1 in -[NSControl sendAction:to:] ()
#12 0x9335ee91 in -[NSCell _sendActionFrom:] ()
#13 0x9335e94c in -[NSButtonCell performClick:] ()
#14 0x97723a6b in Java_com_apple_cocoa_application_NSWindow_performZoom ()

Searching the web for help on this failure was pretty fruitless. I downloaded some other applications that I knew to be Cocoa-Java based, and they didn’t crash on zoom. So what’s the deal? What is my involvement in this problem. Through hard work and luck, I narrowed both crashing problems down to a pretty simple thesis:

“Cocoa-Java will crash on Intel if Cocoa asks Java to return an NSRect.”

It might be overstating the case, but those are the two crashing cases in my application. I suspect there is a byte swapping problem or something. If you return an NSRect, you will crash. Sad story, but fixable. The two method calls in question for me are NSWindowController’s “windowWillUseStandardFrame:defaultFrame:” and NSView’s “rectForPage:”.

Keep It Running

I went to the auto parts store a few months back, around the beginning of autumn. My car had been running hot, and on the verge of overheating all through the summer. I brainstormed with the clerk about what might be wrong, and we finally agreed that it is probably a problem with my thermostat, but could be a problem with the radiator itself. Fine, it’s settled. I’ll replace the thermostat first. But even this involves draining the radiator, perhaps removing it, and a lot of other messy stuff. The shop clerk looked outside at the chilling weather and asked me a few pointed questions: “How old is this car?” “Do you drive it a lot?”, etc. After hearing my responses of “pretty old,” and “not too much,” he gave his pragmatic verdict: “It’s not gonna overheat in wintah, and who knows if it will even run in the spring?”

The point being, it’s not always the right time to do the right thing. I’m confident that that my Cocoa-Java NSRect crashes will disappear if I port the application to Objective C. But can I get away with patching the rough spots today? Moreover, won’t my customers (existing customers of this application) benefit from having the Intel problem solved sooner, rather than later? I decided to attack the problem of crashing NSRect across the bridge.

My first thought was to try to gauge exactly why it crashes. If it’s truly a byte-swapping issue, I could detect the fact that I’m running on Intel, and perhaps massage the data to suit the bridge. The major downside to an approach like this is you’re now at the mercy of Apple leaving the bug in place. In the unlikely event that Apple shows the bridge some love, and fixes the crash, where does that leave me? Now I’m byte-swapping myself into the same crash some ways down the road.

The solution I came up with is simple albeit somewhat of a hack. It’s the duct tape around the muffler, if you will. If you think about the crashing problem with returning an NSRect, an obvious solution would be for Java to not return an NSRect. To work around this problem, I modified the affected Java classes to return NSString instead of NSRect:

public String javaSafeStandardFrame(theWindow, defaultFrame)
public String javaSafeRectForPage(pageNumber)

Now the program doesn’t crash. Of course, it doesn’t zoom or print correctly, either. The problem of course is that Cocoa doesn’t know to call these “javaSafe” versions of the methods, and even if it did, it wouldn’t know to expect a string in return.

This is where the hacking comes in to play. The fact that I’m working with a Cocoa-Java project doesn’t mean I can’t also have Objective-C classes in my project. In fact, at runtime the majority of classes in memory are Objective-C, coming from Apple’s frameworks. What’s to keep me from selectively implementing some classes of my own to help soothe this problem? Using a custom class and the powerful technique of class posing, I’m able to effectively patch out standard versions of the above methods, and replace them with code that calls through to Java, gets a string representation of the rect, and converts it to NSRect for the benefit of Cocoa. Since the NSRect is now returned from Objective C, there’s no more crash. In my main.m source file:

[RectSafeWindowController poseAsClass:[NSWindowController class]];

And in my custom NSWindowController implementation:

- (NSRect)windowWillUseStandardFrame:(NSWindow *)theWindow defaultFrame:(NSRect)newFrame
{
  // Your java class instead of "JavaController"
  Class myJavaClass = NSClassFromString(@"JavaController")
  if ([self isKindOfClass:myJavaClass])
  {
    NSString* goodRectString = [self javaSafeStandardFrame:theWindow :newFrame];
    return NSRectFromString(goodRectString);
  }
  else if ([super respondsToSelector:@selector(windowWillUseStandardFrame:defaultFrame:)])
  {
    return [super windowWillUseStandardFrame:theWindow defaultFrame:newFrame];
  }
  else
  {
    // Just return the default
    return newFrame;
  }
}

A similar approach was used in the NSView override of rectForPage.

Take Stock

Now that my application builds and runs without crashing, it’s time to ship it, right? Possibly, but in this case I chose not to. Although the application has a mature feature set and benefits from years of evolutionary design, it has become a little bit dated in some ways that I grew increasingly uncomfortable with.

Now that the application is running crash free, I can breathe a sigh of relief. I could ship this thing, but I choose not to. I want to be prudent about what I change for the “1.0 relaunch,” but there are some nuances of the appearance and performance that need tweaking. And to comfortably tweak those things, I’m going to have to selectively port some of the Java classes to Objective C. But after all that hard work getting the application to “shippable,” I want to be careful not to backslide too much. Tune in next time for a detailed look at migrating Java classes to Objective C without disrupting the overall functionality of the application.

Continue reading – Step 2: Life Support.