Windows of Perception

February 11th, 2006

(Note: The title of this entry is an homage to Aldous Huxley’s classic Doors of Perception.)

Most Mac users and many developers don’t realize how extensively windows are used in the composition of the overall UI experience on Mac OS X. In the years since Mac OS X was first released, the basic “Quartz Window” has been increasingly used as the workhorse element in providing the glitzy UI features we take for granted. An average user looking at their desktop, with all Finder windows closed, would answer “none” to the question “how many windows are open?” But there are dozens if not hundreds of windows being displayed in part or full on a typical Mac user’s screen. They just don’t look like windows.

A fun and easy way to experiment with this phenomenon is to put the Mac’s built-in screen capture facility into “window grabber” mode.

  1. Press Cmd-Shift-4 (a shortcut whose history can be traced to FKEY resources, lightly documented in the old-ish Inside Macintosh books).
  2. Unless you’ve configured otherwise, press the space bar once to switch to “window grabber mode.” You’ll see the mouse cursor change into a clunky (definitely film, not digital) camera icon.
  3. Without clicking the mouse, simply move this camera icon around your Mac’s UI, watching as it highlights the various windows that it sees.

Move the cursor up to the menu bar and observe that most of the menu bar is a single window. While Spotlight’s icon gets its own window, all of Apple’s menu extras share a single window (within which you can cmd-drag such extras around to sort them). Any applications you’ve installed that use the NSStatusItem API will appear to the left of Apple’s menu extras, and each of these status items you will see also comes with a window.

The desktop? It’s a window. Every icon on the desktop? Windows. (Thanks to Will Koffel for pointing this out at the last Boston CocoaHeads).

With the Dock visible, cruise over to just touch its edge. Notice how it lights up? It’s a window. Every single icon in the dock is also a window, and if you bring up a dock item’s contextual menu, you’ll see that it’s a window, too. Not just contextual menus – click the menu bar and then invoke the camera icon. All menus are windows. And all submenus are windows.

While in screen capture mode, press Cmd-Tab to bring up the task switcher UI. It’s harder to observe here because the two modal tasks are competing with each other, but the task switcher UI is a window, as are each of the application icons included within it. Don’t believe me? Hold the screen capture camera over an item as you cmd-tab to it and release it. Screen grabber artifacts don’t lie!

A Top-Down Approach

When you get tired of the interactive method of browsing for the ever-present window object’s dominance of your computer, you might enjoy double-checking your work with the Quartz Debug application, installed as a default part of the Xcode tools. In my experience, even developers who have used Quartz Debug extensively are often unaware of the “Window List” feature hidden away in its “Tools” menu. There is much more than the simple “flash updates” type functionality you find in Quartz Debug’s main window. Check out the menus!

With the Window List open you can see every single Quartz window being employed by the system to deliver your smashing UI experience. This is great for simply gasping at the impressive speed with which all of these windows are (usually) being managed. It’s also a nice way to sneak up on surprising uses of windows in areas where you might not have thought to apply the screen capture trick. This window list includes all windows, whether they are currently visible or not. But if the window is visible, then selecting it in the Quartz Debug window list causes it to pulsate in a very unignorable way. Even better: the pulsing continues while Quartz Debug is in the background, so select a window in Quartz Debug and then switch to the owning application and try to figure out where the heck it is.

My curiosity was peaked just now when I noticed Omni Outliner Pro was sporting nine windows to accommodate my single open document. Two of them are somewhat obvious: I’ve got a document window and a drawer coming out of that window. Drawers are fairly commonly known to be windows. I’ve also got a single (or so I think) utility/inspector window:

(Update: While the UI above is overall very slick, notice how the jpg conversion reveals the shadow artifacts that result from sneaking several independent windows together. While this isn’t so noticeable in real-time, it’s an interesting flaw, and another example of why it’s tricky to get away with abusing the windowing system.)

This nifty little piece of UI work consists of a list of inspectors, each of which can be expanded or collapsed as the user sees fit. This is a familiar paradigm probably made popular by the use of just such a technique in the Finder’s “Get Info” window. I switched to OmniOutliner and turned on screen capture mode. Sure enough, each of those collapsible sections is its own, independent window! Well shoot howdy! I wondered what could have motivated the developers to do this. It’s true, dynamically growing and shrinking the subviews of a window and adjusting the overall window size at the same time isn’t exactly a newbie task, but I wouldn’t guess keeping the frames of several independent windows in sync and animating at the same time is either.

It became clear when I started playing around with this “Inpector Window” more carefully. I’m a pretty new user to OmniOutliner, and I don’t read many manuals. So it came as a complete surprise when I clicked and dragged one of these sections and effortlessly “tore it away” from the rest of the items. Each of the items is its own window because it truly is a window in a user sense, not just an implementation sense! As you drag these sections away from each other they take with them any sections attached beneath them. You can just as easily drag them back together and watch them snap into place. At any given time only the top-most section of a group of connected items contains a close box. This is just brilliant. I really must applaud the developers of Omni Outliner for this ingenious implementation, which provides a straight-forward single window metaphor for beginners while allowing power users to easily rearrange and compose their inspectors in a more suitable form. This is “window abuse” at its very best!

Windows, Damn Windows, and Broken Windows

Back in the old days (on the Mac), a window was a window was a window. The notion of a window to a user was the same as it was to a developer: a draggable chunk of screen real-estate, usually rectangular, usually containing a title bar, a close box, and a zoom box. A developer could go out of her way to design a window that defied the logical expectations of the user, but it would be a waste to do so, because the system (Mac Toolbox) would carry all the baggage of presuming that the window was a “real” window. You know, a “user window.”

As time has passed, and especially with the iterations of Mac OS X and the establishment of Quartz as the “base unit” of UI design on the Mac, the definition of “window” has been divided such that it means different things to different groups of people. To users, it still means “a moveable chunk of my screen with a title, a close box, etc.” To Carbon and Cocoa developers it generally means “the top-level container in which my ‘views’ are implanted.” While to Quartz developers it probably means little more than “a programatically moveable, layerable claim on screen real estate.” This lightweight Quartz interpretation of the concept makes for a much more reusable (and abusable) raw material.

At any given moment in time, the technical definition of a window may be the same or different from the user definition of a window. While they were more or less in sync in the past, we now have at least three distinct varieties:

  1. Windows – For the sake of sanity, let’s reserve the unqualified word for the highest-level concept of a window. This is the “user’s window.” You can talk about these objects in your documentation. Users know how to navigate with their mouse to the target of your description. Example: “Click the red bubble in the upper-left corner of the browser window to close the web-page document.”
  2. Damn Windows – Only a nerd could love these. They make you say, “Damn! That’s a window?” Unfortunately, you can’t call them windows to a user because they will have no idea what you are talking about. Damn windows are the bulk of what I’ve been discussing in this article: surprising uses of the low-level window object. For instance, you can’t ask a user to “move the mouse pointer to the menus window at the top of your screen, hover above the ‘File’ item and scroll down the File window to the ‘Close’ item.” In the case of OmniOutliner’s magic inspector. The five damn windows must be collectively referred to as a window, even though the developers know very well it’s not a single window! They have to bite their tongues and agree that it’s a window. The user may be asked to separate the coordinated group of damn windows into five separate, free-standing windows. Only then may they individually be referred to by the unqualified term.
  3. Broken Windows – These usually start out being classified as damn windows, but soon you learn that they violate some contract between any two of the system, developer, and user. While damn windows are good things – they provide enhanced user experience through creative use of the basic window object – broken windows are disastrous because they lead one party in the dance of window interactions to be deceived, and therefore to treat the window inappropriately.

Broken windows occur when the system or a developer attempts to use a (user level) window to achieve a tricky end, without taping up all the loose ends.

NSStatusItem’s Broken Window

An example of a broken window, and the motivation for this blog entry, is the NSWindow employed by NSStatusItem to display an application’s “menu item icon” in the upper-right corner of the Mac OS X UI. This implementation crosses the line from being a damn window to a broken window by assuming the convenience of NSWindow without responsibly managing the impact that such a decision has on unsuspecting developers and users. NSWindow is a high-level developer’s API. Although an NSWindow ultimately gets displayed by a low-level Quartz window, there are certain high-level expectations that an NSWindow in a Cocoa application must conform to.

When a developer installs an NSStatusItem into the system menu bar, they do so by either deferring to the system’s default icon view, or by installing their own custom NSView subclass to handle user interaction. This is where things must have gotten tricky for the Apple engineers. To give an application access to a piece of real estate in the upper-right corner of the screen, they needed to give it a low-level window. This means Quartz will have to cordon off a bit of space and allocate a window object. This will show up when you hover your mouse over any status items installed in your menu bar. But to facilitate easy adoption by Cocoa developers, a higher-level UI interface was needed, and NSView is the natural choice for such a thing. But NSView can’t be installed in a low-level Quartz window object. NSViews must, as a rule, be installed in other views. The exception to this rule is where the “top level view” for any particular group of UI gets connected to the lower-level window API. This conection happens by way of an NSWindow object. The NSWindow’s “content view” contains any number of NSViews, each of which may contain other NSViews. But without NSWindow, you’ve got no NSView.

So the Apple developers decided to sneak an NSWindow into the equation. While the API advertises the capacity for a developer to install an NSView in the menu bar, it says nothing of the ramifications this will have on the application’s window list, or on whether the installed status item will be vulnerable to strange attacks by virtue of its rather untruthful “status” as a full-fledged NSWindow.

As I said before, the difference between a damn window and a broken window has everything to do with maintaining contracts. If the user’s and developer’s vision of the state of “windows” in an application is never violated, then it’s a damn window. If a contract is broken, well … it’s a broken window.

By assuming the responsibilities of a Cocoa window, NSStatusItem breaks an important user contract. Cocoa windows mean something to the user. For example, AppleScript exports meaningful information about an application to the user space through System Events’ “windows” property. A user can ask for a particular window by index, where a lower index represents a “more frontmost” window. It’s entirely reasonable for users to expect that the focused document of an application can be referred to like this:

tell application "Adium"
	get window 1
end 

I’m running Adium right now, with a single chat window open and activated. Adium is configured to show its NSStatusItem-based window in the menu bar. The above script, instead of returning the focused chat window, returns the sneaky little NSStatusItem window! So a user who relies on the functionality of AppleScript now has to know that “for Cocoa applications with NSStatusItems installed, the secret status item window is window 1.” That’s total BS. This window is broken.

NSStatusItem also breaks a developer contract. It would be one thing to grimace and accept that control over a status item’s window might fall into the hands of the user, but once there, it must be bullet-proofed against harmful manipulation. I don’t really care how NSStatusItem implements its functionality, so long as it doesn’t destabilize my application.

By hiding the NSWindow implementation from me (the developer) but exposing it to the user (through AppleScript), the status item sets developers up to “play the fool” on behalf of any user script that ends up manipulating the wrong window. See, these fake little NSWindows are capable of doing the darned’est things. Apparently Apple didn’t realize just how much manipulation might be attempted through AppleScript. If I run the following script:

tell application "Adium"
	set zoomed of window 1 to true
end 

I end up with this little beauty of a UI artifact:

I’m pretty sure the Adium developers had no idea they had to guard against such disturbing UI manipulation. I’m using Adium as a convenient demonstration that this affects any scriptable Cocoa app with an NSStatusItem, but obviously my biggest concerns lie in my own products. So I follow the above script up with a similar tweak:

tell application "FastScripts"
	set visible of window 1 to false
end 

Which yields an altogether comically destroyed status bar:

To Apple’s credit, after just a little bit of mouse movement, the status items seem to regain their posture. Visually, at least. In the case of setting an application’s status bar window’s visibility to false, it ironically fixes the visibility but somehow in the process causes the corresponding NSView to be unclickable. In order to resume clickability of my FastScripts icon, I have to go back to AppleScript and manually set the visibility back to true. C’est la vie!

I suppose some of you might be thinking that these violations are minor because they only affect a relatively advanced group of users (AppleScript authors), but the fact that these flaws are exposed through AppleScript doesn’t mean they won’t also rear their ugly heads elsewhere. The problem is a careless abuse of the technology of windows objects to achieve goals that are contrary to the user’s vision of what a window is. Every time you choose to do this, you must consider carefully what the impact will be on the developers and users who rely on your software. In this case, AppleScript reveals at least two obvious problems with the NSStatusItem implementation. Let’s hope there aren’t more to be discovered!

22 Responses to “Windows of Perception”

  1. bill Says:

    Daniel,

    Thanks, it’s insightful.

    If we look at iCal, the violations are quite serious, there’re 8 windows even when iCal is just activated.

    name of windows of application “iCal”
    — {“iCal”, “”, “iSync Alert”, missing value, “”, “”, “”, “”}

  2. Anon Says:

    Hmm… perhaps the Quartz windows should have been called “frames” or some such thing… and NSWindow should be factored into two classes – one that implements user window concepts, and one (NSQuartzFrame?) that just displays NSViews and forwards events up the responder chain… (NSWindow would then own and manage a NSQuartzFrame that displays all of the window’s views).

    BTW – you did file a bug report on this, didn’t you?

  3. Mike Henley Says:

    I’ve learned the hard way to never expect a hard-coded index number to actually work with any kind of reliability in AppleScript. I read with interest your description of the NSStatusItem. The first question that comes to mind is… why? Why must developers birth these funky outcasts from the SystemUIServer anyway? From an AppleScript perspective, I think it makes sense to just address SystemUI when dealing with menu extras and not have to bother monkeying around in every app that owns an NSStatusItem. Of course SystemUI is just as bad if not worse at indexing its elements — complicated by the fact that none of my extras actually have names. A few have unique values for the AXDescription attribute, however. I say, the menubar is a real mess, isn’t it?

  4. cjwl Says:

    Interesting post, thanks. QuartzDebug’s “Show Window List” will show you all the windows in an app, and there is a Cocoa default NSShowAllWindows, which if you set to YES will put off-screen windows on screen, so you can see their contents.

  5. Justin Says:

    If I am not mistaken the lines that represent connections made between outlets, actions and instances et al in Interface Builder are also windows.

  6. alexr Says:

    And of course, we’ll notice that since all the “menus” in the menu bar aren’t the same mechanism, another fundamental UI contract is broken: we cannot click on a menu title and drag across the menu bar to drop down every menu — only the ones using the same mechanism react.

    Try dropping down the Airport menu and then getting the Adium NSStatusItem to react. Mousing down in the NSStatusItem will cause the MenuExtras to appear, but not the other way around. Of course, Spotlight doesn’t react at all.

    The same problem holds for dropping down a regular menu and heading over to this dumping ground on the right end of the menu bar.

    (And because we must always say this — this worked in Mac OS 9 just fine, but we cancelled that.)

  7. Michael Tsai - Blog - Broken Windows Says:

    […] Daniel Jalkut: […]

  8. Ben Says:

    Another side effect of this non-standard menu implementation affects Dashboard. When Dashboard is in the front, clicking on the desktop or any menu should dismiss Dashboard. However, if you click on one of those status menus on the right of the menu bar, Dashboard stays open and status menu draws over the Dashboard layer.

    I should submit a bug on that, come to think of it.

  9. Axel Says:

    Ben’s observations confirmed. But with a little twist. On this machine (10.4.5 Server), left-clicking into the SystemUIServer window closes dashboard. Right-clicking, then moving the mouse, opens, as seen by Ben, menu extras’ menus above the dashboard layer, which must be an implementation error. This just cannot be designed this way.

    And, what baffles me most, because I couldn’t really repeat it, at first just mousing around in the top right area made those menus appear. I have GMail notifier as well as Salling Clicker running with their NSStatusItem menus on. Something is really, really broken here. I wish, Apple would get enough pressure to get their up right architecture right, *sigh*.

    Those other inconsistencies you’ve seen, like you can’t have dropping menus following you mouse are annoying enough alone: click on salling’s menu item, you get a menu, move the mouse over to Gmail notifier, no menus on screen, continue to move into that same direction into SystemUIServer, menus reappear. This is REALLY broken, I think, from a UI consistency perspective. Plus, on my screen, BOTH applications’ NSStatusItems icons are not centered to their blue highlights area. Can this be by chance? Finally, I can reorder my SystemUIServer menus, but not my NSStatusItems. How much sense does it make?

    If it would help, I would try to find time to continue investigating under different OS versions and different amounts of NSStatusItems installed. But I think, bug reports alone can’t fix this, since the design is so flawed. I have completely forgotten, why Apple needs to make SystemUIServer private to the system, but, man, NSStatusItems either belong there or off the screen.

    Other things are broken since long, too. My mom used to have such a blurry iMac G3, she needed to run with 800 pixels across. And guess what, over several major versions of Mac OS X, if she does just that, updating the SystemUIServer is totally broken. Visually, the positions of the menus seem to JUMP AROUND or sometimes even appear TWICE. Together with really long Applications menus like Word’s, even menus I kept fixing to the very right disappear. Once you click into the region, everything updates fine. Now tell a 65 year old to click on something that appears to be something else… None of this happens if you change resolution to 1024. Crap. She got a 20″ iMac now, so, these problems should be finally gone, but this is far away from being software engineering.

    So, I would like to help, have a bit of an angle, but I fear sending bug reports alone won’t let Apple redesign the whole thing to make it right once again. Feel free to contact me at axel.behr AT gmail.com

  10. Neill Says:

    Gosh, what elegant design! I wonder when Microsoft Windows will catch up with this slick new object-oriented window system.

  11. Daniel Jalkut Says:

    Interesting points everybody has been bringing up. It’s true that the NSStatusItem/MenuExtra architecture is broken, but let’s not let that topic dominate this post too much :)

    Neill: Do you care to elaborate on your point? As it stands it sounds like a mutant mix of comment spam and flame bait. But if I had to guess I would say you are suggesting that Microsoft has been implementing their UI with windows for ages, so this is nothing new. Well, if you look at the content of my article you’ll see that I’m not exactly celebrating this implementation detail – I’m merely observing and commenting on the ramifications. If you’re really just trying to stir up Microsoft vs. Apple debate here, then please find another place to target your comments.

  12. CM Harrington Says:

    Anything with it’s own Quartz buffer is considered a “window”. Because the entire UI is double buffered, every discrete element is considered a window. Every icon on your desktop is another buffer, and therefore a “window”. It’s rather common knowledge that if you want to speed up general Finder usage, especially on start-up, get rid of all your icons on your desktop.

  13. Matt Says:

    This is why they really need to get rid of NSStatusItem and just go back to NSMenuExtras. Stupid things like not being able to command-drag and not reacting like the rest of the menu bar.

  14. Robert Says:

    Axel: I think that perhaps the strange right-click behavior in Dashboard may be a feature, not a bug. This behavior is the only way to allow users to change language/keyboard settings while in Dashboard. Still a bit odd, of course.

  15. tonyarnold.com Says:

    Damn windows (NSWindows that is)

    Don’t get me started on the weird things Apple are doing to the desktop windows (or as Daniel would call it, a “Damn window”). Try coding a virtual desktop manager when Apple sets windows at certain NSWindowLevels to be sticky (i.e. seen across all ‘de…

  16. Sune Says:

    This is no news. It’s very wordy – DUH – and there is no news. Things like are known to developers everywhere. BFD.

    There is a difference between systems like OS X and Windows. On the latter, literally EVERYTHING is a window; but on OS X, things can be ‘controls’ or ‘windows’. So you won’t see everything that’s running.

    And of course turning on Quartz debug on the command line shows you all the hidden windows on a per-application basis.

    But again, this is no news and should not be treated as such – and if it’s treated, it should be treated in a more professional and dignified manner.

    Nothing here folks – maybe a snake oil salesman, but surely nothing else. Everybody go home now. Let’s keep order on our neighbourhood streets.

  17. Daniel Jalkut Says:

    Well Sune – that’s pretty rude. I admit my inclination was just to delete your comment, but I suppose I’ll let you show off your inappropriate behavior. Don’t bother responding or writing comments on any other of my entries if you’re going to assume that tone. I will not be so hesitant to delete future comments in which I’m described as a “snake oil salesman.”

  18. Adolph Trudeau Says:

    UIElementInspector.app is a very helpful tool in identifying windows for use in AppleScript UI scripting. Or rather, I almost found it useful. I forgot what I was working on that caused me to learn about it, but I do remember that I haven’t finished whatever it was, so it wasn’t magically helpful.

  19. debm Says:

    Great stuff… I knew that all of it was windows but I had no way to prove it to the programmers who are new to Mac that we are talking about windows in a different parlance. I can show that now.

    BTw, Did you notice that the shadow for the menu bar is also a window?

  20. Mac-arena the Bored Zo Says:

    It’s true, dynamically growing and shrinking the subviews of a window and adjusting the overall window size at the same time isn’t exactly a newbie task, but I wouldn’t guess keeping the frames of several independent windows in sync and animating at the same time is either.

    this can be done rather easily with NSWindow’s “child windows” feature. it probably took some extra hacking by Omni to get inspector windows to be owned and disowned as they’re moved near other inspectors, but the actual lockstep facet of the feature was already done in AppKit.

  21. Daniel Jalkut Says:

    Adolph: I agree the UIElementInspector app is cool for poking around. Another cool app (though not free) that may help you get to the “finished point” with your next attempt is the PreFab UI Browser.

    Deb: Thanks – glad you enjoyed it. I hadn’t noticed the shadow being its own window, but using the screen capture trick I was just able to easily confirm it! Thanks for pointing that out.

    Bored: Haven’t really looked into child windows but I will have to follow up on that. Anything that makes fireworks like these even a tad easier is worth knowing about.

  22. Programaticamente Inceptor » Blog Archive » Windows of Perception. Says:

    […] I’m sorry! It’s an old link… But it’s worth it, specially, if you come from the Mac OS Classic Good Times… <G> ( edited for Rick) when Finder Windows were Finder Windows or so I…, Well… I’m not a coder but I can figure out how much things have changed… […]

Comments are closed.

Follow the Conversation

Stay up-to-date by subscribing to the Comments RSS Feed for this entry.