The Software Side of Retina: Making it All Work

OS X, similar to iOS, uses points to represent display coordinates. Traditional OS/display combinations had a 1:1 mapping between points and pixels. Points in OS X are now floating point values, as a single point can be represented by multiple pixels on a high density display.


Images are the same size, but made of 4x the number of pixels on Screen 1 compared to Screen 0

How pixels map to points is determined by the backing scale factor. The backing scale factor can either be set to 1.0 or 2.0. In the case of the former you get 1:1 point to pixel mapping, while in the latter each point is backed by four pixels. The backing scale factor isn’t a global value, it can be set on a per element basis, allowing controls to to be legible while you get the benefits of a higher resolution for additional screen real estate. This aspect of OS X is key to enabling good behavior in applications as you’ll soon see.

Apple does a lot of the display handling for you so you don’t have to think about any of this. All vector based graphics and text using Apple’s APIs are automatically scaled up. Unmodified dialog boxes, toolbars, menus, etc.. should all look “normal” sized and just be ridiculously sharp on the Retina Display. Bitmapped images are scaled up using linear interpolation, but if higher resolution assets are provided OS X can simply swap and use those on a Retina Display.

Applications that render vector graphics, text and other elements to their own backing store will need hand tuning to look good on the Retina Display. These elements will receive the same linearly interpolated upscale I mentioned above.

It’s a bit complicated and confusing so let me try my best to explain what’s going on here in a practical sense. The 15.4-inch Retina Display has a native resolution of 2880 x 1800, that’s 2880 pixels across and 1800 pixels down for a total area of 5,184,000 pixels.

On the Retina MacBook Pro, Apple has done away with conventional resolution settings. Instead you get a horizontal list of scaling options (this applies to external displays as well):

In the default “best for Retina Display” setting, the desktop, menu bar, icons and Finder windows are drawn at 2880 x 1800, but they are drawn larger than they would normally be at 2880. Apple draws everything at 4x the size to make the desktop behave exactly as it would on a 15.4-inch 1440 x 900 display - this is the backing scale factor (2.0) at work. This approach provides the best image quality as there’s integer mapping from pixels on the panel to pixels on the desktop. No interpolation or filtering is necessary.


The default "Best for Retina Display" setting, 2880 x 1800 but everything is scaled by 2.0 (4x resolution)

Third party applications without specific Retina Display support also operate in this same “looks like 1440 x 900” mode. If you fire up Chrome, Photoshop or Word you’ll see that everything looks identical to how it would look on a standard resolution 15-inch MacBook Pro. Again, the screen is drawn at 2880 x 1800 but everything is scaled up to be the same size it would be at 1440 x 900.

If third party applications use Apple’s standard methods of drawing text and windows, all of these windows will look super sharp. If they don’t however, whatever routines they use to display windows and text will need to be Retina aware otherwise they run the risk of not scaling text properly. The famous example at this point is Google’s Chrome which has its own offscreen text rendering buffer, even though it uses Apple’s text rendering APIs:


Google Chrome (left) vs. Safari (right) on the rMBP

Chrome Canary fixes the text rendering issue but it has a similar problem displaying images, they simply look better in Safari:


Google Chrome Canary (left) vs. Safari (right) on the rMBP

Even though Adobe had a Retina-aware version of Photoshop running at Apple’s WWDC keynote, the publicly available version of CS6 doesn’t feature the same support. Here even open dialog boxes look bad:

Many of you asked about Office 2011. These apps just work like they would at at 1440 x 900, just with blurrier text unfortunately:

MS Excel 2011
MS Word 2011

It’s not just third party applications that need updating however, even Apple’s own iWork suite has yet to be updated to take advantage of the Retina Display. As a result text in Pages is incredibly blurry. It has been roughly three years since Apple last updated the iWork suite, so the applications are definitely due for an overhaul. I am a bit surprised Apple didn’t update them at the launch of the rMBP to be honest. It’s quite possible that a major iWork update is imminent and Apple didn’t see the need to update 3 year old software in lieu of that.

Cocoa applications can be forced to open in magnified low resolution or high resolution modes by looking at the app's info window (cmd + i on a selected app in Finder):

Eventually most apps will by default open in high resolution, such as those that have been optimized for Retina Display operation (e.g. iMovie above). Those applications that are not yet Retina aware may default to opening in low resolution mode (e.g. Adium, Pages), in which case they'll look and behave like they would at 1440 x 900 but with all UI elements upscaled to fit the 2880 x 1800 panel. Non-Cocoa applications will have the resolution scaling option greyed out (e.g. MS Office apps).

Where things get really exciting is when you have an application that not only handles scaling properly, but also takes advantage of the added resolution. Take Aperture 3.3 for example. With OS X set to its “best for Retina display” mode, this is what Aperture looks like with a 2880 x 1800 image open and displayed at full size:

Here Apple is scaling the UI elements like the menus and widgets on the screen (backing scale factor = 2.0), but displaying the open image unscaled (backing scale factor = 1.0). As a result we can fit almost an entire 2880 x 1800 image on the screen without zooming out. Remember the backing scale factor isn’t global, individual elements on the screen can be scaled independently depending on their purpose.

The same thing happens when you look at applications like iMovie or Final Cut HD. The UI elements are scaled up but the video window is displayed unscaled, thus allowing us to display a full 1080p video alongside text and tools that are still legible.

It’s all handled amazingly well. It just works.

Oh but there’s more.

If you select the 1680 x 1050 or 1920 x 1200 scaling modes, Apple actually renders the desktop at 2x the selected resolution (3360 x 2100 or 3840 x 2400, respectively), scales up the text and UI elements accordingly so they aren’t super tiny (backing scale factor = 2.0), and downscales the final image to fit on the 2880 x 1800 panel. The end result is you get a 3360 x 2100 desktop, with text and UI elements the size they would be on a 1680 x 1050 desktop, all without sacrificing much sharpness/crispness thanks to the massive supersampling. The resulting image isn’t as perfect as it would be at the default setting because you have to perform a floating point filter down to 2880 x 1800, but it’s still incredibly good.

The same rules as above apply to Retina-aware applications. Take the Aperture example again, this time at “1680 x 1050”:

Note that we can fit the entire 2880 x 1800 image at 100% almost without having to scroll. This is possible because our screen is actually rendered at 3360 x 2100, with the text and UI elements scaled up so they aren’t super tiny, yet the image is left unscaled.

Here’s the same Aperture setup but at “1920 x 1200”:

The 2880 x 1800 image looks downright small since our desktop is rendered at 3840 x 2400. Despite the fact that we’re able to fit everything into a single screen, the text and other UI elements are totally usable at this setting. You get the benefits of additional application real estate without any of the downsides.

What happens if you decide to take Aperture full screen? The image is displayed almost completely at 2880 x 1800. You do lose a little vertical real estate but not much at all.

The 3D gaming experience is even simpler. Here you just choose the appropriate resolution and you get the same scaling you normally would in the game. I’ve already demonstrated support for 2880 x 1800 in titles like Diablo 3, although there is still a need for developer support as we see with the console window in Portal 2:

The flexibility offered by Apple’s handling of the Retina Display in OS X is unparalleled. What applications like Aperture, iPhoto, iMovie and Final Cut HD offer, is unbridled resolution independence. What Apple has done here is so much more difficult than what it pulled off in iOS with the Retina Display. It will take time for third party application developers to get on board, but with the power of the Mac app store and Apple’s growing install base of Mac users I suspect we will see incredibly quick adoption of support for the MacBook Pro’s Retina Display.

The Retina Display in Numbers Achieving Retina
POST A COMMENT

470 Comments

View All Comments

  • OCedHrt - Sunday, June 24, 2012 - link

    He missed another important point. All of that was in 3 lbs. Now, the current generation starting from last summer has an external discrete graphics and optical drive connected via a thunderbolt based connector (because Apple had exclusivity) with the laptop being only 2.5 lbs.

    This isn't going to compare to the retina macbook pro though - at 15 inches 4.5 lbs is pretty impressive though I think if Sony wanted to do it they could do 4 lbs or less.
    Reply
  • deathdemon89 - Saturday, June 23, 2012 - link

    I agree completely, I'm a proud owner of the old Z, and even today it doesn't feel the least bit dated. And the 1080p screen is holding up admirably well, with no signs of pixellation at normal viewing distances. This device was truly innovative for its time. I still don't understand why it received such mixed reviews by the press. Reply
  • Spunjji - Monday, June 25, 2012 - link

    Mainly the price. Only Apple are allowed to charge that much for a laptop. Also, only Apple can have hot systems. Repeat ad infinitum. Reply
  • mlambert890 - Wednesday, November 28, 2012 - link

    Really ridiculous comment. I can see you are bitter, as is the other mega z fan, but come on already. I worked for Sony for 5 years and love the company. I have owned probably a dozen Vaios including the top of the line last gen Z (with the SSD RAID)

    Instead of ranting and raving you need to ask yourself *why it is* that "only Apple can charge so much" and why "Anand only gives a free pass to Apple"

    You feel what exactly? That there is some grand conspiracy in play? Do you realize how ridiculous that sounds?

    WHY has Sony *lost the ability to charge a premium*? In other words WHY have they *burned all customer loyalty and good will*? I left the company back in 1999 because I saw the writing on the wall.

    You (and the other Z guy) are no different than any other apologist. Companies dont bleed marketshare and fail to sell cancer curing products (you guys are presenting the Z as "truly revolutionary" right?) for no reason. Sorry to tell you there is no "big conspiracy".

    Sony sells super high priced products into a super commoditized market and then they layer on a CRAP TON of bloatware dragging the machine to a stop, make idiot decisions like the HDMI one, and push proprietary crap *worse* than Apple ever has. All of that into the Wintel space which, sorry to tell you, was *always* driven by the cheapest possible parts at the best possible price.

    The PC industry grew *because it was cheap*. Apple *always* occupied a premium niche. I vividly remember the initial release of the Apple I, the Lisa, the Mac 128. These were all always premium products and the competition at the time (be it Commodore, Atari, Ti, or the wintel ecosystem) *always* captured share by being cheap.

    That may annoy you for some reason, but Apple has pretty much *always* held a core premium audience. The only exception was the period of insanity when Jobs had been pushed out and Scully destroyed the company. Even then, the core fans stayed.

    You two make it sound like poor Sony is a victim because the world doesnt all run out and by the Vaio Z.

    Even *without Apple* Sony would be going under, hate to tell you. Sony's problems are Sony's and the whole is *not* the sum of its parts with a laptop.
    Reply
  • solipsism - Saturday, June 23, 2012 - link

    None of that makes sense and is, in fact, rubbish.

    Sony added 1080p because it was it was popular not because it made sense. You have a 168 PPI display on 13" machine which makes text too small to be a good experience for most users.

    They also didn't use a good quality display or add anything to the SW to make the experience good (unlike what Anand talked about in this review), they just added the single metric that was trending because of HDTVs.

    Blu-ray in a notebook has always been a silly option for most users. There is a reason the BRD adoption failed on PCs and it's not because everyone is stupid... except you. ODDs are long overdue for being removed since they take up 25% of the chassis, require them to placed at an edge reducing over 5" of port real estate and restricting design, require a lot of power, are noisy, more prone to break due to the many moving parts, are slow, are just too expensive to be feasible, and add nothing visually that most users trying to watch a movie can discern.

    Quad-SSDs? Really? That's a sensible solution for a consumer notebook?
    Reply
  • EnzoFX - Saturday, June 23, 2012 - link

    and that really is what people don't get. It isn't just about raw specs. The package needs to be complete, polished, what have you. In this case of high dpi screens, is good scaling support, and Apple did it. Support on the software side is something they never get credit for by the Apple haters. All they can see is numbers and think "I've seen numbers like that before". Reply
  • mabellon - Saturday, June 23, 2012 - link

    No Apple didn't do it. Just like on the iPad, they increased resolution by doubling width and height. Their software simply doesn't scale well to arbitrary higher resolution. If it was done right then Chrome would work out of the box - instead the OS 2x scales everything without increasing resolution/quality.

    To the consumer, the choice means a good experience without breaking apps. But claiming that Apple was successful simply bc of software? HA!
    Reply
  • Ohhmaagawd - Saturday, June 23, 2012 - link

    Did you actually read the retina part of the review?

    Chrome doesn't work right because they do their own text rendering. Read the review. If an app uses the native text rendering, the app will look good (at least the text portion). They will have to update the graphical assets of course.

    BTW, Chrome Dev builds have this issue fixed.

    Windows DPI setting isn't default, so few use or even know about the setting and devs haven't made sure they work properly in the high DPI settings.

    Apple has made a move that will be short-term painful in having apps that aren't updated look a bit fuzzy. But since they made it default, this will force devs to update.
    Reply
  • OCedHrt - Sunday, June 24, 2012 - link

    What do you mean Windows DPI setting isn't default? You can change it in a few clicks, but the same thing applies - if your app does not read the DPI values, then Windows can't help you. This is because windows UI is not vector based (I don't know about now, but older apps definitely not) and many applications back then were coded with hard coded pixel counts.

    When the DPI is changed, windows scales the text but the UI dimensions is controlled by the application implementation.
    Reply
  • KitsuneKnight - Saturday, July 07, 2012 - link

    On Windows, changing the DPI will generally mean a huge amount of applications will become simply unusable.

    On this Retina MBP, the worst case appears to be slightly blurry text (which was quickly updated).

    Apple's solution is a good one, because it does things in a way that should keep existing apps working fine, while allowing future developers to leverage new APIs to take advantage of the increased resolution.
    Reply

Log in

Don't have an account? Sign up now