The Software Side of Retina: Making it All Work

OS X, similar to iOS, uses points to represent display coordinates. Traditional OS/display combinations had a 1:1 mapping between points and pixels. Points in OS X are now floating point values, as a single point can be represented by multiple pixels on a high density display.


Images are the same size, but made of 4x the number of pixels on Screen 1 compared to Screen 0

How pixels map to points is determined by the backing scale factor. The backing scale factor can either be set to 1.0 or 2.0. In the case of the former you get 1:1 point to pixel mapping, while in the latter each point is backed by four pixels. The backing scale factor isn’t a global value, it can be set on a per element basis, allowing controls to to be legible while you get the benefits of a higher resolution for additional screen real estate. This aspect of OS X is key to enabling good behavior in applications as you’ll soon see.

Apple does a lot of the display handling for you so you don’t have to think about any of this. All vector based graphics and text using Apple’s APIs are automatically scaled up. Unmodified dialog boxes, toolbars, menus, etc.. should all look “normal” sized and just be ridiculously sharp on the Retina Display. Bitmapped images are scaled up using linear interpolation, but if higher resolution assets are provided OS X can simply swap and use those on a Retina Display.

Applications that render vector graphics, text and other elements to their own backing store will need hand tuning to look good on the Retina Display. These elements will receive the same linearly interpolated upscale I mentioned above.

It’s a bit complicated and confusing so let me try my best to explain what’s going on here in a practical sense. The 15.4-inch Retina Display has a native resolution of 2880 x 1800, that’s 2880 pixels across and 1800 pixels down for a total area of 5,184,000 pixels.

On the Retina MacBook Pro, Apple has done away with conventional resolution settings. Instead you get a horizontal list of scaling options (this applies to external displays as well):

In the default “best for Retina Display” setting, the desktop, menu bar, icons and Finder windows are drawn at 2880 x 1800, but they are drawn larger than they would normally be at 2880. Apple draws everything at 4x the size to make the desktop behave exactly as it would on a 15.4-inch 1440 x 900 display - this is the backing scale factor (2.0) at work. This approach provides the best image quality as there’s integer mapping from pixels on the panel to pixels on the desktop. No interpolation or filtering is necessary.


The default "Best for Retina Display" setting, 2880 x 1800 but everything is scaled by 2.0 (4x resolution)

Third party applications without specific Retina Display support also operate in this same “looks like 1440 x 900” mode. If you fire up Chrome, Photoshop or Word you’ll see that everything looks identical to how it would look on a standard resolution 15-inch MacBook Pro. Again, the screen is drawn at 2880 x 1800 but everything is scaled up to be the same size it would be at 1440 x 900.

If third party applications use Apple’s standard methods of drawing text and windows, all of these windows will look super sharp. If they don’t however, whatever routines they use to display windows and text will need to be Retina aware otherwise they run the risk of not scaling text properly. The famous example at this point is Google’s Chrome which has its own offscreen text rendering buffer, even though it uses Apple’s text rendering APIs:


Google Chrome (left) vs. Safari (right) on the rMBP

Chrome Canary fixes the text rendering issue but it has a similar problem displaying images, they simply look better in Safari:


Google Chrome Canary (left) vs. Safari (right) on the rMBP

Even though Adobe had a Retina-aware version of Photoshop running at Apple’s WWDC keynote, the publicly available version of CS6 doesn’t feature the same support. Here even open dialog boxes look bad:

Many of you asked about Office 2011. These apps just work like they would at at 1440 x 900, just with blurrier text unfortunately:

MS Excel 2011
MS Word 2011

It’s not just third party applications that need updating however, even Apple’s own iWork suite has yet to be updated to take advantage of the Retina Display. As a result text in Pages is incredibly blurry. It has been roughly three years since Apple last updated the iWork suite, so the applications are definitely due for an overhaul. I am a bit surprised Apple didn’t update them at the launch of the rMBP to be honest. It’s quite possible that a major iWork update is imminent and Apple didn’t see the need to update 3 year old software in lieu of that.

Cocoa applications can be forced to open in magnified low resolution or high resolution modes by looking at the app's info window (cmd + i on a selected app in Finder):

Eventually most apps will by default open in high resolution, such as those that have been optimized for Retina Display operation (e.g. iMovie above). Those applications that are not yet Retina aware may default to opening in low resolution mode (e.g. Adium, Pages), in which case they'll look and behave like they would at 1440 x 900 but with all UI elements upscaled to fit the 2880 x 1800 panel. Non-Cocoa applications will have the resolution scaling option greyed out (e.g. MS Office apps).

Where things get really exciting is when you have an application that not only handles scaling properly, but also takes advantage of the added resolution. Take Aperture 3.3 for example. With OS X set to its “best for Retina display” mode, this is what Aperture looks like with a 2880 x 1800 image open and displayed at full size:

Here Apple is scaling the UI elements like the menus and widgets on the screen (backing scale factor = 2.0), but displaying the open image unscaled (backing scale factor = 1.0). As a result we can fit almost an entire 2880 x 1800 image on the screen without zooming out. Remember the backing scale factor isn’t global, individual elements on the screen can be scaled independently depending on their purpose.

The same thing happens when you look at applications like iMovie or Final Cut HD. The UI elements are scaled up but the video window is displayed unscaled, thus allowing us to display a full 1080p video alongside text and tools that are still legible.

It’s all handled amazingly well. It just works.

Oh but there’s more.

If you select the 1680 x 1050 or 1920 x 1200 scaling modes, Apple actually renders the desktop at 2x the selected resolution (3360 x 2100 or 3840 x 2400, respectively), scales up the text and UI elements accordingly so they aren’t super tiny (backing scale factor = 2.0), and downscales the final image to fit on the 2880 x 1800 panel. The end result is you get a 3360 x 2100 desktop, with text and UI elements the size they would be on a 1680 x 1050 desktop, all without sacrificing much sharpness/crispness thanks to the massive supersampling. The resulting image isn’t as perfect as it would be at the default setting because you have to perform a floating point filter down to 2880 x 1800, but it’s still incredibly good.

The same rules as above apply to Retina-aware applications. Take the Aperture example again, this time at “1680 x 1050”:

Note that we can fit the entire 2880 x 1800 image at 100% almost without having to scroll. This is possible because our screen is actually rendered at 3360 x 2100, with the text and UI elements scaled up so they aren’t super tiny, yet the image is left unscaled.

Here’s the same Aperture setup but at “1920 x 1200”:

The 2880 x 1800 image looks downright small since our desktop is rendered at 3840 x 2400. Despite the fact that we’re able to fit everything into a single screen, the text and other UI elements are totally usable at this setting. You get the benefits of additional application real estate without any of the downsides.

What happens if you decide to take Aperture full screen? The image is displayed almost completely at 2880 x 1800. You do lose a little vertical real estate but not much at all.

The 3D gaming experience is even simpler. Here you just choose the appropriate resolution and you get the same scaling you normally would in the game. I’ve already demonstrated support for 2880 x 1800 in titles like Diablo 3, although there is still a need for developer support as we see with the console window in Portal 2:

The flexibility offered by Apple’s handling of the Retina Display in OS X is unparalleled. What applications like Aperture, iPhoto, iMovie and Final Cut HD offer, is unbridled resolution independence. What Apple has done here is so much more difficult than what it pulled off in iOS with the Retina Display. It will take time for third party application developers to get on board, but with the power of the Mac app store and Apple’s growing install base of Mac users I suspect we will see incredibly quick adoption of support for the MacBook Pro’s Retina Display.

The Retina Display in Numbers Achieving Retina
Comments Locked

471 Comments

View All Comments

  • Fx1 - Monday, June 25, 2012 - link

    Whine Whine Whine.

    In the UK we have the best warranty going thanks to UK Regulations and EU protection. Plus 3 year free on education discounts of which i have bought 2 MBP's and never set foot in a university in my life.

    Just because in NZ you don't have that protection then blame your gov for not passing legislation.

    Also why bitch about Ram? just put the 16GB in and then your good for the next 10 years i expect. £13 a year cost
  • Spunjji - Tuesday, June 26, 2012 - link

    You're so full of shit it's bursting out at the seams. The NZ government have imposed much better consumer warranty protection than we have here in the UK. You might know that if you weren't a fucking cretin who spouts off nonsense before thinking. The fact that you seem to think it's his fault that he lives in NZ where Apple give shitty support just shows what a jingoistic little fanboy you are.

    That's also an interesting "£13 a year" theory. It'd be neat if it wasn't a complete and utter crock. If more than 10% of these laptops are used regularly for 10 years, Apple pixie magic included, then I will eat my goddamn chair. Maybe you should have gone to university, they could have slammed some sense into your tiny little skull.
  • Solidstate89 - Sunday, June 24, 2012 - link

    "Apple’s impact on the industry has already been felt. The threat of Apple bringing Retina Displays to its entire lineup forced ASUS’ hand and gave us 1080p IPS panels in the new Zenbook Primes."

    Isn't that a little fucking arrogant to say with absolutely no proof to back that claim up? You know better than anyone how much design work goes into producing these products, and just how LONG the process takes. You really think in just a few months time of rumors, ASUS could have completely revamped their product line-up? Or maybe they were just doing it, because they had already started with the higher resolution displays on their tablet line-up? Or maybe they were even trying to match the absolutely superb and high density display that Sony has been using in their Vaio Z series for all these years.

    Claiming just the RUMOR of Apple doing it was UNDOUBTEDLY the influence behind ASUS' IPS, high resolution screens is baseless bullshit. And you know it.
  • Iketh - Sunday, June 24, 2012 - link

    what i don't get is how people could care so much to get pissed off like you... get a hobby!

    and i agree 100% with Anand's statement... you're blind to not agree, and i'm a PC developer
  • wfolta - Sunday, June 24, 2012 - link

    ASUS may have decided to carry higher resolution displays from their tablets to their laptops, but Apple beat them to the punch with the retina iPad and the rumors of a retina Macbook have been around since at least that far back.

    Considering most manufacturers have reduced pixel density over time, and considering it was Apple that pushed the retina display first to the iPhone, then the iPad, I think the burden of proof is on you to say that these netbook makers were just itching for higher resolution.

    In fact, most of them stopped at the stupid 16:9 1080p (1920x1080) resolution because that's what HD TVs have and they've wanted to tout their ability to play BluRay's at full resolution. There was no concept of making a beautiful display: it was all about a marketing checkbox. And that's what they guy is saying.
  • kmmatney - Sunday, June 24, 2012 - link

    I'm a windows developer - have been for almost 20 years now. I also travel a lot so all my development is done on a laptop, across several virtual machines. I'm currently using a 2007 Dell 17" 1920 x 1200, and have not upgraded yet, waiting for another windows 1920 x 1200 laptop to surface. I've run out of hope now - if Apple comes out with a 17" version of this I'll have to try and convince my company to buy one. I've never used a Mac in my life, but will pay the "Apple tax" to get what I need hardware-wise. I currently have about 20 VMs packed onto a Samsung 256GB 830, so I'm good with storage space.

    The Windows laptop makers just don't get it - not everyone watch's movies all day - some of us want to use our machines for real work and want to have extra vertical space. Is it that hard to make a 16:10 screen? An ugraded screen is a "3D" screen - really?! The "True-life" screen on my old 2007 Dell is still better than anything you can get nowadays - I've directly compared it with co-corkers brand-new laptops.
  • KPOM - Sunday, June 24, 2012 - link

    Is it possible that Apple's solution for making the rMBP work will scale to the new iPhone? In other words, does it offer Apple a way to increase the screen size without causing a break in compatibility between older and newer apps?
  • gstrickler - Sunday, June 24, 2012 - link

    The iPhone and iPad already handle the HiDPI displays, Apple deployed Retina displays there first. The scaling to non integer resolutions is definitely something Apple could do on the iPhone or iPad, but since it takes significantly more CPU/GPU power and affects both performance and battery life, I would not expect it anytime soon. If Apple goes to a larger screen on either device, it's likely that it will support the same total resolution as the current devices and have slightly lower dpi. If we see a 3.7"-4.3" iPhone display, I expect it to be 960x640, just like the current iPhone with Retina display.

    They already have 4 native resolutions to support on iDevices, 480x320 and 960x640 on iPhone, 1024x768 and 2048x1536 on iPad. I don't believe they'll want to increase that as it has an impact on the hardware, battery life, and every developer, so while it's possible, I think it's unlikely.
  • raclimja - Sunday, June 24, 2012 - link

    Ingadget
  • hrrmph - Sunday, June 24, 2012 - link

    This one is *extremely* close to allowing me to replace a full tower workstation and my current laptop. Apple has built a very, very impressive machine that should be a wake-up call to the industry: Move forward, or be left behind.

    Even so, it falls slightly short of my ideal specs. If Apple would make the next one a 17" or 18" version (please make it Retina, I love the foresight there :), add a 10-Key section to the keyboard, upgrade to Giga-WiFi-ac-1.3Gbs, upgrade the quantity of internal drive ports (yes plural - I can use up to 5 immediately, if they will fit), and pretty please give us a standard drive connector.

    While we're at it, can we have all of the drives running at speeds of at least 6.0Gbs? You know, like putting the drives on the current standard mSATA 6.0Gbs connectors, or even hooking them up to a better, newer, smaller, open standard connector at 12.0Gbs.

    That's a lot to ask of course, but if they 'just' managed to make it a 17" Retina display, with three drives, with standard connectors, and WiFi-ac-800Mbs, and managed to do it this year, then I would feel they have done as much as anyone could do in in the time available.

    If they miss out on delivering a 17" model this year, then it becomes more of a wide open competition where almost anyone could deliver a competitive product in the 17"+ space. After all, the display actually only has to be better than 1920 x 1200 to exceed the specs of my current laptop display.

    Right now Apple appears to have the hardware lead and a fairly big one at that. If such a 17"+ machine appears, as described in any of the scenarios above, then it would solidify Apple's hardware lead. With a move like that, Apple would firmly and aggressively cement its position in every laptop screen size category that matters. They could have a compelling position in perhaps 99% of the laptop market.

    It would also satisfy my need to have a portable large display machine that can go on infrequent, but very long trips, where I am sometimes remote, and I often need to be fairly self-sufficient and fault tolerant (ie: fix any of the inevitable data glitches myself).

    It would improve my situation very significantly if there were sufficient physical drives to have separate system, data, and auto-backup drives internally. Also, having extremely capable external ports is important. So both keeping the existing wired ones just released on the current 15" Retina model, and adding the nascent WiFi-ac wireless internal hardware that is debuting this year in non-Apple equipment, would be super helpful for enhancing my ability for doing redundant off-machine data backups - just in case the internal auto-backup gets thrashed.

    In fact if Apple were able to produce such a machine, I would be virtually forced - in a very pragmatic sense - to shell out the cash, move to the dark side, and purchase my first Apple product, ever.

    Yes, after 25 years of computing I would lose my patience with the laggards that comprise the rest of the laptop market. With deep, naughty feelings of dark misgiving, I would both splurge and switch allegiance at the same time. I wonder how many others would do the same?

    If this should come to pass, then I would inevitably wonder if Apple's real intentions are as wicked as I fear: To eventually try to lock all of us as 'sheeple', into a proprietary hell on Earth. You know... like Sony's thankfully failed attempts to force 'works only here' on us. Umm... let's see... MMC Memory Sticks, anyone?.

    Well, back to Apple: Just sayin'... I would do it... even though I know better.

    Hey... isn't Microsoft supposed to be getting back into the hardware business in a big way?

    What do you think the chances are of a company like Microsoft and a machine like that...

    So you're telling me there's a chance... *YEAH!*

    -

Log in

Don't have an account? Sign up now