Cocoa

[Note: This is a guest blog post written by Thomas Zoechling (@weichsel on twitter), a macOS developer based in Austria, about his Mac app Claquette]

Claquette Mac App Icon

Animated GIF is not a good video format.
Actually it even isn’t a proper video format because it lacks random access and audio support. Nonetheless animated GIFs experienced a rennaisance in recent years.
My theory is, that the format’s success doesn’t stem from the features it has, but from the ones it lacks:

  • No random access: Defaults to autoplay – No playback interaction needed
  • No sound: Guarantees silence – Always “Safe for Work”
  • No support for 32bit colors: Moderate file sizes and a characteristic look

Given those constraints, GIFs are a great way to communicate simple concepts or interactions.
Want to showcase an animation or an application feature? Instead of a static screenshot with paragraphs of text, a 3 second GIF works equally well and also draws more attention.

In my daily life as software developer, I often need a quick way to capture graphical bugs. Those clips can be easily shared with colleagues or included with bug reports.
Sometimes it’s also simpler to attach a GIF to customer support requests instead of explaining how a certain feature works.
To cover my own needs, I wrote a small macOS application that allows me to record the screen and export the result as animated GIF. The app uses a custom recording codec and also covers the rest of the GIF creation workflow like crop, trim and file size optimization.

You can download Claquette from the Mac App Store. For detailed product information, please visit our website.

Development

When I started to implement the Animated GIF feature for Claquette, I began with a naïve approach.
Armed with several years of experience in writing Objective-C and some knowledge about video APIs on the Mac, I wrote a small prototype.
That first version just read a video file frame by frame and sent the resulting frames to ImageIO. ImageIO is a system framework that supports reading and writing several file formats.
It also comes with a basic Animated GIF encoder and so I decided to skip any third party libraries and just use the built-in mechanisms of macOS.
I was able to come up with a working prototype in a single afternoon. The program was just a simple command line utility, but it was able to turn an arbitrary video file into an animated GIF.

There was just one problem… Or actually there were several of them: Due to the inner workings of ImageIO, the program used vast amounts of memory. Also, the encoding took very long and the files it created were huge. On top of all that, the resulting GIFs looked horrible.
So while it only took me one Sunday afternoon to create a working prototype, it took me several months to fix the above issues. Especially the large file size and the bad visual appearance of the resulting GIFs required a lot of research.

Getting the most out of a 30 year old file format

The original GIF specification (GIF87a) was written in 1987 – almost 30 years ago. Animation features were added in GIF89a, which is still the most recent version of the format.
So how is it possible that a file format designed for low resolution screens and 256 colors is still in use today?
It turns out that the GIF specification contains some sections that open room for exploitation. Additionally, the visual nature of GIFs allows for optimizations to trick human color perception.

The format is simple and has the following basic structure:

  1. Header
  2. Logical Screen Descriptor
  3. Global Color Table
  4. Graphic Control Extension (Frame #1)
    – Delay
    – Disposal Method
    – Local Color Table
    – Image Descriptor
    – Image Data
  5. Graphic Control Extensions (Frame #2)
  6. Graphic Control Extension (Frame #3)
  7. … (1 GCE for each animation frame)
  8. Trailer

Header and Trailer are basically just magic numbers that mark the start and the end of the file. The Logical Screen Descriptor contains some general image information like width, height and background color. The Global Color Table is a simple list of colors that may contain a maximum of 256 entries.
Main image information is stored in one or more Graphic Control Extension blocks.

Color Table Generation

The color table sections of the format specification are a good starting point to optimize the visual quality of an animated GIF.
Both palettes are restricted by the format’s infamous 256 color limit. When reducing an image that uses 32bit (16.777.216 colors, 256 alpha values) to 8bit (255 colors, 1 alpha bit) it becomes really important which colors you leave out. The process of reducing large palettes to small ones is called Color Quantization. Choosing a good quantization algorithm is crucial when aiming for visually similar images with a reduced palette.
Naïve quantization implementations are based on occurrence, where seldom used colors are left out in the final image. More elaborate algorithms use techniques like dimensional clustering or tree partitioning.

When developing for Apple devices there are several libraries that provide color quantization functionality. macOS and iOS even have basic color quantization algorithms built-in. Apple’s implementation is part of the ImageIO framework’s CGImageDestination API.

The following sample images were created using different quantization techniques. They illustrate the quality impact of the used algorithm on the final image.

CGImageDestination Quantization

The first image shows the output of CGImageDestination. The resulting colors are noticeably off. Apple’s encoder also messes up the transparency color in the GIF palette, which leads to holes in some areas of the image (e.g. the titlebar).

FFmpeg Quantization

The open source library FFmpeg also includes a color quantizer. FFmpeg produces way better results than CGImageDestination. The colors are more vibrant and the transparency color is set correctly.

Claquette Quantization

The color quantization library used by Claquette also outputs a good choice of colors. Additionally the app performs color matching to avoid color shifts and to guarantee correct gamma values.

Frame Difference Calculation

Another important factor during GIF generation is efficient frame difference calculation.
The disposal mode in the Graphic Control Extension allows an encoder to specify how the context is set up before the next frame gets rendered.
GIF89a defines 4 disposal modes:

  • Unspecified: Replaces the existing canvas with the full contents of the current frame.
  • Do not Dispose: Leaves the existing canvas as-is and composites the current (sub)frame over it.
  • Restore to Background: Sets a defined background color and draws the current frame. Areas outside of the subsection in the Image Descriptor shine through.
  • Restore to Previous: Fully restores the canvas to the last frame that did not specify a disposal method.

The Image Descriptor section can be used to define a sub-image which does not provide pixel data for a full frame. Instead it contains coordinates and pixel data for a subsection of the full image.
By using frame differences and sub-images with a proper disposal mode, redundant image data can be avoided. Depending on the nature of the input video, this can greatly reduce the file size of the final GIF.
Modern video codecs like H.264 use techniques like macro blocks and motion compensation. Those methods introduce small errors that propagate from frame to frame. Propagated errors show up as noise when calculating frame diffs and eventually lead to unnecessary large files.
Claquette uses a custom lossless codec, that only stores regions that actually changed. This guarantees exact frame diffs.

The following images show the difference between frame #1 and frame #2 of a screen recording. The only effective change between those frames is a change in the mouse cursor location. An exact diff should therefore only contain an offsetted cursor image.

FFmpeg Diff

The above diff image was created between 2 frames of an H.264 encoded movie. The visible noise is a result of intra-frame encoding errors.

Claquette Diff

The second image was created by Claquette’s image diff algorithm. It only contains the mouse cursor – The only image element that actually changed between frame #1 and #2.

Finishing Touches

After implementing technical details like encoding and optimization, there were still some features missing. Claquette needed an editing UI to handle the whole GIF creation workflow.
As I wanted to keep the app lightweight and simple to use, I decided to add only a small set of editing capabilities: Trim and Crop.
Claquette uses AVFoundation, and therefore I was able to use the AVPlayer class, which comes with a built-in trim tool.
Crop functionality was a bit harder to implement. AVFoundation doesn’t provide any UI component to display crop handles so I had to implement my own.
Besides the standard drag & move interface, my implementation also provides some unique features. It offers alignment guides with haptic feedback and the possibility to enter crop coordinates.

You can see the final implementation of the crop utility in the animated GIF below:

ClaquetteCrop

Launch

To launch the animated GIF feature, I prepared a press kit and wrote mails to review sites that mostly cover Mac software.
Additionally I submitted the app to Product Hunt and informed Apple’s App Store Marketing team.
I can really recommend to launch on Product Hunt: Claquette received over 400 upvotes and finished in the top 3 on launch day. The site also hosts a friendly community, that makes extensive use of the Q&A section below each hunted product.
I was also lucky to get some mentions from high profile Twitter users and good App Store reviews.
Two weeks after launch, Apple suddenly moved Claquette into the “New and Noteworthy” feature section on the Mac App Store front page. Of course this also lead to a noticeable spike in the sales charts.
Overall I was very pleased with the release and the reception of Claquette 1.5.

—
Thomas Zoechling is an independent software developer living in Vienna, Austria.
Besides working on his own products, he is also doing contract work. Currently he is helping IdeasOnCanvas to develop the excellent mind mapping software MindNode (macOS, iOS).

Read more

I like to be prepared for worst-case-scenarios.
And when I found out my Mac app ScreenFloat didn’t work anymore on Apple’s upcoming macOS Sierra because of a new sandbox restriction (you can read the backstory here), I knew fixing it could have gone one of two ways:

  1. Apple fixes it for me in a new beta of macOS Sierra (which, as I now know, is what happened in the form of a new sandbox entitlement), or
  2. I’d have to write my own solution for creating screenshots, basically reimplementing macOS’ screencapture command line utility

At the time, I didn’t know Apple would provide a new sandbox entitlement, so for me, the only choice was to take a couple of days and reimplement macOS’ screencapture‘s functionality.
Now, when I say “reimplement”, I mean I looked at the features I needed in ScreenFloat and implemented them, leaving those I didn’t need aside (fullscreen screenshots, screenshot sounds or capturing windows’ shadows, to name a few).
Time was of the essence, after all, and I didn’t know how long it would take me to implement this stuff.

Deconstructing screencapture

Before I started working on my own solution, I thought it’d be good to understand how macOS’ screencapture utility was implemented.
I executed ‘strings /usr/sbin/screencapture’ in Terminal thinking I could find a clue as to how capturing the screen was done, but all I could find were references to private APIs, like CGSGetScreenRectForWindow or CGSGetWindowLevel:

…
kCGSMovementParent
CGSGetScreenRectForWindow failed: %d
CGSGetWindowLevel failed: %d
…

I could not find any references as to how capturing the screen is done, but I suspect the CGDisplay* APIs are used, and ‘nm /usr/sbin/screencapture’ seems to confirm that theory:

…
U _CGDisplayBounds
U _CGDisplayCreateImage
U _CGDisplayCreateImageForRect
…

Next, I wanted to know how screencapture draws its selection rectangle and cursor.
Knowing there’s basically no drawing on screen without an NSWindow, I created a small app that would filter out screencapture‘s windows during an interactive screenshot, create an image of each and write them to disk.
In doing so, I learned the following:

  • screencapture uses 5 windows to display its selection rect: 4 for the edges and 1 for the fill
  • These windows are present even if you’re not currently making a selection (albeit transparent), and they follow around your mouse cursor
  • The selection cursor isn’t drawn in its own window, it’s an ordinary NSCursor, using private APIs to set it
screencapture CLI's windows during selection

The windows surprised me.
Why would you need 5 windows to draw something you could draw in one, using Core Animation or an NSView?

screencapture has existed since Mac OS X Jaguar (10.2), and there was no Core Animation framework yet, so that’s out of the running.
That leaves NSView’s -drawRect:. Why not use that? Frankly, I don’t know. But I suspect it’s a performance thing – perhaps drawing 5 individual single-colored windows was faster on Mac OS X Jaguar (and still might be on today’s macOS) than one NSView’s -drawRect: and they just kept going with it over the years.
A friend and indie-colleague of mine, Andreas Monitzer (@anlumo1 on twitter) confirmed my suspicions: “Single-colored windows don’t need -drawRect:, and that’s probably just way more efficient.”

Also interesting is that screencapture adds specific Spotlight metadata tags to the screenshot files it creates:

  • kMDItemIsScreenCapture – a boolean value indicating whether the image file is a screenshot (YES). Only present when the file is a screenshot, so there’s no case where NO would be specified – the tag would be missing instead.
  • kMDItemScreenCaptureType – the type of screenshot: “display” for a fullscreen screenshot, “window” for a window-selected screenshot and “selection” for an interactive screenshot.
  • kMDItemScreenCaptureGlobalRect – Not really a rectangular value. As far as I can tell, it only contains the interactive screenshot’s origin point’s x value (where on the screen the screenshot was taken).

To set the cursor, I suspect there’s some private API magic at play.
An app can set its cursor in different ways: via cursor rectangles or directly via NSCursor. But it only works if the app is active and its window is frontmost – something that isn’t the case with screencapture.
‘nm’ reveals the private API CGSRegisterCursorWithImages, and several more:

U _CGSCreateRegisteredCursorImage
U _CGSGetCurrentCursorLocation
U _CGSHardwareCursorActive
U _CGSHideCursor
U _CGSRegisterCursorWithImages
U _CGSSetSystemDefinedCursor
U _CGSShowCursor
U _CoreCursorSet

So much goodness I’m not able to use in a Mac App Store app, just because it’s hidden away in a private API…

screencapture also does another thing I was interested in: When you press ⌘-⇧-4, followed by the space bar, you can move your mouse cursor over individual windows to screenshot them exclusively, and it’ll tint it to let you know about the selection.
I have no idea how it’s done – it’s either a window that gets painted over the selected window, inserted at the correct hierarchy level, or it’s a private API that lets you paint over any NSWindow.

Behind-Window-Window-Selection

Now I was ready to begin

Reimplementing screencapture

In reimplementing the features I needed, I had to hit several milestones:

Milestone #1: Actually capturing the screen somehow

Because I was using screencapture via NSTask to do all the heavy lifting for me, I had no idea where to start for creating user-selected screenshots.
I started with giving AVFoundation a try, as I remembered a couple of WWDC sessions mentioning capturing the screen – for video. Soon enough, though, issues popped up.
Like the image’s compression. A screenshot created with AVFoundation wasn’t anywhere near the quality a screencapture screenshot has – although it was to be expected, since it’s for videos, and videos are heavily compressed.
When text is involved, it’s especially painful:

Comparison of AVFoundation capturing and screencapture's outputLeft: AVFoundation’s output. Right: screencapture‘s

There are different quality settings you can try to play with to improve the shot a little, but it’ll never come close to anything screencapture produces. That’s just unacceptable.
After poking around screencapture, I learned Apple provides APIs for exactly this purpose: CGDisplayCreateImage, to capture an entire screen, and CGDisplayCreateImageForRect, for a manual selection, which is what I was interested in.

#Milestone 2: Drawing into a completely transparent NSWindow

For NSWindow to react to mouse events, it needs to have a colored background with an transparency value of at least 0.05. That sounds very low, but it’s still very noticeable when it’s suddenly put over your screen. You may not be able to pinpoint it, but you know something happened.

A white window with an alpha value of 0.05Left: The desktop. Right: The desktop below a white window with an alpha value of 0.05

That’s why I’m very grateful to Nick Moore, indie developer of PilotMoon fame.
He discovered that you can have a completely transparent NSWindow accept mouse events, by setting its contentView’s layer’s contents to a transparent NSImage.

#Milestone 2: Selection Drawing

With that out of the way, I could move on to actually drawing a selection.
For what the selection would look like, I didn’t have to think much. I wanted to keep it consistent with screencapture: white borders with a white-transparent fill.
But the APIs I’d use to draw the selection were up to me.
Since I didn’t want to use 5 windows like screencapture does, I briefly experimented with NSView’s -drawRect:, but discarded that in favor of something more modern and more performant: CAShapeLayer.
In my tests, it’s looks and feels just as the original, and even if it’s not, it’s indistinguishable to my eyes (on a retina MacBook Pro Mid-2012).

The selection’s functionality would have to be the same as screencapture‘s – drag to select, keep the space bar pressed to move the selection, release the space bar to continue the selection, press the space bar once to enter window-selection mode, press it again to exit again. Nothing that couldn’t be done with NSResponder’s ordinary methods.

Moving a selection with the space barMy reimplementation’s behavior when using the spacebar to move the selection.

Milestone #3: Custom Selection- and Window Capturing

Because of CGDisplayCreateImageForRect, creating a screenshot of a custom selection is fairly straightforward.
Window capturing is a little more work. You *could* do it with CGDisplayCreateImageForRect, but you’d have remnants beneath the window’s rounded corners of anything that was below it at the time of the capture.
It’s good to know Apple provides an API for those cases as well, then: CGWindowListCreateImage. It will let you define the window you’d like to capture and the “features” it should have (shadows, no shadows, include windows below it, don’t include them, etc).
You can get a reference to the window you’d like to capture using CGWindowListCopyWindowInfo – it’ll give you its rect on screen, its window level and more information about it.

Milestone #4: Window Selection Drawing

That last API, CGWindowListCopyWindowInfo, also comes in handy when drawing the window selection (for when you hit the spacebar).
After all, you need to know where all the windows are and their dimensions on the screen, so you can draw the selection accordingly.
Once I have that information, it’s easy to put up another CAShapeLayer above the selected window.
But wait. What if the selected window is beneath another window, or several windows?

Faulty Window SelectionA first try at implementing window selection, clearly failing for windows that are beneath other windows.

That’s where it gets tricky and why I assume Apple is using private APIs here, which lets it either insert a new, selection-color-colored, translucent window into the window hierarchy at the right position or draw directly onto the window.

My solution was to use another CAShapeLayer.
It’s based on CGPath (which I convert to from an NSBezierPath) to draw the colored overlay.
I use the results from CGWindowListCopyWindowInfo to find out which windows are atop the currently selected window, create an NSBezierPath from their bounds, subtract them from my initial NSBezierPath and feed that to the CAShapeLayer.
It works pretty well:

Working Window Selection

It didn’t work like this right away, though. There was a lot of trial and error, sweat and, yes, tears involved in this. But I think it was worth the effort. Doesn’t it look just like the original?

Milestone #5: Drawing the Cursor

The cursor in screencapture has a unique feature: it displays the coordinates next to it, or if a selection is being made, the dimensions (width and height) of that selection.
I wanted the very same thing, so custom drawing was necessary.

I started out using NSCursor, but for every mouse move I’d need to create a new NSImage with the coordinates (or dimensions) and set it as the cursor – that seemed pretty inefficient to me.
I then moved on to using part NSCursor, part CATextLayer. The NSCursor part would be the crosshair, a constant image, set-it-and-forget-it.
The text layer would be updated in -mouseMoved:, updating its position to be next to the cursor and its contents to reflect the cursor position or dimensions.
Sure enough, it worked, but when moving the mouse cursor around fast, the text layer would “lag behind”, not correctly sticking to the cursor. It’s nothing major, but it bothered me.
With this, we come to my final approach.
I hide the system cursor entirely and draw both the crosshair and the text using CALayers. Since they both are now updated inside the same -mouseMoved: (or -mouseDragged:) call, there’s no noticeable “lag” for the text layer – they now move together nicely, as if drawn in one NSCursor object.

Milestone #5: Focus without focus

The tricky thing about screencapture is that it has mouse and keyboard focus without it taking you out of the app you’re in (the window’s close, minimize and fullscreen buttons are not greyed out, for example).
Again, in my desire to be consistent with the Apple-provided command line utility, I needed the very same thing.

It’s difficult to get certain NSResponder method calls if your window is not key, and it prompted another trial and error session to get the right combination of -isKeyWindow, -acceptsFirstResponder, etc.
It now works, but it’s not as nice as Apple’s implementation.
Which brings us to the Caveats section.

Caveats of my implementation

Caveat #1: “Across-Screens Screenshots”

With screencapture, you can make a selection that spans several displays, due to the way it’s implemented (using NSWindows to draw the selection).
With my implementation, that’s not possible, as I put up one transparent window for each screen that’s connected to your Mac. But I settled – I think it’s very rare you’d make that sort of a selection.

Caveat #2: The Cursor

In some cases, the system cursor will pop up again, and it won’t go away until a new screenshot session is started.
This is the hideous result:

The system cursor drawn above the custom cursor

Caveat #3: Capturing Fullscreen Windows

When you click the green fullscreen button on a window, it transitions from being one window into being two windows without you knowing about it – one for the the titlebar/toolbar (which moves down a little when you move your mouse to the top edge of your screen to reveal the menu bar), and one for the actual window.
screencapture is somehow aware of this, and when you do a window selection, it will properly draw its selection rectangle above the entire window.
My implementation doesn’t know about fullscreen windows and treats those two windows separately:

My implementation's fullscreen window selection bugInstead of the entire window, just the titlebar and its shadow are selected.

I haven’t found a solution to this, yet.

Caveat #4: Exposé

When you start Exposé to show all open windows, screencapture can be used to screenshoot them individually.
My implementation falls short of that, as Exposé seems to be a semi-modal mode where other windows can not be moved over it.
Of course, screencapture can.

Pros of my implementation

Apart from these caveats, I also see a couple of upsides to having your own implementation:

Pro #1: Control over the UX

With a custom implementation, I can change anything I want at any time, be it any cursor icon or the behavior in general. I don’t have that luxury with Apple’s built-in tool.

Pro #1 and a half: Timed Shots

Speaking of UX:
Yes, Apple’s built-in tool features timed shots (a screenshot that is not created immediately, but after a small delay). But I think it’s clumsily implemented.
When you start a timed shot:

  • You don’t know how much time there’s left until the shot is taken
  • You can’t click anything beneath the selection rectangle

With a custom implementation, I can provide a better experience here.

Pro #2: Screenshot placement

Something that concerns ScreenFloat exclusively is the placement of the floating shot after taking it.
Using screencapture, I just centred the shot at the mouse cursor. It works, but it’s not very nice.
With a custom implementation, I know exactly where the screenshot was taken and can place it accordingly:

Screenshot Appearance

or I can do a little animation to make it more clear that a screenshot was created:

Screenshot Appearance Animation

Pro #3: Sandbox

With a custom implementation, I don’t need to worry about temporary entitlements – it works without any.

Open Source?

I’m planning on making the source available at some point, but before I do, there’s still a couple of things I need to do, like implement timed shots, which I haven’t gotten around to yet.
Also, so I don’t make a fool of myself, I need to clean up the code, and that’s dependent on the free time I get, which recently is little (I’m not complaining – I love being busy).

Anyway, this has been quite the journey, and if I can manage to fix some of the caveats I described above, I might use my own implementation instead of Apple’s built-in screencapture CLI in ScreenFloat at some point. But for now, I’m happy I’m again able to use Apple’s built-in tool.

Eternal Storms Software Logo– – – Do you enjoy my blog and/or my software? – – –
Stay up-to-date on all things Eternal Storms Software and join my low-frequency newsletter (one mail a month at most).
Thank you 🙂

Read more

In my app ScreenFloat, I use the command line tool screencapture via NSTask to create screenshots. On OS X El Capitan and earlier versions of Apple’s operating system, this worked perfectly fine. Now, on macOS Sierra, I’ve been getting reports that screenshot creation didn’t work anymore, so I investigated.

At first I thought it might be the keyboard shortcut API that has undergone some changes, but that doesn’t seem to be the case, as I saw something actually occurred when I pressed the keyboard shortcut – Xcode’s console printed:

screencapture: cannot run two interactive screen captures at a time

Since I was absolutely sure I’m not launching screencapture via NSTasktwice, I took to Console.app to see if anything unusual was reported there. And there it was:

Sandbox Violation of ScreenFloat on macOS SierraThe output in Console.app when trying to launch an interactive screencapture with NSTask.

deny mach-register

So registering a global Mach service is denied on macOS Sierra. In the back of my mind, I remembered a temporary exception entitlement, but it wasn’t quite the same – com.apple.security.temporary-exception.mach-lookup.global-name. I tried adding it to ScreenFloat’s entitlements file, with com.apple.screencapture.interactive as its value (this temporary entitlement expects an array of string values), but that didn’t help – the same denial and console output occurred. On a hunch, I tried using …mach-register.global-name instead of …mach-lookup.global-name and – tada – it worked!

So I’m all set, right? Well…

Temporary Exception Entitlements

Apple offers a couple of temporary exception entitlements. They may or may not be granted to your app during Apple’s review process. But going through the list, it’s clear that …mach-register.global-name is nowhere to be found, so it’s kind of a private entitlement – which makes it even less likely for it to be granted to your app.

Digging Further

Seeing as the sandbox denial points explicitly to com.apple.screencapture.interactive, not just com.apple.screencapture generally, I tried creating a non-interactive screencapture session with NSTask. To my surprise, it worked – without the entitlement.

So I tried a different command line utility – which. (which will return the executable path to the given command line utility, for example, which screencapture would return /usr/sbin/screencapture). Again, it worked. And again, without the entitlement.

It makes me believe (and hope) that the behavior we see for com.apple.screencapture.interactive is not desired, so I’ve filed a bug report with Apple in the hopes that they can set the record straight soon.

For now, I hope ScreenFloat will be granted the temporary entitlement just so it is functional again on macOS Sierra for the time being. However, if this is in fact the desired behavior, I will have to write my own screencapture utility so ScreenFloat can remain on the Mac App Store.

Bug Reporting

For anyone who’s interested or in a position to view it, here’s the bug report I’ve filed with Apple: rdar://27610157. I do hope to get an answer soon.

Update August 2nd, 2016

As I stated above, com.apple.security.temporary-exception.mach-register.global-name isn’t documented anywhere. Which is also the reason you get an error when trying to submit an app with such an entitlement to iTunes Connect:

ERROR ITMS-90285

So, no dice on the temporary exception. Having to write my own screenshot utility seems more and more likely. I hope I can make it in time for macOS Sierra.

Update September 9th, 2016

The temporary exception is now valid and will go through to Apple’s App Review without a hitch. My own solution is not necessary at this time, but I’m still going to be working on it – you never know.

 

Eternal Storms Software Logo

– – – Do you enjoy my blog and/or my software? – – –

Stay up-to-date on all things Eternal Storms Software and join my low-frequency newsletter (one mail a month at most). Thank you 🙂

Read more

Yoink 3.2 comes with a handful of new features. In this post, I will … NOT tell you about those. Instead, I’d like to focus on what changed under the hood.
I think it can be interesting to reflect on how things change from release to release and what improvements were made. And how.

I’ll write about performance improvements and perceived speed-ups in the app and how I managed to improve Yoink’s handling of resources within the OS X sandbox to be able to hold more files than it was previously able to.

Performance Improvements and Speed-Ups

Let’s begin with low-hanging fruit, shall we?

I improved the responsiveness and perceived performance of Yoink in a couple of subtle, but nonetheless important places:

Make “Clear All” Faster

This is the perfect case of “put work on another thread to make the app seem faster”.

Files that don’t exist on the disk yet (like images from websites, text snippets from documents, a file from an FTP client, etc.), Yoink has to create and therefor, when they’re removed from Yoink, they need to be deleted from disk as to not waste space.

For this, I have a method named -prepareForRemoval. It checks if the file was created  by Yoink itself and deletes the file from the disk.

Now, the way “Clear All” works is fairly straight forward:

  1. Iterate over each item in Yoink
  2. Check if the file is currently pinned in Yoink. If it is, skip it; if it isn’t, continue with
  3. Call -prepareForRemoval that cleans up any files that we don’t need anymore
  4. Update the tableView
There’s a few things I’ve improved here:
For starters, for removing files out of the array that powers the tableView, I had used another array that held the items to be removed and after the loop was done, removed the files from the main array.
This I’ve replaced with -filterUsingPredicate:. It’s not only much cleaner to read, but also more performant and using less resources (getting rid of the superfluous array and the second loop for iterating over it for removal).
Secondly, I took step #3 (calling -prepareForRemoval for each item) out of the filtering method and put it on a background thread that gets executed concurrently with everything else, making the filtering much faster.
It doesn’t make any difference for just a few files, but once there are more than 20 files in Yoink and they’re all cleaned out at once, you can definitely begin to notice it.
It’s almost instant now, no matter how many items there are in Yoink.

Make Splitting Up Stacks Faster

A Stack in Yoink is an item that encapsulates multiple files, created by dropping more than one file to Yoink in one single file-drag. The Stack can be split up so individual files can be dragged out of it.

A Stack in YoinkA Stack in Yoink

Splitting up works fine, but again, if a Stack holds more than just a few files, there might be slow-downs. I can’t entirely get rid of those because of the work that needs to be done, but I can speed up the process.

It used to be a simple for-loop, iterating over the files in the Stack and creating a new item in Yoink for each one.
Now, the method uses concurrent enumeration, by calling
[array enumerateObjectsWithOptions:NSEnumerationConcurrent usingBlock:^(aBlock)].
It makes the system create as many threads as it sees fit under the current system load and configuration and loops over the array concurrently, making splitting up larger Stacks noticeably faster.

Accepting a Drop

Accepting a drag in Yoink

Because of changes equivalent to brain surgery (which I’ll discuss below), I had to speed up accepting files in Yoink, otherwise it would be perceived as slower than the previous version, and that’s unacceptable.

Again, NSEnumerationConcurrent to the rescue. It can do wonders.

On the other hand, sometimes it can be slower than an ordinary for-loop, because there’s the overhead of the thread-creation that goes on behind the scenes.
For example, the equivalent to NSEnumerationConcurrent for sorting is NSSortConcurrent. If you use it for something very simple, though, like comparing two strings, it doesn’t do much good. For more complex comparing, it might speed up your method. Another example, NSEnumerationConcurrent would not have given me any speed-ups before the brain-surgery in Yoink.

Removing Sandbox Limitations in Yoink (a.k.a Brain Surgery)

In older versions of Yoink, the app would start behaving incorrectly once a certain amount of files were added to it over the course of the app’s session lifetime. It’s a direct result of how the sandbox hands NSURLs to your app.

Files can be added to an app via the Open dialog (known inside the sandbox as Powerbox) or via drag and drop – which is what Yoink is doing.
These files come in the form of NSURLs. In the sandbox, NSURLs that hold references to files can’t just be used nilly-willy, access to them has to be started (-startAccessingSecurityScopedResource) and stopped (-stopAccessingSecurityScopedResource).
When a file is added through the powerbox or a drag, the system automatically calls -startAccessing(…) for that NSURL for you before it is passed to you in code (without telling you. Also, it causes problems: see Addendum #1).

The documentation has the following to say about -startAccessing(…):

If you fail to relinquish your access to file-system resources when you no longer need them, your app leaks kernel resources. If sufficient kernel resources are leaked, your app loses its ability to add file-system locations to its sandbox, such as via Powerbox or security-scoped bookmarks, until relaunched.

So, -startAccessing(…) and -stopAccessing(…) have to be balanced, otherwise resources are leaked and no further files can be handled by the app – which is exactly what caused earlier versions of Yoink to misbehave once a certain amount of files have been added to it, because I didn’t balance the -startAccessing(…) call the system made before handing the NSURL to me.

The obvious solution, then, is to indeed call -stopAccessing(…) once you receive the NSURL and call -startAccessing(…) once you really need to access the file. But there’s where things go downhill. Once you stop accessing an NSURL, the sandbox believes you never had access to it in the first place, and any subsequent call to -startAccessing(…) fails. Unless…

NSURL’s Security-Scoped Bookmarks (the actual Brain Surgery Part)

Unless you create a security-scoped bookmark of the NSURL first and then call -stopAccessing(…). In versions prior to 3.2, I created bookmarks only when the app quit, in order to keep access to files over relaunches and restarts.
While the app was running, though, I only handed around and worked with NSURLs (with the limitation that once ~1.500 files were added to it, the app would seize to work properly and needed a restart).

This is what I do now, to allow for many, many more files than ~1.500 to be added during the app’s run:
The first thing I do when I receive an NSURL from the drop is create a bookmark from it. Subsequently, I call -stopAccessing(…) (important: without first calling -startAccessing(…), as the system has already done that for me at that point) and the resource is freed – the app is now in a state just like right from launch – as if no file had been added yet.

But now I have an NSData object where an NSURL object used to be – pretty incompatible. That’s where the brain surgery came in. It sounds trivial, but switching from NSURL to the bookmark and making sure the resource is accessed in all the right places (and not unnecessarily) and also cleaned up was an unexpected amount of work.

But long story short, the brain surgery worked, and the patient is up and well. No spasms. No drool.

Eternal Storms Software Logo

– – – Do you enjoy my blog and/or my software? – – –

Stay up-to-date on all things Eternal Storms Software and join my low-frequency newsletter (one mail a month at most).
Thank you 🙂

Resource Usage Improvements

In the second part of this post, I’d like to explain how I was able to reduce not only Yoink v3.2’s memory footprint, but its overall resource usage.

In Yoink, I want to be able to reflect changes in filenames and be aware of file deletion – it’s a better user experience when, if a file is renamed in Finder, that file also updates its filename inside Yoink.
And there’s not much use in having a file in Yoink that’s been deleted a while ago.

Watching Files for Renames

For this, I use dispatch_sources to watch NSURLs.

GCD provides a suite of dispatch sources—interfaces for monitoring (low-level system objects such as Unix descriptors, Mach ports, Unix signals, VFS nodes, and so forth) for activity.

I want all the information I can get right from the start – probably a habit I picked up during the development of flickery, where flickr counts the number of API requests and once you hit a certain limit, you’re cut off.
So up until now, I created a dispatch_source for every single file that was added to Yoink – better safe than sorry, right?

Well, wrong! dispatch_sources are limited as well, as you need to call open() on the files you’d like to watch. The limit lies around 2.500 files (which, if the NSURL-issue I described above hadn’t already limited the amount of files Yoink could accept, would limit the amount of files Yoink could accept).

No. I needed another, saner and resource-friendlier approach.
Sane and resource-friendly? Bye-bye, polling. You’re out of the race. Good riddance.
Now, three facts are painfully obvious, in the case of Yoink. There are three promising places for improvement, as I don’t actually need to watch:

  • … files that were created by Yoink (images from websites, text snippets, any files that haven’t actually been written to disk yet…)
  • … files that are scrolled out of view in Yoink
  • … files that are encapsulated in a Stack

Files that were created by Yoink don’t need to be watched for name-changes

Text Snippet in Yoink

Files Yoink creates itself are inside Yoink’s sandbox container (at /Users/name/Library/Containers/) and managed by Yoink. No need to watch those.
Text snippets especially – their names are created based on their contents.

Files that are scrolled out of view don’t need to be watched for name-changes

Out of view files in Yoink

Imagine this: You have 9 files in Yoink, separately. You have it set up to show 3 at a time. Currently, you see files 4, 5 and 6. You definitely want to be notified when the name of any of these three files changes in Finder, as the user can currently see them in Yoink, so the changes need to be reflected right away.
But what about files 1, 2, 3 and 7, 8, 9? Do they need to be updated if the user can’t see them right now, anyway?

No!

We watch only visible files, and already we’ve reduced the files to watch for name changes from 9 to 3, saving ~67% of valuable resources – which also means more performance (no extra code execution for those 6 files) and less memory usage. (API side-note: In an NSTableView, you can get the currently visible rows by passing the rect you get from -visibleRect to -rowsInRect:).

If a user scrolls, though, the visible files change and the watched files might now be out of view, being completely useless.
Due to this, Yoink needs a mechanism to update which files need to be watched for filename changes. They are updated when:

  • files are added
  • files are dragged out or removed
  • the window’s size changes
  • Stacks are split up
  • a scroll ends
The update mechanism discards the dispatch_sources for previously visible files and sets up watching the newly visible files.

Files that are in a Stack don’t need to be watched for name-changes

Stacks in Yoink

Yoink displays Stacks like you see in the picture above. An icon, with the number of files inside the Stack. There’s actually no need to watch those files, as file name changes wouldn’t be reflected anyway.
This has “opportunity” for resource-savings written all over it.

When to actually update filenames for files that aren’t being watched

So, when are those filenames updated?
Files that are currently not visible might be visible later because the user scrolls to them or files are removed and dragged out.
Files in Stacks might be exposed by splitting them up.
There needs to be some way to update those filenames, even if they’re not actually being watched by the app right now.

Files in a Stack are actually easy – the filenames are created as soon as the user decides to split the Stack up, as Yoink gets the NSURLLocalizedName for each file.

Files currently not visible require a bit more work, but nothing too difficult.
OS X thankfully provides notifications for when a scroll occurs. Depending on the version of OS X you’re targeting, there’s either NSViewBoundsDidChangeNotification (OS X 10.8 and earlier) or the more fine-grained NSScrollViewWillStartLiveScrollNotification, NSScrollViewDidLiveScrollNotification and NSScrollViewDidEndLiveScrollNotification (OS X 10.9 and newer). For a caveat regarding “legacy” mice, see Addendum #2.

When the *WillStart* notification is sent, I start updating all filenames in Yoink in a background thread (all filenames, because I don’t know where the scroll is going to end).
Once *DidEnd* is called, I update the files to be watched with my usual routine of getting the -visibleRect. Then I update the filenames for the now visible files, just in case the background-thread is still running and hasn’t come to those files yet.

Watching For File Deletion

A warning message when a file residing in Yoink is moved to the Trash in FinderYoink 3.2 displays a warning once a session when a fie is moved to the Trash.

Watching for files being moved to the trash or deleted altogether is another beast entirely.
For this alone, I’d have to watch every single file – being in a stack, not currently visible, etc. -, effectively subverting every single improvement I had hoped for above.
There had to be a better way.

Luckily for me, there is.

I had the idea that, if a file or folder could be watched, the Trash itself should be watchable too, meaning that instead of having to watch every single file if it was moved to the trash or deleted, I’d only have to watch one Trash folder for each volume attached to the Mac (because external volumes have their own Trashes).

Since dispatch_source is my go-to-API for watching files (and I have experience with it), I set up a test-app that watched all Trash(es) folders on the Mac. Surprisingly, it worked only for the internal volume, external volume’s Trashes folders couldn’t be watched. Busted.

FSEvents to the Rescue

Just when it looked like all hope was lost, I remembered the FSEvents API:

[The FSEvents] API provides a mechanism to notify clients about directories they ought to re-scan in order to keep their internal data structures up-to-date with respect to the true state of the file system. (For example, when files or directories are created, modified, or removed.)

It was worth giving it a shot, and – again, surprisingly – watching external volume’s trashes for changes worked this time around. Success.
Now I can tell whether a file in Yoink was moved to the Trash or deleted, without having to set up a dispatch_source for every file and without having to do any polling.

In Closing

Having had code in Yoink that could render the app useless after a couple of thousands files had been added makes me uncomfortable and is pretty embarrassing to admit, but I’m glad I got rid of it.
I’m also very happy with the improvements made with the upcoming update.
In many places, Yoink feels much more responsive because of them.

Addenda

Addendum #1

An NSURL you receive from the powerbox or a drag’n’drop operation already has started access to the resource it points to. This causes the problem that file drags with more than ~1.500 files behave incorrectly (your mileage may vary). The first ~1.500 files are added just fine, but suddenly you start seeing something like this in your logs:

Yoink[74545]: Consume sandbox extension for itemIdentifier (9035) with path: /some/file/path from pasteboard failed!

And worse, the NSURLs you receive beyond that point are useless, you can’t access the files they point to.
My understanding is that this would be a non-issue if the system didn’t automatically call -startAccessing(…) before it hands the NSURLs to you and lets you decide when to startAccessing the NSURLs.
As far as I can tell, this is an issue that can not be worked around at this time, and I’ve tried a couple of things. I hope Apple is working on it. One should believe since 10.7, they’ve had enough time.

Addendum #2

A legacy mouse, as in, a mouse that has a traditional scroll wheel, will only call NSScrollViewDidLiveScrollNotification, not *WillStart* or *DidEnd*.
I handle that in Yoink by calling the update method delayed, and canceling the delayed call should another *DidLiveScroll* notification occur in the meantime, this way, I “simulate” *WillStart* and *DidEnd*:

[NSRunLoop cancelPreviousPerformRequestsWithTarget:self selector:@selector(scrollingDidEnd:) object:nil];
[self performSelector:@selector(scrollingDidEnd:) withObject:nil afterDelay:0.15];

Eternal Storms Software Logo

– – – Do you enjoy my blog and/or my software? – – –

Stay up-to-date on all things Eternal Storms Software and join my low-frequency newsletter (one mail a month at most).
Thank you 🙂

Read more