[Development Journal: ScreenFloat 2] Entry 3 – Busy Doing Nothing* – Eternal Storms Software

ScreenFloat lets you keep visual references to anything you see on your screen floating above other windows using screenshots. It’s also a screenshot organizer.

I’m now working on ScreenFloat 2, and I thought it would be fun to chronicle my progress, struggles, successes, failures and break-throughs, as well as random stuff while developing it.

Disclaimer: Estimated Time of Arrival, Pricing

I don’t do ETAs for my own products.
I’m a solo developer, I have multiple apps that need maintenance and updates, there are just too many moving parts for me to be able to estimate basically anything. And while that may be a serious lack of managerial skill: I accept that flaw and ignore it 🤷‍♂️.

Regarding pricing, I don’t know what ScreenFloat 2 will cost yet. But I am resolved on its upgrade path: existing customers of ScreenFloat 1 will receive ScreenFloat 2 for free.

Entry 3 – Busy Doing Nothing*

I feel like I got nothing done in regards to ScreenFloat 2 over the last two weeks. But it’s not because I didn’t work on it. It’s much rather that I’m not quite sure if what I worked on will make it into ScreenFloat at all. It was all a bunch of UI experiments and refinements. *Refining something that might not make it into the final product does sometimes sadly feel like doing nothing.

Getting Shots to Float

After having finished the migration from the old library format of ScreenFloat 1 to ScreenFloat 2’s new Core Data-based library, the logical next step for me was to get started on making those shots float.
This functionality will pretty much work the same way it did in v1: a floating shot will float above other windows, apps and spaces, to always be visible whatever you do.

– Low-hanging fruit

I made a much needed change to the way I render screenshots in these floating windows. Whereas in v1, I used to set NSWindow‘s backgroundColor to a pattern color of the image, I now use an NSImageView, like a sane person. I have no clue why I used to do it like that before.
Perhaps I couldn’t figure out how to make a window move by its background with an NSImageView on top (see NSView’s mouseDownCanMoveWindow)?
Maybe it was just “easier” or quicker than adding an NSImageView to the window in code?
Maybe it was left over from early prototyping and I never refactored it, because it worked well enough?
Whichever you think it is, rest assured that they all do sound like something I would do. If you ask me, the answer is: “all of the above”.

Another important performance improvement I made is that, when a floating shot is hidden (not closed), instead of only reducing its alphaValue to 0.0 (lazy me), I now call orderOut on the window, effectively removing it from the list of windows the WindowServer has to manage. And though the window is still in memory and can be re-shown any time (at which point I’ll order it back in and animate its alphaValue to what it was before hiding it), while it’s hidden, it doesn’t affect performance as much as a hidden floating shot in v1 does. And the more hidden floating shots you have, the more you’d notice.

^{Preliminary hiding and re-showing of a shot}

Resizing Shots

Floating shots can already be resized in v1 of ScreenFloat. Just like any other window, you can grab one of its edges and drag it to resize it.
Version 2 will provide feedback during resizing: it’ll show the shot’s size as a percentage, and in absolute values:

^{Resizing a floating shot}

I also removed the upper limit, which was 200% in ScreenFloat 1. It snaps to 100% when you’re close to it, which can be toggled off by keeping the command (⌘) key pressed.

Shot Transparency

ScreenFloat 2 will continue to allow you to make a floating shot transparent by scrolling up or down within it, with a couple of improvements.
1) It will remember the transparency value over restarts of the app, or when hiding/closing and re-floating a shot.
2) It provides feedback while changing the transparency.
3) For convenience, the opacity can be changed all the way down to 0%, and when the scrolling ends (you lift your finger from the trackpad), it bounces back up to a minimum value of 40%.

^{Changing the transparency of a floating shot}

– Introducing: The Periodic Table of NSVisualEffectView

About “Busy Doing Nothing”: this is part of that “nothing”.

While working on that feedback panel that comes up when resizing a shot, or changing its transparency (see above), I needed an overview of the different appearances an NSVisualEffectView can have.
During testing, I discovered that NSVisualEffectView accepts material values from 0-37, of which only a few are documented.
With that in mind, this sample app shows 152 NSVisualEffectViews. 76 light, 76 dark, each consisting of 38 vibrant and 38 non-vibrant variants. Some of them look like they produce duplicate results, but all I needed was a brute-force way of showing all variants at once for comparison, so I didn’t bother filtering out anything.

You can download the source code here, if you’d like to play around with it yourself.
By default, it uses your desktop image as its background, but can be changed to a basic color easily – just follow the instructions in the code’s comments.

I’ve only tested it on macOS 12 Monterey for now, so it might not work on earlier versions of macOS because of the undocumented material usage. Alas, backwards-compatibility is something I’ll get to a bit later in ScreenFloat 2’s development, as I don’t want to restrict myself from the get-go from which APIs I am able to use.

Floating Shots UI

Most of the time I spent on ScreenFloat 2 in the last two weeks was on the floating shot’s UI – and even though I’m pretty happy with it now, I’m still not sure if it’ll make it into the final product.

Here’s what the UI looks like in ScreenFloat 1 when you move your mouse over a floating shot:

The top left button closes or deletes a shot, the top right button gives access to various functionality, and the bottom left image-file-icon allows you to drag a file-representation of the shot to other apps.

My goal for ScreenFloat 2 is to not overlay too much stuff over the screenshot itself, while at the same time offering access to more functionality.
Here’s what that looks like right now:

I like it because it keeps the floating shot itself clean, and gives all sorts of options to the user.
It can also be extended, like a bottom bar, for example.
And while it looks simple enough, implementing it was quite a journey. And painful, at times.
My first attempt at implementing it was to extend the actual floating shot’s window. I’d extend the NSLayoutConstraints at all edges and animate-in the additional UI. It worked, but it moved the floating shot out of place, a tiny bit every time. I’d counter that with another animation of the shot’s window’s frame to keep it in place, but that made things even more janky. Fail.
What you see in the video above is my second, better-but-more-complex approach.
The additional UI that appears is a separate, second window, placed underneath the floating shot window. This way, when the UI appears, the shot keeps in place perfectly still. In fact, the shot’s window’s frame isn’t affected by it at all. So far so good, but what about moving the window? What about resizing it? What about adjusting the transparency? That’s where the pain starts.

The “additional UI” window is a child of the floating shots window, so that when you move the shot, the UI window moves in relation to it. But that doesn’t go the other way around – when you move the UI window, the shot would remain where it is. And the docs say not to set the parent of a child as the child of that child, which could perhaps solve this (I didn’t try. It sounded too weird to even attempt it. It’s one of those things that destroy the space-time continuum).
Instead, you have to hook into the frame-updating functions. The trick here is to find the right one to hook into, so that moving the child window does not result in the parent’s movement to lag behind. In my testing, overriding setFrame(_ : display: ) was the sweet point where I couldn’t notice a lag.

^{Moving the Shot, and moving the “additional UI” window}

Resizing the shot with the UI window visible was (and continues to be – I’m not completely done with that yet) another pain point and again a potential source of laggy UI. When the shot is resized, the UI window needs to be resized in accordance. Not only that, the shot has a specific aspect ratio I don’t want to ruin, which is easy enough to implement for the shots window alone, but resizing the encompassing UI window in accordance took a bit more tinkering (and coffee ☕️).
Finally, the whole thing has to work the other way around again, so that when I resize the UI window, the shot has to resize in accordance, with both respecting their own, respective aspect ratios.

^{Resizing the floating shot window directly, and resizing the “additional UI” window. You’ll notice that, when resizing from the “additional UI” window, the info panel doesn’t come up, nor does it snap to 100%. It’s a work-in-progress.}

Adjusting the transparency was the easiest thing to do, on the other hand. I just “hand over” the scrolling event to the floating shot’s window, and it does its thing.

Being able to drag a file-representation of the floating shot out of the app right away is very convenient, so it has to be available in v2.
For now, I settled on this simple approach: Click and hold onto the floating shot, then drag the file to wherever you want it:

Once you know how it’s done, it’s very convenient. But would you have known?
If this remains, there’s got to be some introduction to it. Work in progress.

– Still to be figured out

A video says more than words in this case, so, for your amusement, here are a few kinks that still need working out:

This actually shows a cute, little implementational detail: The “additional UI” window is not completely “solid”. The part where it’s obstructed by the floating shot’s window is “chiselled” out.
When changing the transparency of the shot, it would be a shame to have what is revealed underneath darkened by the “additional UI” window – even if that, too, is changed to a lower transparency.
The only solution was to create a maskImage for the NSVisualEffectView and remove the part that is underneath the floating shot. Yet another thing that has to be updated when the shots window or the UI window is resized.

Screen Capture

I spent all day yesterday on being able to actually capture new screenshots.
Like I did in ScreenFloat v1, v2 will use the screencapture tool macOS provides.
Long-time-readers of this blog might remember my attempt to re-implement that very CLI myself. Frankly, it’s too much trouble. So as long as Apple allows 3rd party developers to use the screencapture CLI, I will continue to do so.

The new implementation, even though it’s now Swift, is more or less the same as v1’s, but there are a few improvements I was able to make.
Most prominently, the placement of the shot.
Here’s the placement of a newly created shot in ScreenFloat 1:

^{In ScreenFloat v1, newly created shots are placed in relation to your mouse cursor.}

And here’s how newly created shots are placed in ScreenFloat 2:

^{In ScreenFloat v2, newly created shots are placed using screencapture‘s capture frame.}

In recent versions of macOS, screencapture outputs the capture frame in a couple of ways: stdErr, the resulting image file’s extended file attributes (“com.apple.metadata:kMDItemScreenCaptureGlobalRect“), and, if indexed by Spotlight, as a general metadata value of the resulting file. If all fails, ScreenFloat 2 reverts to ScreenFloat 1’s behavior.
As usual, there’s something to be aware of:
screencapture‘s captureFrame originates at the top left of your screen, and the x/y coordinate of the frame is at the top left of the frame, whereas many of macOS’ drawing APIs are bottom-left based (but not all – that’s the fun!). So before this can be used to place the floating shot properly, it needs to be converted to “bottom-left” based coordinates instead of “top-left” ones.

This leads me to the second improvement I was able to make: re-take previous screenshots. Using the captureFrame, and feeding it back into screencapture, I’m able to re-create a screenshot of the same area, without user interaction. I haven’t implemented it yet, but tested it in Terminal. What could possibly go wrong?

That’s it for this time.
Thank you for joining me. Feedback, input and questions are welcome: mail me, tweet me.
Take care! 🤗