For a (still in-progress) digital archiving project I wanted to create a Git repository with some classic Mac OS era software. Such software relies on resource forks, which sadly Git does not support. I looked around to see if others had run into this, and found git-resource-fork-hooks, which is a collection of pre-commit and post-checkout Git hooks that convert resource forks into AppleDouble files, allowing them to be tracked. However, there are two limitations of this approach:

• The tools that those hooks use (SplitForks and FixupResourceForks) do not work on APFS volumes, only HFS+ ones.
• The resource fork file is generated is an opaque binary blob. While it can be stored in a Git repository, it does not lend itself to diffing, which would ruin the “time machine” aspect of the archiving project.

I remembered that there was a textual format for resource forks (.r files) which could be “compiled” with the Rez tool (and resource forks could be turned back into .r files with its DeRez companion). This MacTech article from 1998 has more details on Rez, and even mentions source control as a reason to use it.

I searched for any Git hooks that used Rez and found git-xattr-hooks, which is a more specialized subset that only looks at icns resources (incidentally a resource I am very familiar with). That seemed like a good starting point, it was mostly a matter of removing the -only flag.

The other benefit of Rez is that it can be given resource definitions in header files, so that it produces even more structured output. Xcode still ships with resource definitions, and they make a big difference. Here’s the output for a DITL (dialog) resource without resource definitions:

$ DeRez file.rsrc
data 'DITL' (128) {
    $"0003 0000 0000 0099 002F 00AD 0069 0405"            /* .......?./.?.i.. */
    $"4865 6C6C 6F00 0000 0000 0099 007F 00AD"            /* Hello......?...? */
    $"00B9 0405 576F 726C 6400 0000 0000 000C"            /* .?..World....... */
    $"0056 002C 0076 A002 0080 0000 0000 0032"            /* .V.,.v?..?.....2 */
    $"0012 008F 00C5 8816 5759 5349 5759 4720"            /* ...?.ň.WYSIWYG  */
    $"6C69 6B65 2069 7427 7320 3139 3931"                 /* like it's 1991 */
};

And here it is with the system resource definitions (the combination of parameters that works was found via this commit):

$ DeRez -isysroot `xcrun --sdk macosx --show-sdk-path` file.rsrc Carbon.r
resource 'DITL' (128) {
    {    /* array DITLarray: 4 elements */
        /* [1] */
        {153, 47, 173, 105},
        Button {
            enabled,
            "Hello"
        },
        /* [2] */
        {153, 127, 173, 185},
        Button {
            enabled,
            "World"
        },
        /* [3] */
        {12, 86, 44, 118},
        Icon {
            disabled,
            128
        },
        /* [4] */
        {50, 18, 143, 197},
        StaticText {
            disabled,
            "WYSIWYG like it's 1991"
        }
    }
};

Putting all of this together I have created git-resource-fork-hooks, a collection of Python scripts that can be used as pre-commit and post-checkout hooks. They end up creating parallel .r files for each file that has a resource fork, and combining it back on-disk. I briefly looked to see if I could use clean and smudge filters to implement this in a more transparent way, but those are only passed in the file contents (the data fork), and thus can't read or write to the resource fork.

The repo also includes a couple of sample files with resource forks, and as you can see, the diffs are quite nice, even for graphical resources like icons:

I’m guessing that the number of people who would find this tool useful is near zero. On the other hand, Apple keeps shipping the Rez and DeRez tools (and even provided native ARM binaries in Big Sur), thus implying that there is still some value in them, more than two decades after they stopped being a part of Mac development.

An elegant [format], for a more... civilized age.

All of this thinking of resource forks made me a bit nostalgic. It’s pretty incredible to think of what Bruce Horn was able to do with 3K of assembly in 1982. Meanwhile some structured formats that we have today can be so primitive as to not allow Norway or comments. I have a lot of fond memories of using ResEdit to peek around almost every app on my Mac (and cheat by modifying saved tank configs in SpectreVR).

Once I started to develop for the Mac, I appreciated even more things:

• Being able to use TMPL resources to define your own resource types and then have them be graphically editable.
• How resources played nicely with the classic Mac OS memory management system - resources were loaded as handles, and thus those that were marked as “purgeable” could be automatically unloaded under memory pressure.
• Opened resource forks were “chained” which allowed natural overriding of built-in resources (e.g. the standard info/warning/error icons).

While “Show Package Contents” on modern macOS .app bundles has some of the same feel, there’s a lot more fragmentation, and of course there’s nothing like it on iOS without jailbreaking, which is a much higher barrier to entry.

Like many others I’ve spent a lot of time over the past year playing the New York Times’ Spelling Bee puzzle. For those that are not familiar with it, it’s a word game where you’re tasked with finding as many words as possible that can be spelled with the given 7 letters, with the center letter being required. I have by no means mastered it — there are days when getting to the “Genius” ranking proves hard. Especially at those times, I’ve idly thought about how trivial it would be to make a cheating program that applies a simple regular expression through a word list (or even uses something already made). However, that seemed both crude and tedious (entering 7 whole letters by hand).

When thinking of what the ideal bespoke tool for solving Spelling Bee would be, apps like Photomath or various Sudoku solvers came to mind — I would want to point my phone at the Spelling Bee puzzle and get hints for what words to look for, with minimal work on my part. Building such an app seemed like a fun way to play around with the Vision and Core ML frameworks that have appeared in recent iOS releases. Over the course of the past few months I’ve built exactly that, and if you’d like to take it for a spin, it’s available in the App Store. Here’s a short demo video:

Object Detection

The first step was to be able to detect a Spelling Bee “board” using the camera. As it turns out, there are two versions of Spelling Bee, the print and digital editions. Though they are basically the same game, the print one has a simpler display. I ended up creating a Core ML model that had training data with both, with distinct labels (I relied on Jason to send me some pictures of the print version, not being a print subscriber myself). Knowing which version was detected was useful because the print version only accepts 5-letter words, while the digital one allows 4-letter ones.

To create the model, I used RectLabel to annotate images, and Create ML to generate the model. Apple has some sample code for object detection that has the scaffolding for setting up the AVCaptureSession and using the model to get VNRecognizedObjectObservations. The model ended up being surprisingly large (64MB), which was the bulk of the app binary size. I ended up quantizing it to fp16 to halve its size, but even more reduction may be possible.

Print edition	Digital edition

Text Extraction

Now that I knew where in the image the board was, the next task was to extract the letters in it. The vision framework has functionality for this too, and there’s also a sample project. However, when I ran a VNRecognizeTextRequest on the image, I was getting very few matches. My guess was that this was due to widely-spaced individual letters being the input, instead of whole words, which makes the job of the text detector much harder. It looked like others had come to the same conclusion.

I was resigned to having to do more manual letter extraction (perhaps by training a separate object detection/recognition model that could look for letters), when I happened to try Apple’s document scanning framework on my input. That uses the higher-level VNDocumentCameraViewController API, and it appeared to be able to find all of the letters. Looking at the image that it generated, it looked like it was doing some pre-processing (to increase contrast) before doing text extraction. I added a simple Core Image filter that turned the board image into a simple black-and-white version and then I was able to get much better text extraction results.

Captured board image	Processed and simplified board image

The only letter that was still giving me trouble was “I”. Presumably that’s because a standalone capital "I" looks like a nondescript rectangle, and is not obviously a letter. For this I did end up creating a simple separate object recognition model that augments the text extraction result. I trained with images extracted from the processing pipeline, using the somewhat obscure option to expose the app’s Documents directory for syncing via iTunes/finder. This recognizer can be run in parallel with the VNRecognizeTextRequest, and the results from both are combined.

Board Letter Detection

I now had the letters (and their bounding boxes), but I still needed to know which one was the center (required) letter. Though probably overkill for this, I ended up converting the centers of each of the bounding boxes to polar coordinates, and finding those that were close to the expected location of each letter. This also gave me a rough progress/confidence metric — I would only consider a board’s letter fully extracted if I had the same letters in the same positions from a few separate frames.

Dictionary Word Lookup

Once I knew what the puzzle input was, the next step was to generate the possible words that satisfied it. Jason had helpfully generated all possible solutions, but that was for the print version, which did not support 4-letter words. I ended up doing on-device solution generation via a linear scan of a word list — iOS devices are fast enough and the problem is constrained enough that pre-generation was not needed.

One of the challenges was determining what a valid word is. The New York Times describes Spelling Bee as using “common” words, but does not provide a dictionary. The /usr/share/dict/words list which is commonly used for this sort of thing is based on an out-of-copyright dictionary from 1934, which would not have more recent words. I ended up using the 1/3 million most frequent words from the Google Web Trillion Word Corpus, with some filtering. This had the advantage of sorting the words by their frequency of use, making the word list ascend in difficulty. This list does end up with some proper nouns, so there's no guarantee that all presented words are acceptable as solutions, but it was good enough.

Word Definition Display

To make the app more of a “helper”, I decided to not immediately display the word list, but to have a “clue” in the form of each word’s definitions. iOS has a little-known helper for displaying word definitions - UIReferenceLibraryViewController. While this does display the definition of most words, it doesn’t allow any customization of the display, and I wanted to hide the actual word.

Word list	Definition (with word hidden)

It turns out it’s implemented via a WKWebView, and thus it’s possible to inject a small snippet of JavaScript to hide and show definition. The whole point of this project had been to learn something different from the “hybrid app with web views” world that I inhabit at Quip, but sometimes you just can’t escape the web views.

Polish

Now that I had the core functionality working end-to-end, there were still a bunch of finishing touches needed to make it into an “app” as opposed to a tech demo. I ended up adding a slash screen, a “reticle” to make the scanning UI more obvious, and a progress display to show the letters that have been recognized so far.

This was a chance to experiment with SwiftUI. While it was definitely an improvement over auto-layout or Interface Builder, I was still disappointed by the quality of the tooling (Xcode previews would often stop refreshing, even for my very simple project) and the many missing pieces when it comes to integrating with other iOS technologies.

Getting it into the App Store

Despite being a long-time iOS user and developer, this was my first time submitting one of my own apps to the App Store. The technical side was pretty straightforward — I did not encounter any issues with code signing, provisioning profiles or other such things that have haunted Apple platform developers for the past decade. Within a day, I was able to get a TestFlight build out.

However, actually getting the app approved for the App Store was more of an ordeal. I initially got contradictory rejections from Apple (how can app both duplicate another and not have “enough” functionality) and all interactions were handled via canned responses that were not helpful. I ended up having to submit an appeal to the App Review Board to get constructive feedback, after which the app was approved without further issues. I understand the App Store is appealing target for scammers, but having to spend so much reviewer bandwidth on a free, very niche-y app does not seem like a great use of limited resources.

Peeking Inside

If you’d like to take a look to see how the app is implemented, the source is available on GitHub.

I was recently investigating suspiciously high CPU usage while typing in documents in Quip’s Mac app. We had recently switched to WKWebView, and this was happening in the “web content” process, suggesting that it was a JavaScript-related problem, or perhaps something related to the CSS/HTML layout computation for documents. Further examination showed that this happened even in small or simple documents, thus likely ruling out layout complexity.

I hooked up the JavaScript profiler in the web inspector’s timeline view (for something so useful, it’s rather buried), and was able to produce this inverted call graph:

What was surprising was that the bulk of the time was spent in the native toString function, as opposed to any code that we had authored. However, it was unclear which toString this was (Number’s, Object’s, Date’s, etc.), or who was calling it — this profile is from the minified JavaScript in our production app, and there’s no easy way to get a full stack trace to apply a source map to. While it is possible to work with the minified code directly, especially in a codebase you’re familiar with, I was hoping to avoid that.

Hoping to get more visibility into the calling code, I switched to running the JavaScript in development mode, where the code is un-minified and served directly as ES6 modules (see our recent blog post for more details). Surprisingly, the behavior went away entirely — toString was nowhere near the top functions. While this was no doubt a clue, it didn’t help with understanding the problem directly.

I decided to take a different debugging strategy, and switched to profiling the web content process directly via the Xcode Instruments tool (along the lines of past investigations, it’s possible to determine the PID of the process and attach debugging tools to it).

Happily, the profile lined up with what I saw on the JS side: a lot of time is spent in a toString function. It’s now possible to see which object’s method this is, and surprisingly, it’s Function’s. Or perhaps it should not be surprising: as specified, it’s supposed to return the source code for the function, which may be an expensive operation. In the WebKit/JavaScriptCore implementation, this ends up doing a substring operation on the entire file’s source (and there appears to be no caching). This explains the difference that was observed when looking at prod vs. development mode — in prod mode there’s a single large (multi-megabyte) file, while in development the function happened to live in a smallish (a few kilobytes) file.

Now that I knew where the problem was, I added a small monkey-patch to see where the stringification was happening:

const originalToString = Function.prototype.toString;
Function.prototype.toString = function () {
    const startTime = performance.now();
    const rv = originalToString.call(this);
    const runTime = performance.now() - startTime;
    console.groupCollapsed(
        `Function.prototype.toString call: ${runTime.toFixed(3)}ms`);
    console.trace();
    console.groupEnd();
    return rv;
};

This yielded the following trace:

Following through on the button.tsx stack frame pointed to a line with an if (button.type != Button) {...} expression. At first glance it does not appear to be doing any string conversions, but that is what the != operator ends up doing. This is because button is a React element, and the type property is either a string (if it’s a built-in DOM element like div or span) or a class/function (if it’s a custom element). Originally, all elements on this code path (which is invoked whenever the user types) were custom elements, thus the != was comparing one function to another. This is fast since it uses reference equality. However, at some point some DOM children were introduced, and we were comparing a string and a function. Since we’re using !=, this is a loose equality comparison, and we end up stringifying the function operand to make it comparable to the string one.

The fix turned out to be trivial: change the comparison to use !== so that strict equality is used. We’ve been preferring strict equality for most new code, but had not enabled the linting rule to enforce it because it’s a slog to work through all of the legacy violations. This experience shows that it’s not just a matter of preferring strict equality to avoid WAT moments, but that there are performance benefits as well.

Quip did a wholesale migration to TypeScript around this time last year. Now that the dust has settled and we've lived with the consequences for a while, Rafael and I wrote a couple of blog posts about it all: Part one describes the process that we chose, and part two has some anecdotes about how we solved specific problems.

I wrote a post on the Quip blog about the various in-product debugging tools that we've developed over the years. It's been very satisfying to make and use our tools over the years, I'm glad we're finally sharing some details about them.

One of the many overlays we've created

A couple of weeks I was migrating some networking code in Quip's Mac app from NSURLConnection to NSURLSession when I noticed that requests were being made significantly more often than I was expecting. While this was great during the migration (since it made exercising that code path easier), it was unexpected: the data should only have been fetched once and then cached.

After some digging, it turned out that we had a bug in the custom local caching system that sits in front our CDN (CloudFront), which we use to serve profile pictures and other non-authenticated data. Due to a catch-22 in the cache key function (which made it depend on the HTTP response), all assets would initially not be found in the local cache, and would incur a network request. The necessary data was then stored in memory, so until the app was restarted the cache would work as expected, but in the next session they would get requested again.

It turned out that this bug had been introduced a few months prior, but since it manifested itself as a little bit of extra traffic to an external service, we didn't notice it (the only other visible manifestation would be that profile pictures would load more slowly during app startup, or be replaced with placeholders if the user happened to be offline, but we never got any reports of that).

This chart (of CloudFront requests from “Unknown” browsers, which is how our native apps are counted) shows the fix in action; the Mac app build with it was released on November 30th and was picked up by most users over the next few days.

This kind of low-grade accidental DDoS reminded of a similar bug that I investigated a few years ago at Google, while working on Chrome Extensions. A user had reported that the Gmail extension for Chrome (which my team happened to own, since we provided it as sample code) would end up consuming a lot of memory (and eventually be terminated) if the Gmail URL that it tried to fetch data from was blocked by filtering software. After some digging it turned out that the extension would enqueue two retries for every failed failure response, due to code along these lines:

var xhr = new XMLHttpRequest();

... // Send off request

function handleError() {
    ... // schedule another request
}

xhr.onreadystatechange = function() {
    .. // Various early exits if success conditions are met
    
    handleError();
};

xhr.onerror = function() {
   handleError();
};

The readystatechange event always fires, including for error states that also invoke the error event handler. This behavior meant that it would quickly escalate from a request every few minutes to almost one request per second, depending on how long it remained in the blocked state. The fix turned out to be trivial, and since this was a separate package distributed via the Chrome Web Store that gets auto-updated, we could quickly fix the millions of users that had it installed.

It then occurred to me that this would not just affect users where the Gmail URL was blocked, but any user that had spotty connectivity — any HTTP failure would result in a doubling of background requests. I then called up a requests-per-second graph of the Atom feed endpoint for Gmail (which is what the extension used), and saw that it had dropped by 20,000 requests per second over the day or so that it took for the extension update to propagate.

The upshot of all this is that Google Reader at its peak had about 10,000 requests per second, thus making my overall traffic contribution to Google net negative.

JavaScriptCore (JSC) is the JavaScript engine that powers WebKit (and thus Safari). I was recently browsing through its source and noticed a few interesting things:

ARM64_32 Support

Apple Watch Series 4 uses the S4 64-bit ARM CPU, but running in a mode where pointers are still 32 bits (to save on the memory overhead of a 64 -bit architecture). The watch (and its new CPU) were announced in September 2018, but support for the new ARM64_32 architecture was added in December 2017. That the architecture transition was planned in advance is no surprise (it's been in the works since the original Apple Watch was announced in 2015). However, it does show that JSC/WebKit is a good place to watch for future Apple ISA changes.

ARMv8.3 Support

The iPhone XS and other new devices that use the A12 CPU have significantly improved JavaScript performance when compared to their predecessors. It has been speculated that this is due to the A12 supporting the ARM v8.3 instruction set, which has a new floating point instruction that operates with JavaScript rounding semantics. However, it looks like support for that instruction was only added a couple of weeks ago, after the new phone launch. Furthermore, the benchmarking by the Apple engineer after the change landed showed that it was responsible for a 0.5%-2% speed increase, which while nice, does not explain most of the gain.

Further digging into the JSC source led to my noticing that JIT for the ARMv8.3 ISA (ARM64E in Apple's parlance) is not part of the open source components of JSC/WebKit (the commit that added it references a file in WebKitSupport, which is internal to Apple). So perhaps there are further changes for this new CPU, but we don't know what they are. It's an interesting counterpoint to the previous item, where Apple appears to want extra secrecy in this area. As a side note, initial support for this architecture was also added several months before the announcement (and references to ARM64E showed up more than 18 months earlier), thus another advance notice of upcoming CPU changes.

Fuschia Support

Googler Adam Barth (hi Adam!) added support for running JSC on Fuschia (Google's not-Android, not-Chrome OS operating system). Given that Google has its own JavaScript engine (V8), it's interesting to wonder why they would also want another engine running. A 9to5 Google article has the same observation, and some more speculation as to the motivation.