When doing web development (or really any development on a complex enough platform), sometimes the best course of action for understanding puzzling behavior is to read the source. It's therefore been fortunate that Chrome/Blink, WebKit (though not Safari) and Firefox/Gecko are all open source (and often the source is easily searchable too).

One of exceptions has been mobile WebKit when accessed via a UIWebView on iOS. Though it's based on the same components (WebCore, JavaScriptCore, WebKit API layer) as its desktop counterpart, there is enough mobile-specific behavior (e.g. interaction with auto-complete and the on-screen keyboard) that source access would come in handy. Apple would periodically do code dumps on opensource.apple.com, but those only included the WebCore and JavaScriptCore components¹ and in any case there hasn't been one since iOS 6.1.

At WWDC, as part of iOS 8, Apple announced a modern WebKit API that would be unified between the Mac and iOS. Much of the (positive) reaction has been about the new API giving third-party apps access to faster, JITed, JavaScript execution. However, just as important to me is the fact that implementation of the new API is open source.

Besides browsing around the source tree, it's also possible to track its development more closely, via an RSS feed of commits. However, there are no guarantees that just because something is available in the trunk repository that it will also be available on the (presumed) branch that iOS 8 is being worked on. For example, [WKWebView evaluateJavaScript:completionHandler:] was added on June 10, but it didn't show up in iOS 8 until beta 3, released on July 7 (beta 2 was released on June 17). More recent changes, such as the ability to control selection granularity (added on June 26) have yet to show up. There don't seem to be any (header) changes² that live purely in the iOS 8 SDK, so I'm inclined to believe that (at least at this stage) there's not much on-branch development, which is encouraging.

Many thanks to Anders, Benjamin, Mitz and all the other Apple engineers for doing all in the open.

Update on July 21, 2014: The selection granularity API has shown up in beta 4, which was released today.

1. IANAL, but my understanding is that WebCore and JavaScriptCore are LGPL-licensed (due to its KHTML heritage) and so modifications in shipping software have to distributed as source, while WebKit is BSD-licensed, and therefore doesn't have that requirement.
2. Modulo some munging done as part of the release process.

I've written up a quick guide for getting ASan (Address Sanitizer) working with iOS apps. This is the kind of thing I would have put directly into this blog in the past, but:

1. Blogger's editor is not pleasant to use — I usually end up editing the HTML directly, especially for posts with code blocks. Not that Quip doesn't have bugs, but at least they're our bugs.
2. Quip has public sharing now, so in theory that doc should be just as accessible (and indexable) as a regular post.

However, I still like the idea of this blog being a centralized repository of everything that I've written, hence this "stub" post.

Back in iOS 6 Apple added the ability to remotely inspect pages in mobile Safari and UIWebViews. While I'm very grateful for that capability, the fact that it's buried in a submenu in Safari's “Develop” menu means that I have to navigate a maze with a mouse every time I relaunch the app. I decided to investigate adding a way of triggering the inspector via a keyboard shortcut.

The target

My first thought was that I could add a keyboard shortcut via OS X's built-in support. After all, “mobile.html” is just another menu item. Something like:

If only it were so easy

Unfortunately, while that worked if I opened the “Develop” menu at least once, it didn't on a cold start of Safari. I'm guessing that the contents of the menu are generated dynamically (and lazily), and thus there isn't a “mobile.html” item initially for the keyboard shortcut system to hook into.

Inspired by a similar BBEdit script, I then decided to experiment with AppleScript and the System Events UI automation framework. After cursing at AppleScript for a while (can't wait for JavaScript for Automation), I ended up with:

tell application "Safari" to activate
tell application "System Events" to ¬
    click menu item "mobile.html" of menu ¬
        "iPhone Simulator" of menu item "iPhone Simulator" of menu ¬
        "Develop" of menu bar item "Develop" of menu bar 1 of process "Safari"

That seemed to work reliably, now it was just a matter of binding it to a keyboard shortcut. There apps like FastScripts that provide this capability, but to make the script more portable, I wanted a way that didn't depend on third-party software. It turned out that Automator can be used to do this, albeit in a somewhat convoluted fashion:

1. Launch Automator
2. Create a new “Service” workflow
3. Add a “Run AppleScript” action¹
4. Change the setting at the top of the window to “Service receives no input in any application“
5. Replace the (* Your script goes here *) placeholder with the script above (your workflow should end up looking like this)
6. Save the service as “Inspect Simulator”

I wanted to attach a keyboard shortcut to this service when either Safari or the simulator were running, but not in other apps. I therefore then went to the “App Shortcuts” keyboard preferences pane (pictured above) and added shortcuts for that menu item in both apps (to add shortcuts for the simulator, you need to select the “Other…” option in the menu and select it from /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/Applications/iPhone Simulator.app).

One final gotcha is that the first time the script is run in either app, you will get a “The action 'Run AppleScript' encountered an error.” dialog. Immediately behind that dialog is another, saying “'Safari.app' would like to control this computer using accessibility features.” You'll need to open the Security & Privacy preferences pane and enable Safari (and the simulator's) accessibility permissions.

1. Not be confused with the “Execute AppleScript” action, which is a Remote Desktop one — I did that and was puzzled by the “no computers” error message for a good while.

Quip's Android app recently ran into the Android DEX/Dalvik 64K method limit. We suspected that this was due to code generated by the Protocol Buffer compiler¹, but we wanted to get more specific numbers, to both understand the situation better and track our progress. As a starting point, we figured per-package method counts would give us what we needed.

The Android SDK ships with a dexdump tool that disassembles .dex (or .apk files) and dumps certain information out of it. Running it with the -f flag generated a method_ids_size line that showed that we were indeed precariously close to the limit. The script supports an XML output and per-class output of methods, so it seemed like a straightforward task to group methods and classes by package. However, once I actually processed its output, I got a much lower number than expected (when I did a sanity check to add up all the per-package counts). It turned out that the XML output is hardcoded to only output public classes and methods.

I then held my nose and rewrote the script to instead parse dexdump's text format. Unfortunately, even then there was some undercounting — not as significant, but I was missing a few thousand methods. I looked at the counts for a few classes, and nothing seemed to be missing, so this was perplexing. After some more digging, it turned out that the limit counts referenced methods too, not just those defined in the DEX file. Therefore iterating over the methods defined in each class was missing most of the android.* methods that we were calling.

Mohammad then pointed me at a script that used the smali/baksmali assembler/disassembler to generate per-package counts. However, when I ran it, it seemed to overcount. Looking into it a bit more, it looked like the script disassembled the .apk, re-assembled it to generate a .dex per package, and then ran dexdump on each one. However, this meant that referenced methods were counted by each package that used them, thus the overall count would include them more than once.

I briefly considered modifying dexdump to extract the information that I needed, but it didn't seem like a fun codebase to work in; besides being in C++ it had lots of dependencies into the rest of the Android tree. Looking around for other DEX format parses turned up smali's, dexinfo, dexinsight, dexterity, dexlib and a few others. All seemed to require a bit more effort to build and understand than I was willing to put in late on a Friday night. However, after browsing around through the Android tree more, I came across the dexdeps tool². It is designed for separating referenced and defined methods (and classes), but its DEX file parser looked simple enough to modify to extract the data that I was interested in. Better yet, it had no other dependencies, and looked straightforward to build.

Sure enough, it was pretty easy to modify it to create a per-package method counting tool. After a few more commits, I ended up with a dex-method-counts tool that can be pointed at an APK (or DEX file) and provide a package hierarchy tree-view of defined and referenced method counts. The README has a few more details, including a few flags that I've found useful when looking at protocol buffer compiler-generated code.

As for how we solved our actual method count limit problem, we've so far managed to stave off doom by refactoring our .proto files to include fewer messages in our Java build (we were picking up some that were for other platform or server use only). That is, nothing too crazy yet.

1. For others in this situation, Square's Wire library may be an alternative.
2. Somewhat amusingly, this is not the only Java-based DEX parser in the Android source tree, there is also dex-tools in the Compatibility Test Suite area.

No, Quip is not shutting down. But we did just launch an API, so I thought I would take my experience with doing data export to write a backup tool that exports as much data as can be obtained via the API into a local folder. It's missing a few things (images most notably), but you do end up with a folder with all your documents (rendered to HTML) and conversations.

The tool is one of the samples¹ in our API repository, feel free to give it a spin. Pull requests are also welcome.

1. The Webhook one is also mine. Party like it's 2009!

Yesterday was the release day of the Veronica Mars movie. As a Kickstarter backer, Ann got a digital copy of the movie. For reasons that I'm sure were not entirely technical, it was only available via Flixster/UltraViolet¹, so getting access to it involved registering for a new account and jumping through some hoops.

To actually download the movie for offline viewing, Flixster said it needed a “Flixster Desktop” client app. It was served as a ~29 MB .zip file, so it seemed like a straightforward download. I noticed that I was only getting ~ 30K/second download speeds, but I wasn't in a hurry, so I let it run. The download finished, but with only a ~21MB file that was malformed when I tried to expand it. I figured the WiFi of the hotel that we were staying at was somehow messing with the connection, so I tried again while tethered to my phone. I was still getting similarly slow download speeds, and the “completed” download was still too small. Since 30K/second was definitely under the expected tethered LTE throughput, I began to suspect Flixster's servers as the root cause. It certainly seemed plausible given that the file was served from desktop.flixster.com, which did not seem like a CDN domain². I guess ~60,000 fans were enough to DDoS it; from reading a Reddit thread, it seemed like I was not the only one.

The truncated downloads were of slightly different sizes, but it seemed like they finished in similar amounts of time, so I decided to be more scientific and timed the next attempt. It finished in exactly 10 minutes. My hypothesis was now that Flixster's server (or some intermediary) was terminating connections after 10 minutes, regardless of what was being transferred or what state it was in.

Chrome's download manager has a Pause/Resume link, so my next thought was to use it to break up the download into two smaller chunks. After getting the first 10 MB, I paused the download, disconnected the laptop from WiFi (to make sure the connection would not be reused) and then reconnected and tried resuming. Unfortunately, the download never restarted. I did a HEAD request on the file, and since the response headers did not include an Accept-Ranges header, I assumed that the server just didn't support resumable downloads, and that this path was a dead end.

After spending a few minutes trying to find a mirror of the app on sketchy download sites, a vague memory of Chrome's download manager not actually supporting HTTP range requests came to me. I did some quick tests with curl and saw that if I issued requests with --range parameters I got different results back. So it seemed like despite the lack of Accept-Ranges headers, the server (Apache fronted by Varnish) did in fact support range requests³.

I therefore downloaded the file in two chunks by using --range 0-10000000 and --range 10000000- and concatenated them with cat. Somewhat surprisingly, the resulting zip file was well-formed and expanded correctly. I put a copy of the file in my Dropbox account and shared it on the Reddit thread, it seemed to have helped a few others.

Of course, by the end of all this, I was more excited about having successfully downloaded the client app than getting or watching the movie itself⁴.

1. As opposed to say, iTunes or Amazon.
2. Now that I check the download page a day later, it seems to be served from static.flixstercdn.com, so I guess Flixster realized what was going on and fixed the problem.
3. A closer reading of section 14.5 of the HTTP 1.1 spec showed that servers MAY respond with Accept-Ranges: bytes, but are not required to.
4. Downloading the actual movie worked fine, I assume it was distributed over a CDN and thus wasn't quite so easily overwhelmed.

tl;dr: Search Gmail for “is:spam -label:^os” to find messages that you manually marked as spam (as opposed to ones that Gmail automatically marked for you).

Gmail recently had a bug where some emails were accidentally moved to the trash or marked as spam. Google “encouraged” users that might have been affected to check their trash and spam folders for any messages that didn't belong. Since I get a lot of spam (one of the perks of having the same email address since 1996), I didn't relish the thought of going through thousands of messages to see if any of them were mislabeled¹.

I figured that Gmail must keep track of which messages were explicitly marked as spam by the user versus one that it automatically classifies (though I get a lot of spam, almost all of it is caught by Gmail's filters). Gmail (like Google Reader) keeps track of per-message state via internal system labels. For example, others have discovered that Gmail's Smart Labels are represented as ^smartlabel_type labels while Superstars uses names like ^ss_sy. Indeed, if you try to use a caret in a label name, Gmail says that it is not allowed.

It therefore seemed like a reasonable assumption that there was a system label that would tell us how a message came to be marked as spam. The problem was to figure out what it was called.

Thinking back to Reader (where all label operations went through an edit-tag HTTP API call, which listed the labels to added or removed), I figured I would see what the request was when marking a message as spam. Unfortunately, it looked like Gmail's requests were of slightly higher abstraction level, where marking a message as spam would send a request with an act=sp parameter (while marking as read uses act=rd, and so on).

I then figured I should look at HTTP response when loading the spam folder. There appeared to be a bunch of system label names associated with each message. One that I explicitly marked as spam had the labels:

"^a", "^ad_1391126400000", "^all", "^bsm"," ^clu_group", "^clu_unim", "^cob-processed-gmr", "^cob_pevent", "^oc_group", "^os_group", "^s", "^smartlabel_group", "^u"

Meanwhile, another that had been automatically marked as spam used:

"^ad_1391126400000", "^all"," ^bsm", "^clu_notification", "^cob-processed-gmr", "^oc_notification", "^os", "^os_notification", "^s", "^smartlabel_notification", "^u”

^s was present on all of them, and indeed doing a search for label:^s shows all spam messages (and the UI rewrites the search to in:spam). Others could also be puzzled out based on name, for example ^u is for unread messages. The more mysterious ones like ^cob_pevent I figured I could ignore².

After looking at a bunch of messages, both automatically and manually marked as spam, ^os stood out. It only seemed to be present on messages that Gmail itself had decided were spam. Doing the search is:spam -label:^os seemed to show only messages that I had marked as spam. Indeed, each of the messages in the result displayed the header: "Why is this message in Spam? You clicked 'Report spam' for this message." Thus I was able to go through the much shorter list and see if any where mistakenly marked (they weren't).

Seeing the plethora of labels that were present on all messages, I got curious what other internal labels there were. Between examining HTTP responses, looking through Gmail's JavaScript for strings that start with ^ and a simple dictionary attack for two-letter names, here's some others that I've found (those that are marked as “unknown” are ones that match some messages in my account, but with no apparent pattern):

• ^a: archived conversations
• ^b: chat transcripts (equivalent to is:chat, presumably the “b” is for “Buzz”, Google Talk's codename)
• ^f: sent messages (equivalent to is:sent)
• ^g: muted conversations (equivalent to is:muted, the “g” is most likely for “ignore”)
• ^i: inbox (equivalent to in:inbox)
• ^k: trashed messages (equivalent to in:trash, unclear why “k” is the abbreviation)
• ^o: unknown
• ^p: messages that were marked as phishing attempts
• ^r: drafts (equivalent to is:draft)
• ^s: spam (equivalent to is:spam)
• ^t: starred messages (equivalent to is:starred, the “t” is most likely for “to do”)
• ^u: unread messages (equivalent to is:unread)
• ^ac: Google Buzz messages (equivalent to is:buzz)
• ^act: Google Buzz messages (unclear how it's different from ^ac)
• ^af: unknown
• ^bc: unknown subset of chat transcripts
• ^p_cc: another unknown subset of chat transcripts
• ^fs: unknown
• ^ia: unknown
• ^ii: unknown
• ^im: unknown
• ^iim: Priority Inbox (based on Android's documentation)
• ^mf: unknown
• ^np: unknown
• ^ns: unknown
• ^bsm: unknown
• ^op: messages that were automatically marked as phishing attempts
• ^os: messages that were automatically marked as spam
• ^vm: Google Voice voicemails (equivalent to is:voicemail)
• ^pop: unknown, seems to match some (very old messages) that I imported via POP
• ^ss_sy, ^ss_so, ^ss_sr, ^ss_sp, ^ss_sb, ^ss_sg, ^ss_cr, ^ss_co, ^ss_cy, ^ss_cg, ^ss_cb, ^ss_cp: Superstar stars
• ^sl_root, ^smartlabel_promo, _receipt, _travel, _event, _group, _newsletter, _notification, _personal, _social, _receipt and _finance: Smart Labels
• ^io_im: important messages (equivalent to is:important)
• ^io_imc1 through ^io_imc5, ^io_lr: unknown, possibly more degrees of importance (“Info Overload” was the project that resulted in the importance filtering)
• ^clu_unim: unknown, possibly unimportant messages
• ^unsub and ^hunsub: messages where an unsubscribe link has been detected (when marking one as spam, the “In addition to marking this message as spam, you can unsubscribe...” dialog appears). ^unsub seems to be for messages where there's an unsubscribe link you have to click while ^hunsub is for ones where Gmail offers to unsubscribe on your behalf.
• ^cff: sender is in a Google+ circle (equivalent to has:circle)
• ^sps: unknown (no matches in my account, but it was referenced in the JavaScript next to ^p, if I had to guess I would say it's something related to spear phishing)
• ^p_esnotif: Google+ notifications ("es" presumably being "Emerald Sea", Google+'s code name)

1. Of course, in deciding to automate this task, I doomed myself to spend more time that I would have if I'd just gone through the messages by hand.
2. It's somewhat interesting to see how features that were developed later (like Smart Labels — ^smartlabel_group) use longer system label names than ones of medium age (like Superstars — ^ss_sy) which are in turn longer than the original system labels (^u for unread, etc.). Bytes 10 years ago were clearly more precious.