Two Hard Things #

Inspired by Phil Karlton's (possibly apocryphal) Two Hard Things, here's what I struggle with on a regular basis:

  1. Getting information X from system A to system B without violating the N layers of abstraction in between.
  2. Naming X such that the name is concise, unique (i.e. greppable) and has the right connotations to someone new to the codebase (or myself, a year later).

Gmail's HTML Tag Whitelist #

I couldn't find a comprehensive list of the HTML tags that Gmail's sanitizer allows through, so I wrote one up.

The Modern WebKit API is Open Source #

When doing web development (or really any development on a complex enough platform), sometimes the best course of action for understanding puzzling behavior is to read the source. It's therefore been fortunate that Chrome/Blink, WebKit (though not Safari) and Firefox/Gecko are all open source (and often the source is easily searchable too).

One of exceptions has been mobile WebKit when accessed via a UIWebView on iOS. Though it's based on the same components (WebCore, JavaScriptCore, WebKit API layer) as its desktop counterpart, there is enough mobile-specific behavior (e.g. interaction with auto-complete and the on-screen keyboard) that source access would come in handy. Apple would periodically do code dumps on, but those only included the WebCore and JavaScriptCore components¹ and in any case there hasn't been one since iOS 6.1.

At WWDC, as part of iOS 8, Apple announced a modern WebKit API that would be unified between the Mac and iOS. Much of the (positive) reaction has been about the new API giving third-party apps access to faster, JITed, JavaScript execution. However, just as important to me is the fact that implementation of the new API is open source.

Besides browsing around the source tree, it's also possible to track its development more closely, via an RSS feed of commits. However, there are no guarantees that just because something is available in the trunk repository that it will also be available on the (presumed) branch that iOS 8 is being worked on. For example, [WKWebView evaluateJavaScript:completionHandler:] was added on June 10, but it didn't show up in iOS 8 until beta 3, released on July 7 (beta 2 was released on June 17). More recent changes, such as the ability to control selection granularity (added on June 26) have yet to show up. There don't seem to be any (header) changes² that live purely in the iOS 8 SDK, so I'm inclined to believe that (at least at this stage) there's not much on-branch development, which is encouraging.

Many thanks to Anders, Benjamin, Mitz and all the other Apple engineers for doing all in the open.

Update on July 21, 2014: The selection granularity API has shown up in beta 4, which was released today.

  1. IANAL, but my understanding is that WebCore and JavaScriptCore are LGPL-licensed (due to its KHTML heritage) and so modifications in shipping software have to distributed as source, while WebKit is BSD-licensed, and therefore doesn't have that requirement.
  2. Modulo some munging done as part of the release process.

Using ASan with iOS Applications #

I've written up a quick guide for getting ASan (Address Sanitizer) working with iOS apps. This is the kind of thing I would have put directly into this blog in the past, but:

  1. Blogger's editor is not pleasant to use — I usually end up editing the HTML directly, especially for posts with code blocks. Not that Quip doesn't have bugs, but at least they're our bugs.
  2. Quip has public sharing now, so in theory that doc should be just as accessible (and indexable) as a regular post.

However, I still like the idea of this blog being a centralized repository of everything that I've written, hence this "stub" post.

Adding Keyboard Shortcuts For Inspecting iOS Apps and Web Pages in Safari #

Back in iOS 6 Apple added the ability to remotely inspect pages in mobile Safari and UIWebViews. While I'm very grateful for that capability, the fact that it's buried in a submenu in Safari's “Develop” menu means that I have to navigate a maze with a mouse every time I relaunch the app. I decided to investigate adding a way of triggering the inspector via a keyboard shortcut.

iPhone Simulator menu
The target

My first thought was that I could add a keyboard shortcut via OS X's built-in support. After all, “mobile.html” is just another menu item. Something like:

iPhone Simulator menu
If only it were so easy

Unfortunately, while that worked if I opened the “Develop” menu at least once, it didn't on a cold start of Safari. I'm guessing that the contents of the menu are generated dynamically (and lazily), and thus there isn't a “mobile.html” item initially for the keyboard shortcut system to hook into.

Inspired by a similar BBEdit script, I then decided to experiment with AppleScript and the System Events UI automation framework. After cursing at AppleScript for a while (can't wait for JavaScript for Automation), I ended up with:

tell application "Safari" to activate
tell application "System Events" to ¬
    click menu item "mobile.html" of menu ¬
        "iPhone Simulator" of menu item "iPhone Simulator" of menu ¬
        "Develop" of menu bar item "Develop" of menu bar 1 of process "Safari"

That seemed to work reliably, now it was just a matter of binding it to a keyboard shortcut. There apps like FastScripts that provide this capability, but to make the script more portable, I wanted a way that didn't depend on third-party software. It turned out that Automator can be used to do this, albeit in a somewhat convoluted fashion:

  1. Launch Automator
  2. Create a new “Service” workflow
  3. Add a “Run AppleScript” action¹
  4. Change the setting at the top of the window to “Service receives no input in any application“
  5. Replace the (* Your script goes here *) placeholder with the script above (your workflow should end up looking like this)
  6. Save the service as “Inspect Simulator”

I wanted to attach a keyboard shortcut to this service when either Safari or the simulator were running, but not in other apps. I therefore then went to the “App Shortcuts” keyboard preferences pane (pictured above) and added shortcuts for that menu item in both apps (to add shortcuts for the simulator, you need to select the “Other…” option in the menu and select it from /Applications/

One final gotcha is that the first time the script is run in either app, you will get a “The action 'Run AppleScript' encountered an error.” dialog. Immediately behind that dialog is another, saying “'' would like to control this computer using accessibility features.” You'll need to open the Security & Privacy preferences pane and enable Safari (and the simulator's) accessibility permissions.

  1. Not be confused with the “Execute AppleScript” action, which is a Remote Desktop one — I did that and was puzzled by the “no computers” error message for a good while.

Per-Package Method Counts for Android's DEX Format #

Quip's Android app recently ran into the Android DEX/Dalvik 64K method limit. We suspected that this was due to code generated by the Protocol Buffer compiler¹, but we wanted to get more specific numbers, to both understand the situation better and track our progress. As a starting point, we figured per-package method counts would give us what we needed.

The Android SDK ships with a dexdump tool that disassembles .dex (or .apk files) and dumps certain information out of it. Running it with the -f flag generated a method_ids_size line that showed that we were indeed precariously close to the limit. The script supports an XML output and per-class output of methods, so it seemed like a straightforward task to group methods and classes by package. However, once I actually processed its output, I got a much lower number than expected (when I did a sanity check to add up all the per-package counts). It turned out that the XML output is hardcoded to only output public classes and methods.

I then held my nose and rewrote the script to instead parse dexdump's text format. Unfortunately, even then there was some undercounting — not as significant, but I was missing a few thousand methods. I looked at the counts for a few classes, and nothing seemed to be missing, so this was perplexing. After some more digging, it turned out that the limit counts referenced methods too, not just those defined in the DEX file. Therefore iterating over the methods defined in each class was missing most of the android.* methods that we were calling.

Mohammad then pointed me at a script that used the smali/baksmali assembler/disassembler to generate per-package counts. However, when I ran it, it seemed to overcount. Looking into it a bit more, it looked like the script disassembled the .apk, re-assembled it to generate a .dex per package, and then ran dexdump on each one. However, this meant that referenced methods were counted by each package that used them, thus the overall count would include them more than once.

I briefly considered modifying dexdump to extract the information that I needed, but it didn't seem like a fun codebase to work in; besides being in C++ it had lots of dependencies into the rest of the Android tree. Looking around for other DEX format parses turned up smali's, dexinfo, dexinsight, dexterity, dexlib and a few others. All seemed to require a bit more effort to build and understand than I was willing to put in late on a Friday night. However, after browsing around through the Android tree more, I came across the dexdeps tool². It is designed for separating referenced and defined methods (and classes), but its DEX file parser looked simple enough to modify to extract the data that I was interested in. Better yet, it had no other dependencies, and looked straightforward to build.

Sure enough, it was pretty easy to modify it to create a per-package method counting tool. After a few more commits, I ended up with a dex-method-counts tool that can be pointed at an APK (or DEX file) and provide a package hierarchy tree-view of defined and referenced method counts. The README has a few more details, including a few flags that I've found useful when looking at protocol buffer compiler-generated code.

As for how we solved our actual method count limit problem, we've so far managed to stave off doom by refactoring our .proto files to include fewer messages in our Java build (we were picking up some that were for other platform or server use only). That is, nothing too crazy yet.

  1. For others in this situation, Square's Wire library may be an alternative.
  2. Somewhat amusingly, this is not the only Java-based DEX parser in the Android source tree, there is also dex-tools in the Compatibility Test Suite area.

Getting ALL your data out of Quip #

No, Quip is not shutting down. But we did just launch an API, so I thought I would take my experience with doing data export to write a backup tool that exports as much data as can be obtained via the API into a local folder. It's missing a few things (images most notably), but you do end up with a folder with all your documents (rendered to HTML) and conversations.

The tool is one of the samples¹ in our API repository, feel free to give it a spin. Pull requests are also welcome.

  1. The Webhook one is also mine. Party like it's 2009!