Image-based masking was first introduced by WebKit a few years ago, and has proven to be a useful CSS feature. Unfortunately browsers without a WebKit lineage do not support it, which makes it a less than appealing option for cross-browser development. There is however the alternative of SVG-based masking, introduced by Firefox/Gecko at least partly in response to WebKit's feature. My goal was to find some way to combine the two mechanisms, so that I could use the same image assets in both rendering engines to achieve the masking effect. My other requirement was that the masks had to be provided as raster images (WebKit can use SVG file as image masks, but some shapes are complex enough that representing it as a bitmap is preferable).
SVG supports an <image> element, so at first glance this would just be a matter of something like:
Unfortunately, depending on your mask image, when trying that, you will most likely end up with nothing being displayed. A more careful reading shows that WebKit's image masks only use the alpha channel to determine what gets masked, while SVG masks use the luminance. If your mask image has black in the RGB channels, then the luminance is 0, and nothing will show through.
SVG 2 introduces a mask-type="alpha" property that is meant to solve this very problem. Better yet, code to support this feature in Gecko landed 6 months ago. However, the feature is behind a layout.css.masking.enabledabout:config flag, so it's not actually useful.
After more exploration of what SVG can and can't do, it occurred to me that I could transform an alpha channel mask into a luminance mask entirely within SVG (I had initially experimented with using <canvas>, but that would have meant that masks would not be ready until scripts had executed). Specifically, SVG Filters can be used to alter SVG images, including masks. The feColorMatrix filter can be used to manipulate color channel data, and thus a simple matrix can be used to copy the alpha channel over to the RGB channels. Putting all that together gives us:
I've put up a small demo of this in action (a silhouette of the continents is used to mask various textures). It seems to work as expected in Chrome, Safari 6, and Firefox 21.
It occurred to me that I never blogged about my Source Quicklinks
extension (source). I created it back in 2010, when I started working on the Chrome team, focusing on the WebKit side of things. I was spending a lot of time in Code Search trying to understand how things worked. Chromium's code search instance searches not just the Chromium repository itself, but also all its dependencies (WebKit, V8, Skia, etc.). A lot of times the only way to understand a piece of code was to look at the commit that added it (especially in WebKit code, where comments are scarce but the bug associated with the commit often provides the back story). I was therefore doing a lot of URL mangling to go from Code Search results to Trac pages that had "blame" (annotation) views.
The extension made this process easier, adding a page action that provides back and forth links between Code Search, the Chromium repository (both the ViewVC and the Gitweb sites), WebKit's Trac setup and V8. Later, when the omnibox API became available, it also gained a "sql" search shortcut for the Chromium repository and commits.
Though I no longer work on Chrome, I still find myself going through those code bases quite often. For example, the best way to know whether something triggers layout in Blink/WebKit is by reading the code. I've therefore revved the extension to handle the Blink fork, in addition to cleaning up some other things that had started to bitrot. I also attempted to add cross-links between the WebKit and Blink repositories that takes into account the Blink reorganization, though we'll see how useful that ends up being as the codebases diverge more and more.
If Google Reader were a movie or TV show, at the end of the spectacle the credits would roll and you would get to see who was responsible for what you just saw. But in today's age, software "about box credits" are no longer common.
I thought it might be nice, for the sake of posterity, to list all those who worked on Reader over the years. There's been a lot of discussion about Reader's imminent shutdown, but most of it focused on Google (the corporate entity) and its strategy. However, at the end of the day, Reader was built by people. I and a few others have been lucky enough to be more visible, but everyone involved deserves credit and thanks. This is especially the case since as Chris and Brian have described, Reader faced quite a few internal struggles. As I remember it, nearly everyone on the Reader team explicitly requested to join it, and often had to fight to keep their role.
Coming up with this list was difficult, both technically (can you name all your coworkers going back 8 years?), and because it was tough to decide where to draw the line. Google is a big company, and many people in many supporting roles helped to Reader out. First, here's a list of all full-time Reader team members:
Ye Zhou (2009-2010): Software Engineer, still at Google.
Devin Kennedy (summer of 2009): Software Engineer Intern, still at Google.
Arif Siddiquee (2009-2011): Software Engineer, still at Google.
Alan Green (2011-2012): Software Engineer, still at Google.
Mick Hollins (2011-2012): Software Engineer, still at Google.
Additionally, here's others who contributed to Reader in various roles at Google:
Design: Micheal Lopez
Executives: Greg Badros, Jeff Huber, Pavni Diwanji
Legal: Halimah DeLaine
Localization: Gabriella Laszlo, John Saito, Katsuhiko Momoi, Sasan Banava
PR: Nate Tyler, Oscar Shine, Sonya Boralv
Product Management: Bruce Polderman, Sabrina Ellis
Product Marketing: Kevin Systrom, Louis Gray, Peter Harbison, Robby Stein, Tom Stocky, Zach Yeskel
Quality Assurance: Amar Amte, Jan Carpenter, Kavitha Venkatesan, Madhuri Kulkarni, Thanh Le
Site Reliability Engineering: Chen Wang, Christoph Pfisterer, David Parrish, Ed Bardsley, Eric Weigle, Gary Luo, Huaxia Xia, James Long, Jerry Zhiwei Cen, Keith Brady, Lantian Zheng, Liren Chen, Matthew Eastman, Nadav Samet, Niall Sheridan, Olivier Beyssac, Patrick Scott, Paul Chien, Pereira Braga, Petru Paler, Sara Smollett, Scott Lamb, Sebastian Adamczyk, Vladimir Filipović, Wensheng Wang, Yu Liao
20% time and additional engineering: Aaron Boodman, Abdulla Kamar, Akshay Patil, Aman Bhargava, Brad Fitzpatrick, Brett Bavar, Brett Slatkin, Charles Chen, Ed Ho, John Pongsajapan, Olga Stroilova,
Peter Baldwin, Steve Jenson, Steve Lacey, T.V. Raman, Wiktor Gworek
User Experience Designers: Jonathan Terleski, Sean McBride
User Experience Research: Anna Avrekh, David Choi, Nika Smith, Theresa Sobczak
User Support: Graham Waldon, Paul Wilcox, Wen-Ai Yu
I'm sure I'm missing names and got got things wrong, so don't hesitate to contact me with corrections. And to everyone that I worked with on Reader, it was a pleasure!
A few months ago, as I was making yet another test app to demonstrate a Chrome packaged app API, I wished for a REPL. In theory the Chrome Dev Tools would fit the bill, since the console lets you run arbitrary JavaScript statements. However, using the dev tools would still involve making a manifest with the right permissions and loading an unpacked app, and at least a background page if not an actual window to inspect. Once you inspected the right page, invoking and inspecting the results of asynchronous APIs would be tedious, with a lot of boilerplate to type every time.
I started to think about creating a purpose-built REPL for Chrome apps APIs. A generic REPL seemed out of the question, due to eval being disallowed due to the strict Content Security Policy used by apps. My initial thought involved a dropdown listing all functions in the chrome.* namespace and a way to invoke them with canned values (eval may be disallowed, but dynamic invocation of the form chrome[namespace][methodName](arg) is still possible). However, that seemed clunky, and wouldn't help with APIs like the socket one that need to chain several method calls with the parameters for one depending on the results of another.
I then thought more about the eval limitation, and if I could use sandboxed pages to create the REPL environment. In some ways that seemed contradictory; the whole point of sandboxed pages is that they don't have access to Chrome APIs (unlike the main frame/page). In exchange they can use less safe mechanisms such as eval (a form of privilege separation). However, sandboxed pages can communicate with the containing page and get data from them via postMessage1. In theory the input code could be eval-ed in the sandboxed frame, and when it tried to invoke Chrome APIs, the sandboxed frame would postMessage to the main frame, ask it to run that API method, get the result, and plug it back in the expression that was being evaluated.2
This plan hinged on fact that nearly all Chrome apps APIs are asynchronous already, thus it should be possible to create seemingly functionally identical proxies in the sandboxed frame. That way, as far as the user is concerned, they're running the original API methods directly. There would need to be some additional bookkeeping to make callback parameters work, but there was no technical barrier anymore.
Before talking about that bookkeeping, since we're now five paragraphs into the blog post, I should cut to the chase and give you a link to the REPL app that I ended up building: App APIs REPL
(source). And if you'd like to see it in action, here's a screencast of it showing basic JavaScript expression evaluation and then a more complex example playing around with the socket API to mimic HTTP requests to www.google.com.
Here's how eval-ing the following statement works:
The main frame (also referred to as the "host" in the source code) gets the input and sends it to the sandboxed frame via a EVAL message. The sandbox dutifully evals it.
chrome.socket.create is a stub that was created in the sandboxed frame: at application startup, the main frame walks over the chrome.* namespace and gathers all properties into a map and sends them to the sandbox (via a INIT_APIS message). The sandbox re-creates them, and for function properties and events a stub is generated.
When the stub is invoked, it sends a RUN_API_FUNCTION message to the main frame with the API method (chrome.socket.create in this case) that should be run and its parameters. Most parameters can be copied directly via the structured clone algorithm that is used by postMessage.
However, the second parameter is function that cannot be copied. Instead we generate an ID for it, put it in a pending callbacks map, and send the ID in its place.
On the main frame side, the list of parameters is reconstructed. For function parameters, we generate a stub based on the ID that was passed in. Once we have the parameters, we invoke the API function (via dynamic invocation, see above) with them.
When the stub function that was used as the callback parameter is invoked, it takes its arguments (if any), serializes them and then sends them and its function ID back to the sandboxed frame via a RUN_API_FUNCTION_CALLBACK message.
The sandboxed frame looks up the function ID in the callbacks map, deserializes the parameters, and then invokes the function with them.
The callback function uses the log() built-in function. That ends up sending a LOG message to the main frame with the data that it wants logged to the console.
Events work in a similar manner, with stubs being generated for add/removeListener() in the sandbox that end up adding/removing listeners in the main frame. There are two maps of listener functions, one in the sandboxed frame from ID to real listener, and one in the main frame from ID to stub/forwarding listener. This allows removing of listeners to work as expected.
The console functionality of the REPL is provided by jqconsole, which proved to the very easy to drop in and hook up input and output to. History of the console is persisted across app restarts via the storage API. Additional built-in commands like help and methods (which dumps a list of all available API methods) as implemented as custom getters getters in the global JavaScript namespace of the sandboxed frame. There's also a magic _ placeholder that can be used as a callback parameter or event listener; it will be replaced with a generated function that logs invocations.
In addition to being a useful developer and leaning tool, I hope that this REPL also helps with thinking with a sandboxed mindset. I know that the Content Security Policy that's used in apps has been controversial, with some taking it better than others. However, I think that privilege separation, declarative permissions, tying capabilities to user gestures/intent and other security features of the Chrome apps runtime are here to stay. CSP is applicable to the web in general, not just apps. Windows 8 requires sandboxing for store apps and its web-based apps are taking an approach similar to CSP to deter XSS. Sandboxing was one of the main themes for Mac desktop developers this year, with Apple finally pulling the trigger on sandbox requirements. Developers of large, complex applications were able to adapt them to the Mac OS X sandbox. That gives me hope that the Chrome app sandbox will not prevent real apps from being created. It's is starting with the even more restrictive web platform sandbox and relaxing it slightly, but is generally aiming for the same spot as the Mac one.
I'm also hopeful that there will be improvements that make it even easier to write secure apps. For example, the privilege isolation provided by sandboxed pages was inspired by a USENIX presentation (the paper presupposed no browser modifications, the Chrome team just paved the cowpath).
pkg.js is a library that's cropped up recently for making such main/sandboxed frame communication easier.
Note that this not the desired pattern for communication between the main and sandboxed frames. Ideally messages that are passed between the two should be as high-level as possible, with application semantics, not low-level Chrome API semantics. For example, if your sandboxed frame does image processing, it shouldn't get to pick the image paths that it reads/writes from; instead it should be given (and return) a blob of image data; it's up to the main frame to decide where it gets that image data (by reading a path on disk, from the webcam, etc.). Otherwise if the code in the sandbox is malicious, it could abuse the file I/O capability.
Snapchat's (and now Facebook Poke's) main claim to fame is that it lets you send "self-destructing" image messages. Setting aside the debate about the uses of this beyond sexting, the key vulnerability in both apps is the built-in ability to take screenshots. Both take a reactive approach, where you're notified if the recipient took a screenshot, but can't really do anything about it.
I was thinking about ways of mitigating this issue, and figured that perhaps turning the image into an animation where individual frames are not (or at least less) recognizable would be the right path. This is a variant of temporal dithering, except we're intentionally pretending like each frame has a limited amount of precision, and only when averaged together is the original image re-created.
I've created a proof of concept (source) of this. It loads the image into a <canvas> and generates a "positive" and "negative" frame out of it. The positive frame has a random offset added to each pixel's RGB components, while the negative one has it subtracted. When displayed in quick sequence (requestAnimationFrame is used to do this every time the screen refreshes) the two offsets should cancel out, and the resulting image should re-appear.
Source
Temporal dithering1
Positive frame
Negative frame
The resulting flicker is unfortunate, but perhaps that only enhances the furtiveness of an image that will disappear in a few seconds. It also seemed fitting to include Lenna as a source image, given its origin.
Obviously this technique is meant to deter only casual attempts at screen capture. Beyond the analog hole, screen recording software (albeit not an issue on non-jailbroken iOS devices) can easily reconstruct the original image.
Potential areas of exploration are doing the offsets in a different color space (RGB is not linear) and using more than two frames. One option for generating more frames is to keep making random positive and negative pairs. That way repeated screen captures are less likely to yield a pair that can be combined to reconstruct the original image. Another option for generating more frames is to create three or more images that need to be averaged together to yield the original. However, that will result in more image degradation, since the frames are less likely to be perceived as one image; persistence of vision lasts for 1/25th of a second, which is between 2 and 3 frames at 60Hz.
The embedded example in this blog post is rendered as an animated GIF. Due to clamping of GIF frame rates, lack of precision (the delay is specified as an integer counting hundredths of a second) and guessing at the screen's refresh rate (it assumes 60Hz), it will exhibit even more flicker than the programmatic version.
Ann and I have been using Avocado for the past few months to stay in touch while we're apart. It's been working well for us, but it didn't have a great workflow for sharing links (beyond copy-and-paste). Since it now has an API, I thought I could perhaps fix this with a Chrome extension.
Cilantro is that extension (source). It's a straighforward browser action that lets you share the current tab's URL to Avocado. It relies on you being signed into Avocado to make requests (and scrapes a bit of the homepage's JavaScript to get the current user's signature -- Taylor and/or Chris, please don't rename your variables).
About the only clever thing that the extension does is that if you're currently in a Google Reader tab, it will pick up the currently selected item's link to share. It relies on Reader exposing a getPermalink() JavaScript hook1. Chrome extension code normally runs in an isolated world when pointed at a page, so it can't see that function. However, by "navigating" the tab to a javascript: URL, it's able to invoke it, since that code runs in the "main" world. To get at the result, it adds a message listener and then has the javascript: snippet do a postMessage (since the main world can invoke the isolated world's event handlers). This is described in more detail in the extensions documentation.
For the sake of full disclosure, lest you think my graphics skills have suddenly improved, Avocado co-founders Jenna and Chris are friends of mine and they spruced up Cilantro's UI to look less programmer-y minimalist.
This was originally added so that Google Notebook (R.I.P.) could extract the current item when clipping from Reader. Since then, Instapaper's bookmarklet has also started to use it.
At work I frequently switch between quite a few computers: two Macs, one Linux box, one Windows machine, and the occasional Chromebook. Between a KVM (for the desktops) and a swiveling chair (for the laptops), this isn't so bad. The one part that felt awkward was the lack of unified clipboard support, which would be handy when IM-ing links or checking out URLs on multiple computers.
I initially used Syncopy to solve this problem, but it didn't fully solve my problem (due to being Mac-only). Additionally, I didn't really like the idea of my clipboard contents ending up on the developer's server (or having to run a local, could-be-doing-anything binary). Eventually, Syncopy stopped being a viable solution altogether since the developer abandoned it (see the reviews for the iPhone client).
It occurred to me that I could build a replacement as a Chrome extension. Extensions can access the clipboard, and the storage API provides a synced key-value store. Some quick experimentation showed that changes were synced within a few seconds, which was good enough for my needs. There was some rate-limitting, but it didn't seem like it would affect any day-to-day use.
Clipboard Sync (source) is my implementation of that idea. Its main UI surface is a browser action icon. When you want your clipboard synced, click the icon. On all the other Chrome instances that are running and are synced with the same account, you'll get a notification saying that the clipboard data has been pushed. Click on the notification, and the local clipboard will be updated (and the notifications on the other instances will be dismissed).
Clipboard syncing was actually filed as a feature request for Chrome a couple of years ago. It was (rightfully) WontFix-ed since it's a pretty niche feature. It's quite reassuring that the extension system is now flexible enough to allow it to be implemented pretty seamlessly (especially once the commands API hits the stable channel).
