Making git faster on large repositories on Mac OS X #

Update on 12/9/2010: Tim Harper, the maintainer of the pre-built Git OS X binaries, has switched to providing both i386 and x86_64 packages, thus there's no need to build your own anymore.

tl;dr version: Install 64-bit git binaries for Mac OS X for a performance boost when dealing with large repositories.

WebKit uses Subversion upstream, but thanks to long code review times and ChangeLog files (which must be updated for every commit, making it hard to have concurrent changes in the same checkout), using a Git checkout is necessary to maintain sanity.

Git is generally slower on Mac OS X, and this especially seems to be the case with WebKit. The most likely culprit is the repository size -- a WebKit checkout has more than 100K files (most of which are layout tests). For example, running git status at the root of the repository takes 6.58s using the standard pre-built binaries. This does a lot of stats and getdirentries. However, even doing this on a much smaller part of the repository wasn't as fast as it could be (comparing with a Linux machine), e.g. git status WebCore/fileapi takes 1.45s, even though that directory only has 110 files.

Running git status WebCore/fileapi through dtruss showed that it was doing lots of mmaps of 32 MB chunks of the main packfile (this was after a git gc, so there was only one packfile). Thanks to pointers from Evan Martin and Junio Hamano, I was pointed at the core.packedGitWindowSize config, which defaults to 32 MB on 32-bit platforms and 1 GB on 64-bit platforms. mmap is slower on Mac OS X, so having to do it more frequently (with a smaller window size) was problematic. Separately, it turns out that I was running 32-bit binaries, which are the only easily available ones for Mac OS X (64-bit binaries aren't provided since due to concerns about bloat.)

After I built a 64-bit binary, git status WebCore/fileapi went from 1.45s to 0.75s, and git status from 6.58s to 2.50s (these are averages of 4 consecutive runs after doing a warmup one, so that disk cache effects should be similar).

You can either download pre-built 64-bit binaries or build them yourself. The actual change was very straightforward.

Post a Comment