It turned out that my hypothesis as to why libopendaap works on (x86) Linux but not on Mac OS X was correct - it is an endian issue. Thanks to a tip from jfpoole, the problem was traced to authentication/md5.c, specifically lines 47 - 51 that attempt to pick a byte reversal routine (if any). Upon running the ./configure script, WORDS_BIGENDIAN is #define'd to 1 (line 72). However, line 58 of authentication/md5.c does a #ifndef WORDS_BIGENDIAN test to see if the byte reversal routine should be set to a dummy one. By changing this to #if WORDS_BIGENDIAN and rebuilding libopendaap, one is able to successfully connect to severs. This still does not fix the auto-discovery issue (given its non-deterministic behavior, I still think it's a threading issue), so I'll still have to use CFNetServices for that.

On the Java front, One2OhMyGod became orphaned, and has since been resurrected as AppleRecords. A new release of the latter incorporates the necessary changes in order to support connections to iTunes 4.5 clients, thus this seems like a viable option as well. However, last I checked building this seems to be a pain (since it requires libraries like JRendezvous, a.k.a. JmDNS and JavaLayer). I also haven't looked through the source, to see how well separated the DAAP connection stuff is from the GUI and MP3 playback code.