Weather Text: My Own Bespoke Weather App #

I was a happy user of the Dark Sky weather app for many years. Even more than the localized and timely notifications (I live in a place with predictable weather) I appreciated its Apple Watch complications. I specifically used the three-line textual summary shown in the middle of the Modular face.

Apple Watch Modular face with Dark Sky complication
My preferred Apple Watch face, circa 2022

Apple acquired Dark Sky in 2020, incorporated many of its features into the built-in Weather app (with iOS 16/watchOS 10), and shut down the original app at the end of 2022. Unfortunately the complication layout was not carried over – the only large complications are multi-hour/day graphs that have too much information and are hard to see at a glance. On my iPhone, I first switched to Weather Line (RIP) and later to Weather Strip, but neither had a watch app. I considered CARROT Weather, which has customizable complications, as a possible replacement. However, it couldn’t replicate the same layout, and paying for a separate subscription just for a watch app seemed wasteful.

On the other hand my time is free, and I figured that with WeatherKit (the one good thing to come out of the Dark Sky acquisition), I could build my own complication that showed exactly what I wanted. I procrastinated doing this for more than a year (squinting at the Weather complication every morning and thinking “I should do something about this”) but I reached a lull in Infinite Mac development and decided to once again work on an Apple platform developed in this century. It turns out that my procrastination paid off: watchOS 9 introduced WidgetKit for complications and Xcode 15 made wireless debugging (the only option for Apple Watch development) more reliable.

After fighting a bit with my old nemesis, provisioning profiles, and figuring out how to get location access in a widget (which may change in the future), I had something up and running. WeatherKit makes it particularly easy to replicate the Dark Sky complication – there’s even a property to get an SF Symbol for the current conditions and pre-computed sunrise/sunset times

Weather Text is the resulting app - it has a minimal watch UI to request location access and show a preview, but the main focus is the complication/widget (which also shows up in Smart Stack).

Weather Text app screen Apple Watch Modular face with Weather Text complication

I decided not to put the app on the App Store. WeatherKit has a limited free quota, and while it was unlikely that the app would suddenly boom in popularity, I didn’t want to worry about it. I could have made the app paid up-front to deter casual installs, but my App Store account is not set up for payments, and I didn’t want to go through the hassle of that.

The app is instead available via TestFlight, which is hopefully enough of a deterrent that I’ll remain under the free limit (and builds expiring every 90 days also gives me additional control). I would have thought that only going through the lightweight TestFlight approval would exempt me from review shenanigans. However, I still had to change my original icon, despite the presence of plenty of other dark background icons, including the original Dark Sky one.

Weather Text is a home cooked meal/hammer kind of app, mostly meant for me (though I’m not the only one who misses Dark Sky’s complications). I don’t have grand ambitions for it, it’s not a forever project (modulo minimal upkeep to make sure it works with new OS releases). Though modern platforms don’t have quite as much end-user programmability as they used to (though widget.json/Widget Construction Set and Scriptable look interesting), as a middle-aged gentleman programmer it’s nice to still be able to make them fit exactly to my needs.

Infinite Mac: Turning To The Dark Side #

tl;dr: No, not Windows¹, but I’ve decided to expand Infinite Mac to the the black hardware that Mac users secretly lusted after in the 80s and 90s: NeXT.  There is now a runnable collection of all notable NeXTStep versions, going from the initial 0.8 preview in 1988 to the final OPENSTEP 4.2 release in 1997.

NeXTSTEP 3.3 running DOOM, DoomEd and WorldWideWeb

Porting Previous

About a year ago I came across the Previous emulator – it appeared to be a faithful simulation of the NeXT hardware and thus capable of running NeXTStep. While including it in Infinite Mac would be scope-creep, NeXT’s legacy is in many ways more relevant to today’s macOS than classic Mac OS. It also helped that it’s under active development by its original creator (see the epic thread in the NeXT Computers forums), and thus a modern, living codebase².

Previous is the fifth emulator that I’ve ported to WebAssembly/Emscripten and the Infinite Mac runtime, and it’s gotten easier. As I'm doing this work, I’m developing more and more empathy for those doing Mac game ports – some things are really easy and others become yak shaves due to the unintended consequences of choices made by the original developers. Previous is available on multiple platforms and has good abstractions, so overall it was a pretty pleasant experience.

The main challenge was the pervasive use of the SDL library – I don’t want any of it as a platform abstraction, since I have my own. I chose to replace the entry-point and several subsystems to avoid bringing in too much SDL code. Even with that, I still needed to have some stub files for dependencies that I couldn’t omit altogether. On the other hand, it was a pleasant surprise that adding sound support was trivial — the APIs that it needed mapped 1:1 with the ones I’d already exposed from the Infinite Mac runtime.

With the initial emulator being brought up, there were some more fun tasks, like adding a NeXT-style monitor frame and a NeXT appearance to the Infinite Mac controls (working on them is giving me Kaleidoscope scheme flashbacks).

Infinite Mac custom instance configuration dialog, rendered with a NeXTStep appearance

Once I had Previous running I did some profiling and was discovered the same excessive wasm-to-js/js-to-wasm ping-ponging that I noticed in DingusPPC – it also uses setjmp/longjmp to implement an exception handling mechanism for the MMU. The fix was also very similar: intentionally add an extra wrapping function that contains the CPU emulation loop.

The performance impact of the fix was not as noticeable because Previous tries to accurately simulate the performance characteristics of the original hardware (this is also why boot times are much longer). In fact, I had originally missed the speed checking that the native main loop does, and the emulator was running too fast, leading to timing-dependent things like double-clicking not working.

Speaking of mouse input, Previous uses relative mouse input — it accurately simulates real hardware so all it can do is behave as though a physical mouse is moving and generating updates. This was the case with DingusPPC too, and while the Pointer Lock API makes this work reasonably well on desktop browsers, it’s not a good fit for the mobile web. To make input more accessible on touch-enabled devices, I added a virtual trackpad mode that turns swipes and taps on the screen into mouse movements and clicks. This also helps with double-clicking and making pre-Mac OS 8.0 pull-down menus more usable, though it is somewhat finicky to trigger the drag-lock gesture (I now see why it’s so hard for other touchpad vendors to replicate all of Apple’s gestures).

As part of doing some of the integration work I became aware of just how much of a minority player NeXT was in the 80s and 90s computing world (even more so than Macs). For example, they had their text encoding/character set, and while Python (and other systems) have native support for the contemporary MacRoman encoding, I had to add my own to handle NeXT’s. Similarly, there is a convenient Python library for dealing with HFS disk images, but I could not find any for the UFS variant that NeXT used³.

The disk images with NeXTStep installs are quite large (hundreds of megabytes per version), and the Workers KV-based storage that I had been using was becoming increasingly ill-suited for them (and expensive). As part of this project I also switched to Cloudflare R2 (their S3-like object storage system). On one hand I’m not thrilled with the site becoming dependent on additional services, but on the other I do want to keep costs down. Luckily the dependency is not invasive — it would be trivial to store the disk chunks anywhere else (or to keep storing them as static files, the way I do in the local dev server).

Exploring NeXTStep

Although I was familiar with NeXTStep at a theoretical level and had spent a few hours using a slab in the mid 1990s, this was my first time using it in-depth. My first observation was that having it side-by-side with the contemporary Mac system software makes it even more apparent how much of an advance it was.

Notable OS releases of 1988: System 6.0 and NeXTStep 0.8
System 6.0 and NeXTStep 0.8: two very different releases from 1988

As I was preparing the system images with the various NeXTStep/NexTSTEP/OPENSTEP releases⁴, it was fascinating to see NeXT’s various strategy shifts over the years represented in the canned email from Steve Jobs that was present in the Mail app:
Steve Jobs email from NeXTStep 2.0 Steve Jobs email from NeXTStep 3.0 Steve Jobs email from NeXTSTEP 3.3
Strategy du jour: productivity (NeXTStep 2.0), multimedia and internationalization (NeXTStep 3.0), object-oriented development (NeXTSTEP 3.3)

I was also amused to notice a parallel with the early Mac system releases: the initial release is hard to find in pristine form (a copy of System 0.97 was only tracked down late last year). In both cases it’s it because of the very small number of copies produced, and because they shipped on read/write media (floppy disks for the Mac, magneto-optical disks in NeXT’s case), and thus ended up being used for personal storage too. For NeXTStep 0.8 the archive of images that forum member mikeboss created was the best source that I could find, though I’m happy to switch things over if a better one turns up.

As part of installing all of the OS versions I discovered that starting with NeXTStep 3.0 there is support for the HFS file system used by contemporary Macs. This means the Saved HD drive that has persistent storage also ends up working here. It can even be used as a virtual sneakernet to move data between NeXT and Mac installations on Infinite Mac (which means that it’s possible to get data in and out of the machines via Macs which have “The Outside World“ drive).

To make the systems more than a quick curiosity, I also created a NeXT version of the “Infinite HD” drive with pre-installed software. I had read stories about Doom and WorldWideWeb being developed on NeXTStep, and it was nice to make them accessible. I also added a NeXT version of the CD-ROM library feature, with one-click access to disks that have been preserved at the Internet Archive.

NeXT CD-ROM Library on Infinite Mac

As part of finding more NeXT software to try out, I came across Daydream, which allows Mac software to run on black hardware. It’s technically not an emulator, instead it takes advantage of the fact that NeXT and Macs both used the Motorola 68K CPUs. It effectively turns your NeXT machine into a Mac that you dual-boot via the help of an adapter kernel. The original version needed a hardware dongle with genuine Mac ROMs, but a few years ago the NeXT Computers forum resurrected it such that it can run entirely via software. It’s this variant (known as Darkmatter) that is included on Infinite HD. In some ways this justifies the whole Previous side-quest — it’s just a more convoluted way to run the same classic Mac software as the rest of the Infinite Mac emulators.

Darkmatter running System 7.1 emulated on NeXT hardware emulated in Previous in a browser

Things happen

Christopher Nolan (sort of) knows about Infinite Mac. I did a minor site redesign. After recalling the KanjiTalk New Year’s Day easter egg, I added custom date/time support so can you see it anytime you want. Infinite Mac is being used as a tool to make Playdate game art and other toys. You can now load files with resource forks straight from the macOS Finder. Apeiron now works. And most excitingly, DingusPPC has progressed to point where the Mac OS X 10.2 installer starts up.

Update: See also the discussion on Hacker News.

  1. Also, PCjs already has the Windows niche well-covered
  2. andreas_g is the Previous author, and he was very helpful in the thread I started about my porting work
  3. I did take a stab at writing a HexFiend template for UFS/NeXT partition maps
  4. See these pages for details on the capitalization rules.

Infinite Mac: The Case Of The Missing Text #

I had previously mentioned that I ended up fixing some DingusPPC CPU emulation bugs. This was my first time working at this (lower) level of the Infinite Mac emulators, and I thought it would be interesting to write up such an investigation.

There was an issue that was immediately apparent when I first got DingusPPC building: while starting up from the 7.1.2 Disk Tools floppy would result in a functional-looking Finder, doing the same from the 7.1.2 install CD would end up with many rendering glitches — notably no text, but also missing icons.

System 7 Finder window
7.1.2 Disk Tools Finder
System 7 Finder window with no text or icons drawn
7.1.2 Install CD Finder

Classic System Software is mostly a closed-source black box (outside of leaks), and while DingusPPC is open source, it’s a codebase that is still pretty foreign to me. In the absence of other ideas, I figured that creating a more reduced test case would be a good use of time. This reminded me of my investigations into Chromium/WebKit rendering bugs, where half the battle was reducing a complex web page down to a minimal test case.

I examined the two system folders and observed that the CD-ROM one was quite a bit larger, with additional extensions, a different enabler¹, and a bigger System suitcase.

System Folder from the 7.1.2 Disk Tools floppy, list view System Folder from the 7.1.2 CD, list view

I created a read-write copy of the CD-ROM system folder and used a native build of Basilisk II to operate on it, removing or switching out files one a time to morph it into the floppy version, and looking for any differences when booting it in DingusPPC. It turned out that switching out the “Minimal PowerPC Enabler” on the floppy with the full “PowerPC Enabler” one from the CD-ROM was the key difference — with the full enabler text would not render.

I then opened the two enablers in ResEdit and compared them — the full one had a bunch more resources. I thus repeated the process, deleting extra resources one at a time to morph the full enabler into the minimal one.

ResEdit view of resources in the Minimal PowerPC Enabler ResEdit view of resources in the regular PowerPC Enabler

It turned out that the culprit was the ntrb resources, specifically removing ID 16 (StdTextTraps) and 18 (QuickDrawTextTraps) would restore text rendering. They contain PowerPC executable code — the giveaway is the Joy!peff prefix that is the header of the Preferred Executable Format. “Traps” refers to the system/Toolbox calls. My assumption is that the ROM of the first generation of Power Macintoshes was finalized relatively early, and contained mostly 68K code that would need to be emulated². The PowerPC enabler contained native implementations of more of the Toolbox, which were loaded in via these ntrb resources. These particular resources had native implementations of the text subsystem, which was very much in line with where the buggy behavior was.

Between resource_dasm and some help from DingusPPC Discord member joevt I was able to get a list of the 14 Toolbox traps that were being reimplemented as native PowerPC versions. The one that immediately stood out was StdTxMeas – here’s a description of it from Inside Macintosh: Text:

The QuickDraw text routines use two of the bottleneck routines extensively—one to measure text (StdTxMeas) and one to draw it (StdText). Most of the high-level QuickDraw text routines call the low-level routines. The use of bottleneck routines provides flexibility to QuickDraw and applications that need to alter or augment the basic behavior of QuickDraw.

While I had significantly narrowed where things were going wrong, I was reaching the limits of what I could via basic file system and ResEdit hacking. Ideally I would have a simple test program that just called StdTxMeas, so that I could examine its behavior in isolation.

It was thus time to crack open CodeWarrior and write a simple test program. I used the SillyBalls sample code as a starting point — there certainly used to be a lot of boilerplate in getting a basic window up on the screen. I created a simple test function that draws a string and then its bounding box as determined by StdTxMeas. Lacking the ability to render debugging text, I used FrameRect as the output mechanism.

void TestStdTxMeas(WindowPtr mainWindowPtr) {
    Rect measureRect;
    Str255 testString = "\pHello World";
    Point number = {1, 1};
    Point denom = {1, 1};
    FontInfo fontInfo;
    int width;
    
    SetPort(mainWindowPtr);
    MoveTo(20, 20);
    TextSize(12);
    
    DrawString(testString);
    
    width = StdTxMeas(testString[0], testString + 1, &number, &denom, &fontInfo);
    
    // +1 for the width so that if we get a zero we can still see the rect.
    SetRect(&measureRect, 20, 8, 20 + width + 1, 24);
    FrameRect(&measureRect);
}

I set the program as the startup item (so that I wouldn’t have to navigate blindly in a half-drawn Finder window) and observed its behavior under both emulated and native implementation of the text traps.

QuickDrawTester window showing expected output of "Hello World" surrounded by a rectangle QuickDrawTester window showing incorrect output with just a black line

Sure enough, it looked like when using native PowerPC code, StdTxMeas would end up returning 0 instead of the expected value. I suspected a bug in DingusPPC’s PowerPC CPU emulation, but it was unclear where. I had some false starts:

  1. I tried to disassemble the StdTxMeas implementation – I got lost when going through too many layers of indirection due to the calling convention.
  2. I looked at its 68K implementation and checked some of the intermediate building blocks – the width table looked correct.
  3. I tried to get MacsBug running so that I could single-step through it – I was not able to get it to succesfully suspend execution under DingusPPC.

As a variant of attempt 3, I decided to make DingusPPC output the instruction stream that it was executing, hopefully there would be a clue where the problem was. To signpost the instructions that represented the implementation of StdTxMeas, I did two otherwise no-op divisions by specific literals (the MoveTo call is to ensure that the compiler didn’t decide to optimize them out).

GetPen(&pen);
pen.h = pen.h / 32749;

width = StdTxMeas(testString[0], testString + 1, &number, &denom, &fontInfo);

pen.v = pen.v / 32719;
MoveTo(pen.h, pen.v);

Checking the CodeWarrior dissembly I could see the expected li (load immediate) and divw (divide word) op codes:

// pen.h = pen.h / 32749;
    
000000A0: A861003A  lha      r3,58(SP)
000000A4: 38807FED  li       r4,32749
000000A8: 7C6323D6  divw     r3,r3,r4
000000AC: B061003A  sth      r3,58(SP)

I could thus look for the 32,749 divisor in DingusPPC’s divw implementation and then set a flag to trigger its built-in disassembler (and turn it off when the divison by 32,719 occurred). 

Out of this I had a list of 635 instructions that were executed, which was a lot, but still made for a starting point. Looking at counts of the types of instructions, there were 59 unique ones, with loads and stores being the most common. I figured that the most popular instructions were least likely to be buggy, otherwise more things would have been broken. Looking at the bottom of list, two stood out: nabs (negative absolute) and doz  (difference or zero). They both had no coverage in DingusPPC’s test suite³ and were obscure — they are actually part of the POWER instruction set that was the predecesor of PowerPC, and were only included in the PowerPC 601 chip to help with IBM’s migration to it.

Looking at the implementation of nabs, it seemed relatively straightforward — it would always set the sign bit to to 1, thus making the number always negative:

ppc_result_d = (0x80000000 | ppc_result_a);

However, negative numbers are not actually represented as positive numbers with the sign bit set, they use two’s complement. Using Julia Evans’s handy integer.exposed we can see the representation of 123, -123 and 123 with its sign bit flipped, and thus why this implementation is not right.

I switched it out to the more readable explicit sign test and negation:

ppc_result_d = ppc_result_a & 0x80000000 ? ppc_result_a : -ppc_result_a;

Things still weren’t working, but as I had previously mentioned, doz was also suspicious (and it was the instruction that was executed right before nabs). I noticed a copy/paste error in which register the result was being stored in, and once I fixed that, everything began to work!

Correctly rendered System 7.1.2 Finder, showing the contents of the "Power Macintosh CD" drive

This whole investigation took about a week’s worth of evening hacking time, and there was a bit of luck involved in the end. I very easily could have ended up on a wild goose chase looking at some of the other infrequent instructions, and given up before I got to nabs and doz. Or the implementation bugs could have been much more subtle. However, it worked out, and it was really satisfying to be able to contribute to the DingusPPC core.

  1. System enablers were used to patch in support for Macintosh models released after a base version of the OS shipped.
  2. Much of the Toolbox code was written in 68K assembly and so could not be easily ported when Apple made the 68K-to-PowerPC transition in 1994. Emulation was thus the solution not just for older software, but also for large parts of the OS too. It was not until the transition to Mac OS X in 2001 that the need for 68K emulation went away.
  3. The DingusPPC test suite is partially based on the one for the Dolphin emulator. Since that emulates a G3-class CPU, it does not need to worry about the quirks of first-generation PowerPC CPUs.

Infinite Mac: DingusPPC Explorations #

One of my goals for Infinite Mac is to learn more about computer architecture and other fundamentals. While fiddling with actual old hardware is not as interesting to me (the one classic Mac I own is a PowerBook 550c that I turn on about once a year), I do enjoy operating at lower levels of the stack than I normally encounter. Interacting with the emulators that I’ve ported to Emscripten has definitely exposed me to more of this world, but it’s still been pretty superficial. Even the FPU fidelity tweaks from last year amounted to just changing some build flags once I understood what was going on.

That being said, I’m not sure that spending more time with the current set of emulators would be the best use of my time. Basilisk II, SheepShaver and Mini vMac are mature (old enough to drink or rent a car) codebases. That in and of itself is not bad, but the original authors are gone, the vision has drifted, they have fragmented into forks or states of semi-abandonment and have fundamental limitations (aggressive ROM patching and lack of an MMU).

With that in mind, I went looking for other options. The Mac Emulation Matrix spreadsheet lists many choices, and I was particularly interested in options that would eventually allow (early) Mac OS X to be runnable. The obvious choice was QEMU – it has very broad guest OS support and is very actively developed. However, it’s also a beast of a project to build and navigate around; it didn’t seem like it would be something I would able to make much progress on while working on it for a few hours a week. There is also some work on Power Macintosh support in MAME, but it’s a similar large codebase, and it seemed to be much further behind QEMU, compatibility-wise. I also briefly considered PearPC (which is much more focused on early PowerMacs than QEMU), but it’s also in a state of abandonment (development mostly stopped in 2005, with a brief resurrection in 2015).

I then came across DingusPPC, development of which has been going on for a few years (chronicled in this Emaculation thread and these Reddit posts). It had picked up steam recently, getting to a somewhat usable state, with some hope of booting Mac OS X too. The codebase was much smaller than QEMU’s (as was the resulting binary - 1.5MB vs. 17MB) and seemed reasonably modern. I crossed my fingers that this was a KHTML-vs.-Gecko-in-2003 decision and decided to give DingusPPC a try.

My first task was figuring out what worked under DingusPPC — it emulates lots of machines but some are more fleshed out than others. I settled on the Power Macintosh 6100 running System 7.1.2 (via the Disk Tools floppy) both because it seemed to have had the most attention from the developers and because it’s a combination that’s unsupported by SheepShaver or QEMU (both of which emulate PCI-era Macs). Once I had a working native build and machine/OS configuration, I started to work on getting it to build under Emscripten. DingusPPC doesn’t have much cross-platform support beyond what’s provided for free by SDL, and SDL’s Emscripten compatibility layer does not work for Infinite Mac (it assumes direct DOM access is possible from the generated code, but I run all emulators in a worker for better performance). I ended up with an approach inspired by Chromium’s, where some classes have per-platform implementations which are included automatically based on filename suffixes.

Usually when bringing up a new emulator I implement video output first, because it makes debugging much easier — if the ROM is being read correctly I should at least see a happy or sad Mac. I did the same thing with DingusPPC (DPPC from here on out), but I was disappointed to end up with a black screen.

Black screen when tryig to boot DingusPPC
Black screens - it’s not just for graphics programmers.

To get a high-level overview of what was happening, I enabled DPPC’s CPU profiling support and added a periodic dump when host events were read (i.e. every 11ms). I could then compare the counters when booting the same configuration in both the native and Emscripten builds, and see where the divergence occurred. It became apparent that after around 300ms the Emscripten build was stuck in a loop, since the number of supervisor instructions remained flat, while it would occasionally increase in the native one.

I was stumped for a while until I increased the logging level to maximum verbosity, and noticed that a AMIC: Sound out DMA control set to 0x0 line was logged right before the divergence started. I had seen that the native build starts a background thread to pull data from the sound DMA channel, but I had left all of the sound support in the Emscripten build stubbed out, since it’s usually something that I tackle much later. In this case sound turns out the load-bearing — if the DMA channel does not get drained, the boot process hangs. I added a basic polling loop to read from the DMA channel and send it to the JS side, and I got the flashing question mark I was expecting:

Flashing question mark boot screen

The reason why the flashing question was expected is because DPPC had no way to read the startup floppy disk image — it needs to be bridged to the JS APIs that handle streaming of content over the network. I moved all of DPPC’s disk image reading behind an abstraction, added a JS-backed implementation, and then I had a successful boot from the floppy.

Welcome to Power Macintosh startup screen
System 7.1.2 was the only release to say “Welcome to Power Macintosh” when starting up one of those machines.

Once I had the system booting, input was the next task. I was able to send mouse updates pretty easily, but it became immediately apparent that the position was handled differently. Other emulators I’ve ported operate in terms of absolute coordinates, but DingusPPC pretends to have a physical mouse attached, and only sends deltas to the guest OS. While adding support for deltas was easy, I ran into the issue that the guest OS treats them as raw inputs, and then computes an extra acceleration curve based on them, thus movements were not 1:1. For now I chose to implement pointer lock support (another case where a modern web API makes things easier), but the likely better long term fix is to make DingusPPC support the extended ADB protocol, which includes the ability to send absolute coordinates (originally meant for graphics tablets).

I then went to make keyboard input work, which seemed easy enough, but it didn’t seem to work. A quick check in the native build showed that it wasn’t working there either, because the ADB keyboard implementation was actually a stub. I figured this was a good first contribution to upstream to the DPPC project, so I began to spend some quality time with tech note HW01 and the relevant chapter of Inside Macintosh. The protocol was straightforward enough, but it took me a while to catch on to the fact that there was a separate polling loop that needed to be triggered. I did eventually succeed, and the keyboard now works in both the native and Infinite Mac builds of DPPC (though I did have to make some followup fixes).

Key Caps desk accessory

During all of this development I observed that DingusPPC was extremely slow (as in, more than 3 minutes to go from boot to the Finder running). While this could be attributed to its CPU emulator being a pure interpreter and the fact that it more accurately emulates the timings of a floppy drive, the Emscripten build was particularly slow. I ran a profile and was surprised to see that a lot of time was being spent going between JavaScript and WebAssembly.

Bottom-profile results showing top functions
Top Functions
Flame graph showing call stacks
Sampled call stacks

While I do end up calling to JavaScript from the input, disk and display implementations, those hops are relatively rare. These were happening in the core CPU emulation loop, and thus every instruction. The js-to-wasm and wasm-to-js functions and the bouncing back-and-forth is done by Emscripten to implement setjmp/longjmp support, and DPPC uses those to implement exceptions in the CPU. While there is an option to use native Wasm exceptions, it’s only supported by relatively modern browsers, and I try be pretty conservative about which browsers are supported (the non-SharedArrayBuffer fallback mode is still being used by 5% of users).

I made an attempt at retrofitting DPPC to not use setjmp/longjmp, but it became extremely invasive due to the need to abort execution in all of the codepaths that exceptions could happen. I then took a closer look at the stacks that that the profiler showed and noticed that ppc_exec_inner was not in them — it was getting inlined into the ppc_exec outer loop, which is also where the setjmp call was. I had a hunch that forcing the function to not be inlined would change when the JS ↔ WASM hops would happen, and I was right — they only happened once when starting the main loop, and boot times became 2.5 times faster with this 1-line change.

All my work until this point had been from floppy or CD-ROM images, but I was hoping to make hard drive images work too. It turned out that DPPC expects full device images (with a partition map), whereas I had previously operated only on bare HFS partitions/disk images. Coincidentally around this time the creator of BlueSCSI filed an issue asking for device image support, and had some technical pointers. I was able to automatically wrap my existing images and have them work for both BlueSCI uses and as boot devices for DPPC. Unfortunately DPPC’s SCSI hard drive support is still nascent, so while the hard drive boot process begins, it’s not able to complete.

The current status is that there are a few experimental DPPC-based machines that can be run, currently only accessible via the custom instance option. Some notable combinations:

Only the first three boot successfully, but it’s still pretty satisfying to to have gotten this far. Code-wide, I still have my own fork, but I’ve been able to upstream 20 or so changes, including quite a few that help the native build too. I hope to blog about the CPU emulation ones soon — they do scratch my itech of learning more about computer internals. The DPPC tracking issue has some notes on remaining work, and I’ll be rebasing my fork as the project makes progress.

On the non-DingusPPC front, I’ve made a few small improvements to Infinite Mac:

  • The aforementioned device image support means that it’s possible to import and export .hda images for use with BlueSCSI and other physical devices, here’s a demo from a user that tried it.
  • I added custom display size support (for Basilisk II and SheepShaver-based machines), so that you can pretend to have Portrait Display or have the Mac take up your entire window or screen.
  • I kept running into instability with the Infinite HD drive under System 6, seemingly due to a corrupt desktop database. The simplest solution turned out to be to generate a smaller disk image for older OSes (since a lot of 90s-era software didn’t run on them anyway). Chances are the Finder from this era was not really set up for a 2GB disk with tens of thousands of files.

Finally, I would remiss if I didn’t mention Software Applications Incorporated’s website, which takes Infinite Mac in a very interesting direction.

Infinite Mac: Improved Persistence #

Infinite Mac has supported a limited form of persistence since its initial launch. This was done by exporting the contents of the “Saved” folder in “The Outside World” to IndexedDB when the emulator was shut down. While this worked, it had several limitations:

  • Saving was best-effort during the page unload process. The beforeunload event that it relied on does not always fire, and it required an asynchronous postMessage step by which point the process for the page may have been terminated.
  • It relied on Basilisk II and SheepShaver’s ExtFS support (which creates a virtual drive using the File System Manager). This meant that it was not available in Mini vMac-based emulators (i.e. anything for System 6 or earlier).
  • Even when ExtFS/File System Manager is available, the virtual drive has several limitations: some software thinks it’s a networked drive and refuses to run on it, and it’s not bootable.

With the improved disk abstractions that enabled remote CD-ROM support, I realized that that I could mount another hard disk image in the emulated Mac, intercept read and write calls to it, and handle them from a persisted store. This was made possible by the new-ish origin private file system (OPFS) API, and specifically the ability to do synchronous file operations from a worker (without heroics). It’s supported in all modern browsers, including WebKit-based ones. The synchronous part is key because it allows all changes to be persisted as they happen, instead of during page unload.

The specific approach that I settled on was to have a mostly empty 1GB HFS disk image that serves as a “Saved HD”. Any modifications to it (tracked using the same 256KB chunk granularity that’s used for disk image streaming) are persisted in an OPFS file (a second file maintains the indexes of all modified chunks). The “mostly” empty part is because there is some metadata (desktop database, custom icon) as well as a Read Me file that are in the initial state of the disk image. This system makes the actual OPFS space that’s consumed to be proportional to the amount of data written, instead of ballooning to 1GB from the get go. It could also be extended to support a snapshotting/copy-on-write system down the line.

While OPFS is pretty broadly available, Infinite Mac still has enough visitors from older browsers that I wanted to detect support for it (the API is also not available in private Safari windows). One annoying gotcha is that the synchronous file API is only available in workers, and there’s no way to detect its presence from the main browser process. I therefore have to create a temporary worker to check for the existence of the createSyncAccessHandle and related functionality. The check is done as early as possible and the results are cached, so that most of the time we can synchronously determine the support level.


Settings dialog showing options to import and export the Saved HD

With all that in place, it’s possible to use the Saved HD as a “real” disk in the emulated Mac. This enables quite a few capabilities:

  • You can install system software on Saved HD (either by copying it from an existing disk or via a System Software CD-ROM from the library). It can then be used as a startup disk (if running a custom instance with no other disks), allowing custom extensions and control panels to be installed and persisted across sessions.
  • The contents can be exported and imported (from the emulator settings dialog), allowing backups and sharing across browsers and machines.
  • The contents can also be exported to a .dsk disk image file, so that they can be used in Basilisk II, SheepShaver and other native emulators.
  • The same disk is mounted in all instances, so it's also a way to move data from one to system disk to another.

System 7.5 install with a Kaleidoscope scheme and ResEdit editing an icon
Your own private System 7 install

The end result is a way to make Infinite Mac feel like “your Mac” while still keeping everything local and fast. That also lines up well with my goal of keeping the site a fun hobby project — while persisting disks on a server would be neat, it would also be a more expensive and complex operation.

Update on October 1, 2023: This has been extended to also support generating of disk device images (.hda files), which allows Infinite Mac prepare disks for BlueSCSI devices. Ron's Computer Videos has a walkthrough of the process.

20 Years of Blogging #

The first proper blog post on this site was 20 years ago today. It lived under a different URL, I didn’t register persistent.info until later (when .info was a new-and-shiny top-level domain). It also looked different (I don’t have a screenshot of the first version, but I switched themes a few months later) and was published by very different software (Movable Type 2.64), but there is continuity from then to today.

"10 years, man!" scene from Gross Pointe Blank with the 10 crossed out and replaced by 20

Later that day I imported a bunch of older programming journals, thus the entries go back almost 25 years. The first few years are not really blog posts, but it’s been nice to have snapshots of early work, even if they’re not all gems. I’ve been backfilling a projects page, and the posts also serve as tangible artifacts to link to, even if the project itself is long-dead.

Year 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023
Posts 13 3 4 2 4 1 2 5 7 12

The trend for posts per year is looking encouraging when looking at the past decade, here’s to a few more (while computers still work).

Infinite Mac: Disks, CD-ROMs and Custom Instances #

Here are a few highlights of what’s been happening with Infinite Mac since my last update.

Improving Disk Abstractions

My broad goal this time around was to make it easier to load external software. To lay the groundwork for this, I first needed to improve some of the disk abstractions that the project used. One of the things that makes the emulated Macs relatively fast to boot is that they stream in their disk images, since much of it is not needed at startup (e.g. Mac OS 9 only reads 75MB out of a 150MB install). I had implemented this streamed/chunk approach pretty early in the project, when I understood the emulator internals (and Emscripten) less well. It worked by monkey-patching the read/write operations in Emscripten’s MEMFS filesystem (the default when doing an Emscripten build).

Besides being somewhat hacky, this had the downside of requiring that all mounted disks be loaded into memory as ArrayBuffers (hence the name MEMFS). While the system software disks are relatively small, the Infinite HD disk with pre-loaded software is 1 GB, thus this was leading to significant memory use. I had added some basic instrumentation of emulator errors, and running out of memory was surprisingly common (6.5% of emulator starts), presumably due to low-memory iOS devices or 32-bit Chrome OS devices. If I was going to add the ability to load additional (CD-ROM-sized) disk images, the memory use would increase even more.

I briefly explored Emscripten’s other file system options, but they all didn’t seem like quite the right fit. Fundamentally the issue was that all disk access was going through too many layers of abstraction, and the intent was lost. The emulator was operating against the POSIX file system API (open, read, on a file descriptor etc.) and by the time Infinite Mac’s code ran, it required extra bookkeeping to know which file was being requested (it could be a non-disk file), and the choice for the backing store for that file (as an ArrayBuffer) had already been made by Emscripten.

5 layers of abstraction
Before
3 layers of abstraction
After

While many problems in computer science can be solved with a layer of indirection, in this case it was a matter of removing one or two. The emulators are cross-platform systems themselves, and they have a way of swapping out the POSIX file backing for disks with others for other platforms. By adding more direct “JS” disk implementations in both Basilisk II/SheepShaver (hereafter “Basilisk”) and Mini vMac and then implementing the JS APIs on the Infinite Mac side, the entire POSIX and Emscripten layers could be bypassed. We now only need to use as much RAM as the number of 256K chunks that have been accessed (and there is enough flexibility in the system that a LRU system could also be implemented to keep the working set fixed in size). After this change the out-of-emory error rate went to 0.3%.

The next thing to tackle was avoiding the need to restart the emulators when mounting additional disks. This only affected the Basilisk version, since Mini vMac already had built-in support for mounting disks on demand. I assumed that this was not a fundamental limitation, all the emulators supporting mounting (real) removal media on demand already. It turned out to be a missing feature in Basilisk’s disk abstraction, and once I added that it was possible to mount disks immediately there too.

CD-ROM Library

With this in place, I was able to start working on the CD-ROM “library” feature I had envisioned. This was based on the observation that the Internet Archive has a large collection of disk images in the .iso format, include many for the Mac. Furthermore, the files are accessible without any kind of authentication, and support HTTP range requests. This meant that it’s possible to take the same chunked/streaming approach I was using for my disk images and use it for these files without any kind of pre-processing. It is rather amusing that download speeds and latencies over the Internet are better than those of 1-2x CD-ROM drives from the early 90s (150-300 KB/sec and 100-200ms seek times).

I do these range requests in simple handler in a Cloudflare Worker, mostly so that they end up being same-origin requests. I later found that there is a CORS gateway for the Internet Archive, but it appears to be a secondary service, and doing the fetching myself means that I can benefit from Cloudlare’s edge caching. However, switching to doing direct client-side loading is something to consider if the bandwidth costs become prohibitive (though for now donations are more than covering the costs).

One gotcha that I ran into this that some disk images are compressed. While there ways to make .zip files seekable, they require re-compressing, which would defeat the purpose of making this be a minimal setup/low overhead feature. For now I’ve chosen to skip over compressed files (or support other disk image formats that support compression, such as .dmg).

Continuing with the project’s curation philosophy, I wanted to have a nice browsable UI of CD-ROMs, ready to be mounted in the emulated Mac. I ended with a collection of metadata files that specify disk image URLs and cover art that is processed by a simple importing script. The generated catalog is rendered in a pop-up folder-inspired UI:

Infinite Mac with CD-ROM library pop-up window open

I added some of my favorites and got a few more suggestions via Mastodon. Around this same time a pretty throughly researched article on the first CD-ROMs appeared, and I hoped to include the ones referenced too. However, most of the early CD-ROMs were actually just taking advantage of the ability to include audio CD tracks, which is something that I have not gotten to work yet, so I was not able to add them.

I did also add the ability to mount arbitrary CD-ROMs via a dialog, so if you come across something on the Internet Archive that catches your eye, you can give it a try (you can even drag-and-drop the link from another tab). I also made the Infinite HD disk image 2GB (i.e. with 1GB of free space), so that it’s available as a destination for any software that requires installation to a local hard drive. This is another way in which the change to not require disk images to be loaded into RAM paid off.

Custom Instances

Around the time that I was wrapping up CD-ROM support, I came across the recently-launched Classic Macintosh Game Demos site. It’s a “kindred spirit”, in that it’s trying to make things easily accessible and has a curatorial bent. Jules Graybill (the author) was sourcing the game demos from CD-ROMs that came with magazines, images of which were also uploaded to the Internet Archive. I added the ability to specify a CD-ROM URL via a query parameter and reached out to him to see if he wanted to make use of it (to allow the games to be played in-browser). He quickly implemented it, which is a great example of the flexibility of web-based projects — they can be made to work together (some might even say “linked” or “mashed up”) with minimal coordination.

Shortly after, Jules filed an issue asking about an additional system extension he needed included on the system disk. Extrapolating from the ability to load CD-ROMs from URLs, I wondered if the same could be done for the system disk, so that he could provide his own image with exactly the extensions that he needed. And while we’re at it, why not allow the type of machine to be chosen via a URL parameter too, and control over other Infinite Mac features (AppleTalk support or the extra Infinite HD disk). Getting perhaps a bit carried away, I ended up building a Calculator Construction Set equivalent for emulated Macs, available at infinitemac.org/run.

Custom configuration dialog, matching a System 7 startup disk

Custom configuration dialog, matching a Mac OS 8.1 startup disk

To reduce the cognitive dissonance (and to have a bit of fun), I made the UI resemble the look-and-feel of the OS that is being booted. I had already added some classic- and Platinum-style controls for other bits of configuration UI, but this dialog also required implementing popup menus and checkboxes (the Appearance Sample app from the Appearance SDK was invaluable in getting a view of all “modern” Mac OS controls). There’s still a bit of a an uncanny valley feel, perhaps due to fonts not being the same and modern screens having higher logical DPI (thus everything feeling smaller than it used to), but hopefully it will get closer over time.

You can also use this customization mode to have “Frankenstein” instances. For example, you can load the System 1.0 image while booting with System 7, allowing you to use a modern version of ResEdit to poke around the initial versions of the System and Finder. Or you can use this with any system disks that you may happen to come across (e.g. if you wanted to see what the Slovenian version of System 7.5 looked like).

Odds and Ends

While the goal of the site is to make as many versions of classic Mac OS available in the browser, I realized some time after launch that 40+ releases is a lot to scroll through, and that more notable ones might get lost among all of the point releases. I therefore added a simple filter toggle to make the list that’s initially shown more manageable.

A longstanding issue was that dissolve animations in HyperCard would run very slowly, something that affected the native build of Basilisk II too. It turned out to be due to a missing implementation of high-resolution timers, which was recently fixed on the native side. While simply updating my Basilisk II fork did not get the fix for free, I did now have enough clues to implement my own version of it.

There was the usual round of upgrades for the software stack: a new way to install Emscripten, transitioning from create-react-app to Vite (that one was a yak shave), and switching to the latest version of TypeScript, React, and other dependencies.  Though site performance hasn’t really been an issue, the switch to Vite made the initial bundle size more visible, so I spent a bit of time adding moving some things to dynamic imports and some preloading to pick up some easy wins.

I haven’t entirely decided what I’m going to focus on next, but I’m leaning towards improving the ability to persist changes to disks — it’s a shame to spend time installing software from CD-ROMs and then lose it all when closing the tab. The improved file system abstractions should make it easier to implement some of my ideas here.