If your only tool is a hammer, they say, all problems resemble nails.

My main skill is computer programming (although I have a few others, like playing piano). I'm an idealist: I want to improve the world, and I don't care a whit about making money for myself or anyone else. So I've spent my life developing software for purposes that I view as worthwhile. This essay describes these efforts.

High school

In the spring of 1973, I was a high school senior in Brookline MA, a suburb of Boston. I played classical piano, ran track, and was socially awkward.

My brother Stephen was a sophomore at Harvard. He had access to a computer there, via a Teletype terminal, which had a clunky keyboard and printed onto a roll of paper. I don't know what kind of computer it was. It had a BASIC interpreter. I read the manual.

I saw that programming lets one create logical machines of limitless complexity and beauty. This idea resonated strongly with me. I knew at once that my future lay in this direction.

My first program simulated a game of pool. The program kept track of the positions of the pool balls. To make a shot, you entered a direction and velocity for the cue ball. The program then simulated the movement of the balls, taking into account rolling friction, and the ball/ball and ball/cushion collisions. Once all the balls had stopped, it printed a crude ASCII picture of the table. Writing this program was way harder than I imagined at first.

My GF of that era (Sonia Sultan) had an uncle who ran a software-related company. I applied for a job, with no qualifications other than my considerable hubris, and was turned down. Their loss!

College: 1973-1977

That fall, having failed to get into Harvard, I enrolled at Wesleyan University, which had (for that time) great computing resources:

• A DEC PDP-10 running TOPS-10. It had compilers for FORTRAN and BLISS (a DEC system programming language). The administration used the PDP-10 for accounting etc., but they also let students use it for whatever they wanted. The accounting stuff was done using punched cards. There were also six or so CRT terminals (24x80, cursor-addressable, Hazeltine and Wyse).
• A DEC GT40. This was the combination of a PDP-11 16-bit computer (no floating point hardware) and a graphics system. The computer didn't have a disk or operating system. There was a bootstrap program in ROM that loaded code over a serial line, and a PDP-11 assembler that ran on the PDP-10. To use the GT40, you had to write an assembly-language program on the PDP-10, assemble it there, then (using front-panel switches on the GT40) load it into the GT40 memory and run it. The GT40 had 32KB of core memory (little magnets); later they added another 32KB at significant cost (like $30K).

Wesleyan had purchased two audio-related add-ons for the GT40:

• Banks of eight 12-bit A/D and D/A converters;
• A fancy custom analog synthesizer, with lots of oscillators, filters, and ring modulators, and hundreds of patch cords.

I don't know who built these; Ron Goldman may have been involved.

The purchase of the GT40 and audio equipment was (I think) the work of Alvin Lucier, a professor in the music department who specialized in electronic music. The GT40 was kept in a locked room near the computer center; I managed to get a key.

The GT40's graphics system had a green vector display. To draw things, you created (using the CPU) a 'display list' in shared memory, describing points, vectors, and ASCII characters. The graphics system would loop over this list and draw things. You could dynamically change the list, changing the graphics instantly.

The GT-40 came with a game called Lunar Lander, a real-time 2-D game where you try to land a LEM by rotating it and firing engines.

I soon met fellow students who had huge influences on me. Matt Ginsberg was a sophomore. He was a math/physics/computer genius; he later got a PhD in physics from Oxford, and developed the world's best software for scheduling huge projects (like building aircraft carriers).

Matt idolized and emulated Star Trek's Mr. Spock. He had written a real-time multi-player game called Trek that ran on the PDP-10. It was a 3D space combat game. A CRT terminal acted as your 'starship console'. It showed the relative position and velocity of your opponents, and the derivatives. You could fire phasers and photon torpedoes; you had to 'lead' them to get a hit. There was a Comms area where you could chat with and taunt other players.

A group of 10 or so nerds met in the computer center most nights and played Trek until the wee hours. This included me, Chris Terman, Phil Tong, Bobbi Spellman, and Linda Hayes. We might form teams (with occasional back-stabbing) or have a free-for-all.

One of these 'Trekkies' was Alex Strong, a junior. He knew a lot about everything STEM-y. He taught me the basics of 3D graphics (e.g. the perspective transformation) and audio DSP. Alex went on to invent the Karplus-Strong Plucked String Algorithm, a simple and efficient way to produce a realistic guitar sound.

Earlier (during orientation week) I had met Ron Kuivila, who was interested in many kinds of modern music, as well as mixed-media things like sound installations. Ron and I become close friends and took many math and music classes together. He was into technology mostly in service of his pieces. He was more into analog electronics than computers, though when he needed computers to do something he had no problems figuring out how.

I was into crossword puzzles even back then. I got hold of a word-list file, and wrote a FORTRAN program that tried to fill in an arbitrary (black-square) grid with words, using a back-tracking algorithm. My program worked for small grids but couldn't finish larger ones (e.g. 15x15).

I told Matt about this, and the problem interested him too. Within a week or two he'd written a program that worked for big grids. Later in life he continued to work on crosswords, and developed a program, Dr. Fill, that solves actual clued puzzles (not sure how this works). He's also published 50 crossword puzzles, including some in the NY Times!

UPDATE: writing this essay inspired me to have another try at the grid-filling problem (see below).

Meanwhile, I was exploring the potential of the GT40. The display-list architecture facilitated making psychedelic Matrix-style ASCII graphics, and I spent some time playing with this.

I wrote a 2-player, real-time 2D game called Tanks where you maneuver your tank around obstacles and try to shoot the other tank. It used the front-panel switches as input; the keyboard didn't work well with 2 people using it at once.

Having learned about 3D graphics from Alex, I wrote various simulators where you fly around and dock with space stations.

Although the PDP-11 had a great orthogonal instruction set, programming in assembly language could be tedious. I wanted to be able to program it in FORTRAN. The PDP-10's FORTRAN compiler came with source code (in BLISS). This used an intermediate representation with a stack-based virtual machine. Instead of generating PDP-11 code from this, I wrote some PHP-11 assembly code that implemented the virtual machine. I got all this working - kind of amazing, since at the same time I was taking a full course load, running track, and experimenting with various improprieties. But I don't think I ever used it for anything.

I was an ambitious pianist. I played hard pieces, and some were so hard I knew I'd never be able to play them the way I wanted. It occurred to me that I could use computers and synthesizers to create virtual performances of such pieces. This idea has remained in the back of my mind, and I've pursued it to this day. Unfortunately the equipment at Wesleyan was not well-suited to this; it would have been hard just to play equal-tempered notes on the synthesizer.

Alvin Lucier taught a class in Electronic and Computer music, which I took as a freshman. The GT40 and the synthesizer had been bought in part to use for projects in that class.

Except for Ron Goldman (who wasn't around much) I was the only one who had figured out how to use the GT40. So in my sophomore (and possibly junior) year I acted as TA for Alvin's class. I taught the students (mostly humanities majors) about computer architecture and assembly language, with varying (mostly low) levels of success. I helped them think up projects. Unfortunately, these were often way too ambitious, and in many cases I ended up implementing them myself.

When it came time for graduation, Ron Kuivila and I were commissioned to make some computer music for the ceremony. He had the idea of recording the spoken first names of all the seniors, and playing this back together with a slowed-down and pitch-shifted version (i.e. ~10X slower but at the some pitch). I wrote code to do the pitch-shifting on the PDP-11 in a crude (non-FFT) way. It worked pretty well and sounded cool.

We did a lot of great stuff at Wesleyan, and we did it just for fun. It didn't occur to us to sell or publicize any of it, or show DEC our various games. There was no Internet back then. Later, Alex Strong licensed the plucked-string algorithm to Mattel, who used it in a cheesy plastic toy guitar.

Math grad school: 1977-1979

In fall 1977 I enrolled in grad school in the Math Dept. at U. of Wisconsin - Madison. For the first year all I did was math, focusing on logic (model theory, recursion theory). I didn't touch a computer.

I loved all my math classes (and aced them) but by the end of the year I realized that I didn't have the desire or talent for a career in math. So the next year I took some CS classes. I don't remember exactly what. Possibly AI, which at that point was mostly about rule-based systems, knowledge representation, and alpha-beta pruning. I wrote a program in Pascal to play Hex. I used a VAX and learned Unix, vi and probably some C.

UW Chemistry Department: 1979-1982

In summer 1979, after getting my MS in Math, I was at loose ends. I applied for a system programming job at Intel and did an interview in Santa Clara (lots of questions about distributed file systems, about which I knew nothing). They turned me down after a 2 month delay. F*** 'em.

At that point, UW had a 'campus mainframe' computer (Honeywell). You typed your program on punched cards, handed the deck to a guy behind a window, and got your output from the line printer an hour or two later.

Departments were discouraged from getting their own computers. But the Chemistry department had bought one anyway: a Harris /7, a 24-bit machine that ran a proprietary OS called Vulcan (written in assembler; I had a source code listing, which was about 4" thick).

The job of System Administrator in the Chem Dept. was opening up, and I applied for and got it, in spite of a total lack of qualification.

I worked there for 3 years and had a great time. My only official duty was to back up the disks onto 9-track tape (which took about 5 minutes per week of my time) and to call the Harris repairmen when something broke.

I hung out and talked to the chemists about their computing, and tried to figure how to help them. There were two main groups: the X-ray crystallographers, who used a program called OR FLS, and the physical chemists, who did ab initio quantum calculations.

The Harris had a crappy line-oriented text editor built into the OS. The first thing I did was to develop (in FORTRAN) a WYSIWYG editor modeled after vi; in fact I called it vi. This was wildly popular with the chemists.

The Harris computer, it turned out, made sporadic floating-point errors. The physical chemists - who ran multi-hour jobs - had taken to running each job twice to make sure the answers agreed. This problem got worse until eventually we had to shut down the system for several weeks while I harrassed the repair people. They eventually found that the floating-point board (yes, an entire board) would occasionally overheat when the mantissas contained too many ones.

The Chem Dept computer center had a 36" Calcomp pen plotter, which they used (with canned software) to make 3D hidden-line pictures of molecules. I got interested in using it to make shaded 3D pictures using cross-hatching and other shading approaches. Mostly I used this to make computer art, late at night. (I got away with a lot in this job.)

I did a lot of other graphics-related stuff. At some point someone (Joel Horn?) suggested to me that my work might be publishable. So I wrote 3 papers:

• Pen plotter optimization (traveling salesman heuristic).
• Hidden-line elimination in projected grid surfaces.
• Optimizing viewer orientation (computational spherical geometry).

Computer Science grad school: 1982-1985

Encouraged by the success of my papers, I decided to pursue a PhD in CS, so I left the Chemistry Dept. job and went to CS grad school full time.

Official work

I took the standard intro grad courses, including Olvi Mangasarian's course in optimization. He was a GREAT teacher, and I idolized him so much that I spent a year trying to do research with him, in large-scale linear optimization using non-linear techniques. Unfortunately I lacked any talent in that area.

I took Bob Cook's class in operating systems and idolized him too. He walked us through the Bell Labs Unix source code (in C), and I learned about virtual memory, processes, file systems, etc., all from the ground up. He also taught me how subtle and complex synchronization can be (and to avoid the need for it if possible).

In my first year of CS grad school (1982) I taught undergrad classes (for engineers, not CS majors). One was on PDP-11 assembly-language programming. The second was FORTRAN programming. I enjoyed this a lot but my programming assignments were too ambitious. For example, I had them write a simulator of a set of N elevators in a skyscraper, calculating the performance of various algorithms for elevator movement. I though this would be straightforward but it wasn't, and I had to greatly simplify it.

I started grad school vaguely in the theory end of CS, but after taking Bob Cook's class my interests shifted to operating systems and distributed systems. I took Charlie Fischer's class in compilers, and Larry Landweber's class in networks, which covered protocols like TCP/IP and the ISO stack. I combined these into the idea of using context-free grammars to specify transport protocols, and to write software to convert the specifications to (Unix kernel-level) implementations. For the latter I hacked on some VAX 11/750s running Unix (I don't remember what kind, maybe BSD).

Landweber was my advisor. Our weekly meetings were almost entirely gossip.

My protocol-specification idea never went anywhere. It was pretty wank, which I realized halfway through writing my thesis. But it had a nice theory/practice blend that got me some academic job offers (UC Berkeley and Brown; U. Wash and U. Toronto turned me down).

Unofficial work

At the same time, I was fiddling around with home computers. I bought an Apple II and wrote some pretty good animated graphical games (e.g. 'Space Worms') in 6502 assembler.

I bought an Epson dot-matrix printer, which was dot-addressable using control characters. I used this (in BASIC, I assume) to do 3D shaded graphics with various kinds of dithering, including ordered dithering. I sent Xmas cards with 3D shaded pictures of ball-ornamented trees. Pretty geeky.

I bought a Mountain Music synthesizer card, which shared the Apple's RAM and could play waveforms stored there (this used half the bus cycles, which slowed the computer down by 2X).

I implemented the Karplus-Strong algorithm on this (in assembler) and played around a little. But it was clear that I needed more computing power to do interesting algorithmic composition.

I had a friend named Bruce Holmer, a Chemistry grad student. Bruce knew about digital design, i.e. how to read data sheets for various chips and figure out how to connect them up.

Bruce and I decided to build a microcomputer using the Motorola 6809 16-bit processor, with 64KB of static RAM, an RS232 interface, and a floppy disk interface for an 8" Shugart drive. We designed it (he did 99% of the work) and built two of them using wire-wrap (I did most of the work). Bruce wrote a bootstrap loader, which we burned onto a UV-erasable EPROM chip.

Miraculously, it all worked: my wire-wrapping had zero errors, and the bootstrap loader worked the first time.

We bought a rudimentary operating system (on floppy) called OS-9. I bought a Wyse terminal (amber), connected by RS-232.

I connected the 6809 system to the Apple II via a 2nd RS232 port. I used the Apple like a MIDI synthesizer; it ran a program that listened for key-down commands and played them on the Mountain Music card, using the plucked-string algorithm.

Sometime around then I learned about Forth, a programming language developed by astronomers for telescope control. I wanted to use Forth for computer music because

• It uses little memory and runs fast.
• It lets you add new language features.
• It's interpreted, so the edit/run cycle is fast.

The downside of Forth is that it's extremely difficult to write or debug. It uses a stack-based, reverse-Polish model, so to read a piece of code, you need to keep a mental model of what's on the stack. It's exhausting.

I bought a Forth system that ran on OS-9. In Bob Cook's class I had learned all about processes, and I was eager to put this into practice. I imagined a language where the different components of a piece of music (pitch sequences, rhythmic patterns, tempo and volume fluctuation) are generated by separate processes.

This was pretty easy to do in Forth. I called the result FORMULA (Forth Music Language) At first, scheduling was non-preemptive, so process state just consisted of a stack pointer and program counter.

Around this time I reconnected with Ron Kuivila, who was now teaching music at Wesleyan. He got interested in FORMULA, and immediately started helping with its design and development. Over the next several years, we had a series of visits (Ron to Madison or Berkeley, me to Middletown) where we worked with extreme intensity of FORMULA.

I spent way more time developing FORMULA than making music with it. By this time (1985) I was focusing on my PhD thesis. The only piece I recall making was a rendition of Steve Reich's Piano Phase, which used only basic features of FORMULA.

UC Berkeley CS department: 1985-1992

In spring 1985 I applied for CS faculty jobs at various places. I got offers from Brown and UC Berkeley. I chose Berkeley because I grew up there and had family there.

This was probably the wrong choice. My 7 years in the UCB CS department were hellish. I mostly wanted to work on computer music and FORMULA, but that's wasn't going to lead to tenure, so I led a sort of double life; I'll describe the two parts separately.

Unofficial work

In 1985 Atari introduced the 520ST. It had a 32-bit Motorola 68000 and 512KB of memory: a giant step up from the Apple II and its kin.

Ron and I learned that Sun (which also used 68000s at that point) used Forth for their boot logic (detecting devices, etc.). The guy who developed this, Mitch Bradley, had written his own Forth implementation. He packaged it with an Emacs text editor and distributed it as Forthmacs; it ran on various platforms including the Atari ST.

Around the same time, Yamaha released the FB-01, a MIDI-controlled synthesizer module (no keyboard). It used frequency modulation (FM) with linear ADSR envelopes. It could make a wide range of sounds. None of them sounded much like actual instruments, and the 'piano' was especially bad. But hey, it worked, and it was cheap. I bought a couple of them.

The planets seemed to have lined up. Ron and I began working on FORMULA in earnest. We added preemptive scheduling, which means you could interrupt a process, in response to a keystroke or other event, and switch to another one. Ron needed this in order to use FORMULA for interactive pieces. This was a little tricky because you save and restore all the registers, not just the PC and stack pointer.

I met Rob Vaterlaus, a CS grad student who was a Mac expert and a musician. As his MS project, he ported FORMULA to the Mac and made a nice GUI where you could edit FORMULA code in multiple windows. Another student, Marc Sabatella, did a music-related master project with me; I can't remember the details.

FORMULA was the only computer music software with multiprocessing. But everyone used MAX, which ran on the Mac and had a nice GUI for representing chains of function calls. Mills College had their own Forth-based language, HTML.

So almost no one used FORMULA. Just Henricus Holtman. Truth be told, we didn't use it much ourselves. Ron he used it for a few pieces, including one called Tuff Stuff with cyclical pitches, I made a rendition of a Liszt piano piece, 'Les jeux d'eaux a la Villa d'Este'. It had tempo and dynamic variation, but was pretty lame, largely because the FB-01 piano sound was cheesy.

For the first few years at UCB I organized a weekly computer music seminar. It was attended mostly by non-UCB people like Henricus, John Bischoff, Tim Perkis, and Jim Horton.

Later, I met Jeff Bilmes and we developed a C++ library called MOOD (Music Object-Oriented Dialect) that implemented some of the FORMULA features using coroutines. We had a demo but that was it.

Official work

At the same time, I tried to do some real research. I was assigned a 'mentor', Domenico Ferrari, who was nominally in charge of the Berkeley Unix (BSD) project but knew little about system software. We started a foolishly over-ambitious project called DASH whose goal was to create a post-UNIX operating system. We had a bunch of graduate students but none of them knew much system programming. So the whole thing was a fiasco. I think we got as far as writing code to do context switches on a 68000. We wrote some technical reports proposing vague communication architectures.

It gradually became clear that tenure was not in my future. But I decided to keep doing research anyway, in the area of operating system support for continuous media (digital audio and video), with performance guarantees (delay and throughput). I invented something called the 'linear bounded model', which I think is the ideal way to describe performance guarantees for digital streams.

For a year or two, OS support for audio/video was a hot topic. There was even a conference or two. But - although I had founded the area - I was ostracized; Ferrari, who by that point had soured on me, was involved.

I had a couple of grad students - Pamela Chan, Ramesh Govindan, and a bit of George Homsy - working on this, and we developed a (simulated) audio/video filesystem, and an actual audio/video I/O server (kind of like a window server) called Comet. We published some pretty legit papers about these.

I gave a farewell talk about this work at UC. Mike Stonebraker attended this and said (in front of the audience) that my work was 'myopic' because digital video bandwidth was too large to stream from a disk (hint: it's not; think about YouTube).

Getting denied tenure was a slap in the face, but it was all good; I'd been miserable at UCB/CS, and I was glad to get away from ass-hats like Stonebraker and Alan Jay Smith.

Cleaning out my office at Berkeley was cathartic. I threw away years worth of CS journals I'd never looked at. I left the key on the desk and went home.

Sonic Solutions: 1992-1995

Soon after that I was contacted by Sonic Solutions, a company in San Rafael, 20 minutes from Berkeley. They made the SonicSystem, a high-end audio editing system that ran on Mac. It used custom DSP and DAC boards that plugged into the Mac bus, and were connected via SCSI to transfer audio data. Audio data was stored in the Mac OS file system.

Editing was non-destructive. Audio processing (cross-fades, reverb, filters etc.) was done in real time by Motorola DSP chips.

The customers included music and movie studios; movie sound tracks could involve hundreds of input files and lots of output channels. To handle this you needed to stuff lots of boards into one Mac.

The CTO was Andy Moorer, a pioneering figure in DSP, and an all-purpose genius (and slight eccentric). He wrote a paper about shaping sampling truncation noise called 'Whither Dither?', perhaps the greatest title ever.

When I arrived there, the SonicSystem application was a single 30,000-line landfill-type C++ file. The first thing I did after got my Mac working (they gave me a broken one) was to figure out what all the code did, divide it into files, and add comments. I taught my co-workers a bit about software engineering; they were receptive.

I'd been brought in to develop a very ambitous next-generation system: SonicNet. This was a distributed system in which multiple SonicSystems could be connected by an FDDI token-ring network (100 Mbps - fast for that time).

The goal was to support editing of distributed and shared audio data, e.g. several editors working on a movie soundtrack at the same time. A given editing 'schedule' could pull data from multiple disks around the network. Editors could share disks, network, and SCSI busses.

This involved a new 'MediaNet' board that had

• An FDDI interface.
• A processor, running a small real-time OS.
• SCSI interfaces that could connect to DSP/DAC boards in the same system, or to disk drives.

We hired a great hardware guy, Jeff Mock, to design the MediaNet board. I don't know the details, but a lot of the functionality (like the network and SCSI interfaces) was implemented in a Xilinx FPGA.

The plan was to create our own filesystem (MOFS - Media-Oriented File System), running on the MediaNet boards, with large allocation units so you could read or write close to disk rotation speed. We'd also write drivers so that you access MOFS files from the Mac OS.

Andy Moorer designed the disk layout of MOFS, which had to accommodate directories and small files as well as large media files. He did this in a few days, and his design was nearly perfect. He handed this off to me, to implement and debug.

My job was to get this all working: to write code (running on both the Mac and the MediaNet board) to route data from one device to another, as fast as the hardware would allow, and to integrate this into the SonicSystem. I was joined in this by Dean Jacobs. He lived in Berkeley; we carpooled and became fast friends.

Writing performance-driven software for complex hardware systems is a unique challenge. At any point lots of stuff is going on in parallel: disk transfers, network packet transmission, DSP operations. Interrupts happen, processes are scheduled, eventually they run. Synchronization is necessary but it can cause slowness. You need to have a clear mental model of the system.

To see what's going on, you need logs. Writing them to files is too slow. I used in-memory logs. Sometimes I spent an hour reading the log for a few milliseconds of activity.

Eventually, it all worked beautifully, at about 99% of the speed of the raw hardware.

After that, I focused on guaranteed performance. In the pro audio world, you can't have glitches or dropouts in the middle of a playback. Playing a complex playlist could involve lots of potential bottlenecks: disks, SCSI buses, DSPs, and network bandwidth. To guarantee correct playback, you have to keep track of the usage of all these resources, and you have to schedule them properly. A new playback can start only if you can reserve enough resources for it.

This was putting into practice the last couple of years of my research at UCB, so that was great.

In retrospect, SonicNet was revolutionary. It was audacious (and maybe foolhardy) for a small company like Sonic to take on something so big and difficult.

Parts of it - like the protocols for resource reservation - were potentially useful for any system that wanted to provide guaranteed high performance over networks. We tried to interest other computer companies (like Silicon Graphics) in developing these into standards, but failed.

Writing filesystems is demanding because if something fails, people can really get screwed. Normally I didn't have to deal with customers, but one day our support people put me in touch with a guy whose MOFS filesystem had failed at a point where it stored a month's worth of un-backed-up work. There was near-hysterical panic in his voice. Although I had written a repair utility for MOFS, it didn't work in his case. I had to tell him there was nothing we could do. This made me feel pretty shitty.

Apple Records in London used SonicNet, and I was sent there to chat with them and attend a conference about audio archival. I ran into George Martin and shook his hand!

But there weren't many other customers for SonicNet. Once we finished it, there wasn't much for us to do, and Dean and I took to spending lots of time playing Frisbee in the parking lot.

In fact, the company's flagship product, the SonicSystem, was doomed for various reasons. It was anchored to Mac. The DSP-based architecture became outdated once CPUs were fast enough to do real-time audio stuff. And the competition was bigger and better: Avid (video) and Digidesign Protools (audio).

Sonic Solutions lost their lease in San Rafael and moved to Novato, a 20 min longer commute for me. I quit at this point. Later they shifted to making consumer systems for burning CDs and DVDs.

After I left Sonic, I decided to write a research paper about the scheduling and reservation mechanisms in SonicNet; I thought it was the best work I'd done to that point. Writing this paper turned out to incredibly hard in terms of motivation and productivity. There was a long period when I was addicted to Tetris (a mindless computer game). I'd force myself to work on the paper, imagining a gun held to my head. After writing a sentence or two I'd reward myself with Tetris.

This stretched on for months. But eventually I finished the paper and it was published in ACM Transactions on Computer Systems, the top journal. This was a personal triumph for me, even though I knew that few people would look at the paper, and that it would have little impact.

Unofficial work

When I left Berkeley, I took with me a fancy Silicon Graphics graphics workstation that I had bought with grant money. It ran SGI's version of Unix and had a 3D graphics library similar to OpenGL. I used it to play around with graphics.

I did a lot of work with random-dot autostereograms (images where you cross or wall your eyes and things pop out in 3D). I did animations, e.g. a bubbling 3D surface. I developed gray-scale images where the apparent depth of the 3D surface was continuous rather than discrete.

I worked on this with George Homsy (an all-around genius who had taken my CS 162 Operating Systems class at UCB). He proposed making images that would have a translucent surface (like a sheer curtain) in front of a solid background, but we didn't get this working.

I also wrote programs that made animated fractal-like structures out of triangles (I don't remember the details), and that could choreograph them in response to input (such as MIDI).

Dean Jacobs was working with baton-like controllers called 'MIDI Shakers' and had developed software on his Mac that would generate synthesized music, shaped by input from the shakers.

We got together and rigged things up so that the MIDI shakers would also control my fractal-graphics program. We got a paying gig for this: some sort of office party at a fancy house in SF. I can't say it went all that well.

Rare.com: 1995

After leaving Sonic Solutions I had a few months of voluntary unemployment. The Web was taking off, and I realized that I needed to upgrade my technical skills. So I learned HTML and SQL.

In 1995 if you wanted to watch a movie at home, you went to Blockbuster and rented a VHS tape. If you didn't have a movie in mind, you browsed boxes and guessed what you'd like. This didn't work well, and things were much worse if 2 or 3 people tried to find a movie they'd all like.

So I (and it turned out, other people) came up with the idea of getting lots of people to rate things (like movies), and then looking for correlations in the data to predict how well person X would like movie Y. This became known as "collaborative filtering".

I created a web site, http://rare.com (RARE = Rating and Recommendation Engine). It let you rate movies, and it would then suggest movies you might like. It had a 'group' feature: if several people had each entered ratings, it would recommend movies they'd all like, i.e. that maximized the minimum of the predicted ratings.

I got the movie data from a web site called IMDB, which released it as a CSV file. At that point there were about 30,000 movies and TV shows.

I developed a clever way to quickly extract lots of ratings from a given person:

1. Show them a list of the 20 most-rated movies. Usually they'd rate 4 or 5 of these.
2. Then show them the 20 movies most rated by other people who had seen movies they've rated so far. Repeat.

This would quickly ferret out the obscure movies they'd seen. Typically you'd get 40 or 50 ratings pretty quickly.

RARE got a lot of users - a couple thousand or so. I announced it in an email to everyone I knew even slightly. One guy - Rick Sayre - responded 'is this what they call spam'? What a douche.

I can't really say how well RARE worked. It tended to recommend 'Citizen Kane'. I experimented with a couple of algorithms, including a clever one in which people and movies are modeled as vectors and the predicted rating is the dot product; it used optimization to fit the vectors to rating data.

In any case, after a few months IMDB sold itself to Amazon and stopped releasing their data (boo! hiss!). So I couldn't update RARE's movie list. At some point I shut it down.

RARE was my first web site. I wrote it in C++, using a technology called CGI that let you write a C/C++ program that takes a URL and parameters, and generates HTML for a web page.

These days there are frameworks, like Django and Ruby on Rails, for handling the details of URLs and arguments, dispatching requests to handler functions, and talking to databases. I did all that stuff myself; it wasn't hard.

Tunes.com: 1995-1998

Somehow I met Kamran Mohsenin, a smart and glib 30-something, who was starting a company to develop a CD-selling web site, http://tunes.com.

I signed on, somewhat against my better judgement: this was Kamran's first startup and he didn't have funding yet. But it seemed like an ideal place to experiment with music discovery on a large scale. I was the only developer.

We wanted to offer streaming audio samples of all recorded music. Kamran had worked out a deal with Valley Music (a CD distributor) where we could borrow all existing CDs from them (roughly 100K). At that point record companies weren't comfortable with the Internet. They had a rule that you could offer only a 30-second sample of each song (60 seconds for jazz and classical).

My first task was to build a system to rip audio from lots of CDs. Kamran had found a fancy CD player that you could load with 100 CDs and control with RS-232. On a Mac, I wrote a C++ program that could interface to this player, take a bunch of CDs, and record a 30- or 60-second segment of each track. We took the 2nd 30 seconds, since that was usually where the hook was. The system would then compress the audio in Real Audio format (that was the only option back then; MP3 came along a bit later).

I got this working and we set up an operation in the back room to process CDs: scan the cover art and collect audio samples.

We found a source of CD metadata: title, artist, track titles, and track durations. I can remember what this was, but if must have been free; we didn't have money to buy anything.

BTW: at that point, CD themselves contained no metadata, not even an ID saying what CD this is. Instead, you had to look at the list of track lengths and use that as a sort of fingerprint. But when they re-issue a CD sometimes track lengths change a little! So you have to include a slop factor. What a mess.

For its web server, Tunes had a DEC Alpha computer (RISC-based) that was pretty beefy. We used a demo copy of Informix. I wrote the web code as a C++ CGI program. The web site let you browse CDs, see their cover art, play samples of their tracks, and rate them.

I implemented collaborative filtering in Tunes. When you listened to a track, you could rate it. With enough ratings, it would recommend things. I had the idea that this would lead people to new genres; e.g. that if you liked Bach, you'd like a subset of Indian music that had similar contrapuntal properties.

However, these features never worked very well. And maybe people don't really want this; at one point we got an email from a user who was outraged that the system had recommended a country/western song.

Each Tunes web page was personalized to the user, which meant a bunch of DB accesses. When we opened the site to the public the server was overloaded. Kamran (who had some tech skills) hastily installed a squid proxy to speed things up. This worked, though personalization was lost.

Kamran followed trends. At one point everyone was using HTML frames to make web pages that looked like desktop apps. He made a deal with a nearby graphic design company to create a design that looked sorta like a jukebox. With a tremendous amount of work, I made our site use this design.

But then Amazon started up, and they used a clean full-page design - no frames. So Kamran - to my relief - told me to make our site look like that.

We were constantly trying to get investment. There were lots of meetings with low-level VCs. Some showed lukewarm interest, but none bit. So we all just kept on working for nothing, hoping that someday we'd get investment, or become profitable, or get acquired.

We looked for ways to enrich the site, e.g. providing more information about releases and artists. Kamran discovered the All Music Guide, a company based in rural Wisconsin that amassed information about music (mostly rock and jazz):

• Expert reviews of releases.
• "Influence" relation between artists.
• A detailed genre hierarchy.
• A "tag" system: words like 'energetic', 'calming', etc.

I integrated most of this into Tunes. When you looked at a CD, you could see the artist's influences, other CDs in that subgenre, and so on.

I was interested in the spatial and temporal continuity of music. I worked on an interface with a world map and a Year knob. You could set the knob to 1920, hover over Romania, and hear music from that place and time. Then you could move the cursor to Hungary and hear slightly different music. Or you could turn the knob to 1930.

Kamran hired a guy named Kevin Stein, whose background was in TV game shows. He was a smart guy (he used the word 'intersticial' a lot). For Tunes he didn't bring much to the table, but he got me thinking about games we could create with our assets. I made a couple of them:

• You hear a song sample and guess the artist and/or genre.
• Musical "Concentration": each cell in a 4x4 grid has a music sample. You have to find the matched pairs. This is actually a pretty cool game because it exercises audio memory, kinda like Simon.

Kamram hooked us up with Musclefish, an audio DSP consulting company in Berkeley where Thom Blum and Erling Wold (who I knew from my UCB days) worked. We discussed various 'acoustic fingerprints' for music - rhythm, timbre, harmonic complexity - that could be used as a basis for a 'music that sounds like this' exploration tool. This never got off the ground. (Note: later, Spotify and MusicBrainz developed such tools).

The fact that Tunes had only 30 second samples limited what we could do. We couldn't stream songs like Pandora and Spotify. And I'm not sure you can tell if you like a song from hearing a short sample. It's like rating a book after reading a page or two.

Kamran wanted Tunes to be laboratory for musical exploration and events, which of course I did too. I fiddled with a web-based system where a user could move around in sort of harmonic grid. Server-side software would send MIDI commands to a synth (an FB-01) and its output would be digitized, compressed to Real Audio, and streamed to the user and other listeners. I got this working but not to the point of usability.

At some point, we started actually selling CDs. This involved interfacing to a credit card processing system, and adding various DB tables.

We were profoundly disappointed by the results. We sold 40 or 50 CDs a day, with about a dollar profit on each one. Kamran quickly looked into getting Tunes acquired by someone - anyone. He tried Amazon, but their offer was basically zero; they were already in the process of adding CD sales to their site.

Kamran eventually found a buyer in a company in Chicago (I can't remember their name) that operated the Rolling Stone web site. Their CTO was a young Microsoft zealot. Everything they did ran on Windows and was written in proprietary MS languages. The guy was weirdly confrontational. I have no idea why they bought us; the CTO had no intention of using anything we had done.

At this point we all kind of stopped working. The Chicago people would occasionally call us, asking what we were doing. Finally it ended.

After the acquisition, my stock in Tunes was worth something, and I immediately sold it. It came out to something like $60K/year for the 3 years I'd worked there.

So Tunes was disappointing financially. But I was also bummed out that the music discovery features never got off the ground.

Amazon: 1995

At the beginning of Tunes, I was still operating RARE, which collected movie ratings. I was developing Tunes, which collected music ratings. Both sites had some form of collaborative filtering. I was curious: if we combine the ratings, can we get better recommendations? Does taste in movies correlate with taste in music?

This seemed possible, and it also seemed possible that taste in books could correlate with both. Amazon was just getting started at that point; it had maybe 20 employees, and it was just selling books.

I tracked down Jeff Bezos's email address and wrote him, proposing that we form a consortium of sites involve personal taste (Amazon, Tunes, RARE), pool our rating data, and develop a shared collaborative filtering engine that we could all use to make better recommendations.

Bezos was interested in this and invited me up to Seattle to talk. A few weeks later I went back for a 2nd visit.

About a week after this he emailed me. There was no mention of the consortium idea; instead he wanted to hire me as CTO of Amazon, with an offer of 2.9% of their common stock (with the usual 4-year vest). I declined.

You're probably thinking that I'm is an idiot for not taking the job. But keep in mind that:

• It wasn't clear that Amazon was going to succeed.
• I almost certainly would not have done well as their CTO. I wouldn't have lasted 4 years; maybe just a few months.
• I felt an obligation to Kamran and my Tunes co-workers.
• I didn't want to live in Seattle, and I didn't want a 60-hr/week job.

Also I'm genuinely not interested in money. I care about my social, musical, and athletic life, and lots of money doesn't necessarily improve these.

Blake Systems

As Tunes spiraled downward, I had the (optimistic) idea that I could make money with collaborative filtering. I created a corporation (Blake Systems) and hired a lawyer (Kat McCabe, wonderful) who had some connections. Through her I got a customer: Wired Magazine, which hired me as a consultant. For a few months I'd go to their SOMA offices once a week, work at a computer there, and imagine that I was on the cusp of something big. I can't remember what I planned to do or what data they gave me. It never went anywhere.

Tunes had hired a young Wesleyan grad named Charlie Kilpatrick. When Tunes folded I hired him at Blake Systems. We got hired to develop an e-commerce web site using ColdFusion (an HTML template language - dumb idea). Details are vague. We never got paid, and I wasn't able to pay Charlie, which I feel bad about.

SETI@home: 1997-2004

In Dec. 1994 Dave Gedye (who I had known at UCB, and was a good friend) thought up volunteer computing: getting lots of home computers to do scientific computing, using the Internet to distribute inputs and collect outputs. He identified radio SETI as a good application of volunteer computing, though it could be used for lots of other things too.

As Tunes was winding down (1998) SETI@home was ramping up. We got some money (from Paul Allen's Starwave company, with matching money from a CA state program). This was enough to hire a Mac expert (Charlie Fenton) and a Windows expert (Kyle Williams). We also found Eric Korpela, an astrophysicist with major computer skills. He (with help from Jeff Cobb) designed and implemented the guts of things: splitting data into 'workunits', and doing the front-end analysis.

I wrote most of the rest, including

• A database (Informix) that kept track of volunteer accounts and hosts. We used the same DB for storing science info.
• A CGI-based server for dispatching jobs.
• The client logic for fetching jobs, running them, and uploading the results.
• Graphics code for showing an animated visualization of the client data analysis. I did this on Windows, so I had to learn Visual Studio. I can't remember what graphics library I used but I think it worked on Mac too.
• Web code (I believe in C++) for showing things like leader-boards.

At some point Eric Person came along and wrote some web code in PHP. I learned PHP, loved it, and converted all the web code to it.

The client was monolithic: it was a single program that had both the science code and the framework code (managing jobs, acting like a screensaver). It didn't have automatic update, so every time we changed the science code, all users had to manually download and install a new version (and we'd lose lots of them in the process).

It became clear that we needed a new framework, one in which the science code could be updated automatically. The first attempt, unsuccessful, was United Devices (see below). After that we switched to BOINC (see below).

Once things were working on BOINC, SETI@home went into a weird stasis where it kept going, analyzing data and accumulating results, and dealing with a sequence of mundane problems like database size limits, but with no viable plan for actually doing science, i.e. analyzing the results and finding ET. This lasted from 2005 to 2016; I kind of checked out of the project during that period. The stasis ended when I started Nebula in 2016.

United Devices: 2000-2002

In 2000 I was approached by a startup, United Devices, that wanted to develop a framework of the sort we needed for SETI@home, and to monetize volunteer(ish) computing. Although I had profound misgivings (the founders were not the sharpest tools in the shed) I signed on as CTO. The other founders had some money so at least I got paid.

I wrote (in C++) the server and client for a general system for volunteer computing. This worked, and I got SETI@home working on it. But soon the company hired other developers who wanted to do things differently, and they chucked what I had done. Details are here.

Stardust@home and Bossa

Space Sciences Lab was home to many interesting projects. One of these was Stardust. A satellite with an aerogel collector had flown in space and collected interstellar dust particles. The collector had been parachuted back to Earth, and the aerogel was sitting in a clean room just down the hall. The job was to locate the dust particles (buried somewhere in the aerogel), extract them, and analyze them.

Andrew Westphal (a brilliant astrophysicist) was in charge of this, and he (inspired by SETI@home) had the idea of using Internet volunteers to look at microphotographs of the aerogel and identify the dust particles. He created a web-based project, Stardust@home, to do this.

I got involved. My contribution was a sort of web-based microscope. The web page (using Javascript) would load a 'stack' of images, one per focal depth. Using a slider, you could move the focal plane in and out, and click on a dust particle if you thought you saw one. While you were doing this, the page would pre-fetch the next stack of images, to reduce waiting (the Internet as slow back then).

I was especially interested in validation: estimating the skill of each volunteer (in this case, rates of false positives and false negatives) and use this to figure out how much to replicate each job to achieve a given overall error rate.

Stardust@home had a mechanism for making these estimates: volunteers would occasionally be given 'test jobs' with known results. But they had so many volunteers that they didn't have to get fancy; each job was done by at least a dozen volunteers, and that was enough to guarantee a low error rate.

Andrew looked around for other applications of 'distributed thinking'. He found Tim White, a UCB professor of archaeology whose research involved finding early-hominid fossils in the Olduvai Gorge in Africa. This was done by having grad students walk back and forth, looking down at the ground, looking for little pieces of fossilized bones and teeth. This was inefficient, and it trampled stuff.

Tim and Andrew had the idea of making a rig - maybe 10' across - with a GPS device and several cameras. The grad student, carrying this, would walk over the area of interest. It would take pictures every few feet.

Internet volunteers would then look at these images - maybe in some kind of 3D viewer - looking for fossils. The grad students could then go back and retrieve these.

This was a fabulous concept, but there were lots of technical issues; e.g. the rig would have to carry big batteries, and would be heavy. We had lots of meetings but things never got off the ground.

But - inspired by the success of BOINC - I had the idea of creating a software framework for distributed thinking. It would store - in a database - lots of jobs and test jobs. It would estimate volunteer ability and would do optimal replications. I wrote it in PHP, and there was a demo app.

For a brief moment (before modern AI made humans irrelevant) distributed thinking was a thing. Someone created an academic journal and a conference. Amazon came out with The Mechanical Turk, which was like Bossa but with micro-payments for doing tasks.

GalaxyZoo came along - volunteers classified galaxies - and this was generalized to Zooniverse, which got funding and offered support. At one point I talked with them; their software was ad-hoc, and I tried unsuccessfully to interest them in Bossa.

In any case, no one used Bossa. At some point Daniel Gonzalez (who I knew from BOINC) made something called pyBossa that he said was a translation of Bossa into Python (he didn't notify or involve me, which pissed me off). I think pyBossa was used for a few projects.

An academic bandwagon formed around 'citizen science', which included distributed-thinking projects like GalaxyZoo, and hands-on projects where volunteers counted birds and stuff. At some point NSF funded SciStarter, a directory of citizen science projects. There was an event where they tried to get Congress to fund citizen science more.

This community, it turned out, did not consider volunteer computing to be citizen science. At one point, at a conference, I talked with someone from SciStarter and suggested that we partner somehow: at that point BOINC had about 200K science-enthusiast volunteers, and SciStarter had about 1K. There was no interest or follow-up.

One thing I've learned in my career: most people are protective of their turf. Grid and supercomputer people, citizen science, education, you name it. If presented with something that seems like it might threaten their domain or shift the paradigm somehow, they often react with surprising and irrational hostility.

Teachpoint and Bolt

I'm interested in how knowledge can be optimally acquired by humans. The amount of human knowledge (science in particular) has exploded, but the educational system still mostly uses the traditional model, based on classrooms, teachers, courses, semesters, exams etc. I think this model is inefficient; most people could learn 10X faster than they do. As a result most people don't end up knowing much, and this has negative societal consequences (e.g. fascism, which feeds off ignorance).

Note: I'm concerned here with the knowledge acquisition part of learning, not stuff like problem-solving, experimentation, creativity, socialization, etc. All super important, but not what I'm talking about.

There's a lot of research in learning:

• People learn best in different ways (visual, symbolic, experiential etc.) and at different paces.
• Students need to use or re-learn concepts at certain intervals for the concepts to enter long-term memory.
• For information presented as text, the typography (e.g. spacing between words and sentences) can affect comprehension. This effect varies between people.
• For information presented as speech, the voice, inflection, speed etc. can effect comprehension.
• A person's ability to learn at a given moment depends on their emotional state, how much sleep they got, etc.

and so on. All these things depend on the individual. People learn differently; their goals, background, and interests are different. But the traditional model is not personalized, and it can't accommodate these differences (also, most teachers are pretty mediocre). I discuss this in more detail here.

Let's imagine that you have the best possible private tutor; call her Mary. What would your learning sessions be like?

• Mary somehow senses your mental and physical state: whether you're sleep-deprived, upset, distracted, etc. She takes this into account in deciding, for example, whether to teach you new material or review old material. Or she might say 'take a walk outside and come back in an hour'.
• She might begin a session by having you play games that warm up your brain.
• She knows when you need a break, and for how long.
• Mary keeps track of everything you know, and when you last learned or reviewed it. She reviews things at the optimal times for long-term memorization.
• When Mary teaches you something new, she knows what the prerequisite concepts are, and she makes sure you know them.
• Mary has many different lessons to teach a given concept. She picks the one that's likely to work best for you.
• When Mary presents text, she uses the medium that will work best for you. Maybe it's spoken, in Patrick Stewart's voice; you see a video image of him as well if that improves your retention. Maybe it's displayed on a screen, with a particular font and spacing.
• Mary constantly quizzes you. After each new concept there's a quiz. If you don't get the concept, she teaches it differently or slower. More quizzes, an hour/day/week later, move the concept from your short- to medium- to long-term memory.
• Mary knows what your interests are. When she teaches something abstract - say, a math concept - she uses examples that apply it to things you're interested in.
• Mary knows your long-term learning goals. Maybe you want to learn Calculus, organic chemistry, and world history. She makes sure that you eventually learn them all, but she doesn't dictate a particular schedule. It's not always an hour of each, every day. If you're on a history jag, she might teach mostly history for a week or two.

The traditional model can't do any of these things. And Mary doesn't exist. But I can imagine a computer-based system that would approximate Mary, teaching you in an adaptive and personalized way: It works something like this:

• There's a 'concept graph' with 'prerequisite' dependency edges.
• For a given student, the system keeps track of their knowledge of each concept, and the recency of this knowledge. Thus it knows when concepts need to be refreshed for long-term retention, and, for a given concept, which prerequisite concepts need to be learned first.
• The system has multiple lessons for each concept, in different modes.
• The system measures the effectiveness of lessons for individual students using assessments; this data provides a characterization of both lessons and students.

Essentially, it uses data mining to discover the best way to teach individual students, and to move information to their long-term memory.

I had this idea around 1995. I discussed it with Rich Kraft, and he liked it so much that he wanted to create a company - we called it Teachpoint - to develop it. I pitched the idea, over dinner, to a Silicon Valley executive type. His wife's reaction was: why on Earth would anyone want to keep learning stuff and taking tests?

Later, in ~2005, it occurred to me that volunteer computing provided an ideal context for such a teaching system: There's a constant stream of arriving 'students'; they're highly diverse, and they're interested and motivated to learn about the science that they're participating in.

I developed a PHP-based prototype, BOLT (Berkeley Open Learning Technology) implementing some of the ideas above. It used a database to keep track of students, concepts, lessons, results, etc. As an example I made a course on identifying California conifers. It was clunky but it worked.

I really wanted to explore this further; it was more interesting to me than computing. NSF had a 'learning technologies' division. I drafted a grant proposal.

I talked with one of the BOINC projects - Rosetta@home - and pitched the idea adding a BOLT-based education component to their web site. They agreed.

I talked to people in the UC Berkeley Education department. I quickly realized that the research being done there was mostly bogus, and they were threatened by the ideas behind BOLT. But one post-doc - Nate Titterton - was interested, and he agreed to be a co-PI on the proposal.

In the end, I didn't submit the proposal. It turned out that in the NSF Education program, all proposals had to include hiring an outside entity to verify the validity of the data and the statistical methods. I guess this was because there had been so many fraudulent projects in the past.

With about a month before the deadline, I scrambled to find such an entity. Someone at Lawrence Hall of Science could possibly do it. But I realized that I didn't have time to pull the whole thing together adequately. I also had the impression that the world of academic education research was an old-boy network, and I was a complete outsider with no credentials, and that the chances of the proposal getting funded were probably close to zero. Also Nate and the Rosetta people were not as enthusiastic about the whole thing as I was. So I dropped it.

BOINC: 2002-present

After the United Devices debacle I created BOINC and got an NSF grant to pay my salary and a couple of others to work on it. This became the center-piece of my career. It's documented here so I won't say more about it.

Nebula: 2016-2002

By 2005 we had moved SETI@home to BOINC. We continued to have weekly SETI@home meetings but they became boring. They were 95% sysadmin stuff, mostly centered around the Informix DB: running out of IDs, disk space, shards, fragments etc. I got in the (bad) habit of sitting at my desk, listening but not participating.

The BOINC part of SETI@home is the 'front end'. It finds 'detections': spikes, Gaussians etc. But to find ET you also need a 'back end' which:

• Removes RFI from the detections.
• Identifies 'signal candidates': groups of detections that are plausibly from a single cosmic source: with roughly the same sky position and frequency, and possibly spread out of years.
• Ranks these signal candidates (of which there might be millions) so that we can examine them manually and decide which ones to re-observe (and make a final yes/no decision).

Detections continued to accumulate in the DB. As the amount data increased, doing back-end analysis became almost impossible; it would take weeks just to pull the data from the DB. In 2005, Eric and Jeff developed something called NTPCKR that analyzed the most recent data as it came in. This worked, but just barely, and not well enough to let us see if the analysis algorithms actually worked as intended. So they probably didn't.

So SETI@home just kept on going, year after year. We had told the world that by running SETI@home they were doing SETI. People had believed and trusted us. They had paid electric bills for years. This had produced big carbon emissions. Lots of them even donated money to us. And, by not finishing the project, we had discarded all of this effort, and we hadn't even bothered to tell them.

Everyone seemed OK with this, but I was not. In 2016 I was running out of grant money for BOINC, and also running out of things to work on. So I sounded an alarm for SETI@home. I proposed that I (paid from our war chest of donations) build a back end that let us do science and finish the project. The new back end code would ideally complete the analysis in roughly a day, allowing us to experiment with algorithms and find ones that worked. I called this project Nebula.

There were two main reasons for the slowness of the existing code:

• The programs read detections from the Informix database, which was slow.
• The code was written in an excessively object-oriented way. When I profiled it, it was spending most of its time in constructors and destructors.

So the first things I did were to:

• Dump the database contents into CSV-format files, and henceforth use those rather than the DB. Sometimes the data needed to be accessed in a particular order (time or frequency). I used the Unix 'sort' program, which is good at sorting huge files.
• Use efficient C++ data structures. In particular, the Boost library had 'R-trees', which are multidimensional indices that let you efficiently access rectangles in e.g. time/frequency space.

It became clear that Nebula needed more computing power than we had locally. Some things (like scoring pixels) could be distributed across lots of computers. We couldn't use BOINC because the data was too big.

First I tried using AWS, the Amazon cloud. Amazon gave us a 'starter' account with $5K credit. I hired an undergrad. Unfortunately he hard-wired the AWS account key in the code, and we were keeping the code on Github, and a couple of days later hackers found this and used up all our credit. Also AWS was a pain in the ass to use, so I dumped it.

UC Berkeley had its own private 'research cloud'. I looked into using this, but it was an even bigger PITA than AWS, and even more expensive.

I described the situation to Bruce Allen, who ran the Einstein@home BOINC project, and who had built a giant computing cluster (Atlas) at MPI Hanover. He said I could use Atlas for Nebula if I put his name on the resulting papers. Deal!!!

Atlas had several thousands nodes, some of which were beefy, and a lot of NFS-mounted storage. It used Condor as the batch scheduler. Condor turned out to be pretty easy to use. Way easier than BOINC.

I did file sorting and RFI removal in parallel on a big (96-core) machine, and pixel scoring using Condor. Each of these took a few hours, so I was able to do the back-end pipeline on the entire data set in a little over a day. I'd then ship the output files (candidate lists etc.) back to the UCB servers.

There, I developed web web interfaces for looking at waterfall plots (which showed RFI) and showing lists of candidates and other stuff.

Once this was working, we got into a very productive work cycle for a year or two. I'd do a back-end run. Eric and Dan would look at the results and find something that needed improvement (like undetected RFI). We'd meet and discuss algorithms, often thinking up completely new approaches. I'd go off and implement these, which typically took a day or two, then do a back-end run. We did this over and over.

As this neared completion, we were working on two papers: one on the front end, one on Nebula. At first Eric was first author of both, but as time went by more of the Nebula ideas were mine (plus I wrote the paper) so I became first author of that one.

We were almost done when Eric blew up at me, using some magic swear words. I think he felt that I had usurped the project. In any case, I was in no mood to be talked to like that, so everything stopped.

Time went by. The Nebula paper was maybe 70% done, and I couldn't do the last 30%. For a while I tried to get Dan to work on the paper, and to act as an envoy to Eric. He agreed, but didn't actually do anything. More time went by (like a year). My frustration grew.

Finally I called Eric and suggested we get back to work on the papers. He agreed; there was no mention of the blow-up. We met in person, twice a week (my house and his house) for several months. It was extremely productive; we even came up with some new and important ideas. We finished the papers, and they were both really good.

Nebula was the hardest software project I've done, for several reasons:

• It involved taking a huge piece of existing software (NTPCKR) and performing delicate brain surgery on it.
• It was highly distributed; things ran on lots different computers, some of which I couldn't log into directly.
• To get the code to run fast enough I had to use lots of tricks - parallelism, fancy data structures - that made it complex.
• Things took hours to finish; the debugging cycle could be extremely slow.
• The notion of 'correctness' was fuzzy. The output was a mountain of data. I spent lots of time generating visualizations and having colleagues use these to decide if we were on the right track.

But the effort was worth it. Without Nebula, 20 years of effort, by lots of people, would be down the drain.

HERA: 2015

In 2015 my BOINC grant was running out and I needed money to pay my salary. Aaron Parsons had been an office-mate at SSL, and he was by now an astronomy professor at UCB. He hired me as a short-term programmer for his HERA project, which was building an array radio telescope (in South Africa) that used some kind of fancy DSP algorithms (Aaron's invention) to see through foreground stuff and all the way to the earliest visible universe.

HERA was a big multi-institution project with lots of partners. I attended a big project-wide meeting in San Diego. They wanted me to build a distributed 'library' system to keep track of all the data they recorded. I was managed by a guy from Arizona State. I told them I could develop the system fastest in PHP, and I wrote a detailed spec and implementation plan, and sent them around.

The project's other software people were at U. Washington. They were Python zealots, and were territorial. None of them responded to my proposals and specs. I implemented the system, and it worked, but that was the end of it. I heard later that they had 'rewritten it in Python', which means that they never looked at it and wrote their own thing instead.

Projects with Noah

In 2015, when my son Noah was 10 or so, I taught him some Python and we worked on a few projects together. Some of these had to do with sports analytics:

• Predicting the scores of NCAA football games based on scores of games to date, using a simple model in which each team has offensive and defensive coefficients. The predicted scores are the product of these, and you find them by data-fitting. This worked pretty well. We scraped data from a web site.
• Estimating the true value of NBA players, based on their plus/minus but also factoring out the effect of the other players on the floor. We used JSON data available from the NBA. This didn't work quite so well.
• A program for picking the optimal batting order for a baseball team, given the batting statistics of the players. This worked by doing a bunch of simulations.

We designed and impemented a 3D real-time game called Space Slalom where you fly through rings floating in space. We first wrote this in Python but there were glitches due to garbage collection, so we rewrote it in Javascript and it worked quite well.

At Dave Gedye's suggestion, we both worked on Project Euler, and made it to assignment 50 or so.

I also (without Noah) wrote some word game stuff, including solvers for Boggle and license plate game, the 5-letter work game (Jotto), and NY Times Spelling Bee. Also the 24 puzzle. My friend Ellen Wang wrote an optimal Wordle solver, and I wrote a web front end for it.

PanoSETI

In 2022 Dan Werthimer told me about his new project, PanoSETI, which was building a telescope designed to detect short (nanosecond) and faint (a few photons) optical pulses. These could be ET beacons or (unknown) natural phenomena. Each telescope can see 100 square degrees, and a "fly's eye" configuration, with ~50 telescopes, can see a whole hemisphere.

There were various technical challenges. Each telescope could spew data at a high rate, close to the write speed of a hard disk. We used Hashpipe on Linux to record data. FITS format wasted space, so I devised a more efficient format.

In a large configuration (lots of telescopes, lots of data-recorder computers) there was the problem of starting and stopping the whole thing coherently, and gathering the output files on one server afterward. I wrote some Python control scripts for these purposes, using JSON config files to describe everything. This all worked and was very nice code.

I also developed a back-end system (in PHP) for maintaining an archive of observing runs, their output files, and the analyses of runs using various algorithms, e.g. so that you could compare and refine the algorithms. Something like this would have been useful for Nebula.

But the project turned weird. The titular leader was Shelley Wright, from UCSD. She had a postdoc who had prototyped the whole thing using Matlab; that was the only language he knew. He was possessive of his code and refused to put it on Github, and he was passive-aggressive toward the Python-based efforts.

The project meetings focused on hardware; Shelley treated 'software' as a boring afterthought, like a coat of paint or something.

At one Zoom meeting, where I was presenting the architecture of the Python code (I had written docs, but no one had bothered to look them) Shelley interruted and started talking about the advantages of Matlab to draw graphs. That was enough; I wasn't going to work as volunteer for a project whose leader didn't support me. I quit the project the next day and haven't looked at it since.

Music discovery and IMSLP: 2020-2024

I think the arts are hugely important in human existence. I'm into classical music. I play piano, I go to lots of concerts, and I organize an amateur music group.

I love classical music, but I think the 'ecosystem' of classical music has major problems. Only a few superstar musicians make a living from it, and they mostly play standard repertoire: the same old pieces, over and over.

In my view the fundamental problem is lack of discovery tools. It's hard for performers to find interesting new compositions, to find other people to perform with, and to find audiences. I write about this (at considerable length) here.

The Web has produced ways for people to meet for business and social purposes (Linkedin, OKCupid, etc.) and for people to discover products and entertainment (Amazon, YouTube). But there are no counterparts for classical music.

So I'm interested in creating ways to discover new music. One approach would use collaborative filtering, like I did with RARE for movies. When you're browsing music, you can rate pieces, and say how difficult they are for you. If lots of people do this, for lots of pieces, we could mine the data and predict what pieces and individual would like, and be able to play.

This approach could work for discovering old as well as new pieces.

The natural place to do this would be IMSLP. It has PDF scores, and in many cases recordings, of almost all out-of-copyright (> 75 years old) classical music. Some current composers put their (open-source) pieces there in the hope that performers will find them and play them.

With a bit of work - it seemed to me - we could add collaborative filtering and other discovery tools to IMSLP. This would jump-start the entire classical music ecosystem.

So I tracked down the leader of IMSLP - Edward Guo, from Taiwan - and proposed this to him. He pointed out that there was a problem: IMSLP's data was in an arcane and messy Mediawiki format, with a funky template system for storing metadata. There was no underlying database.

He proposed that I design a SQL database for IMSLP data, and develop software to populate this by 'scraping' the Mediawiki data. He offered - in fact, insisted - on paying me to do this.

So I did it, because it was prerequisite to my music discovery goals. It was tedious and unpleasant. At the same time I developed a rudimentary web interface, and I fiddled with optimizing queries like 'show me all arrangements for piano 4-hands of string quartets by French female composers'.

On my next phone call with Guo, I reminded him of my goal of supporting music discovery. At that point he got weird and evasive (even more than usual; he's a close-to-the-vest type). Then he fired me.

About a year later, IMSLP release 'Clara', which as far as I can tell is a web and app interface based on my database structure. It offers slightly better search capabilities than the Wiki version, but no music discovery features.

Guo had a volunteer, Daniel Miller, who he said was working on releasing the IMSLP data in a better-structured way. I contacted him. He was a 'semantic network model' zealot (ontologies, RDF, SPARQL) and was dismissive of relational databases. He showed no interest in collaboration.

I made one more attempt to establish some kind of collaboration with Guo, and at first he seemed slightly receptive. But his next email slammed the door; he suggested that I find some other project to work on. It became clear that he's interested only in pursuing his own (somewhat myopic) vision.

This is a bad situation. IMSLP is a central resource in classical music - it's by far the best source of scores and metadata. But it's owned by a control freak with no imagination. It's a bummer for all of us.

Classical Music Index

When it was clear that IMSLP was not going to add music-discovery features, I decided to make my own web site with such features, using the snapshot of IMSLP data that I had. The first step was designing a database schema.

• The schema is super general. Its notion of 'work' encompasses movements and arrangements. It represents 'roles' like composer, librettist, arranger. It has a location hierarchy that includes continents, provinces and cities as well as countries.
• It uses JSON and multi-valued indices in all cases where there can be more than one of something (e.g. a work's instrumentations).
• It models concerts, including their programs and venues.
• It has user accounts, and it lets users rate and review things (works, performances, composers, etc.).

• Has fairly general search features. E.g. you can search for 2-piano arrangments of string quartets by French female composers.
• Lets you rate and review things, and lets you browse other user's ratings and reviews ('social discovery' mechanisms). I'll add collaborative filtering when were are enough ratings.
• Lets you add items (works, concerts) in a way that ensures data quality (e.g. you don't have to type composer names).

I did this all in PHP, leveraging some of the BOINC web code.

Recently (Mar 2025) I got around to getting a domain name (classicalmusicindex.org) and using HTTPS. At some point I'll need to promote it somehow.

Music Match

Around the same time I got interested in another music discovery problem: people discovering people. In particular performers discovering composers, and vice versa.

This could potentially tie into Groupmuse, a web site for classical house concerts. They bring together three groups: hosts (people with living rooms and pianos), performers (mostly recent conservatory grads) and listeners. I've hosted and attended many of these concerts, and I slightly know some of the Groupmuse people (Mosa Tsay and Kyle Schmoltz).

I approached them with the idea of involving a fourth group: composers. Perhaps using Music Match, performers could find local composers; Groupmuse events could feature a new work by the composer, and the composer could attend the event and talk about the composition.

It seems to me that this could make classical music into a living, breathing, dynamic thing, rather than a dusty museum.

Anyway, my contacts at Groupmuse listened to my ideas but were not interested in pursuing them. They're focused on making money for performers: a goal which sounds good but is (I think) at cross purposes with reviving classical music on a large scale.

But I implemented Music Match anyway: it's here. It's pretty well done, if I do say so. I told all my music friends about it, and some of them tried it and created accounts, but none of them recruited anyone else. So Music Match is just sitting there unused. At some point I should probably work on promoting it.

Like CMI, I did this all in PHP, using some of the BOINC web code.

Numula

In 2021 I heard about Pianoteq, a piano synthesizer that runs on PCs. It uses physical modeling rather than sampling, and it sounds pretty good. You control it via MIDI; e.g. it can play MIDI files. This got me thinking about my life-long dream of making virtual performances of pieces like Beethoven's Appassionata.

I decided to resurrect FORMULA, but with 21st-century tools. The obvious language to use was Python. I called it Numula (Nuance Music Language).

I looked at building on an existing Python music library. The main candidate was Music21, but this is vast and complex, and it emphasizes musicology rather than performance. I like to keep things simple. So I found a bare-bones MIDI library and built on that.

Numula has mostly the same goals as FORMULA did 40 years earlier:

• Support layered specification of nuance (tempo, dynamics, articulation).
• Keep these specifications separate from the score (pitches and rhythms).
• Allow the nuance to be expressed in code: you can put loops around things, make them into parameterized functions, etc.

In FORMULA, these pieces of code were concurrent processes. In Numula, they're functions that generate data structures; these are combined with the score (also a data structure) to produce the final product. This is as powerful as processes and way simpler.

Numula has two other giant advantages over FORMULA:

• Python is a modern, rich, and easy-to-use language; Forth is not.
• Computers now have double precision floating-point arithmetic. You don't have to worry about accumulated round-off error (an issue we never addressed in FORMULA).

Using Numula, I made virtual performances of several pieces: Berio's 'wasserklavier', Beethoven's Appassionata (3rd mvt), Chopin's Prelude #8, and Robert Helps' Three Homages.

In Numula, the various types of nuance (dynamic change, tempo change, pauses, emphases, articulation, pedaling) are all described by what I call Piecewise Functions of Time (PFT): a data structure consisting of a time-ordered list of 'primitives'. A primitive might be a linear or exponential function over some time interval, or a momentary thing like a Dirac delta.

Numula lets you create scores and PFTs algorithmically, with a lot of function calls. But this is tedious, so I developed a set of 'shorthand notations' that let you describe things compactly, in text. The notations vary according to the purpose, but they share some features:

• You can have nested loops.
• You can do variable substitution using Python's 'f string' feature.
• Long pieces (like the Appassionata) were hard to debug: if you get a time interval wrong, everything is off from that point forward, and it can be very hard to locate the error. I added a feature where you can put measure numbers in shorthand strings, and if you make a timing error, you find out immediately.

In creating a virtual performance with Numula, entering the score is easy; perfecting the nuance is the time-consuming part. The editing cycle is like this:

• Listen to a section.
• Decide what to change.
• Locate it in the code.
• Edit the code.
• Run the program again and listen.

I thought a lot about how to reduce this cycle to the bare minimum, and came up with what I call 'Interactive Parameter adjustment' (IPA). You use it for adjusting parameters: dynamic and tempo levels, pause durations, etc. You select a section and a parameter. Then the cycle is:

• Press the up or down arrow to change the parameter value.
• Press the space bar to play the section.

No mouse required.

Of course, it would be great if Numula could be integrated with score-editing software; you'd see nuance layers as 'tracks' under the score, and you'd edit graphically.

The obvious place to do this is MuseScore, which is open source. In fact, it turned out that much of the development of Musescore was done by Marc Sabatella, a CS grad student who did his Masters project with me at UCB.

I tracked down Marc and visited him in Denver. It turned out that MuseScore had been bought by a Russian company, which made money by selling scores that users had entered (do I have that right?). The technical lead was a guy from Ireland who had bitched a lot about the original version. I emailed this guy and never heard back.

Of all the software I've written, Numula is my favorite. It's elegant, simple, and amazingly powerful. It makes beautiful music, and it fulfills one of my long-term goals.

Crossword grid-filling

I've always wanted to have another try at the grid-filling problem. Recently (Dec. 2024) I finally got around to it. I wrote a C++ program that uses some clever tricks. It works like a charm (fills typical 15x15 grids in a few milliseconds). This success is due to the fact that: a) I have more tricks up my sleeve now; b) the C++ library has handy stuff like hash tables and vectors; c) computers are way faster now (like 10,000X).

Conclusions

Values and goals

I've been incredibly lucky in that I've always been able to choose what to work on. Everything I've done has been driven by my values, or was at least consistent with them. I won't try to describe my value system here; it's reflected in various other essays I've written. To give a rough idea, here are some things I like:

• Human intelligence and creativity.
• Interacting with people face to face; I go to dive bars and talk with strangers.
• Physical experiences like hiking and biking. If it's cold, windy, or rainy, so much the better. I can't keep my hands off rock.
• Nature, and the study of nature (i.e. science).

... and some things I dislike:

• Social media.
• Anything "virtual".
• Artificial intelligence.
• To some extent, technology in general. I'm sort of a Luddite. The start of the Unabomber Manifesto resonates with me.
• Capitalism (the unregulated greed-driven cancerous kind).

I've chosen projects accordingly. I almost never wrote software to do corporate/capitalist stuff like selling product or targeting ads (see Amazon above). There have been some common themes in my work:

• Recognizing and accommodating human individuality; in particular, differences in the ways that people learn, and in their tastes in music and other arts.
• Amplifying human creativity. My music-related work is not about having computers compose music; it's about helping humans make music.
• Enabling science, and involving the public in science.

Elements of style

I have a particular programming style, and a taste in languages and language features. Both are the result of my background:

• I wrote a lot of assembly language. When I code in a high-level language, I like to have an idea of the machine instructions that will result, and I like it if these are fairly close to what I'd write myself in assembler.
• I'm a mathematician at heart. Math proofs are preferably short and clear. I try to write code that's like a good proof.
• Mathematicians strive for generality. Things like linear algebra and group theory were invented to study one thing, but they turned out to describe lots of other things. Similarly, a piece of code created for one purpose may (when appropriately generalized) be useful for others. Like most good programmers, I constantly look for things I can factor out and reuse.

Above all else, I try to keep my code as simple as possible. This has many advantages, but the main reason is that I'm not actually very smart (my dirty little secret) and I can't wrap my brain around things that are too complex.

My process and aesthetics of programming

Languages and environments

I've written more code in C/C++ than any other language. I like it because I know what it compiles to, and I can write efficient code. I don't like it because it offers no protection against memory errors, which can be hellish to track down. I'm surprised that this exists in 2024. Rust supposedly solves this problem.

On Unix, I don't use an IDE with C++. I develop in a terminal window, using Vi to edit, and (when needed) gdb to debug. On Windows I use Visual Studio. I like its debugging features but I don't like mouse-based text editing.

Programming languages come out best when one smart person controls their design. For C++ this was initially Bjarne Stroustrup, but in recent years it's been taken over by committees, zealots, and language wankers. C++ has become a landfill of features that, to me, seem incomprehensible and useless. I mostly don't bother to learn them or use them.

I love Python (Guido von Rossum's brainchild). It offers direct runtime access into its own internals (like Forth) and has a nice readable syntax. I don't use its type system. Developing Numula (in Python) was a joy.

Language purists abhor PHP, but I love it; I'm super productive when using it; I never feel like I'm fighting it. In real life, the libraries available for a language are as important as the language itself. PHP has libraries for almost everything.

I'm not a fan of trends, or of other people telling me how to program. I'm perfectly capable of finding my own Design Patterns, thank you.

There are tons of 'frameworks' for making things like data-driven web sites (e.g. Ruby on Rails). Having done this from the ground up, I find these irritating and useless. In the buy vs. build tradeoff, I usually build.

There's a trend in Web design towards making web pages look like desktop GUI apps. The page never reloads; instead, bits of Javascript handle input, do AJAX requests, and modify the page's HTML. The page becomes a multithreaded program, with all the issues of scheduling and synchronization that entails.

I don't see the point of this. With modern hardware and networks, you can generate a page on a server and send it to a browser in the blink of an eye. With this approach, everything is synchronous and easy to debug, and I think it produces web sites that are easier to navigate.

Using Javascript on both browser and server (node.js) is a thing. I'm not sure why.

I hear that AI tools are useful for both programming and documentation, but I haven't used them and have no plans to.

The act of programming

By 'programming' I mean three interconnected things:

• Designing, writing and debugging code.
• Writing user documentation: what the software does and how to use it.
• Documenting the code (how it works internally) to help other programmers extend or improve it.

You have to do all three of these if you want to write software that has an impact.

When I approach a large programming task, I have a fuzzy mental image of what it's supposed to do, and an even fuzzier image of how I'm going to do it. To clarify the latter I start by thinking about data: what's needed, how it's stored, how it flows around. I write some design docs (informal, for my own benefit). Maybe I write some pseudocode. Eventually I write some real code.

As I do this, the fuzzy images slowly come into focus. Sometimes it turns out that the images needs to be tweaked, or maybe even discarded.

I constantly move from high to low level and back. Sometimes things get really hard and complex at the low level, but if you change the high level a little (i.e. what it's supposed to do ) the difficulties disappear. Sometimes (the ideal case) the little tweak not only simplifies the bottom level, but expands and generalizes the top-level functionality.

A sculptor (Michelangelo?) said that sculpting is, for him, like uncovering something that already exists inside the block of stone. For me, programming is similar: I feel like there's a perfect design and implementation, and I discover it by chipping away one bit at a time.

Of course, my creations are imperfect. When I look at code I wrote 10 years ago, I'm often appalled. I can see the logic of what I did, but I'd do it differently (and better) now. This is especially true with BOINC, particularly the documentation, which is terrible. I think that in general my programming has improved over time.

When I run into something that's really complex and hard - a design decision, or an algorithm I need - I do what I call 'putting on my thinking cap'. This means lying down, closing my eyes, visualizing data structures, and letting ideas come to me. This is similar to how I used to solve math problems. It works remarkably well.

Getting stoned is occasionally useful in these situations, but (especially in recent years) it leaves me fuzzy-headed, so I haven't done it recently.

In general I'm more productive outside my house, because there are no distractions like piano and refrigerator. I'm productive when other people are nearby, because I don't want them to see me goofing off. I've done a lot of programming in crowded and noisy cafes; having to block out the noise somehow heightens my focus.

I was very productive at Space Sciences Lab, in a big room with a few others (Matt Lebofsky and Jeff Cobb, sometimes Aaron Parsons). We all worked super hard, but once or twice a day we'd take a break and chat about the Sierras, prog rock, etc.

My attention span while coding is long. Once, while I was working at Sonic, my dad (Henry) stopped by to visit me. They showed him to my cubicle, where I was working. He could tell I was extremely focused so he stood behind me and waited for me to look up. He waited for around 45 minutes.

For the last few years I've worked at home. I use a 54" 4K TV as a monitor, and it's set up at eye level about 7' away to reduce eye strain and bad posture.

Rewards

My life has four main (somewhat overlapping) parts:

• Music (I play piano).
• Athletics and outdoors (I do almost everything).
• Interpersonal stuff: family, friends, and relationships.
• Programming.

Each of these gives me its own forms of pleasure, purpose, and satisfaction. For programming these include:

• I love it when people people - preferably lots of them - use something I've written, especially if it lets them do creative things or provides some other utility.
• I want my work to change the world in positive ways. BOINC got lots of (maybe a million) people interested in radio SETI other fields of science. It enabled research that helped find drugs and fight disease. Music Match tried to enrich classical music.
• For me, programming is psychotherapy. The world can seem messy, confusing, and unpredictable. People misinterpret what I say and get annoyed with me. When I work on my own software, I'm in a world of order and beauty. I have control; I made the world myself. It's predictable: if something works, it works forever. If it doesn't work, there's a reason waiting to be discovered.

Working on a joint project is a bit different. The other programmers may have coding styles or approaches that differ from mine. There may be conflict and compromise. Egos and feelings can come into play. It's more like the real world.

But joint projects can also be great. There are plenty of people who, compared to me, are smarter, or think more clearly, or know more programming techniques and algorithms. Working with such people - brainstorming, hammering out designs and structures - is one of my greatest pleasures. It typically leads to better solutions than what we could have created individually.

Software mortality

It would be nice if software lasted forever, but it doesn't. What you write might get used by a few people - maybe a lot of people if you're lucky. But eventually something better replaces it. Or it becomes irrelevant. Or the language that it's written in goes away. Or the 5.25" floppies it's stored on can't be read anymore. Of course, everything in life is transient. We all have to come to grips with that.

I have a fantasy that 100,000 years from now, long after humans have become extinct in a huge environmental clusterfuck, aliens arrive on Earth in silvery lenticular vehicles. They find our libraries and our solid-state storage devices; they learn our languages, and read our literature and our journals and our software.

One slobbering tentacled alien says to another: "Y'know, these humans mostly churned out a bunch of crap". The other replies: "Yeah, but check out this guy David Anderson - he wrote a lot of great software, and some nice research papers too".