David P. Anderson
December 2021

Introduction

Traditional classical music scores use Western Music Notation (WMN), the familiar system of staffs, note-heads, stems, ties, slurs, dots, dynamic markings, etc. Scores consist of fixed-format black-and-white printed pages. Things have remained the same for hundreds of years.

This essay describes some ideas about using computer technology (color displays, interactive software, the Web) to expand and enrich musical notation. I focus on classical piano music since that's what I play, but the ideas are relevant in other contexts.

The ideas fall into four separate but related areas:

• Notation of musical nuance: extend WMN to allow more general and precise notation of temporal and dynamic nuance.
• Configurable scores: separate the information underlying a score (semantics) from the way it's displayed to the performer (syntax). Let the performer control how the score looks.
• Layered scores: separate scores into layers. The base layer is what the composer wrote. Additional layers contain things like fingerings, nuance notation, and other annotations. Layers are first-class objects that can be archived and shared. The performer can select layers and create their own.
• Interactive scores: a program "follows" a performance (via audio or MIDI) and displays only the part of the score currently being played.

Notating nuance

By "nuance" I mean the deviations in timing, dynamics, and articulation between a score and a performance. In classical piano music, nuance is everything. If you play a chord with all notes equally loud, it sounds bad; you typically voice to the top and/or bottom. When Rubenstein plays Chopin, no two notes have the same dynamic level, and no two beats have the same length. In Chiara Bertoglio's performance of An Wasserflüssen Babylon, she staggers the voices to separate them; i.e. notes that are simultaneous in the score are not played simultaneously.

These kinds of nuance are difficult or impossible to notate in WMN. The monochrome, typeset nature of WMN doesn't really allow it. But with the features of computers and hi-res color displays, it's easy to imagine ways to notate nuance, either precisely or suggestively; I'll describe some below.

Of course, WMN already has various notations of nuance. You can write 'accelerando' and draw hairpins. These indicate a general idea, but they're vague and imprecise. And there are plenty of things you can't notate at all.

Capabilities of computer displays

Existing scores are white pages with black printing. These pages sit on a music stand some distance above the keyboard. Compared with this, computer displays have several additional capabilities:

• Color: everything can be arbitrary color.
• Movement: everything can move. Noteheads can wiggle. The score can scroll horizontally, always showing the part currently being played (see below).
• In a system with a head-mounted display (like Google glasses), the score can be displayed anywhere in the visual field; e.g. it can hover over the keyboard, allowing you to see the score and your hands at the same time.
• Head-mounted displays can display things in 3-D. Noteheads can "pop out".

Why notate nuance?

When I first talked about this with my pianist friends they were concerned that adding nuance to scores would eliminate spontaneity and unpredictability from live performance, turning pianists into automata. But that's not what I have in mind. Here some situations where notating nuance might be useful:

• A composer (perhaps using MuseScore) could add nuance to a score so that, when played by the computer, it sound more like it's being played by human musicians. The composer could include all, some, or none of these notations in the published score.
• When working on a piece, I could record my interpretive decisions so that I'm reminded of them if I put the piece aside for a while and return to it.
• A chamber group, making decisions about phrasing and interpretation during a rehearsal, could record these and practice them separately.
• Editors who want to add a lot of interpretive stuff (like Schnabel) could do so in concise way, rather than writing long essays at the bottom of each page.
• There are piano pieces for which I have a (nuanced) interpretation in my mind, but I'm not able to render it physically. I'd like to be able to render it using a computer.

Semantics and syntax

Let's say you want to indicate that the top and bottom notes of a chord should be played louder. There are various ways to indicate this visually. You could make the note-heads different shades of gray, or different colors, or different sizes, or they could pulsate in some way.

The idea that top and bottom are louder is semantics; the way it's represented visually is syntax. These should be separated, with some mechanism (sort of like CSS stylesheets in HTML) that maps semantics to syntax. The editor or composer specifies semantics; the performer selects the syntax.

Comments

• The notations I propose below extend WMN; they don't change or replace any part of it. I think WMN is great, and we've all made a huge time investment in learning it at a neural level.
• These ideas are at a brainstorming stage. This document is intended to get the ideas out there and stimulate discussion. To make a new notation 'real' we'd need to
  • Add, to MuseScore, a feature to input and edit its semantic level.
  • Add, to MuseScore, at least one way to render it syntactically.
  • Add, to MuseScore's audio output, an interpretation of the notation.
  • Annotate a few widely-played pieces so that musicians can try out the notation and offer feedback. Perhaps publish these on IMSLP.
• The figure of merit of a notational semantics is the musical relevance of its expressive power. The figure of merit of a notational syntax is the ease with which it can be learned, and visually parsed during performance.
• I hope that people will propose and implement new nuance notations. These will compete, and evolve. Maybe some of the features might catch on and become a de facto standard among composers and editors. Maybe at some point these will be collected into a 'WMN 2.0' standard.

OK, now some possible nuance notations.

Dynamics

Note-level dynamics (including chord voicing); possibilities:

• Note-head color-coding: loud in red, soft in green (or some other color scheme). Or loud in black, soft in gray.
• Note-head size: bigger is louder (already used in WMN sometimes)
• Pulsed note-heads; deeper or faster pulse = louder.

Note: traditional scores are black on a white background. If we're going to use color and gray-scale, we might want to consider black backround; it may be easier to visually distinguish colors that way.

Overall dynamics:

• Vertically scale entire staff, bigger = louder? (might be confusing, especially with abrupt changes).
• Change color or line weight of all notes.
• Change the background color.

Grouping

It should be possible to group notes (e.g. fugue voices), and highlight groups in different ways:

• With a colored background, like a Hi-Liter.
• Drawing the noteheads differently (color, size, pulsation).

We might want to demarcate structural units (e.g. 8 bar phrases), perhaps with some indication of shape within a phrase (e.g. the high point). This could be used in "interactive scores" (see below) in various ways, e.g. show only the current phrase, or highlight it in some way.

Rhythm and timing

We should be able to denote fluctuations in the underlying pulse. One way to do this is with "proportional notation", in which the horizontal space for a note is proportional its notated duration. We can then draw light vertical "grid lines" on quarters or eighths. The distance between lines is the duration of the interval, and the variation in this spacing shows fluctuation.

To make this easier to visually parse, we could tinge these vertical strips with red (faster than average) to white (average) to blue (slower than average). (Although this could conflict with other uses of color.) This color-coding could be used without proportional notation.

It would make sense to center note-heads on the grid line when they occur. Deviations from the underlying pulse (playing ahead of or behind beat) could then be denoted by offsetting note-heads from the grid lines. This is different from measure lines in WMN; if we use this notation, we'd need to change measure lines so that they pass through downbeat note-heads; things would be confusing otherwise.

The WMN notation for rolled chord is a wiggly line with an optional arrow. This can't express the speed or irregularity of the roll, or rolls that are neither up nor down. Proportional notation could express these.

Articulation

A more precise notation of articulation (staccato/portamento/legato) would be good; an indication of how long each note sounds, or is held down. I don't have any ideas here.

Entering and editing nuance

Nuance typically has structure; e.g. a pattern of rubato might repeat over and over. You wouldn't want to have to tweak the timing of individual notes; you'd want to able to define some kind of template that you can then apply repeatedly. We'd need to figure out a GUI approach for doing this.

We'd also need a way to "layer" nuance - e.g. to combine measure-level rubato with a long accelerando.

Aside: in the 1980s I developed a programming language called FORMULA for doing exactly this sort of thing. But no one (except me and a few others) wanted to write programs to generate music.

Configurable scores

The idea is that the performer should be able to control how a score is displayed in terms of

• size
• density: horizontal and vertical spacing
• color
• how nuances are rendered

Layered scores

The idea is to separate the base score (i.e. what the composer wrote) from various sorts of annotations, such as:

• Nuance notations.
• Fingerings.
• Textual annotations.
• Notational edits, such as changing stem directions, moving notes between staffs, fixing accidentals, and so on.

These layers can be created by the composer, by an editor, or by the performer. Performers can add or remove these layers, as well as adding their own. They can use the same notational features as composers and editors; they'll no longer have to scribble in pencil.

Layers would be stored as separate XML files. MuseScore would support the creation, editing, and selection of layers. Performers and editors can upload their layers to IMSLP, which would offer a collection of layers along with each score.

Interactive scores

With traditional scores, performers see 1 or 2 pages at a time. They have to keep track of where they are on the page, and they have to turn pages.

The idea of an "interactive" score" is that software can "listen" to the performer (via sound or MIDI), keep track of where they are, and display only the current part of the score: maybe a few seconds. Advantages:

• "Now" is always at the same place on the display. The performer's eye doesn't have to jump around.
• Because only part of the score is shown, this part can be larger, easier to parse, and can contain more information.
• No page turns.
• The software keeps track of repeats.

This opens up new notational possibilities: for example, the displayed portion could change in size or color to indicate dynamics or tempo.

More generally: a "performance score" (what the performer sees during a performance) doesn't necessarily have to show everything. If the piece is mostly memorized, it might show interpretive info and a few key notes. A performance score could potentially be any sensory input: virtual reality, sound, tactile feedback, etc.

Implementing interactive scores would require integrating performance-following technology (audio or MIDI) with MuseScore.

How to realize these ideas

See a list of resources. The main things I currently use are:

• IMSLP: A web-based library of copyright-free classical music scores. These are PDFs: mostly scans of printed scores, a few generated by score editors.
• forScore: a program (cheap but not free) that displays PDFs on iPads. It interfaces with Bluetooth page-turn pedals. It lets you annotate scores with a pencil.
• MuseScore: an open-source score editor. It can render scores as audio, so it's widely used by composers: they can hear pieces as they compose them. There are similar commercial systems (Sibelius, Finale).
• MusicXML: a standard for machine-readable WMN score representation. The various score editors all use this, and extend it with non-standard elements to support their unique features.

Each one is mature and has a big community of users, developers, and/or contributors. I think any efforts to extend music notation needs to leverage these tools.

To realize some or all of the ideas described here:

• Extend MusicXML to handle a variety of new features.
• Implement these new features in MuseScore.
• Have MuseScore replace forScore as the standard way performers view scores during practice and performance.
• Have MusicXML files (with new features) replace PDF as the standard score file format offered by IMSLP.