|Formalizing nuance in classical music||
random trip report
1 May 2022
This essay is concerned with nuance in western classical piano music. In this context, nuance can be loosely defined as the differences between the score for a piece and an audio rendition of the piece, such as a human performance. These differences can be roughly divided into
We focus on piano. Other instruments (and voice) are more complex because a) their notes have additional parameters (such as attack and timbre), and b) the parameters may change continuously. The ideas presented here apply to such instruments, but would have to be extended to encompass these factors.
For most classical music, nuance is a critical component of rendition. An example: "Wasserklavier" by Luciano Berio. The score is here.
Notice that in Grimaux's performance, no two beats have exactly the same duration, and no two notes have exactly the same volume. The nuance of the performance unlocks the beauty and expression of the piece.
Where does nuance come from? Some scores have indications of nuance: tempo markings, slurs, crescendo marks, fermatas, pedal markings, etc. However:
Some musical styles have associated conventions for nuance. In many styles, for example, upbeats are softer, and pieces end with a ritardando. Performers learn these conventions by osmosis (and in many cases the modern conventions differ from those of the composition's period).
But score markings and stylistic conventions are just guidelines. In the end, nuance is left up to the performer(s). Some nuance may be planned in advance. Some may be spontaneous during a particular performance. Some may be unintentional artifacts of the performer's technique.
I think we need a "formalism" to describe nuance: a language with precisely-defined syntax and semantics. This formalism should have these properties:
I have developed a formalism with these properties: Music Nuance Specification (MNS). Its details are described here, and a Python implementation of it is here.
To many musicians, nuance is ineffable - it's something magical that happens during performances, and to analyze or formalize it would break that magic spell.
This viewpoint is understandable. But as music evolves, and as computers are increasingly important tools for composition, pedagogy, and performance, there are reasons to expand our ability to represent and manipulate nuance: to make it a first-class citizen, along with scores and sounds. Doing so will not replace the human component of nuance, or the spontaneity of performance; rather, it will provide tools that enhance these processes, and that enable new ways of making music.
Let's assume that we have a formalism describing nuance, and that we have software tools that make it easy to create and edit "nuance specifications" for pieces. These capabilities would have several applications:
As a composer writes a piece, using a score editor such as MuseScore or Sibelius, they could also develop a nuance specification for the piece. The audio rendering function of the score editor could use this to produce nuanced renditions of the piece. This would facilitate the composition process and would convey the composer's intentions more clearly to prospective performers.
Performers could create prepared performances of pieces, in which they play the piece using a computer rather than a traditional instrument.
A teacher's instruction to a student could be represented as a nuance specification which guides the student's practice. This could be done in various ways. For example, as a student practices a piece they could be shown "virtual conductor" that expresses (graphically, on a computer display) a simplified representation of the target nuance.
When an ensemble (say, a piano duo) rehearses together, they could record their interpretive decisions as a nuance specification. They could then use this to guide their individual practice (perhaps using the "virtual conductor" described above).
IMSLP lets people share musical scores, and recordings of renditions. It could also include nuance descriptions for pieces. This would provide a framework for sharing and discussing the interpretation of pieces.
What kind of UI (user interface) would facilitate creating and editing a nuance specification - in particular, for transcribing one's mental model of a performance?
This generally involves changing every parameter - start time, duration, volume - of every note. We can imagine a GUI that shows a piano-roll representation of the score and lets you click on notes to change their parameters. This low-level approach would let you do whatever you want, but it would be impossibly tedious.
Desirable properties of a UI for editing nuance:
Some general approaches:
Having a formalism for nuance opens up huge range of possible research in musicology.
The most basic issue is what I'll call the "primitive selection problem". A nuance formalism (like MNS) provides a set of "primitives". Some of these define fluctuations in tempo or volume that affect lots of notes. Others apply random 'jitter' to timing and volume. Others affect sets of notes based on attributes such as position within a measure. Others apply to individual notes.
The goal in designing the set of primitives is to find a small "basis set" of transformations, each with a small number of parameters, that can achieve the desired specifications - for example, that can closely approximate typical human performances.
MNS, for example, has a primitive for linear tempo change. This was easy to implement - but is it a good approximation of ritardandos and accelerandos in practice? There may be better choices.
I can imagine a research program to study this, by calculating the nuance in human performances, and finding the primitives that approximate it best.
The first step is to automatically extract nuance from human performances:
We can then use software to find a transformation that maps the score to the performance. This tranformation would typically have multiple levels. A first level would model large-scale fluctuations. The second level would take the residue from this, and fit it, possibly with different types of primitives. At some point the residue presumably would be noise-like, and its statistical properties could measured.
Each level would consist of a set of primitives. The software would consider various families of primitives: in the case of continuous fluctuations this might include linear, polynomial, exponential, logarithmic, etc. The software would use data-fitting techniques to find an optimal basis set.
It may turn out that the optimal set of primitives depends on
There has been some research in this general area. I've looked at some, but haven't found anything usable.