Sources

• Proceedings of recent ICMCs. I looked at 2013-2018.

The following journals use the rip-off "pay to view" model:

• Computer Music Journal
• Music Analysis
• Music Perception
• Musicae Scientiae
• Journal of new music research

CMJ is on Jstor. I looked at 1990-2019. Instructions for Contributors.

Nuance analysis

I use the following 'tags' for papers:

"Statistical": studies the statistics of the nuance.

"Modeling": identifies parameterized functions that fit the data.

"Perception": studies how listeners perceive music.

"Physiological": studies factors involving performer's physical and cognitive processes, or physical properties of instruments.

"Algorithmic": systems that deterministically add nuance to a work based an (externally defined) structural decomposition e.g. putting volume hairpins over phrases, and putting a ritard at the end. In some cases there are additional high-level controls, e.g. an 'emotion' knob you can turn. Seems to me that this can only generate nuance that is repetitive and predictable. It's not sufficient for e.g. virtual performance. It implies that there is an ideal interpretation of a work, which is contradicted by the fact that people keep playing and recording the work.

Basics

• Time Warping of Compound Events and Signals. Roger Dannenberg. CMJ 21(3) Autumn 97 pp61-70.

  time warping; perform time is integral of 1/tempo

• The Vibrato Problem: Comparing Two Solutions. Honing. CMJ 19(3) Autumn 2005.

  Things like drum rolls and trills shouldn't be affected by tempo change.

• Ensemble Timing in Computer Music. David Jaffe. ICMC 1984 p 185-191.

  Implies that a linear accel produces a quadratic time map; wrong.

Bruno Repp

Most of the important work in this area is by Bruno Repp. His work is amazingly thorough and meticulous.

• A microcosm of musical expression. I. Quantitative analysis of pianists’ timing in the initial measures of Chopin’s Etude in E major . The Journal of the Acoustical Society of America, 1998

  Statistical. Compared 117 performances. Used principal component analysis on the vector of 'inter-onset intervals'. Ignores asynchrony. Found 4 major components; e.g. ritards at end of melodic fragments. Looked for correlation with a) recording date; b) age at time of recording; gender; nationality.

• A microcosm of musical expression: II. Quantitative analysis of pianists’ dynamics in the initial measures of Chopin’s Etude in E major Journal of the Acoustical Society of America, Vol. 105 pp. 1972-1988, 1998

  Similar to the above, but with dynamics.

• A microcosm of musical expression. III. Contributions of timing and dynamics to the aesthetic impression of pianists' performances of the initial measures of Chopin's Etude in E Major. Journal of the Acoustical Society of America, Vol. 106 pp. 469-478, 1999

  Perception, statistics. Studied how the above factors correlated with expert judgement of the performances. Found little correlation.

• XVI. Individual differences in the expressive shaping of a musical phrase: The opening of Chopin’s etude in E major In 'Music, mind, and science', book Suk Won Yi, 1999.

  Rehashes the above 3 papers.

• Detecting deviations from metronomic timing in music: Effects of perceptual structure on the mental timekeeper Perception & Psychophysics, Vol. 61 (3) pp. 529-548, 1999

  Perception, Statistical. The idea of 'timing expectation', and expression as a deviation from this. Subjects press a button if they hear a pause in the music.

• Relationships between performance timing, perception of timing perturbations, and perceptual-motor synchronisation in two Chopin preludes Australian Journal of Psychology, Vol. 51 (No. 3) pp. 188-203, 1999

  Perception, statistical. Whether people hear timing deviations depends on their position in phrase structure.

• Variations on a Theme by Chopin: Relations Between Perception and Production of Timing in Music. Human Perception and Performance, Journal of Experimental Psychology, Vol. 24 (No. 3) pp. 791-811, 1998

  Perception, statistical. Similar to the above. Also studies timing deviations when performer is trying to play metronomically (like expressive deviations but smaller).

• The Detectability of Local Deviations from a Typical Expressive Timing Pattern. Music Perception, 1998

  Perception. More on listeners' detection of timing deviations.

• Effects of tempo on the timing of simple musical rhythms. Repp, Windsor, Desain. Music Perception, Vol. 19 (No. 4) pp. 565-593, 2002

  Statistics. Timing deviations differ little with tempo.

• The timing implication of musical structures. In D. Greer (Ed.), Musicology and Sister Disciplines, Past, Present, Future, pp. 60-70, 1997

  Statistical. Earlier (pre-PCA) version of above Chopin papers.

• Patterns of note onset asynchronies in expressive piano performance. Journal of the Acoustical Society, Vol. 100 (6) pp. 3917-3932, 1996

  Statistical, physiological. Deviations from simultaneity in chords. Top note tends to be earlier, because it's louder and hammer gets there first. Some pianists have left-hand lead (agogic accents). Kinda shallow analysis.

• Pedal timing and tempo in expressive piano performance: A reliminary investigation. Psychology of Music, Vol. 24 pp. 199-221, 1996

  Statistical, physiological. Measures pedal down/up times, how they vary with tempo. Inconclusive.

• The dynamics of expressive piano performance: Schumann’s “Träumerei” revisited. J. Acoust. Soc. Am, Vol. 100 (1) pp. 641-650, 1996

  Statistical, modeling. People played melody louder, and volume increased with pitch and tempo. Volume in crescendi may be a linear function of metrical time!

• Quantitative effects of global tempo on expressive timing in music performance: Some perceptual evidence. Music Perception, Vol. 13 (No. 1) pp. 39-57, 1995

  Statistical. People play with less timing deviation when playing faster or slower than their preferred tempo.

• Expressive timing in a Debussy prelude: A comparison of student and expert pianists . Musicae Scientiae, Vol. 1 (2) pp. 257-268. 1997

  Statistics. 10 people playing the same piece have about the same timing. Experts have more deviation than intermediates.

• Expressive timing in Schumann’s “Träumerei:” an analysis of performances by graduate student pianists. J. Acoust. Soc. Am, Part 1, November, Vol. 98 (5) pp. 2413-2427, 1995

  Like the above but with Schumann piece

• Relational invariance of expressive microstructure across global tempo changes in music performance: An exploratory study. Psychological Research, Vol. 56 pp. 299-284, 1994

  Statistical. Timing nuance scales with tempo, except for grace note timing.

• A Constraint on the Expressive Timing of a Melodic Gesture: Evidence from Performance and Aesthetic Judgment. Music Perception, Vol. 10 (2) pp. 221-242, 1992

  Modeling, perception. Studied 28 expert performances of Traumerei. Ritards were best modeled as parabolic (quadratic), and listeners liked this more than random curves, or time-shifted parabolas.

• Diversity and commonality in music performance: An analysis of timing microstructure in Schumann's “Traumerei”. Journal of the Acoustical Society of America, Vol. 92 (5) pp. 2546-2568, 1992

  Similar to the above.

Physical tempo models

These papers assert that tempo derives from physical movement, e.g. that ritardandi correspond to slowing down with a constant braking force. This implies a quadratic slowness function.

Comment: their curve-fitting is noisy; it's not clear that a quadratic fits better than an exponential or something else.

• The Final Ritard: On Music, Motion, and Kinematic Models. Honing. CMJ 27(3) Autumn 2003. 66-72
• Force Dynamics of Tempo Change in Music Jacob Feldman, David Epstein and Whitman Richards. Music Perception: An Interdisciplinary Journal , Winter, 1992, Vol. 10, No. 2 (Winter, 1992), pp. 185-203
• Stopping in running and in music performance Part II: A model of the final ritardando based on runner's deceleration. . Friberg and Sundberg. TMH-QPSR 38(1) 1997 p.67-73.

  Also Part I: Runners' decelerations and final ritards

  Final ritardando is similar to way runners stop; constant braking power. Inverse tempo is a square-root function of score time. (w/ parameters for braking rate and final tempo).

• Is the musical retard an allusion to physical motion? Kronman and Sundberg. STL-QPSR 25(2-3) 1984 pp. 126-141.

  Retard and physical motion. square root model.

• On the anatomy of the retard. A study of timing in music. Sundberg and Verrillo. STL-QPSR 18 (2-3) 1977 pp44-57.

  Model: ritardandos have two parts. Tested this w/ expert human listeners. They liked less rit than performed.

Algorithms

Algorithms that generate nuance based on some structural aspect (perhaps manually specified) of the score. Goal: automatically generate a human-sounding rendition (why??).

• Generative Rules for Music Performance: A Formal Description of a Rule System. Friberg. CMJ 15(2) summer 1991. higher = louder etc.
• A Model of Expressive Timing in Tonal Music, Neil Todd, Music Perception, 1985.

  Algorithmic. Nuance as a function of musical structure ("time-span reduction"). Model a piece as a hierarchy of gestures. Each gesture is played with hairpins in tempo and dynamics. (with specific functions for the hairpins)

  My conjecture: this produces plausible but boring nuance.

• The dynamics of dynamics: A model of musical expression. Neil P. McAngus Todd. The Journal of the Acoustical Society of America 91, 3540 (1992)

  Algorithmic. Similar to the above. Motion-based physical model. Hairpins. Shape depends on 'structural important'. Faster == louder. Notion of 'intensity'. Three-level hierarchy.

  Talks about nuance inference: make a guess, look at delta, refine guess.

• A computational model of rubato. N. Todd, Contemporary Music Review. 1989. pp 69-88.

  Algorithmic.

• The Extraction of Expressive Shaping in Performance. Muller, Mazzola. CMJ 27(1) spring 2003. 47-58.

  Inference of nuance (as 'performance vector fields') from audio performance.

• Tempo Curves Re-visited: Hierarchies of Performance Fields. Mazzola, G., and 0. Zahorka. Computer Music Journal 18(1):40-52. 1994.

  Need to review. CMJ 27(1) spring 2003 47-58.

• Expressive Timing and Dynamics in Real and Artificial Musical Performances: Using an Algorithm as an Analytical Tool W. Luke Windsor, Eric F. Clarke. Music Perception 15 (2): 127–152. (1997)

  Algorithmic. Compares algorithmic nuance (Todd model) with human performances. Conclusion: algorithmic is a good baseline but doesn't completely explain performances.

• Computational Modeling of Expressive Music Performance with Linear and Non-linear Basis Function Models. Carlos Eduardo Cancino Chacon. PhD thesis, Johannes Kepler Univ. Linz. 2018.

  Algorithmic. Generate a model by analyzing performances.

• What were you expecting? Using Expectancy Features to Predict Expressive Performances of Classical Piano Music Cancino-chacon et al. Machine Learning 106(6) 887-909, 2017

Other

• Make Me a Match: An Evaluation of Different Approaches to Score-Performance Matching. Jeijink, Desain, Honing, Windsor. CMK 24(1) spring 2000. 43-56.

  Survey of 'matching'.

• Timing is of the Essence. Jeff Bilmes' MS thesis from MIT, 1993.

  For an ensemble drum performance, finds deviation from score, then multiplies this by various positive and negative numbers.

• Computational modeling of musical cognition: a cause study on model selection. Henkjan Honing

  Modeling. Compares kinematic and 'perception-based' models of the final ritard. Complex.

• From Time to Time: The Representation of Timing and Tempo. Henkjan Honing. Computer Music Journal 25 (3) 2001 50-61.
• Tempo curves considered harmful. Desain and Honing. Contemporary Music Review 7(2). Jan 1993. Peter Desain and Henjkan Honing. Good source of refs.
• Towards a calculus for expressive timing in music. Desain and Honing, Computers in Music Research 3 pp43-120, 1991
• A Musical Microscope Applied to the Piano Playing of Vladimir de Pachmann . Nigel Nettheim, 2001.

  Subjectively compares several performances of the start of a Chopin Nocturne. Uses interesting score notation that shows timing and dynamics.

• A corpus analysis of rubato in Bach C prelude. Benadon and Zanette. Music Performance Research 7 pp 1-26. 2015. Studies the average lengths of pauses: the larger the phrase unit, the longer the pause. Also compares various performers. Looks at Traumerei also. Worth reading.
• A Three-Dimensional Model for Evaluating Individual Differences in Tempo and Tempo Variation in Musical Performance. Zhou and Fabian. Musicae Scientiae, 2019.

  Subjective comparison of tempo (global and local) in 2 performance of Chopin Ballade 1. Lots of words. Reduces things to 6 parameters (e.g. tempo in slow parts, use of ritards) Models everything as tempo, not pauses. Get 'stylistic profile' of the 2 pianists, but doesn't show this carries over to other pieces.

• Modeling and control of expressiveness in music performance. Canazza et al. PROCEEDINGS OF THE IEEE, VOL. 92, NO. 4, APRIL 2004.
• A Survey of Computer Systems for Expressive Music Performance. Kirke and Miranda, ACM Computing Surveys,Volume 42Issue 1December 2009
• Emotional coloring of computer-controlled music performances. Bresin and Friberg. CMJ 24(4), 2000
• Investigating pianists' individuality in the performance of five timbral nuances through patterns of articulation, touch, dynamics, and pedaling. Bernays and Traube. Frontiers in Psychology, 2014.
• Computational Models of Expressive Music Performance: The State of the Art. Widmer and Goeble. September 2004, Journal of New Music Research 33(3):203-216
• Computational models of expressive music performance: a comprehensive and critical review. Cancino-Chacon et al. Frontiers in digital humanities, 2018.
• Sources of timing variations in music performance: A psychological segmentation model. Penel and Drake. Psychol Res 1998
• Mapping Musical Thought to Musical Performance. Caroline Palmer. J. Exp. Psych 1989
• In Search of the Horwitz Factor. Widmer et al. AI Magazine 2003.
• Timing variations in music performance. Penel and Drake. Perception and Psychophysics. 2004

  Analysis of timing variation.

• Performing Bach's Keyboard Music - Phrasing, George Kochevitsky, Bach 3(4), pp 28-32 Oct 1972.
• Music Performance Research at the Millennium. Gabrielsson. Psychology of Music ; Manchester Vol. 31, Iss. 3, (Jul 2003): 221-272.
• Emotional coloring of computer-controlled musical performances. Bresin and Friberg. CMJ 24(4) winter 2000. pp 44-63
• Methodologies for Expressiveness Modelling of and for Music Performance. De Poli. Journal of New Music Research 33(3) 189-202.

  Does this actually say anything?

• Performance analysis and Chopin's mazurkas. Nicholas Cook. Musicae Scientiae, 2007
• Methodologies for Expressiveness Modeling of and for Music Performance Giovanni De Poli. 2004, Journal of New Music Research.
• A state of the art on computational music performance. Miguel Delgado et al. Expert Systems with Applications, 2010
• Overview of the KTH rule system for musical performance. Anders Friberg et al. Advances in Cognitive Psychology. 2006.

  Algorithm.

• Playing Mozart by Analogy: Learning Multilevel Timing and Dynamics Strategies G. Widmer. Journal of New Music Research 32(3):259-268, September 2003
• Expressive gesture in Grieg’s recordings of two Op. 43 Lyric Pieces: An exploratory principal components analysis. Georgia Volioti, Music Performance Research, 2019
• Expressiveness in Music Performance: Empirical approaches across styles and cultures Dorottya Fabian, (Oxford: OUP 2014), 58-79
• Audio Morphing Different Expressive Intentions for Multimedia Systems Canazza, De Poli, Drioli, Roda. July 2000 IEEE Multimedia 7(3):79-83.
• An Interdisciplinary Review of Music Performance Analysis Pati, Lerch et al. November 2020 Transactions of the International Society for Music Information Retrieval 3(1):221-245
• Investigations of Between-Hand Synchronization in Magaloffs Chopin. Goebl, Flassmann, Widmer. CMJ 34(3) fall 2010 pp 35-44.

  Various types of time-shift nuance, and 'earlier tempo rubato' (hand independence).

• Bézier Spline Modeling of Pitch-Continuous Melodic Expression and Ornamentation. Battey. CMJ 28(4), winter 2004.

  Use splines to model pitch variation in e.g. Indian music

Manfred Clynes

Manfred Clynes had a theory where nuance comes from 'pulse patterns': multiple levels, one for 16ths in a beat, others for beats in 4/4 and 3/4 measures. Separate patterns for volume and duration.

Where do these patterns come from? He did something where he had famous performers tap their fingers along with music, and recorded the result. But mostly he just made them up.

He claimed that each composer (not performer!) had a particular set of pulse patterns that applied equally well to all their work. He had a vague theory of acoustic aesthetics/emotion called "Sentic forms".

This work seems like BS. Repp calls it 'analysis through synthesis': claim that a model is valid because it produces plausible results in a few cases (though in this case the results are not plausible). Also, all of Clynes' stuff (including his self-authored Wikipedia page) has an uncomfortably high level of self-promotion.

He developed a Windows program called SuperConductor (not related to SuperCollider AFAIK). Superconductor does audio synthesis (e.g. of bowed string sounds). It provides control (through a GUI) of high-level (e.g. crescendi) and note-level (e.g. vibrato) nuance. It supports nonlinear curves (exponentials, cubic splines?), and (presumbly) pulse patterns. It looks rigid and limited. It comes with a few pieces by "The Great Composers" (in a proprietary format) that you can add nuance to. It can also import/export MIDI.

There are a bunch of examples on YouTube, mostly from string quartets. These are not bad - they have nuance with appropriate general properties. However, the nuance is excessive, and it's clear within a few seconds that it's computer-generated.

Papers:

• The hidden code of musicality. ICMC 2983
• CLYNES, M. 1986. Generative principles of musical thought: Integration of microstructure with structure. Communication and Cognition 3, 185-223.
• Expressive Microstructure in Music: A Preliminary Perceptual Assessment of Four Composers' 'Pulses'. Bruno Repp. Music Perception: An Interdisciplinary Journal, Vol. 6, No. 3 (Spring, 1989), pp. 243-273 (31 pages)

  An attempt to evaluate Clynes' claims. Mixed results.

• Composers' Pulses: Science or Art?. Bruno Repp. Music Perception, 13, Vol. 7 (4) pp. 423-434, 1990

  Similar to the above

• Expressive Microstructure in Music: A Preliminary Perceptual Assessment of Four Composers' "Pulses". Bruno Repp. Music Perception, Vol. 6 (3) pp. 243-274, 1989

  Similar to the above.

• Patterns of expressive timing in performances of a Beethoven minuet by nineteen famous pianists. Bruno Repp. Published 1 May 1990, The Journal of the Acoustical Society of America

  Tried and failed to find Clynes-type 'pulse patterns' in 19 performances of a Beethoven piece.

• Composer-specific aspects of musical performance: and analysis of Clynes's theory of pulses for performances of mozart and beethoven. William Forde Thompson. Music Perception 7(1) 15-42, 1989.

  Another attempt to evaluate Clynes' claims. Mixed results. When Beethoven is played with the Beethoven patterns, listeners said it sounded Beethovenian. But it was 'more musical' when played with the Haydn pattern. Etc.

• Manfred Clynes, pianist. Bruno Repp. In:, A Festschrift for Manfred Clynes. St. Louis, MO: MMB Music, pp. 70-84

  Homage and/or ass-kissing.

Other

• The art of inaccuracy: Why pianists' errors are difficult to hear. Music Perception, Vol. 14 (2) pp. 161-184, 1996

  Studies mistakes (wrong or added notes). Most of the time they're harmonically appropriate, hence hard to hear even by experts.

Comments

No one has compared multiple performances by the same performer.

Sources of performance data

• MAESTRO: a corpus of piano performances (MIDI file and audio; no score info).

Visualizing nuance, virtual conductor