Timbrenauts: Creative Explorations in Timbre Space
Timbrenauts: Creative Explorations in Timbre Space
Interactive Project Report
Published: July 10, 2024
Authors
Yuval Adler (McGill University) and Berk Schneider (University of California San Diego)
Collaborators
Peter Ko (University of California San Diego), Louis Goldford (Columbia University), and Janet Sit (University of California San Diego)
Overview
“Timbrenauts” is a cross-institutional collaborative student research-creation project funded by the Analysis, Creation, and Teaching of Orchestration (ACTOR) Partnership during the 2022-2023 academic year. We employed experimental designs from timbre perception research to generate data models that will inform the creation of new musical compositions for an atypical instrumental duo of cello and trombone, while leveraging extended techniques rarely explored in traditional musical repertoire. In this TOR module, we discuss our motivations, data collection process, similarity judgment experiment, and the resulting data we want musicians to use as a basis for their creative output. We aim to reveal interesting structures, such as timbral overlaps between the two instruments, which could be employed in exploratory musical compositions. Finally, we'll share an early improvisation session and compositional sketch that demonstrates one of the ways this project can inform new musical works. We also provide a demo of the similarity judgment task for you to try on this page, allowing you to rate the similarities or differences you perceive between the timbres of extended techniques performed on the two musical instruments.
Motivation
Our goal is to test whether methodologies from music psychology and cognition research can be adapted for use in perceptually informed performance, orchestration, and composition practices. This research-creation project aims to utilize timbral similarity judgments between two distinctive musical instruments in a fresh and creative manner, serving as inspiration for composers and performers alike. In addition to demonstrating the relevance of music psychology methodologies for creative purposes, the resulting pieces will uncover novel approaches to the two instruments. Another motivation we have is to integrate contemporary music performance practices into the laboratory setting. Research into orchestral timbres has traditionally focused on the standard playing modes of each instrument. However, these instruments possess much richer palettes of sounds, especially when extended techniques are included. Considering that each instrument occupies its own rich and diverse timbre space, we are interested in exploring how these two timbre spaces overlap. For example, a trombone sordino with a pixie and plunger mute in the upper register may resemble the higher overtones of sul ponticello on the cello more readily than the trombone’s own ordinary recordings in the same register. Similarly, the over-pressurized bowing of a cello string may sound more like the split tones of a trombone rather than an extended timbre of the cello itself.
Imagine a web of timbral relationships that maps these sounds in degrees of similarity, from which composers and improvising performers can readily draw. If we could determine the degree of timbral proximity or distinction of these extended techniques during the early stages of musical exploration, the possibilities for musical creation and orchestration could greatly expand.
The Team
Our team comprises dedicated and diverse individuals, including music scholars, scientists, humanists, composers, and performers. The proposed project directly aligns with Yuval Adler’s doctoral work at McGill University, which explores contemporary practices in instrumental grouping and delves into the expanded orchestration palette available through extended techniques. One orchestration method Adler is investigating involves what composers often refer to as “overlapping timbral territories,” and this project provides an opportunity to empirically test and understand this approach. The two participating experimental performers, Berk Schneider (trombone and co-researcher) and Peter Ko (cello), have been collecting a variety of sonorities on their instruments that have not yet been extensively researched. In addition to psychoacoustic research, we hope these techniques can be incorporated into contemporary music performance, composition, and orchestration practices. Finally, composers Louis Goldford and Janet Sit bring their breadth and depth of experience, gained from their work at innovative research institutions such as IRCAM and the Toronto Creative Music Lab, as well as collaborations with ensembles like the JACK quartet and Talea Ensemble.
First Recording Session
Collecting Sounds
We began with a group discussion among all the members to compile a list of sounds we would like to record on the two instruments. Sounds were selected for the recording session by the performers and composers based on musical interest, speculation of timbral blend between the instruments, and the practicality of execution in different registers and tonalities on either instrument. The audio sample archive includes extended techniques that go beyond typical ordinary playing to include airy overtones and multiphonic gestures, expanding to increasingly complex techniques. Each instrument produced approximately 200 individual samples of various techniques, dynamic ranges, pitches, and registers. Audio samples were recorded with a stereo pair of Josephson C42 cardioid FET condenser microphones for room capture, and a matched pair of Shure KSM137 small-diaphragm condenser microphones were used for close-up capture of each instrument. For the similarity judgment experiment, only recordings made with the KSM137 were used to reduce acoustic reverberations from the specific room where the sounds were captured—the Conrad Prebys Music Center Recording Studio at UC San Diego.
Audio Samples
To give you a taste of the type of recordings we collected from the two instruments, listen to each of the samples below. As an added bit of fun, try to guess which samples come from the trombone and which from the cello. The answers are provided after the samples, so listen before you read further.
Similarity Judgements Experiment
In the experimental portion of this project, we ask participants to rate the similarity they perceive between samples chosen from the recorded sounds. Rather than focusing on the possibility of blends between the sounds, our approach was to select sounds from each instrument that match 10 phenomenological categories of diverse sound production. For example, we might choose a sample from each instrument producing a sound that fits the category of “sustained pitch with heavy vibrato” and another pair of sounds fitting a wholly different category, such as “sustained noise with fluctuating brightness.” The motivation behind this approach is twofold: 1) to map an expansive timbre space using a limited number of diverse samples, and 2) to demonstrate the richness and variety inherent in these instruments’ sound production potential. The first motivation is exploratory, driven by the desire to give the composers in this project an overview or map of the timbre space available to them. This data could then be used by the composers to validate their own models for the entire collection of recorded sounds. The second motivation is connected to our hypothesis that extended instrumental techniques can cover a wide enough range of timbres that each instrument will be better visualized as overlapping regions, like in a multidimensional Venn diagram, rather than single points in a timbre space, as they are typically shown in early experiments into orchestral instrumental timbres.
Participants rated the sounds in an acoustically controlled environment in the Music Perception and Cognition Laboratory at McGill University, with the support of Prof. Stephen McAdams and Bennett Smith. As they rated the 210 pairs of sounds, the participants interacted with an experimental interface, as shown in the image below.
Figure 1: Example interface for the similarity judgment trials.
Try the Experiment Yourself
Below is a basic version of the experiment that you can try for yourself. This webpage does not collect any data from your ratings; the interactive elements are presented only as an example.
Familiarize Yourself with the Set of Sounds
First, listen to the 20 sounds that comprise the samples used in our experiment. This is meant to give you an overall impression of the types of sounds you will encounter in an experimental trial. This step is necessary so that you can use the full range of the slider for your ratings. The sounds that you think are the most dissimilar among this collection of samples should receive the maximum dissimilarity score.
Rate Similarities Between Pairs of Sounds
Now try your hand at rating the similarity of the sound sample pairs below. Listen to a pair of sounds and rate that pair’s similarity before moving on to the next. What rating would you give between 1 and 10, where 1 is very dissimilar, and 10 is identical? There are no wrong answers, and often your first instinct is the best response.
The ratings above might feel awkward at first, and that’s okay. In the actual experiment, a practice phase precedes the main experimental trials. You might also be wondering how your ratings might differ from others’ and if that affects their usability as data. This is where the number of participants becomes important, because when we survey multiple people from a diverse population, the reliability of the ratings increases. Individual ratings might be influenced by the listener’s familiarity or unfamiliarity with these musical instruments, their hearing particularities, or their musical background. However, over a large enough group, these particularities balance out to give a more robust picture of perceived similarity between the recorded samples among the broader population. The resultant data is not, however, a replacement for individual and artistic judgment by musicians and listeners. When you rate these sounds, do you listen more for the spectral characteristics, the pitch information, the attack and decay of the sound, or other parameters?
At this stage, you might be asking what the ten phenomenological categories we used to choose the individual samples are. To avoid bias in your judgments in the example ratings above, we withheld this information until you had the chance to try rating them yourself. If you haven’t tried rating them yet, it is not too late to scroll back before continuing to read our explanation of the sound samples we used.
We chose the ten categories based on the sounds possible on the instruments and the interests of the composers in the project while trying to maximize the diversity of sound production techniques we explored. The ten categories we selected, and the corresponding samples chosen from each instrument, are as follows:
SINGLE PITCH SOUNDS
Sustained pitch
Punctuated pitch
Sustained muted pitch
Sustained granular pitch
Sustained distorted pitch
MULTIPLE PITCHES
Sustained overtone variation
Sustained multiphonic
UNPITCHED SOUNDS
Sustained unpitched
Punctuated unpitched
Granular unpitched
Interpreting the Perceptual Data
As a result of the perceptual judgements we collected, we created three initial models, discussed below. For more information on the mathematical techniques used to generate these models, see Adler’s PhD thesis, expected in late 2024. The underlying data will also be made available as an open-source document to interested parties who want to attempt their own models. A summary of each data model is provided below.
Multidimensional Similarity Space (MSS)
This model presents the data spatially, where distance between the individual samples models perceptual similarity. For example, the sustained unpitched sounds (shown by the light-yellow markers at [0.2, 0.6, 0.1] for trombone and [0.2, 0.5, -0.1] for cello) were closer in similarity ratings compared to the sustained unpitched granulated sounds (dark yellow at [0.7, -0.3, 0.4] for trombone and [0.4, -0.3, -0.6] for cello). It’s interesting to note here some of these relationships can be better predicted by the phenomenological categories we decided on in advance (such as the sustained noises clustering together) while others are better predicted by instrument (like how each of the sustained granulated noises are closer to other sounds from the same instrument rather than to each other). This multidimensional space of extended techniques could be used to reveal atypical sound production technique combinations in orchestration or improvisation for attempting either instrumental blends or greater distinction between the instruments. This model was generated as a multidimensional space using an algorithm called SMACOF with the IDIOSCAL method. Below, Figure 2 shows one view of the MSS model.
Click here for an interactive version of Figure 2, which lets you rotate it, zoom in, and highlight specific data points for more information.
Figure 2: A plot showing one view of the Multidimensional Similarity Space (MSS) model.
Closest Timbral Neighbors (CTN)
This model reveals a network of timbral similarity, connecting between the recorded samples based on the mean similarity rating between each possible sample pair. The graph can be treated as a roadmap for potential musical composition or improvisation, as timbral groupings connected by lines might be used concurrently or consecutively during a musical performance to create a sense of cohesion. For instance, a composer might write a section using unpitched sustained sounds and overtone variations on both instruments, indicated as timbral neighbors in the graph by the lines connecting the light-yellow and light-green markers at the top-center of the plot shown in Figure 3. All sample pairs whose mean similarity rating was below a set threshold were connected. Although this graph was based on the means of raw similarity ratings of participants in the perceptual experiment, and not on the distances computed in the MSS model from before, the samples are plotted on dimensions 1 and 2 from the MSS for ease of reference between the two.
Figure 3: A network graph showing connections between Closest Timbral Neighbors (CTN)
Greatest Timbral Diversity (GTD)
This model emphasizes timbral differences between recorded samples, where connected nodes represent pairs with high mean dissimilarity ratings. Sound pairs that scored a mean dissimilarity rating above a certain threshold are connected. The sounds at the ends of these connections, implied to have large timbral differences, can be used to differentiate two musical segments by having a large shift in sonic character. They can also be used to set apart the two instruments when they play sounds with diverging qualities concurrently. Alternatively, traversing this graph by playing connected sounds one after the other can produce long sequences of maximally varying sounds to produce a kaleidoscopic effect, or to make one instrument sound like multiple sound sources with very different timbres. For example, in Figure 4, we can see that sustained pitch on the cello (lightest-blue on the far left of the plot) is shown to be very different from the unpitched granulated sounds on the trombone and the cello (dark yellow on the bottom right quadrant of the plot). This graph can be used to structure portions of a musical composition or improvisation session, allowing diverse timbres which could expand the texture or style of the music. Similarly to the CTN graph, this graph is based on similarity rating means, but is displayed over the first and second dimension of the MSS plot to make cross-reference easier.
Figure 4: A network graph showing connections for Greatest Timbral Diversity
Creative Explorations and Follow-up Recordings
In response to the first recording session, where we focused on individual instrumental techniques recorded separately, we had a second recording session where we focused on concurrent blends and explorations of combined timbres. In this session the instrumentalists utilized the flexibility afforded when playing together, adjusting to the other instrument’s sounds in real time. For instance, when recording muted pitches on the trombone, it wouldn’t be feasible to record every possible shade of that family of sounds; however, a professional trombone player could rely on their deep knowledge of their instrument to adapt their timbre to match a cello’s particular sound in the moment, something that would be difficult to do based on an incomplete dataset of individual samples. Thus, in the second recording session we were developing the sounds we collected in the first recording session by relying on the musicians’ intuitive understanding of the timbre spaces we were exploring.
Adding More Sounds
As in the first recording session, we met as a group to plan the second recording session. We created a spreadsheet outlining the composer's timbral interests for building compound sounds. The spreadsheet allowed composers and performers to streamline the process of proposing and categorizing timbres that might otherwise be considered unrelated and to highlight unexpected techniques that could create interesting timbral fusions. We grouped sounds by playing technique while also considering the interests of the composers and performers regarding experimental sustained, gradual, distorted, and pulsated timbral textures. This outlined specific hypotheses about timbral blends and orchestration effects. Composer suggestions were cross-checked by performers, who made decisions about which instrumental techniques were feasible and might blend best.
Example 1
For instance, after reviewing recordings from the first session, Goldford speculated that a concurrent blend might be accomplished by asking the cello to play sul ponticello in octaves alongside a whistle tone. This was tested on an F#6 and Bb6 with the trombone utilizing a Harmon mute:
Recording Session No. 2 — Sustained overtone variation
Comparing this blend to the first recording session’s samples of the same phenomenological category—using harmonic overtone gestures on each instrument—reveals some similarities and differences:
Recording Session No. 1 — Sustained overtone variation
Even so, the initial data collected about the similarity of the sounds from the first recording session show that these extended timbral overtone techniques lie relatively close within a similarity space, making them potentially useful for concurrent groupings in contemporary orchestration. We can label this timbral relationship as a Closest Timbral Neighbor (CTN). We conjecture that the whistle tone sounds made in the second recording session would fit in the same timbre space neighborhood of the airy, flute-like sounds from the first recording session. This area is highlighted in the plot in Figure 5.
Figure 5: The CTN network graph, with the highlighted area where new airy, noisy, flute-like sounds are conjectured to belong.
Example 2
In our study, the phenomenological category of distorted pitch was produced on cello by playing with an excessive amount of pressure between the bow and the string. By “crushing” his bow into the strings, Ko produced an especially grainy and distorted sound in the first recording session. Also during the first recording session, pure trombone split tones were recorded. However, in the second recording session it was determined that these techniques did not contribute to the concurrent blend the musicians were seeking. The similarity data from the first session as seen in the CTN graph shows these two distorted pitched sounds did not score as very similar. You can listen for yourself below and see where these sounds are placed in the highlighted region in Figure 6.
Recording Session No. 1 — Sustained distorted pitch
Both composers were curious about how split tones or tones that are “quasi-pitched, spit-filled, or pursed-lips buzzing” (as described by Goldford) on the trombone might complement the cello’s distinct overpressure timbre. Instead of the techniques used in the first session, more grotesque trombone timbres were created using an overfocused embouchure and buzz mute to achieve a deeper concurrent blend with the cello. Ko and Schneider's intuitive expansion of the composers' similarity speculations in the second recording session produced these adapted experimental approaches to create what they judged as a significantly increased concurrent blend within the phenomenological category of distorted pitch:
Recording Session No. 2 — Sustained distorted pitch
The plot in Figure 6 highlights where these new sounds might fit in the timbre space. Of note is that currently, the sounds in the distorted pitch category do not meet the threshold to be marked as timbral neighbors, but the sounds from the second recording session do make a compelling case that this region in the timbre space can be populated with more sounds that could bridge the existing gap.
Figure 6: The CTN network graph, with the highlighted area where new distorted pitched sounds are conjectured to belong.
Example 3
When considering shorter percussive gestures, a cello bow bounce (jeté, spiccato, gettato, etc.) on various strings, combined with airy, fragmented flutter tongue rams and slaps on the trombone, provided a timbre space neighborhood for punctuated unpitched sounds. When exploring this timbral region in the second session, Schneider used an aggressive and forward flutter tongue, which faded into a ram, eventually transitioning to slaps while oscillating his embouchure loosely to match Ko’s rapid bow bounce gestures:
Recording Session No. 2 — Punctuated unpitched
Again, this experimentation expanded upon the timbral pallet of the first recording session and data set which already showed a high degree of similarity between the punctuated unpitched gestures.
Recording Session No. 1 — Punctuated unpitched
Example 4
A final example of concurrent blending between extended techniques on the instruments can be found in the combination of muted pitched sounds. A cello sul tasto with mute alongside pixie-muted trombone with plunger are heard in the first sample below, where the duo is working its way up chromatically from A2, ending on F3. In the second sample, a Harmon mute is used on trombone alongside a cello’s bowed endpin producing an E4. We judged the degrees of timbral similarity were roughly the same in both recording sessions; the samples from the first session are provided below as well.
Recording Session No. 2 — Sustained muted pitch
Recording Session No. 1 — Sustained muted pitch
One interesting thing to note about the sustained muted trombone pitch in Audio Sample 13 (above) is what happens when the trombone changes its mute configuration. As the trombone opens and closes the plunger mute, it becomes clear that there is a cross-fade occurring between certain instrumental timbres. As the trombone closes the plunger, the timbre of the cello becomes more prominent, resulting in a blend that augments the cello sound. When the mute is half open, both timbres blend more equally, creating a new emergent sound. We can represent this possibility on the CTN graph in Figure 7: we suggest the trombone, with a variable mute, can transition smoothly between sounds that are closer in timbre to different muted and unmuted sustained pitched sounds in the highlighted area.
Figure 7: The CTN network graph, with the highlighted area where new pitched sounds with variable mutes applied are conjectured to belong.
Duo Improv
During the second recording session, Ko and Schneider used timbres from the extended spreadsheet as a guide for open improvisation. The duo incorporated elements from the spreadsheet document and comments by the composers to build a loose structure of gestures, including concurrent blends, interpolations via tension and distortion, and crossfades in a research-creation approach. They also introduced new ideas and sounds that expanded beyond the initial collection.
One such example features both instruments pitch bending, adding expressive vibrato and glissandi while crossfading between their natural sustained timbres.:
Recording Session No. 2 — Sustained pitch
Another example combined sul pont with a sung multiphonic in a sustained granular pitch:
The full Improv session can be heard below:
Technique Performance Flow (TPF)
The duo was also able to experiment with what they termed Technique Performance Flow (TPF). For instance, Ko could transition with ease between sul pont, cello bounce, and other more common playing techniques. Sometimes, planning helps certain techniques flow better, such as the behind-the-bridge crush, which can be played either prepared with a clothespin or on its own; the latter allows the performer to fluidly alternate between a 'crushed' or 'natural' cello sound on the same string. On the other hand, adjusting the endpin proved to be the most challenging aspect when considering the flow of the extended techniques used during the project since the cellist needs to bend down to set the pin and tune it for performance if necessary. Likewise, the trombone had similar tendencies when it came to the performance flow of each extended technique that an improviser or composer must consider when considering the data models. For instance, If Schneider decided to use a plunger mute with pixie mute, the F attachment valve on the instrument would not be accessible without preparation. Transitioning smoothly between split tones and more percussive accents, such as using slap tongue or hitting the mouthpiece with the hand, required additional time for adjustments.
Adler Sketch
Another example of how the recordings and data could influence the final creative output of this project can be seen in the sketch that Adler wrote for the duo to record in the second session. The recording is provided below. This short sketch focuses on the possibility of interpolating between various points in the timbre space by incorporating knowledge of the physical performance requirements to make fluid transitions between the sounds, like the TPF discussed above. In places where continuous sound production was impossible, musical continuity could be maintained by using similar timbres in one instrument while the other instrument pauses to switch mechanically to produce the next sound, building on the perceived timbral similarities we found. The etude also goes beyond focusing on individual timbres and blends to explore shifting textures in its orchestration, such as the added musical voices and layers towards the end, combining to finish on a small, dense sound mass grouped by its similar timbres.
Future Steps: Creative Output
The overall directive of the project is to use the collected data to compose full-scale works for the instrumental duo. We hope these works will use as much of the collected data as inspiration, including the preliminary dataset samples (recording session no. 1), perceptual models, and the combined sonorities, improvisations, and sketches (recording session no. 2). We also want to further develop the tools we offer the musicians. First, we plan to interpret the perceptual models to create timbre spaces using computer audio analysis that matches the experimental data. This could then be used to analyze the rest of the recorded samples. This approach could make the recorded sounds usable in a compositional process with FluCoMa or other similar systems. Additionally, we want to formulate a network graph that will show the possible Technique Performance Flow (TPF) between the different sound production techniques based on the performers’ review of the physical requirements of producing these sounds. For instance, can specific extended techniques be played in sequence easily? Can a performer interpolate between them smoothly? This element of performance flow is critical for composition and improvisation, as it dictates the feasibility of musical phrases that include these extended techniques. Finally, we could continuously develop the dataset by carrying out follow-up perceptual experiments. These would add more sounds to be tested perceptually, such as some of the combined blends. These follow-up experiments could be used to expand the similarity space we have and consider each musician's timbral speculations and praxis in a feedback loop between in-lab experimentation and creative explorations.