Page  360 ï~~Implementation of the KL Synthesis Algorithm under Real-Time Control* W. Brent Weeks, W. Andrew Schloss, R. Lynn Kirlin University of Victoria, Canada ABSTRACT KL Synthesis is a novel approach to additive synthesis, using different and fewer oscillators than in traditional Fourier-based synthesis. The algorithm is generally restricted to the synthesis of harmonic tones. Its principal advantages are: 1) reduced computation in real-time synthesis due to the small number of oscillators required (typically 3 or 4), and 2) a reduced number of synthesis parameters allowing essential control over the synthesis by a gestural input.device. We are currently implementing and testing a DSP-based realtime synthesizer, as well as a modified non-real-time analyzer. 1. Introduction KL (Karhunen-Loeve) Synthesis is a novel form of additive synthesis. The algorithm, developed by Stapleton and Bass [1,2], consists of both analysis and synthesis procedures. Traditionally, additive synthesis is based on Fourier analysis. A tone is modelled as the sum of several time-varying sinusoidal oscillators. Fourier-based additive synthesis has been extensively used for the synthesis of both harmonic and non-harmonic tones [3]. Its principal advantages are the existence of straightforward and proven analysis techniques, as well as conceptual simplicity. Its principal disadvantage is the number of oscillators required for a realistic synthesis (typically from 16-30 [1,41). For instance, a typical development platform for implementing synthesis techniques is the popular Motorola DSP56000. In [5], it is estimated that 8 interpolating table lookup oscillators can be implemented at a 50KHz sample rate on a DSP56000. One DSP56000 would not suffice for quality real-time Fourier-based synthesis. The KL Algorithm replaces the sine wave oscillators of traditional additive synthesis with a set of non-sinusoidal, orthonormal basis function oscillators. The KL additive synthesis equation can be written as: N s(t) = A.(t)#.(t(t)), (1) -=1 where: A. (t) are the oscillator amplitude envelopes,/3, are the KL basis function oscillators, and fi(t) is the phase function for all of the oscillators. Typically, only 3 or 4 KL oscillators are required for quality synthesis. A unique set of KL oscillators is derived for an individual instrument or a group of instruments which have highly correlated waveforms. The common phase function is a requirement of harmonic KL synthesis. 2. Analysis Algorithm The analysis algorithm (Figure 1) has been implemented in a Matlab menu-driven program for SUN Sparcstations. Extract Sample Generate KL Oscillators Extract Amplitude Functions from for Ensemble of Envelopes and Phase Instrument Tone(s) Sample Functions Functions for Ensemble Figure 1 - KL Analysis Algorithm - High Level *This work was supported in part by the Natural Sc:iences and Engineering Research Council of Canada. ICMC 360

Page  361 ï~~Extract Sample Functions from an Instrument Tone First, the instrument tone is segmented into attack, sustain and decay regions. This segmentation is specified by the user. An amplitude envelope of the entire tone is presented graphically [6], and the user picks the begin and end points of the sustain region. Second, an automated verification is made of the period (in samples) of the sampled tone. (Data files for the sampled tones include a priori information about the tone pitch.) Next, single period segments of the tone are extracted from the sustain region. It is also possible to extract single period segments from the attack and decay regions. These single period segments are then resampled to a normalized period of 128 or 256. This normalized period represents the number of points that will be specified in the final KL oscillator lookup tables. Finally, the single period segments are normalized to unit energy and circularly low-pass filtered (to make the segments periodically extensible), forming the sample functions. Generate KL Oscillators for Ensemble of Sample Functions Sample functions from several individual instrument tones are grouped in an ensemble by the user. One type of ensemble consists of sample functions for one instrument sampled at several pitches and dynamic levels. An alternative ensemble grouping is to include sample functions from two or more instruments (eg. Trumpet and Cello). In [1], groupings of 5 or 6 instruments were examined; however, the primary focus of that work was to group as many instruments as possible into one ensemble (class) for the purpose of memory efficiency. We intend to restrict ourselves at this point in the research to groupings of one or two instruments only. This will allow us to explore tracking timbre variation with pitch and volume for single instruments, an important issue in realistic synthesis [7]. As well, we will explore cross-timbres between two instruments, using the same KL oscillators, but interpolating the envelopes. It is expected that this will produce cross-timbres different than those achieved when just performing a cross-fade between two instruments. Once the ensemble is assembled, all of the sample functions in the ensemble are phasealigned. This is done to reduce the number of KL oscillators that will be required for quality synthesis. The KL Transform is then applied to the sample functions as follows. Form a data matrix X: X = [XIIX2I...IXM], (2) where: x1..XM are column vectors containing the M sample functions. Then either form the correlation matrix Rx = XXTand compute the eigenspace for Rx, or compute the Singular Value Decomposition (SVD) of X directly [8]. It is more efficient and numerically stable in this case to simply compute the SVD. The SVD decomposes the data matrix as X = USVT. The matrix U contains as its first M column vectors, u1..-UM, the KL basis functions, which are the same length as the sample functions Xl-.."XM. The first N column vectors are chosen as the lookup tables for the KL oscillators: P,, - u,, n = 1l...N. (3) Extract Amplitude Envelopes and Phase Function for an Instrument Tone The final stage in the analysis is to take the KL oscillators derived from an entire ensemble of tones, and compute the amplitude envelopes and phase function for an individual tone. This is accomplished by stepping through the tone at regular intervals (eg. every 2 periods), determining the phase offset between the KL basis function oscillators and the tone at each interval, and finally computing the amplitude weight for each oscillator at each interval. JCMC 361

Page  362 ï~~Letting A~ (k) denote the nth amplitude weight at interval k in the tone, we have an(k) -=Y Kw(k),i (4) where: /3,, is the nth oscillator lookup table, re-sampled to the tone period and phase aligned with w(k), and w(k) is a single-period window of the tone at interval k. After computing these high resolution amplitude envelopes and the phase function, the amplitude envelopes and phase function can be approximated by piecewise linear segments. As well, the phase function can be expressed as a frequency envelope. Modifications to the Analysis A modification to the analysis above which is currently being explored is to compute a separate set of KL oscillators for each of the attack, sustain and decay regions of a tone. This should facilitate a better synthesis of the tone. A smooth transition from one region to the next as the KL oscillators are changed is accomplished by ensuring the KL oscillator functions from region to region are phase-aligned. 3. Synthesis Implementation The synthesis algorithm is simply additive synthesis. Instead of sine-wave oscillators, KL oscillators are used. These oscillators may have to be updated over the course of a tone, if the modification mentioned above is used. The overall synthesis implementation, with a real-time performance interface implemented in Max on a Macintosh II, is shown in Figure 2. Midi Input Device Radio Dru Serial Interface MidijInterface Radio Drum Midi Instrument Receiver Receiver Radio Drum Midi Instrument Synthesis Parameters Select/Merge Synthesis Parameters Parameters High Level Synthesis Parameters KL Synthesis Driver DSP Synthesis Control Parameters DSP KL E Synthesizer Figure 2 - KL Synthesis with Real-Time Performance Interface The Radio Drum Receiver, Midi Instrument Receiver, Parameter Selector/Merger, and KL Synthesis Driver are all implemented in Max. The DSP KL Synthesizer module is implemented in machine code on a Digidesign DSP56000 card (Audiomedia or Sound Accelerator). ICMC 362

Page  363 ï~~The High Level Synthesis Parameters are: " Note On: Initialize Pitch, Volume, (Timbre Ratio) " Note Off " Pitch Update, Volume Update, (Timbre Ratio Update) Vibrato and tremolo initializations and updates may be added to the parameters in the future. The timbre ratio is an optional parameter, which will only be used when exploring crosstimbres between instruments. It will specify the relative weighting of two instrument tones in forming a cross-timbre. Pitch updates may be treated as either smooth glissando transitions, or as quantized legato transitions to another note. The high level synthesis parameters can be generated by a Midi input device and/or the radio drum. Mapping Midi parameters to the high level synthesis parameters is straightforward. Mappings of the radio drum x,y,z parameters will follow those ideas presented in [9]. For instance, the three dimensions of one stick location could be mapped to pitch, volume and timbre ratio. The other stick could be used to define the start and end of notes by striking the drum surface. The DSP Synthesis Control Parameters are: " Load Patch " Start Note: Initial Amplitude Envelopes, Initial Oscillator Phase, Initial Phase Function " Update Amplitude Envelopes and Phase Function " End Note (for non-legato playing) The KL Synthesis Driver periodically updates the DSP synthesis control parameters, in response to both the natural evolution of a tone as well as user input via a Midi device or the radio drum. 4. Results At the time of writing, the analysis algorithm is being tested with a non-real-time synthesis procedure. The analysis/re-synthesis results in tones similar to the originals, with a slight degradation in quality. Cello and trumpet tones are the focus of current testing. The analysis procedure is being debugged and revised to improve the synthesis quality. The implementation of the real-time synthesizer described in the previous section is under way. Results of this work will be given during the paper session. 5. References 1. Stapleton, J.C. and Bass, S.C. Synthesis of Musical Tones Based on the Karhunen-Love Transform. IEEE Transactions on Acoustics, Speech, and Signal Processing 36, 3 (March 1988), 305-319. 2. Stapleton, J.C. Karhunen-Loeve Based Additive Synthesis of Musical Tones, Ph.D. dissertation, Purdue University, December 1985. 3. Smith, J.O. and Serra, X. PARSHL: An Analysis/Synthesis Program for Non-Harmonic Sounds Based on a Sinusoidal Representation.. In ICMC Proceedings, Computer Music Association, 1987, 290-297. 4. Gordon, J.W. System Architectures for Computer Music. Computing Surveys 17,2 (June 1985), 191 -233. 5. Strawn, J. Implementing Table Lookup Oscillators for Music with the Motorola DSP56000 Family. Preprint for the 85th Convention of the Audio Engineering Society, Audio Engineering Society, 1988. 6. Schloss, W.A. On the Automatic Transcription of Percussive Music - From Acoustic Signal to HighLevel Analysis, Ph.D. dissertation, Report No. STAN-M-27, Stanford University, May 1985, 58-60. 7. Moore, F.R. Elements of Computer Music, Prentice Hall (1990), 17-23. 8. Watkins, D.S. Fundamentals of Matrix Computations, John Wiley & Sons (1991). 9. Schloss, W.A. Recent Advances in the Coupling of the Language Max with the MathewsfBoie Radio Drum. In JCMC Proceedings, Computer Music Association, 1990. ICMC 363