This edition first published 2018
© 2018 John Wiley & Sons Ltd

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by law. Advice on how to obtain permission to reuse material from this title is available at http://www.wiley.com/go/permissions.

The right of Ville Pulkki, Symeon Delikaris-Manias and Archontis Politis to be identified as the authors of the editorial material in this work has been asserted in accordance with law.

Registered Offices
John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, USA
John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, UK

Editorial Office
The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, UK

For details of our global editorial offices, customer services, and more information about Wiley products visit us at www.wiley.com.

Wiley also publishes its books in a variety of electronic formats and by print-on-demand. Some content that appears in standard print versions of this book may not be available in other formats.

Limit of Liability/Disclaimer of Warranty
While the publisher and authors have used their best efforts in preparing this work, they make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives, written sales materials or promotional statements for this work. The fact that an organization, website, or product is referred to in this work as a citation and/or potential source of further information does not mean that the publisher and authors endorse the information or services the organization, website, or product may provide or recommendations it may make. This work is sold with the understanding that the publisher is not engaged in rendering professional services. The advice and strategies contained herein may not be suitable for your situation. You should consult with a specialist where appropriate. Further, readers should be aware that websites listed in this work may have changed or disappeared between when this work was written and when it is read. Neither the publisher nor authors shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.

Library of Congress Cataloging-in-Publication Data

Names: Pulkki, Ville, editor. | Delikaris-Manias, Symeon, editor. | Politis, Archontis, editor.
Title: Parametric time-frequency domain spatial audio / edited by Ville Pulkki, Symeon Delikaris-Manias, Archontis Politis, Aalto University, Aalto, Finland.
Description: First edition. | Hoboken, NJ, USA : Wiley, 2018. | Includes bibliographical references and index. |
Identifiers: LCCN 2017020532 (print) | LCCN 2017032223 (ebook) | ISBN 9781119252580 (pdf) | ISBN 9781119252610 (epub) | ISBN 9781119252597 (hardback)
Subjects: LCSH: Surround-sound systems--Mathematical models. | Time-domain analysis. | Signal processing. | BISAC: TECHNOLOGY & ENGINEERING / Electronics / General.
Classification: LCC TK7881.83 (ebook) | LCC TK7881.83 .P37 2018 (print) | DDC 621.382/2--dc23
LC record available at https://lccn.loc.gov/2017020532

Cover Design: Wiley
Cover Image: © Vectorig/Gettyimages

CONTENTS

List of Contributors

Preface

Notes

About the Companion Website

Part I Analysis and Synthesis of Spatial Sound

1 Time–Frequency Processing: Methods and Tools
1. 1.1 Introduction
2. 1.2 Time–Frequency Processing
3. 1.3 Processing of Spatial Audio
4. Note
5. References
2 Spatial Decomposition by Spherical Array Processing
1. 2.1 Introduction
2. 2.2 Sound Field Measurement by a Spherical Array
3. 2.3 Array Processing and Plane-Wave Decomposition
4. 2.4 Sensitivity to Noise and Standard Regularization Methods
5. 2.5 Optimal Noise-Robust Design
6. 2.6 Spatial Aliasing and High Frequency Performance Limit
7. 2.7 High Frequency Bandwidth Extension by Aliasing Cancellation
8. 2.8 High Performance Broadband PWD Example
9. 2.9 Summary
10. 2.10 Acknowledgment
11. References
3 Sound Field Analysis Using Sparse Recovery
1. 3.1 Introduction
2. 3.2 The Plane-Wave Decomposition Problem
3. 3.3 Bayesian Approach to Plane-Wave Decomposition
4. 3.4 Calculating the IRLS Noise-Power Regularization Parameter
5. 3.5 Numerical Simulations
6. 3.6 Experiment: Echoic Sound Scene Analysis
7. 3.7 Conclusions
8. Appendix
9. References

Part II Reproduction of Spatial Sound

4 Overview of Time–Frequency Domain Parametric Spatial Audio Techniques
1. 4.1 Introduction
2. 4.2 Parametric Processing Overview
3. References
5 First-Order Directional Audio Coding (DirAC)
1. 5.1 Representing Spatial Sound with First-Order B-Format Signals
2. 5.2 Some Notes on the Evolution of the Technique
3. 5.3 DirAC with Ideal B-Format Signals
4. 5.4 Analysis of Directional Parameters with Real Microphone Setups
5. 5.5 First-Order DirAC with Monophonic Audio Transmission
6. 5.6 First-Order DirAC with Multichannel Audio Transmission
7. 5.7 DirAC Synthesis for Headphones and for Hearing Aids
8. 5.8 Optimizing the Time–Frequency Resolution of DirAC for Critical Signals
9. 5.9 Example Implementation
10. 5.10 Summary
11. References
6 Higher-Order Directional Audio Coding
1. 6.1 Introduction
2. 6.2 Sound Field Model
3. 6.3 Energetic Analysis and Estimation of Parameters
4. 6.4 Synthesis of Target Setup Signals
5. 6.5 Subjective Evaluation
6. 6.6 Conclusions
7. Note
8. References
7 Multi-Channel Sound Acquisition Using a Multi-Wave Sound Field Model
1. 7.1 Introduction
2. 7.2 Parametric Sound Acquisition and Processing
3. 7.3 Multi-Wave Sound Field and Signal Model
4. 7.4 Direct and Diffuse Signal Estimation
5. 7.5 Parameter Estimation
6. 7.6 Application to Spatial Sound Reproduction
7. 7.7 Summary
8. Notes
9. References
8 Adaptive Mixing of Excessively Directive and Robust Beamformers for Reproduction of Spatial Sound
1. 8.1 Introduction
2. 8.2 Notation and Signal Model
3. 8.3 Overview of the Method
4. 8.4 Loudspeaker-Based Spatial Sound Reproduction
5. 8.5 Binaural-Based Spatial Sound Reproduction
6. 8.6 Conclusions
7. References
9 Source Separation and Reconstruction of Spatial Audio Using Spectrogram Factorization
1. 9.1 Introduction
2. 9.2 Spectrogram Factorization
3. 9.3 Array Signal Processing and Spectrogram Factorization
4. 9.4 Applications of Spectrogram Factorization in Spatial Audio
5. 9.5 Discussion
6. 9.6 Matlab Example
7. Note
8. References

Part III Signal-Dependent Spatial Filtering

10 Time–Frequency Domain Spatial Audio Enhancement
1. 10.1 Introduction
2. 10.2 Signal-Independent Enhancement
3. 10.3 Signal-Dependent Enhancement
4. References
11 Cross-Spectrum-Based Post-Filter Utilizing Noisy and Robust Beamformers
1. 11.1 Introduction
2. 11.2 Notation and Signal Model
3. 11.3 Estimation of the Cross-Spectrum-Based Post-Filter
4. 11.4 Implementation Examples
5. 11.5 Conclusions and Further Remarks
6. 11.6 Source Code
7. Note
8. References
12 Microphone-Array-Based Speech Enhancement Using Neural Networks
1. 12.1 Introduction
2. 12.2 Time–Frequency Masks for Speech Enhancement Using Supervised Learning
3. 12.3 Artificial Neural Networks
4. 12.4 Mask Learning: A Simulated Example
5. 12.5 Mask Learning: A Real-World Example
6. 12.6 Conclusions
7. 12.7 Source Code
8. Notes
9. References

Part IV Applications

13 Upmixing and Beamforming in Professional Audio
1. 13.1 Introduction
2. 13.2 Stereo-to-Multichannel Upmix Processor
3. 13.3 Digitally Enhanced Shotgun Microphone
4. 13.4 Surround Microphone System Based on Two Microphone Elements
5. 13.5 Summary
6. References
14 Spatial Sound Scene Synthesis and Manipulation for Virtual Reality and Audio Effects
1. 14.1 Introduction
2. 14.2 Parametric Sound Scene Synthesis for Virtual Reality
3. 14.3 Spatial Manipulation of Sound Scenes
4. 14.4 Summary
5. References
15 Parametric Spatial Audio Techniques in Teleconferencing and Remote Presence
1. 15.1 Introduction and Motivation
2. 15.2 Background
3. 15.3 Immersive Audio Communication System (ImmACS)
4. 15.4 Capture and Reproduction of Crowded Acoustic Environments
5. 15.5 Conclusions
6. Notes
7. References

Index

EULA

List of Illustrations

Chapter 1

Figure 1.1 Block diagram of a typical parametric time–frequency processing algorithm. The processing operates on three sampling rates: that of the wide-band signal, that of the frequency band signal, and that of the parametric information.
Figure 1.2 General block diagram of a uniform filter bank.
Figure 1.3 Linear-phase and low-delay prototype filters, using K = 64. Filter banks using a low-delay prototype filter are useful in applications where the constraints on delay are so stringent that the performance in terms of frequency response must be compromised.
Figure 1.4 Effect of real and complex modulation in uniform filter banks. (a) A set of band-pass filters, (b) individual band-pass filters, (c) their downsampled spectra, (d) the re-upsampled spectra, and (e) the band-pass synthesis-filtered spectra.
Figure 1.5 Linear-phase prototype filter applied by the MPEG QMF bank, and the square-root Hann window typical in STFT, using K = 64.
Figure 1.6 Block diagram of a forward transform of a robust filter bank employing the FFT.
Figure 1.7 Block diagram of an inverse transform of a robust filter bank employing the FFT.
Figure 1.8 (a) Analysis windowing of subsequent windows; (b) overlap-add at the synthesis stage.
Figure 1.9 Output frequency plot of the Matlab script in Listing 1.1.
Figure 1.10 Cascaded filters applied in MPEG systems to obtain a higher frequency selectivity at the low frequencies. Adapted from Breebaart et al. (2005).

Chapter 2

Figure 2.1 Ideal fourth-order spatial density function, a₄(k, Ω), presented for (a) different directions Ω = (θ, ϕ) at a selected frequency f = 4.7 kHz, and for (b) Ω = (90°, ϕ) over all azimuth angles ϕ and over different frequencies.
Figure 2.2 Magnitude of the radial function b_n(k) with different orders, for a rigid-sphere array.
Figure 2.3 Comparison of the regularization functions c_n(k) using conventional PWD, R-PWD, and DaS PWD with (a) a given SNR (SNR = 40 dB) and for zeroth- to fourth-order regularization functions, and (b) different SNR values for the third-order R-PWD regularization function .
Figure 2.4 Magnitude of the fourth-order spatial density function, |a₄(k, Ω)|, at Ω = (90°, ϕ) for all azimuth angles ϕ over different frequencies: (a) the true spatial density function a₄(k, Ω), and the estimated spatial density function , using (b) conventional PWD, (c) DaS PWD, (d) R-PWD.
Figure 2.5 Magnitude of the fourth-order spatial density function, |a₄(k, Ω)|, at Ω = (90°, ϕ) for all azimuth angles ϕ over different frequencies with (a) the true spatial density function a₄(k, Ω), and the estimated spatial density function , with high sound field order , using (b) conventional PWD, (c) AC-PWD.
Figure 2.6 Magnitude of the fourth-order spatial density function, |a₄(k, Ω)|, at Ω = (90°, ϕ) for all azimuth angles ϕ over different frequencies with (a) the true spatial density function a₄(k, Ω), and the estimated spatial density function , using (b) conventional PWD, (c) optimal BB-PWD.
Figure 2.7 The overall PWD error using the conventional PWD and R-PWD methods: (a) the error for a single plane wave, (b) the mean overall PWD error, , for a diffuse sound field.

Chapter 3

Figure 3.1 Comparison of the accuracy of two different SR-based sound field analysis methods, as a function of the number of sound sources. The signal to noise ratio of the SH signals is 0 dB. The markers and bars represent the median and interquartile range of the energy map mismatch, respectively.
Figure 3.2 Average energy map mismatch resulting from the analysis of SH signals using regularized IRLS, as a function of the number of sound sources and SNR, for SH orders 1–4.
Figure 3.3 Bird’s-eye view diagram of the experiment described in Section 3.6.
Figure 3.4 Comparison of the sound field energy maps obtained by analyzing 4 s of recorded SH signals with: (a) spherical beamforming, (b) MUSIC, and (c) the sparse recovery method presented in this paper. The light, solid lines (blue in the online version) indicate the wall edges and the circles indicate the loudspeaker positions.
Figure 3.5 Comparison of the sound field energy maps obtained by analyzing 1024 samples of recorded SH signals with: (a) spherical beamforming, (b) MUSIC, and (c) the sparse recovery method presented in this paper. The light, solid lines (blue in the online version) indicate the wall edges and the circles indicate the loudspeaker positions.

Chapter 4

Figure 4.1 Basic processing blocks of parametric and non-parametric approaches to spatial sound reproduction. Adapted from Politis 2016.
Figure 4.2 Typical processing blocks of spatial audio coding and up-mixing systems. Source: Politis 2016. Reproduced with permission of Archontis Politis.
Figure 4.3 Basic processing examples for reproduction based on spatial filtering and source separation. Source: Politis 2016. Reproduced with permission of Archontis Politis.

Chapter 5

Figure 5.1 Directional patterns of the four coincident microphones of a B-format microphone. The lighter patterns indicate phase-inversed response. b_w has not been scaled by for this illustration.
Figure 5.2 Temporally subsequent intensity vectors analyzed on one frequency channel from a real 2D microphone setup in highly diffuse [left] and non-diffuse [right] sound fields.
Figure 5.3 Flow diagram of DirAC with single-channel transmission.
Figure 5.4 Square array of four omnidirectional microphones to derive horizontal B-format signals. Opposing microphones are spaced at a distance d apart, with typical values of d between 1 cm and 10 cm.
Figure 5.5 (a) RMS error in estimation of direction using square arrays of four omnidirectional microphones, when the distances between opposing microphones are 1 cm (solid line), 2 cm (dashed line), and 4 cm (dash-dotted line). (b) The effect of diffuse sound on directional analysis with a square array with 4 cm inter-microphone distances. Simulations were conducted with different energy ratios Γ between a plane wave and the diffuse sound field. In (a) and (b) the results for root mean square errors are averaged over various analysis directions. Source: Ahonen 2012a. Reproduced with permission of the Audio Engineering Society.
Figure 5.6 Cylinder microphone array. A rigid cylinder of diameter d casting an acoustic shadow and causing scattering is positioned between four omnidirectional microphones.
Figure 5.7 RMS error in DOA estimation with the pressure–energy gradient method. (a) Results for cylinder arrays with diameters 4 cm, 8 cm, and 14 cm. A white noise sample is reproduced alternatively at 0°, 15°, 30°, and 45° in an anechoic chamber. (b) The effect of diffuse sound on PEG directional analysis with a cylinder array of diameter 8 cm with kHz. The direction estimation is performed with different energy ratios Γ between a plane wave and the diffuse sound. The plane wave arrives at intervals of 5° from 0° to 45°. In (a) and (b), RMSEs are averaged over various directions. Source: Ahonen 2012a. Reproduced with permission of the Audio Engineering Society.
Figure 5.8 A one-dimensional microphone array is used to derive the omnidirectional signal b_w and dipole signal b_x whose directional patterns are shown with dashed lines. A plane wave arrives with lateral angle α. Adapted from Ahonen (2010).
Figure 5.9 ψ as function of the direct-to-diffuse ratio Γ on abscissa, with different direct-to-microphone-noise ratio ϒ values as denoted in the legend.
Figure 5.10 Diffuseness estimated with a two-microphone array as a function of the energy ratio Γ between a plane wave from a given azimuth angle and the diffuse field. (a) Results with energetic analysis, (Equation 5.7). (b) Results with coefficient of variation analysis. Source: Ahonen 2009. Reproduced with permission of IEEE.
Figure 5.11 Setup for spatial teleconferencing with DirAC.
Figure 5.12 Flow graph of virtual microphone DirAC. Source: Vilkamo 2009. Reproduced with permission of the Audio Engineering Society.
Figure 5.13 Mean opinion scores for different reproduction methods in the anechoic chamber with 95% confidence intervals. Source: Vilkamo 2009. Reproduced with permission of the Audio Engineering Society.
Figure 5.14 Mean opinion scores for different reproduction methods in the listening room with 95% confidence intervals. The legend denotes the listening position. Source: Vilkamo 2009. Reproduced with permission of the Audio Engineering Society.
Figure 5.15 DirAC processing from a B-format signal using the covariance-rendering method. The main optimal mixing block uses the available signal components in the input channels to produce the output. The residual signal r is generated to compensate when there are not enough independent signal components available otherwise. Source: Vilkamo 2013. Reproduced with permission of the Audio Engineering Society.
Figure 5.16 A surround microphone array used in DirAC processing. , , and define the dimensions of the setup, θ₁ defines the orientation of the L and R microphones, and θ₂ defines the orientation of the surround microphones. θ is the direction of incidence of the incoming wave. Source: Politis 2015b. Reproduced with permission of the Audio Engineering Society.
Figure 5.17 Four-channel microphone setup in binaural hearing aid.
Figure 5.18 Block diagram of DirAC processing with multiresolution STFT processing. Source: Laitinen 2011b. Reproduced with permission of the Audio Engineering Society.
Figure 5.19 Implementation of decorrelation in the example code. The loudspeaker signals corresponding to the non-diffuse and diffuse streams are produced individually, and the diffuse stream is decorrelated after frequency to time domain transformation.

Chapter 6

Figure 6.1 Some basic sound-field scenarios, and their interpretation and loudspeaker reproduction using FO-DirAC and HO-DirAC. The rows depict: (a) a single plane wave, (b) an isotropic diffuse field, (c) a single plane wave mixed with a diffuse field, (d) two plane waves, and (e) a diffusely distributed extended source. The columns depict (a) the original field, (b) FO-DirAC, and (c) HO-DirAC. The HO-DirAC analysis here is demonstrated with two angular sectors only.
Figure 6.2 Principle of operation of HO-DirAC using second-order signals. The sound field is virtually divided into three directionally emphasized components, called sectors, and modified pressure and velocity signals are computed that would occur in the sectors. The typical energetic analysis of standard DirAC is performed on the modified pressure–velocity signals for the three sectors, and the parameters are combined and utilized during synthesis using the optimal mixing technique with covariance rendering.
Figure 6.3 Example HO-DirAC velocity patterns based on cardioid sectors. The dark and light tones depict positive and negative polarity respectively.
Figure 6.4 Directional distributions of three unit-amplitude plane waves incident from (0, 0), ( − 90, 0), and (90, 30). (a) A 30th-order representation of the amplitude distribution. (b) A 5th-order representation. (c) The reassigned distribution using sector analysis on the 5th-order signals.
Figure 6.5 Sector profiles for analysis orders N = 2 − 5, based on t-designs, illustrating the sector centers and the areas of points closer to them. Source: Politis 2015. Reproduced with permission of IEEE.
Figure 6.6 Basic block diagram of HO-DirAC processing, following the notation introduced in the text.
Figure 6.7 Binaural diffuse-field coherence example, based on the first author’s measured HRTFs. The simplified curve shows an approximation used during parametric processing.
Figure 6.8 Means and 95% confidence intervals of the listening test with idealized reproduction assuming the first or fourth order of available spherical harmonics. P denotes parametric reproduction, and L denotes linear reproduction. Source: Politis 2015. Reproduced with permission of IEEE.

Chapter 7

Figure 7.1 Block scheme of the parametric processing of spatial sound. Source: Thiergart 2015. Reproduced with permission of Oliver Thiergart.
Figure 7.2 Example showing a direct sound field (plane wave), a diffuse sound field, and the sum of both fields, for a frequency of f = 500 Hz. Source: Thiergart 2015. Reproduced with permission of Oliver Thiergart.
Figure 7.3 Example target response functions for the direct sound. The left response function can be used, for instance, in sound extraction applications. The right response functions can be used in spatial sound reproduction employing a stereo loudspeaker (LS) setup. Source: Thiergart 2014b. Reproduced with permission of IEEE.
Figure 7.4 Principal estimation of the target signal Y(k, n) using the indirect approach. Source: Thiergart 2015. Reproduced with permission of Oliver Thiergart.
Figure 7.5 Relation of the different multi-channel filters for direct sound estimation in terms of noise reduction, diffuse sound reduction, and signal distortions. Source: Thiergart 2015. Reproduced with permission of Oliver Thiergart.
Figure 7.6 Solid line: Example desired response function . Dashed line: Resulting directivity pattern of an example spatial filter that assumes L = 2 plane waves with DOAs φ₁ and φ₂. Source: Thiergart 2014b. Reproduced with permission of IEEE.
Figure 7.7 Example sigmoid functions depending on the SDNR. The parameters of the function “sigmoid 1” are: a₁ = 1, a₂ = 0.5, a₃ = −9, a₄ = −18 dB. Source: Thiergart 2015. Reproduced with permission of Oliver Thiergart.
Figure 7.8 Directivity pattern of the SOA filter (left) and approximate average LCMV filter (right) for extracting diffuse sound in the presence of a single plane wave. Uniform linear array with M = 8 omnidirectional microphones (r = 3 cm). Source: Thiergart 2015. Reproduced with permission of Oliver Thiergart.
Figure 7.9 Block scheme of the parametric processing of spatial sound.
Figure 7.10 Left: Loudspeaker setup for a 5.1 configuration (L: left, R: right, C: center, LS: left surround, RS: right surround). Right: Panning functions for a 5.1 surround sound loudspeaker setup using the VBAP panning scheme (Pulkki, 1997). Source: Thiergart 2015. Reproduced with permission of Oliver Thiergart.
Figure 7.11 Left: Example target response functions to extract direct sounds. Right: Measured input long-term spatial power density.
Figure 7.12 MUSHRA listening test results for direct sound extraction. The plot shows the average and 95% confidence intervals. Source: Thiergart 2014b. Reproduced with permission of IEEE.
Figure 7.13 Effect of the different filters for extracting diffuse sound. Plots (b)–(d) show the level difference between X₁(k, n) and the extracted diffuse sound. Source: Thiergart 2014. Reproduced with permission of Oliver Thiergart.
Figure 7.14 MUSHRA listening test results for the diffuse sound extraction. The plot shows the average and 95% confidence intervals. Source: Thiergart 2014. Reproduced with permission of Oliver Thiergart.

Chapter 8

Figure 8.1 Block diagram of the proposed parametric spatial sound reproduction system. A noise-robust beamformer frequency-band signal (W_sx) is processed to have the spatial energy distribution and the inter-channel dependencies according to the spatially more selective analysis beamformer frequency-band signal (W_ax) which are normalized with the overall sound field energy, estimated with beamformer weights w_o. The adaptive mixing and decorrelation technique is summarized in Section 8.4.3.
Figure 8.2 Arbitrary two-dimensional loudspeaker setup and its corresponding VBAP panning functions. A plane wave that arrives at the array from the illustrated angle is reproduced with two loudspeakers with gains g(1) and g(2), which are the panning gains for the corresponding directions. Such gain values form the entries of the target pattern matrix T in Equation (8.5).
Figure 8.3 Comparison of the spatial energy distribution of the tested reproduction modes for two different frequencies for a single sound source at the position of a loudspeaker and a single source between the loudspeakers. The vector length corresponds to the reproduced amplitude. The square sum of the vector lengths, that is, the total energy, is normalized to a constant in each case.
Figure 8.4 Comparison of the spatial energy distribution of the tested reproduction modes for two different frequencies for two non-coherent simultaneous sources with an equal magnitude spectrum, and a diffuse sound field. The vector length corresponds to the reproduced amplitude. The square sum of the vector lengths, that is, the total energy, is normalized to a constant in each case.
Figure 8.5 Means and 95% confidence intervals of the overall results, and the means with each reference scene separately. Source: Delikaris-Manias 2015. Reproduced with permission of IEEE.

Chapter 9

Figure 9.1 Example of magnitude spectrogram redundancy of piano notes. Repeating events are denoted by rectangles of the same shades of gray; the note in the middle occurs only once.
Figure 9.2 Illustration of NMF spectrogram model (a) and its parameters: component basis b_ik in the left row (b), and activations in the right row g_kl (c).
Figure 9.3 (a): Geometry of an array consisting of microphones at locations n and m and look direction vector k_o pointing towards a source located at s. (b): Approximately uniformly distributed points (sparsely sampled grid) on the surface of a sphere.
Figure 9.4 Surround sound and positioning of sound sources in the front channels.
Figure 9.5 Process of upmixing based on the NTF model of a multichannel spectrogram and a single channel downmix. Source: Nikunen 2012. Reproduced with permission of the Audio Engineering Society.
Figure 9.6 Illustration of the microphone array inside a sphere which is defined by the look directions k_o; a few such vectors originating from [0,0,0] are illustrated at the edge of the cut-out area.
Figure 9.7 Initialization of spatial weights based on DOA analysis (contour plot) and final estimated spatial weights z_qo (gray surface) depicted as looking from above the array. The annotated source positions are illustrated as circles.
Figure 9.8 Initialization of spatial weights based on DOA analysis (light contour) and background source initial weights (darker contour). The final estimated direction weights are illustrated as a gray surface.
Figure 9.9 Assisted listening scenario using a compact microphone array for recording.
Figure 9.10 Word recognition accuracies for each tested method.
Figure 9.11 Results of the subjective listening test.

Chapter 10

Figure 10.1 Typical scenario where multiple sound and noise sources are active in a reverberant room. The task is to enhance the direction of sound from a specific area.
Figure 10.2 Signal-independent beamforming where microphone array input signals x can be either in the time or the frequency domain, depending on the weights and .
Figure 10.3 Signal-dependent beamforming. The microphone input signal x is assumed to be in the time domain.
Figure 10.4 Beamforming with post-filtering.

Chapter 11

Figure 11.1 Beam patterns (left) with their corresponding spectrograms (right) of two beamforming signals. The background color indicates phase information: black for positive phase and white for negative phase.
Figure 11.2 Visualization of multiplication between different two-dimensional pick-up beam patterns and the half-wave rectification process (denoted as a max operator). Black indicates positive phase and white negative phase.
Figure 11.3 Block diagram of the static CroPaC method. Two directionally narrow beamformer weights and are formulated offline. The thick lines denote multiple microphone signals, the thin lines denote single-channel signals, and the dashed lines denote parametric information.
Figure 11.4 Visualization of the sum of the energies between two first-order beam patterns. The background color indicates phase information: black for positive phase and white for negative phase.
Figure 11.5 Block diagram of the adaptive CroPaC method. Narrow beam pattern weights are formulated offline and vector v is the steering vector towards the look direction. The thick lines denote multiple microphone signals, the thin lines denote single-channel signals, and the dashed lines denote parametric information. Source: Delikaris-Manias 2016. Reproduced with permission of IEEE.
Figure 11.6 Three-dimensional directivity pattern, projected on the two-dimensional plane, for the first static beamformer (top row), for the second static beamformer (middle row), and for their product (bottom row) in a diffuse field with the presence of different numbers of interferers. The white x denotes the position of the target sound source, and the gray x the positions of the interferers.
Figure 11.7 Three-dimensional directivity pattern, projected on the two-dimensional plane, for the static beamformer (top row), for the adaptive beamformer (middle row), and for their product (bottom row) in a diffuse field with the presence of different numbers of interferers. The white x denotes the position of the target sound source and the gray x the positions of the interferers.
Figure 11.8 Prototype microphone arrays built and tested with the CroPaC post-filtering scheme.
Figure 11.9 Waveforms and spectrograms for the noisy microphone output, the MVDR output, and the CroPaC output. The MVDR provides limited attenuation due to the compact size of the array. The CroPaC post-filter provides additional attenuation, especially in the low-frequency region.

Chapter 12

Figure 12.1 An illustration of building a database for supervised mask learning. The input feature vector x(t) at time t is extracted from the noisy microphone array signal spectrograms m(k, t), k = 0, …, N − 1 at time frame t. The corresponding target mask values H(k, t) are obtained using microphone array signals and a clean reference signal s(k, t).
Figure 12.2 An illustration of training an artificial neural network (ANN) to predict target mask values H(k, t) from features x(t). The weights of the ANN are adjusted based on the feedback error.
Figure 12.3 An illustration of ANN-based mask prediction for beamformed signals. The features obtained from the input signal are used to predict the mask using the trained ANN. The predicted mask is then applied to the beamformed signal as a post-filter.
Figure 12.4 Illustration of a forward pass of a feed-forward neural network (FFNN) with one hidden layer. The contents of the jth hidden unit are depicted on the right: each element of the input vector is multiplied with the corresponding weight of the hidden unit and summed together with the bias value b⁽²⁾_j. The output of this linear operation is denoted as < ?TeX$z⁽²⁾_j$? > [see Equation (12.17)], which is then transformed with an activation function to obtain the output value of the unit, [see Equation (12.18)].
Figure 12.5 Examples of: (a) overfitting with a complex model, (b) underfitting with a simple model, (c) successful fitting with a suitable model, and (d) successful fitting with a complex model with regularization.
Figure 12.6 The effect of different hyper-parameters (network width, number of hidden layers, and ℓ₂-regularization parameter (η_β, refer to text) values for the networks with the best-performing features for the simulated sawtooth data.
Figure 12.7 Example of the first three sawtooth sequences. Panel (a) displays the features and targets, and panel (b) illustrates the corresponding time-domain signals.
Figure 12.8 The six-microphone array mounted to a tablet computer’s surface used to capture signals is depicted. The top-middle microphone faces backwards, and is referred to here as the “noise-capturing microphone.” Source: Courtesy of Jon Barker, University of Sheffield. Reproduced with permission of Jon Barker.
Figure 12.9 Short part of a signal of a WSJ corpus sentence uttered by a female reader in a pedestrian area, used to illustrate magnitude spectrograms in (a), (b), and (c), and phase-based features in (d). Different target masks are illustrated in (e) and (f).
Figure 12.10 Example of a mask and the predicted mask with array spectral and spatial features (BLPN) for the signal in Figure 12.9. The mask values are truncated between −10 dB and 10 dB for illustrative purposes.
Figure 12.11 Average training MSE on different sets of data (lower is better). The test set refers to the randomly drawn 7.5% training data subset (120 sentences), whereas development and evaluation contain 1640 and 1320 recorded sentences, respectively. Refer to Table 12.2 for a description of the feature combinations. Note that the target mask values are logarithms of the ratio masks with zero mean shift and standard deviation scaling. Therefore, values are not comparable to the simulation section’s MSE values of the Wiener filters.
Figure 12.12 Time-domain signal excerpts of the raw input signal of a microphone (top), the delay-and-sum beamforming signal enhanced with TF masking and BLPN features (middle), and the close-talk microphone signal (bottom).
Figure 12.13 Average speech enhancement objective scores for different methods using the three sets of data. All sentences of the training (N = 1600), development (N = 1640), and evaluation (N = 1320) datasets are used. Refer to Table 12.2 for a description of the feature combinations for the post-filtering.

Chapter 13

Figure 13.1 Soundfield UPM-1 front and back panels. Source: Courtesy of TSL Professional Products Ltd. Reproduced with permission of Chris Exelby.
Figure 13.2 The upmix algorithm operates in a subband domain. The input stereo signal is decomposed into direct and ambient sound. Direct sound is rendered as a function of the estimated direction, and ambient sound is rendered with the goal of generating high-quality surround ambience. Source: Faller 2007b. Reproduced with permission of the Audio Engineering Society.
Figure 13.3 Schoeps SuperCMIT 2U digital shotgun microphone. Source: Courtesy of Schoeps. Reproduced with permission of Schoeps.
Figure 13.4 Two microphone elements used in the SuperCMIT. Source: Courtesy of Schoeps. Reproduced with permission of Schoeps.
Figure 13.5 Magnitude response of the subbands (thin, top), filter bank output (bold, top), and group delay (bottom) are shown. Source: Wittek 2010. Reproduced with permission of the Audio Engineering Society.
Figure 13.6 Schematic diagram of the processing that is applied to the forward- and backward-facing signals f(n) and b(n) to generate the output signals. The labels FB, IIR, and MSP denote filter bank, IIR filters, and microphone signal processing, respectively. Source: Wittek 2010. Reproduced with permission of the Audio Engineering Society.
Figure 13.7 Free-field responses of the SuperCMIT at three sound incidence angles (black solid curve at 0°, dashed curve at 90°, solid curve at 180°) for the unprocessed shotgun signal (top) and the two beamformer presets (middle and bottom). Source: Wittek 2010. Reproduced with permission of the Audio Engineering Society.
Figure 13.8 Polar patterns of the SuperCMIT beamformer (Channel 1, Preset 1 and 2) and unmodified shotgun signal (Channel 2). Source: Wittek 2010. Reproduced with permission of the Audio Engineering Society.
Figure 13.9 Diffuse-field responses of SuperCMIT Channel 2, SuperCMIT Channel 1 Preset 1, and SuperCMIT Channel 1 Preset 2. Source: Wittek 2010. Reproduced with permission of the Audio Engineering Society.
Figure 13.10 Directivity index for first-order directivities, conventional shotgun microphones, and the SuperCMIT beamformer Presets 1 and 2. Source: Wittek 2010. Reproduced with permission of the Audio Engineering Society.
Figure 13.11 Sennheiser Esfera microphone and processor. Source: Courtesy of Sennheiser. Reproduced with permission of Sennheiser.
Figure 13.12 Sennheiser Esfera processor control user interface. Source: Courtesy of Sennheiser. Reproduced with permission of Sennheiser.
Figure 13.13 (a) Directional responses of B-format W, X, and Y signals. (b) An example of the directional responses of surround audio channels generated from a linear combination of signals with responses as in (a). Source: Faller 2011. Reproduced with permission of the Audio Engineering Society.
Figure 13.14 (a) Directional responses of a coincident cardioid pair. (b) An example of the directional responses of surround audio channels generated from the signals with responses as in (a). Source: Faller 2011. Reproduced with permission of the Audio Engineering Society.

Chapter 14

Figure 14.1 A virtual world with three virtual sound sources. The subject should localize the sources and perceive the room effect as the avatar would perceive it if the virtual world were real. The single elephant should be perceived as point-like, and the elephant triple and the elephant herd should be perceived as spatially spread.
Figure 14.2 Virtual audio synthesis with DirAC-based processing.
Figure 14.3 A sound signal emitted by a source in a virtual world is transformed to a mono-DirAC stream by attaching directional metadata to it.
Figure 14.4 Transformation from mono-DirAC stream into first-order B-format.
Figure 14.5 A B-format recording is projected onto the surface of a cylinder. Left: Schematic presentation of the process, where the vectors define the surface positions as seen from the origin of the object, and the vectors define the corresponding positions from the viewpoint of the avatar. Right: An illustration of the result, where a spatial audio recording and a 360° image captured from a square are projected onto a cylinder around the avatar. Source: Pihlajamäki 2012. Reproduced with permission of the Audio Engineering Society.
Figure 14.6 DirAC-based method to project spatial sound recording onto the surface of a virtual object as shown in Figure 14.5. Source: Pihlajamäki 2015. Reproduced with permission of the Audio Engineering Society.
Figure 14.7 Inputs with different spatial characteristics are rendered into diffuse and non-diffuse loudspeaker signal streams. Source: Pihlajamäki 2013. Reproduced with permission of the Audio Engineering Society.
Figure 14.8 The spatial modulation effect. An audio signal is spatialized using spatial metadata analyzed from a separate B-format spatial sound recording. Source: Politis 2012. Reproduced with permission of Archontis Politis.
Figure 14.9 The ambient component of a sound field is captured by pointing a virtual microphone in the opposite direction to the direction of arrival of the sound, and modulating its directional pattern with the analyzed diffuseness parameter. Source: Politis 2012. Reproduced with permission of Archontis Politis.
Figure 14.10 Upmixing a monophonic signal to a B-format stream by estimating the diffuseness parameter based on dereverberation methods. Source: Pihlajamäki 2013. Reproduced with permission of the Audio Engineering Society.

Chapter 15

Figure 15.1 The analog microphone array with all the necessary cables and the external multichannel sound card along with its counterpart using digital MEMS microphones. All the bulky and expensive equipment can be replaced by our digital array, which requires only one USB cable. Source: Alexandridis 2016. Reproduced with permission of the IEEE.
Figure 15.2 Uniform circular microphone array configuration. The microphones are numbered 1 to M and the sound sources are s₁ to s_P.The array radius is denoted as q, α and l are the angle and distance between adjacent microphones, and A is the obtuse angle formed by the chord defined by microphones 1 and 2 and the x-axis; θ₁ indicates the DOA of source s₁. Adapted from Pavlidi 2013.
Figure 15.3 Block diagram of ImmACS encoder. Source: Alexandridis 2013a. https://www.hindawi.com/ journals/jece/2013/718574/. Used under CC BY 3.0.
Figure 15.4 Direction of arrival estimation error vs. SNR in a simulated environment. Each curve corresponds to a different number of frequency components used in a single-source zone. Source: Pavlidi 2013. Reproduced with permission of the IEEE.
Figure 15.5 A wide source atom (dashed line) and a narrow source atom (solid line) applied to the smoothed histogram of four sources (speakers). Source: Pavlidi 2013. Reproduced with permission of the IEEE.
Figure 15.6 Examples of: (a) beamformer output signals, and (b) post-filtered signals, for a scenario of three active sound sources.
Figure 15.7 Example of the side-information coding scheme for a scenario of four active sound sources. All frequencies of the mask of the first source (the most dominant) are stored. For the second mask, only the frequencies where the first mask is zero need to be stored, as the other frequencies will definitely have the value of zero in the second and all other masks (orthogonality property). For each next mask, only the frequencies where all previous masks are zero need to be stored. Source: Alexandridis 2013. Reproduced with permission of the University of Crete.
Figure 15.8 Block diagram of ImmACS decoder for (a) loudspeaker reproduction and (b) binaural reproduction. Source: Alexandridis 2013a. https://www.hindawi.com/journals/jece/2013/718574/. Used under CC BY 3.0.
Figure 15.9 Desired directivity patterns for (a) a four-channel, (b) a five-channel and (c) an eight-channel system. Source: Stefanakis and Mouchtaris 2016.
Figure 15.10 Condition number of matrix D(ω)^HD(ω) in dB (black) and variation of the regularization parameter μ (gray) as a function of the frequency for an eight-sensor circular array of radius 5 cm. Source: Stefanakis and Mouchtaris 2016.
Figure 15.11 Actual directivity patterns (solid black) versus desired directivity patterns (dashed gray) at different frequencies for an eight-element circular sensor array. The directivity patterns shown correspond to the first loudspeaker at 45°, considering a symmetric arrangement of four loudspeakers at angles of 45°, 135°, −135°, and −45° degrees. Source: Stefanakis and Mouchtaris 2016.
Figure 15.12 (a) The sensor array. (b) Sketch of the football stadium with the loudspeaker setup used for evaluation. The white dot in the lower-right corner of the football field denotes the array location. Source: Stefanakis and Mouchtaris 2016.
Figure 15.13 Rectified loudspeaker signal amplitudes for (a) the non-parametric method, and (b) the parametric method.

List of Contributors

Ahonen, Jukka: Akukon Ltd, Finland
Alexandridis, Anastasios: Foundation for Research and Technology-Hellas, Institute of Computer Science (FORTH-ICS), Heraklion, Crete, Greece
Alon, David Lou: Department of Electrical and Computer Engineering, Ben-Gurion University of the Negev, Israel
Bäckström, Tom: Department of Signal Processing and Acoustics, Aalto University, Finland
Delikaris-Manias, Symeon: Department of Signal Processing and Acoustics, Aalto University, Finland
Epain, Nicolas: CARLab, School of Electrical and Information Engineering, University of Sydney, Australia
Faller, Christof: Illusonic GmbH, Switzerland and École Polytechnique Fédérale de Lausanne (EPFL), Switzerland
Habets, Emanuël: International Audio Laboratories Erlangen, Germany
Jin, Craig T.: CARLab, School of Electrical and Information Engineering, University of Sydney, Australia
Laitinen, Mikko-Ville: Nokia Technologies, Finland
Mouchtaris, Athanasios: Foundation for Research and Technology-Hellas, Institute of Computer Science (FORTH-ICS), Heraklion, Crete, Greece
Nikunen, Joonas: Department of Signal Processing, Tampere University of Technology, Finland
Noohi, Tahereh: CARLab, School of Electrical and Information Engineering, University of Sydney, Australia
Pavlidi, Despoina: Foundation for Research and Technology-Hellas, Institute of Computer Science (FORTH-ICS), Heraklion, Crete, Greece
Pertilä, Pasi: Department of Signal Processing, Tampere University of Technology, Finland
Pihlajamäki, Tapani: Nokia Technologies, Finland
Politis, Archontis: Department of Signal Processing and Acoustics, Aalto University, Finland
Pulkki, Ville: Department of Signal Processing and Acoustics, Aalto University, Finland
Rafaely, Boaz: Department of Electrical and Computer Engineering, Ben-Gurion University of the Negev, Israel
Stefanakis, Nikolaos: Foundation for Research and Technology-Hellas, Institute of Computer Science (FORTH-ICS), Heraklion, Crete, Greece
Thiergart, Oliver: International Audio Laboratories Erlangen, Germany
Vilkamo, Juha: Nokia Technologies, Finland
Virtanen, Tuomas: Department of Signal Processing, Tampere University of Technology, Finland