iSTFTNet2: Faster and More Lightweight iSTFT-Based Neural Vocoder Using
  1D-2D CNN
Takuhiro Kaneko Hirokazu Kameoka Kou Tanaka Shogo Seki
The inverse short-time Fourier transform network (iSTFTNet) has garnered
attention owing to its fast, lightweight, and high-fidelity speech synthesis.
It obtains these characteristics using a fast and lightweight 1D CNN as the
backbone and replacing some neural processes with iSTFT. Owing to the
difficulty of a 1D CNN to model high-dimensional spectrograms, the frequency
dimension is reduced via temporal upsampling. However, this strategy
compromises the potential to enhance the speed. Therefore, we propose
iSTFTNet2, an improved variant of iSTFTNet with a 1D-2D CNN that employs 1D and
2D CNNs to model temporal and spectrogram structures, respectively. We designed
a 2D CNN that performs frequency upsampling after conversion in a few-frequency
space. This design facilitates the modeling of high-dimensional spectrograms
without compromising the speed. The results demonstrated that iSTFTNet2 made
iSTFTNet faster and more lightweight with comparable speech quality. Audio
samples are available at
https://www.kecl.ntt.co.jp/people/kaneko.takuhiro/projects/istftnet2/.