{"action":"created","issue":{"url":"https://api.github.com/repos/WebAudio/web-audio-api/issues/468","labels_url":"https://api.github.com/repos/WebAudio/web-audio-api/issues/468/labels{/name}","comments_url":"https://api.github.com/repos/WebAudio/web-audio-api/issues/468/comments","events_url":"https://api.github.com/repos/WebAudio/web-audio-api/issues/468/events","html_url":"https://github.com/WebAudio/web-audio-api/issues/468","id":55779414,"number":468,"title":"AudioFrequencyWorkerNode for working in the Frequency Domain","user":{"login":"hughrawlinson","id":829836,"avatar_url":"https://avatars.githubusercontent.com/u/829836?v=3","gravatar_id":"","url":"https://api.github.com/users/hughrawlinson","html_url":"https://github.com/hughrawlinson","followers_url":"https://api.github.com/users/hughrawlinson/followers","following_url":"https://api.github.com/users/hughrawlinson/following{/other_user}","gists_url":"https://api.github.com/users/hughrawlinson/gists{/gist_id}","starred_url":"https://api.github.com/users/hughrawlinson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hughrawlinson/subscriptions","organizations_url":"https://api.github.com/users/hughrawlinson/orgs","repos_url":"https://api.github.com/users/hughrawlinson/repos","events_url":"https://api.github.com/users/hughrawlinson/events{/privacy}","received_events_url":"https://api.github.com/users/hughrawlinson/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":20,"created_at":"2015-01-28T16:31:22Z","updated_at":"2015-01-29T02:12:15Z","closed_at":null,"body":"At the moment, ScriptProcessorNodes and AudioWorkers are operating on the time domain buffer data. At the Web Audio Conference, it seems like there's demand for frequency domain data inside a callback that's going to get called for every audio frame.\r\n\r\nWe're thus proposing an new node called an AudioFrequencyWorkerNode, which gives the developer the option to obtain audio data, perform processing, and output in the frequency domain. This involves passing an options object to the createAudioFrequencyWorker method, specifying the input and output types.\r\n\r\n### Defaults\r\nThe AudioFrequencyWorkerNode should allow access to time domain and frequency domain data concurrently. If no options object is passed to createAudioFrequencyWorker, the input type would default to the amplitude/phase pair, as would the output. However, the options object would allow the user to choose between amplitude/phase, real/imaginary, and time domain data. The dataOut would default to the same as the dataIn, but could be set to a different data type, in case the user wants to read real/imaginary pairs in, and write out to the time domain for example.\r\n\r\n### Proposed processing structure of the AudioFrequencyWorkerNode\r\n\r\nINPUT (time-domain)\r\n↓\r\nwindowing\r\n↓\r\nFFT\r\n↓\r\n~ ~ ~ ~ ~ ~ ~ ~ ~\r\ndataIn\r\n↓\r\nonaudioprocess\r\n↓\r\ndataOut\r\n~ ~ ~ ~ ~ ~ ~ ~ ~\r\n↓\r\nmirror\r\n↓\r\ncomplete data\r\n↓\r\nIFFT\r\n↓\r\nwindowing\r\n↓\r\nOUTPUT (time-domain)\r\n\r\n\r\n### Example code:\r\n\r\n_main JS_\r\n\r\n```javascript\r\nvar aw = createAudioFrequencyWorker(\"worker.js\",{\r\n        dataIn:[\r\n            \"amplitude\",\r\n            \"phase\",\r\n            \"real\",\r\n            \"imaginary\",\r\n            \"signal\"\r\n        ],\r\n        dataOut:\"complex\" || \"amplitude\" || \"phase\" || \"signal\",\r\n        bufferSize:[power of two],\r\n        hopSize:N,\r\n        windowingType,\r\n        zeroPadding:N\r\n    }\r\n// Signal (time domain) would be the default I/O for the AudioFrequencyWorker.\r\n// indicating another datatype would actuate an fft and/or ifft around the user code.\r\n```\r\n\r\n_AudioWorker JS_\r\n\r\n```javascript\r\n// callback to AudioFrequencyWorkerNode\r\nonaudioprocess = function (e) {\r\n  // e.amplitude[0][channel];\r\n  // e.phase[0][channel];\r\n  // e.real[0][channel];\r\n  // e.imaginary[0][channel];\r\n  // e.signal[0][channel]; // replacing the 'input', time domain.\r\n  \r\n  // edit the arrays in place, rather than having a separate output array to copy into\r\n  \r\n  for (var channel = 0; channel<e.amplitude.length;channel++){\r\n    for(var i = 0; i < e.amplitude.length;i++){\r\n        e.amplitude[i][channel] = e.amplitude[i][channel]>0.5 ? e.amplitude[i][channel] : 0;\r\n    }\r\n  }\r\n};\r\n```\r\n\r\n### Use Cases\r\n* Feature Extraction,\r\n* Phase Vocoding,\r\n* Pitch Shifting,\r\n* Pitch Track,\r\n* Time stretching,\r\n* Frequency domain gating effects\r\n\r\nJesse Allison, Hugh Rawlinson, Jakub Fiala, Nevo Segal\r\n@jesseallison, @hughrawlinson, @jakubfiala, @nevosegal\r\n\r\n### Related\r\n#248 \r\n#262 \r\n"},"comment":{"url":"https://api.github.com/repos/WebAudio/web-audio-api/issues/comments/71956860","html_url":"https://github.com/WebAudio/web-audio-api/issues/468#issuecomment-71956860","issue_url":"https://api.github.com/repos/WebAudio/web-audio-api/issues/468","id":71956860,"user":{"login":"cwilso","id":507001,"avatar_url":"https://avatars.githubusercontent.com/u/507001?v=3","gravatar_id":"","url":"https://api.github.com/users/cwilso","html_url":"https://github.com/cwilso","followers_url":"https://api.github.com/users/cwilso/followers","following_url":"https://api.github.com/users/cwilso/following{/other_user}","gists_url":"https://api.github.com/users/cwilso/gists{/gist_id}","starred_url":"https://api.github.com/users/cwilso/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cwilso/subscriptions","organizations_url":"https://api.github.com/users/cwilso/orgs","repos_url":"https://api.github.com/users/cwilso/repos","events_url":"https://api.github.com/users/cwilso/events{/privacy}","received_events_url":"https://api.github.com/users/cwilso/received_events","type":"User","site_admin":false},"created_at":"2015-01-29T02:12:15Z","updated_at":"2015-01-29T02:12:15Z","body":"If you're looking for an FFT that produces sound - a frequency-domain\r\ntransformer - then yes, obvious the Analyser is not for you.\r\n\r\nMy personal take on this is that this would likely be better served by a\r\nseparate FFT library, and someone dealing with the array aggregation in\r\ninteresting ways.  (We could provide a \"buffering node\" that enabled\r\nprocessing of large blocks with corresponding latency - but I'm not\r\npersonally convinced quite enough that it's necessary (=hard/nonperformant\r\nenough to do yourself).  At any rate, this is DEFINITELY separate from the\r\nstraight AudioWorker.\r\n\r\nThat said - I'll defer to the WG, but I don't think this is a v1 feature.\r\nYou can, for the moment, build the buffering semantic yourself in a worker,\r\nand the FFT in JS (or use Web Array Math if there's an implementation).\r\n\r\nOn Thu, Jan 29, 2015 at 12:24 PM, Jakub Fiala <notifications@github.com>\r\nwrote:\r\n\r\n> @rtoy <https://github.com/rtoy> So the difference between the knowledge\r\n> necessary to, say, do amplitude modulation in your AudioWorker, and\r\n> performing a highly optimized Fast Fourier Transform is quite vast. The\r\n> main motivation behind this proposal is really that the majority of Web\r\n> Audio developers aren't able to do that easily, me being one of them. Given\r\n> that it's one of the fundaments of audio DSP and pretty much everybody in\r\n> the field gets to use the FFT spectrum at some point, I think it's more\r\n> than fitting to have a *really damn good*, as well as *really damn fast*\r\n> version of it in WA.\r\n>\r\n> Not to mention that we already *do* have an FFT in WA, and this is just\r\n> an attempt to make it useful for more than just simply visualizing the\r\n> spectrum, which is pretty much all that the AnalyserNode is good for.\r\n> Actually, in this sense I can take your argument even further, and say – if\r\n> there already is a native FFT in Web Audio, why should we have people\r\n> implementing it again in JS? Isn't that just horribly inefficient?\r\n>\r\n> Mind you, we originally envisaged this as an extension to the normal\r\n> AudioWorker (so no extra nodes), but after a consultation with Paul Adenot\r\n> we realised it would actually only work as a separate node mainly because\r\n> of the fixed buffer size constraint.\r\n>\r\n> —\r\n> Reply to this email directly or view it on GitHub\r\n> <https://github.com/WebAudio/web-audio-api/issues/468#issuecomment-71952201>\r\n> .\r\n>"}}