{"action":"created","issue":{"url":"https://api.github.com/repos/WebAudio/web-audio-api/issues/468","labels_url":"https://api.github.com/repos/WebAudio/web-audio-api/issues/468/labels{/name}","comments_url":"https://api.github.com/repos/WebAudio/web-audio-api/issues/468/comments","events_url":"https://api.github.com/repos/WebAudio/web-audio-api/issues/468/events","html_url":"https://github.com/WebAudio/web-audio-api/issues/468","id":55779414,"number":468,"title":"AudioFrequencyWorkerNode for working in the Frequency Domain","user":{"login":"hughrawlinson","id":829836,"avatar_url":"https://avatars.githubusercontent.com/u/829836?v=3","gravatar_id":"","url":"https://api.github.com/users/hughrawlinson","html_url":"https://github.com/hughrawlinson","followers_url":"https://api.github.com/users/hughrawlinson/followers","following_url":"https://api.github.com/users/hughrawlinson/following{/other_user}","gists_url":"https://api.github.com/users/hughrawlinson/gists{/gist_id}","starred_url":"https://api.github.com/users/hughrawlinson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hughrawlinson/subscriptions","organizations_url":"https://api.github.com/users/hughrawlinson/orgs","repos_url":"https://api.github.com/users/hughrawlinson/repos","events_url":"https://api.github.com/users/hughrawlinson/events{/privacy}","received_events_url":"https://api.github.com/users/hughrawlinson/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":27,"created_at":"2015-01-28T16:31:22Z","updated_at":"2015-01-29T19:26:53Z","closed_at":null,"body":"At the moment, ScriptProcessorNodes and AudioWorkers are operating on the time domain buffer data. At the Web Audio Conference, it seems like there's demand for frequency domain data inside a callback that's going to get called for every audio frame.\r\n\r\nWe're thus proposing an new node called an AudioFrequencyWorkerNode, which gives the developer the option to obtain audio data, perform processing, and output in the frequency domain. This involves passing an options object to the createAudioFrequencyWorker method, specifying the input and output types.\r\n\r\n### Defaults\r\nThe AudioFrequencyWorkerNode should allow access to time domain and frequency domain data concurrently. If no options object is passed to createAudioFrequencyWorker, the input type would default to the amplitude/phase pair, as would the output. However, the options object would allow the user to choose between amplitude/phase, real/imaginary, and time domain data. The dataOut would default to the same as the dataIn, but could be set to a different data type, in case the user wants to read real/imaginary pairs in, and write out to the time domain for example.\r\n\r\n### Proposed processing structure of the AudioFrequencyWorkerNode\r\n\r\nINPUT (time-domain)\r\n↓\r\nwindowing\r\n↓\r\nFFT\r\n↓\r\n~ ~ ~ ~ ~ ~ ~ ~ ~\r\ndataIn\r\n↓\r\nonaudioprocess\r\n↓\r\ndataOut\r\n~ ~ ~ ~ ~ ~ ~ ~ ~\r\n↓\r\nmirror\r\n↓\r\ncomplete data\r\n↓\r\nIFFT\r\n↓\r\nwindowing\r\n↓\r\nOUTPUT (time-domain)\r\n\r\n\r\n### Example code:\r\n\r\n_main JS_\r\n\r\n```javascript\r\nvar aw = createAudioFrequencyWorker(\"worker.js\",{\r\n        dataIn:[\r\n            \"amplitude\",\r\n            \"phase\",\r\n            \"real\",\r\n            \"imaginary\",\r\n            \"signal\"\r\n        ],\r\n        dataOut:\"complex\" || \"amplitude\" || \"phase\" || \"signal\",\r\n        bufferSize:[power of two],\r\n        hopSize:N,\r\n        windowingType,\r\n        zeroPadding:N\r\n    }\r\n// Signal (time domain) would be the default I/O for the AudioFrequencyWorker.\r\n// indicating another datatype would actuate an fft and/or ifft around the user code.\r\n```\r\n\r\n_AudioWorker JS_\r\n\r\n```javascript\r\n// callback to AudioFrequencyWorkerNode\r\nonaudioprocess = function (e) {\r\n  // e.amplitude[0][channel];\r\n  // e.phase[0][channel];\r\n  // e.real[0][channel];\r\n  // e.imaginary[0][channel];\r\n  // e.signal[0][channel]; // replacing the 'input', time domain.\r\n  \r\n  // edit the arrays in place, rather than having a separate output array to copy into\r\n  \r\n  for (var channel = 0; channel<e.amplitude.length;channel++){\r\n    for(var i = 0; i < e.amplitude.length;i++){\r\n        e.amplitude[i][channel] = e.amplitude[i][channel]>0.5 ? e.amplitude[i][channel] : 0;\r\n    }\r\n  }\r\n};\r\n```\r\n\r\n### Use Cases\r\n* Feature Extraction,\r\n* Phase Vocoding,\r\n* Pitch Shifting,\r\n* Pitch Track,\r\n* Time stretching,\r\n* Frequency domain gating effects\r\n\r\nJesse Allison, Hugh Rawlinson, Jakub Fiala, Nevo Segal\r\n@jesseallison, @hughrawlinson, @jakubfiala, @nevosegal\r\n\r\n### Related\r\n#248 \r\n#262 \r\n"},"comment":{"url":"https://api.github.com/repos/WebAudio/web-audio-api/issues/comments/72087573","html_url":"https://github.com/WebAudio/web-audio-api/issues/468#issuecomment-72087573","issue_url":"https://api.github.com/repos/WebAudio/web-audio-api/issues/468","id":72087573,"user":{"login":"rtoy","id":663356,"avatar_url":"https://avatars.githubusercontent.com/u/663356?v=3","gravatar_id":"","url":"https://api.github.com/users/rtoy","html_url":"https://github.com/rtoy","followers_url":"https://api.github.com/users/rtoy/followers","following_url":"https://api.github.com/users/rtoy/following{/other_user}","gists_url":"https://api.github.com/users/rtoy/gists{/gist_id}","starred_url":"https://api.github.com/users/rtoy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rtoy/subscriptions","organizations_url":"https://api.github.com/users/rtoy/orgs","repos_url":"https://api.github.com/users/rtoy/repos","events_url":"https://api.github.com/users/rtoy/events{/privacy}","received_events_url":"https://api.github.com/users/rtoy/received_events","type":"User","site_admin":false},"created_at":"2015-01-29T19:26:53Z","updated_at":"2015-01-29T19:26:53Z","body":"On Wed, Jan 28, 2015 at 4:25 PM, Hugh Rawlinson <notifications@github.com>\r\nwrote:\r\n\r\n> I agree with @sebpiq <https://github.com/sebpiq>, the ability to do\r\n> something in AudioWorker doesn't negate the need for a node. You could\r\n> implement every single node in the spec with AudioWorker, if that's the\r\n> reasoning then why not just make people implement their own Oscillators\r\n> with AudioWorker rather than supply OscillatorNode?\r\n>\r\n\r\n​Perhaps if history had proceeded differently and an AudioWorker existed\r\nfrom the beginning, all nodes would have been defined or implemented in\r\nterms of an AudioWorker.  That's not how history went. I'm not sure, but\r\nit's conceivable that implementations could change existing nodes to be\r\nAudioWorkers underneath if so desired.​\r\n\r\n\r\n>  ​Is the intent to supply a huge set of nodes where you can do just about\r\n> any general purpose DSP technique as if you were using, say, Matlab-like\r\n> clone running in a browser? I don't think that's the goal.​\r\n>\r\n> I don't think you need a huge set of nodes to do any general purpose DSP\r\n> technique. I do however think it's good to be able to work in both the time\r\n> and frequency domains, and AudioWorker caters for the time domain, so why\r\n> not have something that caters for the frequency domain? If you want to\r\n> optimise for as few nodes as possible, then surely we should all be\r\n> implementing our own oscillators in AudioWorkers rather than using\r\n> OscillatorNodes, I don't really think having fewer nodes is a great design\r\n> goal. Having general purpose nodes though, is, and I think\r\n> AudioFrequencyWorker is general enough to warrant being a node.\r\n>\r\n\r\n​I grant that it is useful.  But not beyond what an AudioWorker could do\r\njust as well.​\r\n\r\n​And given the fact that even with the few nodes we have today, we *still*\r\nhave failed after several years to specify them completely enough that\r\nsomeone could implement them from the spec. Fewer nodes are good. :-)​\r\n\r\n>  @adelespinasse <https://github.com/adelespinasse> @hughrawlinson\r\n> <https://github.com/hughrawlinson> That seems to be a nice complement for\r\n> AudioWorker.\r\n>\r\n> Yeah, the Web Array Math API functions would definitely be useful inside\r\n> of AudioWorker, but the spec doesn't seem to be progressing, and as it's a\r\n> separate spec it may not be ready for years even after the Web Audio API is\r\n> released.\r\n>\r\n> —\r\n> Reply to this email directly or view it on GitHub\r\n> <https://github.com/WebAudio/web-audio-api/issues/468#issuecomment-71945937>\r\n> .\r\n>\r\n\r\n\r\n\r\n-- \r\nRay"}}