Planet GStreamer en Planet GStreamer - Arun Raghavan: Quantifying Synchronisation: Oscilloscope Edition <p>I&#8217;ve written a bit in my last two blog posts about the work I&#8217;ve been doing in inter-device synchronised playback using GStreamer. I <a href="">introduced the library</a> and then demonstrated its use in building <a href="">video walls</a>.</p> <p>The important thing in synchronisation, of course, is <em>how much in-sync are the streams</em>? The video in my <a href="">previous post</a> gave a glimpse into that, and in this post I&#8217;ll expand on that with a more rigorous, quantifiable approach.</p> <p>Before I start, a quick note: I am currently providing freelance <a href="">consulting</a> around GStreamer, PulseAudio and open source multimedia in general. If you&#8217;re looking for help with any of these, do <a href="">get in touch</a>.</p> <a href=""><img /></a>The sync measurement setup <h2>Quantifying what?</h2> <p>What is it that we are trying to measure? Let&#8217;s look at this in terms of the outcome &#8212; I have two computers, on a network. Using the <code>gst-sync-server</code> library, I play a stream on both of them. The ideal outcome is that the same video frame is displayed at exactly the same time, and the audio sample being played out of the respective speakers is also identical at any given instant.</p> <p>As we saw <a href="">previously</a>, the video output is not a good way to measure what we want. This is because video displays are updated in sync with the display clock, over which consumer hardware generally does not have control. Besides, our eyes are not <em>that</em> sensitive to minor differences in timing unless images are side-by-side. After all, we&#8217;re fooling it with static pictures that change every 16.67ms or so.</p> <p>Using audio, though, we should be able to do better. Digital audio streams for music/videos typically consist of 44100 or 48000 samples a second, so we have a much finer granularity than video provides us. The human ear is also fairly sensitive to timings with regards to sound. If it hears the same sound at an interval larger than 10 ms, you will hear two distinct sounds and the echo will annoy you to no end.</p> <p>Measuring audio is also good enough because once you&#8217;ve got audio in sync, GStreamer will take care of <acronym title="Audio/Video">A/V</acronym> sync itself.</p> <h2>Setup</h2> <p>Okay, so now that we know what we want to measure, but how do we measure it? The setup is illustrated below:</p> <a href=""><img /></a>Sync measurement setup illustrated <p>As before, I&#8217;ve set up my desktop PC and laptop to play the same stream in sync. The stream being played is a local audio file &#8212; I&#8217;m keeping the setup simple by not adding network streaming to the equation.</p> <p>The audio itself is just a tick sound every second. The tick is a simple 440 Hz sine wave (<code>A₄</code> for the musically inclined) that runs for for 1600 samples. It sounds something like this:</p> <!--[if lt IE 9]><script>document.createElement('audio');</script><![endif]--> <a href=""></a> <p>I&#8217;ve connected the 3.5mm audio output of both the computers to my faithful digital oscilloscope (a Tektronix TBS 1072B if you wanted to know). So now measuring synchronisation is really a question of seeing how far apart the leading edge of the sine wave on the tick is.</p> <p>Of course, this assumes we&#8217;re not more than 1s out of sync (that&#8217;s the periodicity of the tick itself), and I&#8217;ve verified that by playing non-periodic sounds (any song or video) and making sure they&#8217;re in sync as well. You can trust me on this, or better yet, get the code and try it yourself! :)</p> <p>The last piece to worry about &#8212; the network. How well we can sync the two streams depends on how well we can synchronise the clocks of the pipeline we&#8217;re running on each of the two devices. I&#8217;ll talk about how this works in a subsequent post, but my measurements are done on both a wired and wireless network.</p> <h2>Measurements</h2> <p>Before we get into it, we should keep in mind that due to how we synchronise streams &#8212; using a network clock &#8212; how in-sync our streams are will vary over time depending on the quality of the network connection.</p> <p>If this variation is small enough, it won&#8217;t be noticeable. If it is large (10s of milliseconds), then we may notice start to notice it as echo, or glitches when the pipeline tries to correct for the lack of sync.</p> <p>In the first setup, my laptop and desktop are connected to each other directly via a LAN cable. The result looks something like this:</p> <a href=""><img width="300" height="200" src="" class="attachment-medium size-medium" alt="" /></a> <a href=""><img width="300" height="200" src="" class="attachment-medium size-medium" alt="" /></a> <a href=""><img width="300" height="200" src="" class="attachment-medium size-medium" alt="" /></a> <a href=""><img width="300" height="200" src="" class="attachment-medium size-medium" alt="" /></a> <p>The first two images show the best case &#8212; we need to zoom in real close to see how out of sync the audio is, and it&#8217;s roughly 50µs.</p> <p>The next two images show the &#8220;worst case&#8221;. This time, the zoomed out (5ms) version shows some out-of-sync-ness, and on zooming in, we see that it&#8217;s in the order of 500µs.</p> <p>So even our bad case is actually quite good &#8212; sound travels at about 340 m/s, so 500µs is the equivalent of two speakers about 17cm apart.</p> <p>Now let&#8217;s make things a little more interesting. With both my laptop and desktop connected to a wifi network:</p> <a href=""><img width="300" height="200" src="" class="attachment-medium size-medium" alt="" /></a> <a href=""><img width="300" height="200" src="" class="attachment-medium size-medium" alt="" /></a> <a href=""><img width="300" height="200" src="" class="attachment-medium size-medium" alt="" /></a> <a href=""><img width="300" height="200" src="" class="attachment-medium size-medium" alt="" /></a> <a href=""><img width="300" height="200" src="" class="attachment-medium size-medium" alt="" /></a> <a href=""><img width="300" height="200" src="" class="attachment-medium size-medium" alt="" /></a> <p>On average, the sync can be quite okay. The first pair of images show sync to be within about 300µs.</p> <p>However, the wifi on my desktop is flaky, so you can see it go off up to 2.5ms in the next pair. In my setup, it even goes off up to 10-20ms, before returning to the average case. The next two images show it go back and forth.</p> <p>Why does this happen? Well, let&#8217;s take a quick look at what <code>ping</code> statistics from my desktop to my laptop look like:</p> <a href=""><img /></a>Ping from desktop to laptop on wifi <p>That&#8217;s not good &#8212; you can see that the minimum, average and maximum <acronym title="Round Trip Time">RTT</acronym> are very different. Our network clock logic probably needs some tuning to deal with this much jitter.</p> <h2>Conclusion</h2> <p>These measurements show that we can get some (in my opinion) pretty good synchronisation between devices using GStreamer. I wrote the <code>gst-sync-server</code> library to make it easy to build applications on top of this feature.</p> <p>The obvious area to improve is how we cope with jittery networks. We&#8217;ve added some infrastructure to capture and replay clock synchronisation messages offline. What remains is to build a large enough body of good and bad cases, and then tune the sync algorithm to work as well as possible with all of these.</p> <p>Also, Florent over at <a href="">Ubicast</a> pointed out a <a href="">nice tool</a> they&#8217;ve written to measure A/V sync on the same device. It would be interesting to modify this to allow for automated measurement of inter-device sync.</p> <p>In a future post, I&#8217;ll write more about how we actually achieve synchronisation between devices, and how we can go about improving it.</p> Thu, 19 Jan 2017 12:09:50 +0000 GStreamer: GStreamer 1.11.1 unstable release /news/#2017-01-12T15:00:00Z <p> The GStreamer team is pleased to announce the first release of the unstable 1.11 release series. The 1.11 release series is adding new features on top of the 1.0, 1.2, 1.4, 1.6, 1.8 and 1.10 series and is part of the API and ABI-stable 1.x release series of the GStreamer multimedia framework. The unstable 1.11 release series will lead to the stable 1.12 release series in the next weeks. Any newly added API can still change until that point. </p><p> Full release notes will be provided at some point during the 1.11 release cycle, highlighting all the new features, bugfixes, performance optimizations and other important changes. </p><p> Binaries for Android, iOS, Mac OS X and Windows will be provided in the next days. </p><p> Check out the release notes for <a href="">GStreamer core</a>, <a href="">gst-plugins-base</a>, <a href="">gst-plugins-good</a>, <a href="">gst-plugins-ugly</a>, <a href="">gst-plugins-bad</a>, <a href="">gst-libav</a>, <a href="">gst-rtsp-server</a>, <a href="">gst-python</a>, <a href="">gst-editing-services</a>, <a href="">gst-validate</a>, <a href="">gstreamer-vaapi</a>, or <a href="">gst-omx</a>, or download tarballs for <a href="">gstreamer</a>, <a href="">gst-plugins-base</a>, <a href="">gst-plugins-good</a>, <a href="">gst-plugins-ugly</a>, <a href="">gst-plugins-bad</a>, <a href="">gst-libav</a>, <a href="">gst-rtsp-server</a>, <a href="">gst-python</a>, <a href="">gst-editing-services</a>, <a href="">gst-validate</a>, <a href="">gstreamer-vaapi</a>, or <a href="">gst-omx</a>. </p> Thu, 12 Jan 2017 15:00:00 +0000 Phil Normand: WebKitGTK+ and HTML5 fullscreen video,2010-08-20:Articles/117/ <p><a class="reference external" href=""><span class="caps">HTML5</span> video</a> is really nice and all but one annoying thing is the lack of fullscreen video specification, it is currently up to the User-Agent to allow fullscreen video&nbsp;display.</p> <p>WebKit allows a fullscreen button in the media controls and Safari can, on user-demand only, switch a video to fullscreen and display nice controls over the video. There&#8217;s also a webkit-specific <span class="caps">DOM</span> <span class="caps">API</span> that allow custom media controls to also make use of that feature, as long as it is initiated by a user gesture. Vimeo&#8217;s <span class="caps">HTML5</span> player uses that feature for&nbsp;instance.</p> <p>Thanks to the efforts made by <a class="reference external" href="">Igalia</a> to improve the WebKit <span class="caps">GTK</span>+ port and its GStreamer media-player, I have been able to implement support for this fullscreen video display feature. So <a class="reference external" href="">any application</a> using WebKitGTK+ can now make use of it&nbsp;:)</p> <p>The way it is done with this first implementation is that a new fullscreen gtk+ window is created and our GStreamer media-player inserts an autovideosink in the pipeline and overlays the video in the window. Some simple controls are supported, the <span class="caps">UI</span> is actually similar to Totem&#8217;s. One improvement we could do in the future would be to allow application to override or customize that simple controls <span class="caps">UI</span>.</p> <p>The nice thing about this is that we of course use the <a class="reference external" href="">GstXOverlay</a> interface and that it&#8217;s implemented by the linux, windows and mac os video sinks :) So when other WebKit ports start to use our GStreamer player implementation it will be fairly easy to port the feature and allow more platforms to support fullscreen video display&nbsp;:)</p> <p><a class="reference external" href="">Benjamin</a> also suggested that we could reuse the cairo surface of our WebKit videosink and paint it in a fullscreen GdkWindow. But this should be done only when his Cairo/Pixman patches to enable hardware acceleration&nbsp;land.</p> <p>So there is room for improvement but I believe this is a first nice step for fullscreen <span class="caps">HTML5</span> video consumption from <a class="reference external" href="">Epiphany</a> and other WebKitGTK+-based browsers :) Thanks a lot to <a class="reference external" href="">Gustavo</a>, Sebastian Dröge and <a class="reference external" href="">Martin</a> for the&nbsp;code-reviews.</p> Wed, 11 Jan 2017 14:42:00 +0000 Jean-François Fortin Tam: Reviewing the Librem 15 <p>Following up on my <a href="" target="_blank">previous post</a> where I detailed the work I&#8217;ve been doing mostly on Purism&#8217;s website, today&#8217;s post post will cover some video work. Near the beginning of October, I received a <a href="">Librem 15</a> v2 unit for testing and reviewing purposes. I have been using it as my main laptop since then, as I don&#8217;t believe in reviewing something without using it daily for a couple weeks at least. And so on nights and week-ends, I wrote down testing results, rough impressions and recommendations, then wrote a detailed plan and script to make the first in depth video review of this laptop. Here&#8217;s the result—not your typical 2-minutes superficial tour:</p> <p></p> <p><span id="more-3436"></span></p> <p>With this review, I wanted to:</p> <ul> <li>Satisfy my own curiosity and then share the key findings; one of the things that annoyed me some months ago is that I couldn&#8217;t find any good &#8220;up close&#8221; review video to answer my own technical questions, and I thought &#8220;Surely I&#8217;m not the only one! Certainly a bunch of other people would like to see what the beast feels like in practice.&#8221;</li> <li>Make an audio+video production I would be proud of, artistically speaking. I&#8217;m rather meticulous in my craft as like creating quality work made to last (similarly, I have recently finished a particular decorative painting after <em>months</em> of obsession&#8230; I&#8217;ll let you know about that in some other blog post ;)</li> <li>Put my production equipment to good use; I had recently purchased a lot of equipment for my studio and outdoors shooting—it was just begging to be used! Some details on that further down in this post.</li> <li>Provide a <em>ton</em> of industrial design feedback to the Purism team for future models, based on my experience owning and using almost every laptop type out there. And so I did. Pages and pages of it, way more than can fit in a video:</li> </ul> <a href=""><img class="wp-image-3491 size-large" src="" width="700" height="276" /></a>Pictured: my review notes <h1>A fine line</h1> <p>For the video, I spent a fair amount of time writing and revising my video&#8217;s plan and narration (half a dozen times at least), until I came to a satisfactory &#8220;final&#8221; version that I was able to record the narration for (using the exquisite ancient techniques of voice acting).</p> <p>The tricky part is being simultaneously concise, exhaustive, fair, and entertaining. I wanted to be as balanced as possible and to cover the most crucial topics.</p> <ul> <li>At the end of the day, there are some simple fundamentals of what makes a good or bad computer. Checking my 14 pages of review notes, I knew I was being <em>extremely</em> demanding, and that some of those expectations of mine came down to personal preference, things that most people don&#8217;t even care about, or common issues that are not even addressed by most &#8220;big brand&#8221; OEMs&#8217; products&#8230; so I balanced my criticism with a dose of realism, making sure to focus on what would matter to people.</li> <li>I also chose topics that would have a longer &#8220;shelf life&#8221;, considering how much work it takes to produce a high-quality video. For example, even while preparing the review over the course of 2-3 months, some aspects (such as the touchpad drivers) changed/improved and made me revise my opinion. The touchpad behaved better in Fedora 25 than in Fedora 24&#8230; until a kernel update <a href="">broke it</a> (Ugh. At that point I decided to version-lock my kernel package in Fedora).</li> <li>I was conservative in my estimates, even if that makes the device look less shiny than it is. For example, while I said &#8220;5-6 hours&#8221; of battery life in the review video, in practice I realized that I can get <em>10-12 hours of battery life</em> with my personal usage pattern (writing text in Gedit, no web browser open, 50% brightness, etc.) and a few simple tweaks in PowerTop.</li> </ul> <p>The final version of my script, as I recorded it, was <strong>59 minutes</strong> long. No jokes. And that was <em>after</em> I had decided to ignore some topics (ex.: the whole part about the preloaded operating system; that part would be long enough to be a standalone review review).</p> <p>I spent some days processing that 59 minutes recording to remove any sound impurities or mistakes, and sped up the tempo, bringing down the duration to 37 and then 31 minutes. Still, that was too long, so while I was doing the final video edit, I tightened everything further (removing as many speech gaps as possible) and cut out a few more topics at the last minute. The final rendered result is a video that is <strong>21 minutes</strong> long. Much more reasonable, considering it&#8217;s an &#8220;in depth&#8221; review.</p> <h1>Real audio, lighting, and optics</h1> <p>My general work ethic is: when you do something, do it <em>properly</em> or don&#8217;t do it at all.</p> <p>For this project I used studio lighting, tripods and stands, a dolly, a DSLR with two lenses, a smaller camera for some last minute shots, a high-end PCM sound recorder, a phantom-powered shotgun microphone, a recording booth, monitoring headphones, sandbags, etc.</p> <p><a href=""><img class="alignnone size-medium wp-image-3494" src="" alt="" width="500" height="334" /></a></p> <p><a href=""><img class="alignnone size-medium wp-image-3496" src="" alt="" width="500" height="334" /></a></p> <p><a href=""><img class="alignnone size-medium wp-image-3495" src="" alt="" width="500" height="334" /></a></p> <p><a href=""><img class="alignnone size-medium wp-image-3497" src="" alt="" width="500" height="314" /></a></p> <p>To complement the narration and cover the length of the review, I chose seven different songs (out of many dozens) based on genre, mood and tempo. I sometimes had to cut/mix songs to avoid the annoying parts. The result, I hope, is a video that remains pleasant and entertaining to watch throughout, while also having a certain emotional or &#8220;material&#8221; quality to it. For example, the song I used for the thermal design portion makes me feel like I&#8217;m touching heatpipes and watching energy flow. Maybe that&#8217;s just me though—perhaps if there was some lounge music I would <a href="" target="_blank">feel like a sofa</a> ;)</p> <h1>Fun!</h1> <p>Much like when I made <a href="">a video for a local symphonic orchestra</a>, I have enjoyed the making of this review video immensely. I&#8217;m glad I got an interesting device to review (not a <a href="">whole chicken in a can</a>) with the freedom to make the &#8220;video I would have wanted to see&#8221;. On that note, if anyone has a fighter jet they&#8217;d want to see reviewed, let me know (I&#8217;m <a href="" target="_blank">pretty good</a> at dodging missiles ;)</p> Tue, 10 Jan 2017 16:00:51 +0000 Jean-François Fortin Tam: Renommer un périphérique audio avec PulseAudio <p>J&#8217;ai découvert par hasard qu&#8217;on peut faire un clic droit sur le &#8220;profil&#8221; d&#8217;un périphérique dans <em>pavucontrol</em> pour renommer le périphérique. Or, cette fonctionnalité n&#8217;est pas disponible par défaut puisqu&#8217;il faut un module supplémentaire (sous Fedora, du moins). Pour faire un essai en temps réel:</p> <p><code>$ pactl load-module module-device-manager<br /> 34</code></p> <p><span id="more-3221"></span></p> <p>&#8220;34&#8221;? Quelle drôle de réponse! Je présume que ça veut dire que ça a fonctionné. Si on réessaie immédiatement la même commande, on obtient:</p> <p><code>$ pactl load-module module-device-manager<br /> Échec : Échec lors de l'initialisation du module</code></p> <p>Ce qui, si on pense comme un développeur un peu paresseux sur la sémantique, dévoile que le module est bel et bien chargé déjà.</p> <p>On peut maintenant donner des noms beaucoup plus courts et pratiques à nos périphériques favoris. Par exemple, ma SoundBlaster au nom beaucoup trop complexe:</p> <div class="wp-video"><!--[if lt IE 9]><script>document.createElement('video');</script><![endif]--> <a href=""></a></div> <p>Ou encore mon bon vieux <a href="">micro Logitech</a> au nom tout à fait cryptique:</p> <div class="wp-video"><a href=""></a></div> <p>Côté interface, tout ceci est plutôt moche et difficile à découvrir, alors j&#8217;ai ouvert <a href="">un rapport de bug à ce sujet</a>. Ce qui est un peu dommage, c&#8217;est qu&#8217;on renomme ici les entrées/sorties (sources/sinks) individuellement, au lieu de renommer le périphérique matériel dans sa globalité. Aussi, les versions renommées ne sont pas prises en compte par les paramètres de son de GNOME Control Center.</p> <p>En tout cas, après ce test concluant, on peut ajouter les lignes suivantes dans le fichier ~/.config/pulse/ pour rendre le changement permanent (je préfère éditer ce fichier dans mon dossier personnel plutôt que le fichier système &#8220;/etc/pulse/;, pour que ça persiste après des &#8220;clean installs&#8221; de distros:</p> <p><code>.include /etc/pulse/<br /> <strong>load-module module-device-manager</strong></code></p> <p>Note: &#8220;;, contrairement à daemon.conf (dans lequel j&#8217;ai simplement mis &#8220;flat-volumes = no&#8221; pour revenir au mixage à l&#8217;ancienne), n&#8217;hérite pas automatiquement des paramètres système, c&#8217;est pourquoi j&#8217;ai inséré la ligne &#8220;.include&#8221;.</p> Tue, 03 Jan 2017 16:00:57 +0000 Sebastian Pölsterl: Announcing scikit-survival – a Python library for survival analysis build on top of scikit-learn at <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even"><div class="tex2jax"><p>I've meant to do this release for quite a while now and last week I finally had some time to package everything and update the dependencies. <a href="" rel="nofollow">scikit-survival</a> contains the majority of code I developed during my Ph.D. </p> <p><strong>About Survival Analysis</strong></p> <p>Survival analysis – also referred to as reliability analysis in engineering – refers to type of problem in statistics where the objective is to establish a connections between a set of measurements (often called features or covariates) and the time to an event. The name <em>survival analysis</em> originates from clinical research: in many clinical studies, one is interested in predicting the time to death, i.e., survival. Broadly speaking, survival analysis is a type of regression problem (one wants to predict a continuous value), but with a twist. Consider a clinical study, which investigates coronary heart disease and has been carried out over a 1 year period as in the figure below.</p> <p><img src="" /></p> <p>Patient A was lost to follow-up after three months with no recorded cardiovascular event, patient B experienced an event four and a half months after enrollment, patient D withdrew from the study two months after enrollment, and patient E did not experience any event before the study ended. Consequently, the exact time of a cardiovascular event could only be recorded for patients B and C; their records are <em>uncensored</em>. For the remaining patients it is unknown whether they did or did not experience an event after termination of the study. The only valid information that is available for patients A, D, and E is that they were event-free up to their last follow-up. Therefore, their records are <em>censored</em>.</p> <p>Formally, each patient record consists of a set of covariates $x \in \mathbb{R}^d$ , and the time $t &gt; 0$ when an event occurred or the time $c &gt; 0$ of censoring. Since censoring and experiencing and event are mutually exclusive, it is common to define an event indicator $\delta \in \{0; 1\}$ and the observable survival time $y &gt; 0$. The observable time $y$ of a right censored sample is defined as<br /> \[ y = \min(t, c) =<br /> \begin{cases}<br /> t &amp; \text{if } \delta = 1 , \\<br /> c &amp; \text{if } \delta = 0 ,<br /> \end{cases}<br /> \]</p> <p><strong>What is scikit-survival?</strong></p> <p>Recently, many methods from machine learning have been adapted for these kind of problems: <a href="" rel="nofollow">random forest</a>, <a href="" rel="nofollow">gradient</a> <a href="" rel="nofollow">boosting</a>, and <a href="" rel="nofollow">support</a> <a href="" rel="nofollow">vector</a> <a href="" rel="nofollow">machine</a>, many of which are only available for R, but not Python. Some of the traditional models are part of <a href="" rel="nofollow">lifelines</a> or <a href="" rel="nofollow">statsmodels</a>, but none of those libraries plays nice with <a href="" rel="nofollow">scikit-learn</a>, which is the quasi-standard machine learning framework for Python.</p> <p>This is exactly where <a href="" rel="nofollow">scikit-survival</a> comes in. Models implemented in <a href="" rel="nofollow">scikit-survival</a> follow the scikit-learn interfaces. Thus, it is possible to use PCA from scikit-learn for dimensionality reduction and feed the low-dimensional representation to a survival model from <a href="" rel="nofollow">scikit-survival</a>, or cross-validate a survival model using the classes from scikit-learn. You can see an example of the latter in <a href="" rel="nofollow">this notebook</a>.</p> <p><strong>Download and Install</strong></p> <p>The source code is <a href="" rel="nofollow">available at GitHub</a> and can be installed via <a href="" rel="nofollow">Anaconda</a> (currently only for Linux) or pip.</p> <p>conda install -c sebp scikit-survival</p> <p>pip install scikit-survival</p> <p>The API documentation is available <a href="" rel="nofollow">here</a> and scikit-survival ships with a couple of <a href="" rel="nofollow">sample datasets</a> from the medical domain to get you started.</p> </div></div></div></div> Thu, 29 Dec 2016 14:08:22 +0000 Arun Raghavan: Synchronised Playback and Video Walls <p>Hello again, and I hope you&#8217;re having a pleasant end of the year (if you are, maybe don&#8217;t check the news until next year).</p> <p>I&#8217;d <a href="">written</a> about synchronised playback with GStreamer a little while ago, and work on that has been continuing apace. Since I last wrote about it, a bunch of work has gone in:</p> <ul> <li><p>Landed support for sending a playlist to clients (instead of a single URI)</p></li> <li><p>Added the ability to start/stop playback</p></li> <li><p>The API has been cleaned up considerably to allow us to consider including this <a href="">upstream</a></p></li> <li><p>The control protocol implementation was made an interface, so you don&#8217;t have to use the built-in TCP server (different use-cases might want different transports)</p></li> <li><p>Made a bunch of robustness fixes and documentation</p></li> <li><p>Introduced API for clients to send the server information about themselves</p></li> <li><p>Also added API for the server to send video transformations for specific clients to apply before rendering</p></li> </ul> <p>While the other bits are exciting in their own right, in this post I&#8217;m going to talk about the last two items.</p> <h2>Video walls</h2> <p>For those of you who aren&#8217;t familiar with the term, a video wall is just an array of displays stacked to make a larger display. These are often used in public installations.</p> <p>One way to set up a video wall is to have each display connected to a small computer (such as the Raspberry Pi), and have them play a part of the entire video, cropped and scaled for the display that is connected. This might look something like:</p> <a href=""><img /></a>A 4&#215;4 video wall <p>The tricky part, of course, is synchronisation &#8212; which is where <code>gst-sync-server</code> comes in. Since we&#8217;re able to play a given stream in sync across devices on a network, the only missing piece was the ability to distribute a set of per-client transformations so that clients could apply those, and that is now done.</p> <p>In order to keep things clean from an API perspective, I took the following approach:</p> <ul> <li><p>Clients now have the ability to send a client ID and a configuration (which is just a dictionary) when they first connect to the server</p></li> <li><p>The server API emits a signal with the client ID and configuration, which allows you to know when a client connects, what kind of display it&#8217;s running, and where it is positioned</p></li> <li><p>The server now has additional fields to send a map of client ID to a set of video transformations</p></li> </ul> <p>This allows us to do fancy things like having each client manage its own information with the server dynamically adapting the set of transformations based on what is connected. Of course, the simpler case of having a static configuration on the server also works.</p> <h2 id="demo">Demo</h2> <p>Since seeing is believing, here&#8217;s a demo of the synchronised playback in action:</p> <p><div class="jetpack-video-wrapper"></div></p> <p>The setup is my laptop, which has an Intel GPU, and my desktop, which has an NVidia GPU. These are connected to two monitors (thanks go out to my good friends from <a href="">Uncommon</a> for lending me their thin-bezelled displays).</p> <p>The video resolution is 1920&#215;800, and I&#8217;ve adjusted the crop parameters to account for the bezels, so the video actually does look continuous. I&#8217;ve uploaded the <a href="">text configuration</a> if you&#8217;re curious about what that looks like.</p> <p>As I mention in the video, the synchronisation is not as tight than I would like it to be. This is most likely because of the differing device configurations. I&#8217;ve been working with <a href="">Nicolas</a> to try to address this shortcoming by using some timing extensions that the Wayland protocol allows for. More news on this as it breaks.</p> <p>More generally, I&#8217;ve done some work to quantify the degree of sync, but I&#8217;m going to leave that for another day.</p> <p><em>p.s. the reason I used <code>kmssink</code> in the demo was that it was the quickest way I know of to get a full-screen video going &#8212; I&#8217;m happy to hear about alternatives, though</em></p> <h2>Future work</h2> <h3>Make it real</h3> <p>My demo was implemented quite quickly by allowing the example server code to load and serve up a static configuration. What I would like is to have a proper working application that people can easily package and deploy on the kinds of embedded systems used in real video walls. If you&#8217;re interested in taking this up, I&#8217;d be happy to help out. Bonus points if we can dynamically calculate transformations based on client configuration (position, display size, bezel size, etc.)</p> <h3>Hardware acceleration</h3> <p>One thing that&#8217;s bothering me is that the video transformations are applied in software using GStreamer elements. This works fine(ish) for the hardware I&#8217;m developing on, but in real life, we would want to use OpenGL(ES) transformations, or platform specific elements to have hardware-accelerated transformations. My initial thoughts are for this to be either API on <code>playbin</code> or a <code>GstBin</code> that takes a set of transformations as parameters and internally sets up the best method to do this based on whatever sink is available downstream (some sinks provide cropping and other transformations).</p> <h3>Why not audio?</h3> <p>I&#8217;ve only written about video transformations here, but we can do the same with audio transformations too. For example, multi-room audio systems allow you to configure the locations of wireless speakers &#8212; so you can set which one&#8217;s on the left, and which on the right &#8212; and the speaker will automatically play the appropriate channel. Implementing this should be quite easy with the infrastructure that&#8217;s currently in place.</p> <h2>Merry Happy <code>*.*</code></h2> <p>I hope you enjoyed reading that &#8212; I&#8217;ve had great responses from a lot of people about how they might be able to use this work. If there&#8217;s something you&#8217;d like to see, leave a comment or <a href="">file an issue</a>.</p> <p>Happy end of the year, and all the best for 2017!</p> Wed, 28 Dec 2016 18:01:34 +0000 Bastien Nocera: Making your own retro keyboard, We're about a week before Christmas, and I'm going to explain how I created a retro keyboard as a gift to my father, who introduced me to computers when he brought back a <a href="">Thomson TO7</a> home, all the way back in 1985.<br /><br />The original idea was to use a Thomson computer to fit in a smaller computer, such as a <a href="">CHIP</a> or <a href="">Raspberry Pi</a>, but the software update support would have been difficult, the use limited to the builtin programs, and it would have required a separate screen. So I restricted myself to only making a keyboard. It was a big enough task, as we'll see.<br /><br /><b>How do keyboards work?</b><br /><br />Loads of switches, that's how. I'll point you to&nbsp;<a href="">Michał Trybus</a>' blog post « <a href="">How to make a keyboard - the matrix</a> » for details on this works. You'll just need to remember that most of the keyboards present in those older computers have no support for xKRO, and that the micro-controller we'll be using already has the necessary pull-up resistors builtin.<br /><br /><b>The keyboard hardware</b><br /><br />I chose the smallest Thomson computer available for my project, the <a href="">MO5</a>. I could have used a stand-alone keyboard, but would have lost all the charm of it (it just looks like a PC keyboard), <a href="">some other computers have much bigger form factors</a>, to include cartridge, cassette or floppy disk readers.<br /><br />The <a href="">DCMoto emulator</a>'s website includes tons of documentation, including <a href="">technical documentation</a> explaining the inner workings of each one of the chipsets on the mainboard. In one of those manuals, you'll find this page:<br /><br /><div class="separator"><a href=""><img border="0" height="301" src="" width="400" /></a></div><br /><br />Whoot! The keyboard matrix in details, no need for us to discover it with a multimeter.<br /><br /><div class="separator"><a href=""><img border="0" height="240" src="" width="320" /></a></div><div><i>That needs a wash in soapy water</i></div><br />After opening up the computer, and eventually giving the internals, and the keyboard especially if it has mechanical keys, a good clean, we'll need to see how the keyboard is connected.<br /><br /><div class="separator"><a href=""><img border="0" height="211" src="" width="320" /></a></div><div><i>Finicky metal covered plastic</i></div><div><br /></div>Those keyboards usually are <a href="">membrane keyboards</a>, with pressure pads, so we'll need to either find replacement connectors at our local electronics store, or desolder the ones on the motherboard. I chose the latter option.<br /><br /><div class="separator"><a href=""><img border="0" height="206" src="" width="320" /></a></div><div class="separator"><i>Desoldered connectors</i></div><br />After matching the physical connectors to the rows and columns in the matrix, using a multimeter and a few key presses, we now know which connector pin corresponds to which connector on the matrix. We can start soldering.<br /><br /><b>The micro-controller</b><br /><br />The micro-controller in my case is a Teensy 2.0, an Atmel AVR-based micro-controller with a very useful firmware that makes it very very difficult to brick. You can either press the little button on the board itself to upload new firmware, or wire it to an external momentary switch. The funny thing is that the Atmega32U4 is 16 times faster than the original CPU (yeah, we're getting old).<br /><br />I chose to wire it to the "Initial. Prog" ("Reset") button on the keyboard, so as to make it easy to upload new firmware. To do this, I needed to cut a few traces coming out of the physical switch on the board, to avoid interferences from components on the board, using a tile cutter. This is completely optional, and if you're only going to use firmware that you already know at least somewhat works, you can set a key combo to go into firmware upload mode in the firmware. We'll get back to that later.<br /><br />As far as connecting and soldering to the pins, we can use any I/O pins we want, except D6, which is connected to the board's LED. Note that any deviation from the pinout used in your firmware, you'd need to make changes to it. We'll come back to that again in a minute.<br /><br /><b>The soldering</b><br /><br /><div class="separator"><a href=""><img border="0" height="253" src="" width="320" /></a></div><div><i>Colorful tinning</i></div><br />I wanted to keep the external ports full, so it didn't look like there were holes in the case, but there was enough headroom inside the case to fit the original board, the teensy and pins on the board. That makes it easy to rewire in case of error. You could also dremel (yes, used as a verb) a hole in the board.<br /><br />As always, make sure early that things would fit, especially the cables!<br /><br /><div class="separator"><a href=""><img border="0" height="205" src="" width="320" /></a></div><div><i>The unnecessary pollution</i></div><br /><b>The firmware</b><br /><br />Fairly early on during my research, I found the <a href="">TMK keyboard firmware</a>, as well as very well written <a href="">forum</a> <a href="">post</a> with detailed explanations on how to modify an existing firmware for your own uses.<br /><br />This is what I used to modify the firmware for the gh60 keyboard for my own use. You can see <a href="">here a step-by-step example</a>, implementing the modifications in the same order as the forum post.<br /><br />Once you've followed the steps, you'll need to compile the firmware. Fedora ships with the necessary packages, so it's a simple:<br /><br /><!-- HTML generated using --><br /><div><pre><span>sudo dnf install -y avr-libc avr-binutils avr-gcc</span><br /></pre></div><br />I also compiled and installed in my $PATH the <a href="">teensy_cli firmware uploader</a>, and fixed up the <a href="">udev rules</a>. And after a "<span>make teensy</span>" and a button press...<br /><br />It worked first time! This is a good time to verify that all the keys work, and you don't see doubled-up letters because of short circuits in your setup. I had 2 wires touching, and one column that just didn't work.<br /><br />I also prepared a <a href="">stand-alone repository</a>, with a firmware that uses the tmk_core from the tmk firmware, instead of modifying an existing one.<br /><br /><b>Some advices</b><br /><br />This isn't the first time I hacked on hardware, but I'll repeat some old adages, and advices, because I rarely heed those warnings, and I regret...<br /><ul><li>Don't forget the size, length and non-flexibility of cables in your design</li><li>Plan ahead when you're going to cut or otherwise modify hardware, because you might regret it later</li><li>Use breadboard cables and pins to connect things, if you have the room</li><li>Don't hotglue until you've tested and retested and are sure you're not going to make more modifications</li></ul>That last one explains the slightly funny cabling of my keyboard.<br /><br /><b>Finishing touches</b><br /><br /><div class="separator"><a href=""><img border="0" height="228" src="" width="320" /></a></div><div><i>All Sugru'ed up</i></div><br />To finish things off nicely, I used Sugru to stick the USB cable, out of the machine, in place. And as earlier, it will avoid having an opening onto the internals.<br /><br />There are a couple more things that I'll need to finish up before delivery. First, the keymap I have chosen in the firmware only works when a US keymap is selected. I'll need to make a keymap for Linux, possibly <a href="">hard-coding it</a>. I will also need to create a Windows keymap for my father to use (yep, genealogy software on Linux isn't quite up-to-par).<br /><br /><div class="separator"><a href=""><img border="0" height="135" src="" width="400" /></a></div><div><i>Prototype and final hardware</i></div><br />All this will happen in the <a href="">aforementioned repository</a>. And if you ever make your own keyboard, I'm happy to merge in changes to this repository with documentation for your Speccy, C64, or Amstrad CPC hacks.<br /><br />(If somebody wants to buy me&nbsp;<a href="">a Sega keyboard</a>, I'll gladly work on a non-destructive adapter. Get in touch :) Thu, 15 Dec 2016 16:48:11 +0000 (Bastien Nocera) GStreamer: GStreamer 1.10.2 stable release (binaries) /news/#2016-11-30T17:45:00Z <p> Pre-built binary images of the 1.10.2 stable release of GStreamer are now available for Windows 32/64-bit, iOS and Mac OS X and Android. </p><p> See <a href="">/releases/1.10/</a> for the full list of changes. </p><p> The builds are available for download from: <a href="">Android</a>, <a href="">iOS</a>, <a href="">Mac OS X</a> and <a href="">Windows</a>. </p> Wed, 30 Nov 2016 17:45:00 +0000 GStreamer: GStreamer 1.10.2 stable release /news/#2016-11-29T14:00:00Z <p> The GStreamer team is pleased to announce the second bugfix release in the stable 1.10 release series of your favourite cross-platform multimedia framework! </p><p> This release only contains bugfixes and it should be safe to update from 1.10.0. For a full list of bugfixes see <a href=" form&limit=0&list_id=168172&order=bug_id&product=GStreamer&query_format=advanced&resolution=FIXED&target_milestone=1.10.2">Bugzilla</a>. </p><p> See <a href="">/releases/1.10/</a> for the full release notes. </p><p> Binaries for Android, iOS, Mac OS X and Windows will be available shortly. </p><p> Check out the release notes for <a href="">GStreamer core</a>, <a href="">gst-plugins-base</a>, <a href="">gst-plugins-good</a>, <a href="">gst-plugins-ugly</a>, <a href="">gst-plugins-bad</a>, <a href="">gst-libav</a>, <a href="">gst-rtsp-server</a>, <a href="">gst-python</a>, <a href="">gst-editing-services</a>, <a href="">gst-validate</a>, <a href="">gstreamer-vaapi</a>, or <a href="">gst-omx</a>, or download tarballs for <a href="">gstreamer</a>, <a href="">gst-plugins-base</a>, <a href="">gst-plugins-good</a>, <a href="">gst-plugins-ugly</a>, <a href="">gst-plugins-bad</a>, <a href="">gst-libav</a>, <a href="">gst-rtsp-server</a>, <a href="">gst-python</a>, <a href="">gst-editing-services</a>, <a href="">gst-validate</a>, <a href="">gstreamer-vaapi</a>, or <a href="">gst-omx</a>. </p> Tue, 29 Nov 2016 14:00:00 +0000 GStreamer: Revamped documentation and switch-off /news/#2016-11-29T12:00:00Z <p> The GStreamer project is pleased to announce its new revamped <a href="">documentation</a> featuring a new design, a new navigation bar, search functionality, source code syntax highlighting as well as new tutorials and documentation about how to use GStreamer on Android, iOS, macOS and Windows. </p><p> It now contains the former SDK tutorials which have kindly been made available by Fluendo &amp; Collabora under a Creative Commons license. The tutorials have been reviewed and updated for GStreamer 1.x. The old site will be shut down with redirects pointing to the updated tutorials and the official GStreamer website. </p><p> Thanks to everyone who helped make this happen. </p><p> This is just the beginning. Our goal is to provide a more cohesive documentation experience for our users going forward. To that end, we have converted most of our documentation into markdown format. This should hopefully make it easier for developers and contributors to create new documentation, and to maintain the existing one. There is a lot more work to do, do get in touch if you want to help out. The documentation is maintained in the new <a href="">gst-docs</a> module. </p><p> If you encounter any problems or spot any omissions or outdated content in the new documentation, please <a href=""> file a bug in bugzilla</a> to let us know. </p> Tue, 29 Nov 2016 12:00:00 +0000 Sebastian Dröge: Writing GStreamer Elements in Rust (Part 3): Parsing data from untrusted sources like it’s 2016 <p><strong>This is part 3, the older parts can be found here: <a href="">part 1</a> and <a href="">part 2</a></strong></p> <p>And again it took quite a while to write a new update about my experiments with writing <a href="" target="_blank">GStreamer</a> elements in <a href="" target="_blank">Rust</a>. The previous articles can be found <a href="" target="_blank">here</a> and <a href="" target="_blank">here</a>. Since last time, there was also the <a href="" target="_blank">GStreamer Conference 2016 in Berlin</a>, where I had a <a href="" target="_blank">short presentation</a> about this.</p> <p>Progress was rather slow unfortunately, due to work and other things getting into the way. Let&#8217;s hope this improves. Anyway!</p> <p>There will be three parts again, and especially <a href="">the last one</a> would be something where I could use some suggestions from more experienced Rust developers about how to solve state handling / state machines in a nicer way. <a href="">The first part</a> will be about parsing data in general, especially from untrusted sources. <a href="">The second part</a> will be about my experimental and current proof of concept FLV demuxer.</p> <h3 id="parsing-data">Parsing Data</h3> <h4>Safety?</h4> <p>First of all, you probably all saw <a href="" target="_blank">a couple</a> <a href="" target="_blank">of CVEs</a> about security relevant bugs in (rather uncommon) GStreamer elements going around. While all of them would&#8217;ve been prevented by having the code written in Rust (due to by-default array bounds checking), that&#8217;s not going to be our topic here. They also would&#8217;ve been prevented by using various GStreamer helper API, like <a href="" target="_blank">GstByteReader</a>, <a href="" target="_blank">GstByteWriter</a> and <a href="" target="_blank">GstBitReader</a>. So just use those, really. Especially in new code (which is exactly the problem with the code affected by the CVEs, it was old and forgotten). Don&#8217;t do an accountant&#8217;s job, counting how much money/many bytes you have left to read.</p> <p>But yes, this is something where Rust will also provide an advantage by having by-default safety features. It&#8217;s not going to solve all our problems, but at least some classes of problems. And sure, you can write safe C code if you&#8217;re careful but I&#8217;m sure you also drive with a seatbelt although you <em>can</em> drive safely. To quote <a href="" target="_blank">Federico</a> about his motivation for rewriting (parts of) <a href="" target="_blank">librsvg</a> in Rust:</p> <blockquote><p>Every once in a while someone discovers a bug in librsvg that makes it all the way to a CVE security advisory, and it&#8217;s all due to using C. We&#8217;ve gotten double free()s, wrong casts, and out-of-bounds memory accesses. Recently someone did fuzz-testing with some really pathological SVGs, and found interesting explosions in the library. That&#8217;s the kind of 1970s bullshit that Rust prevents.</p></blockquote> <p>You can directly replace the word <em>librsvg</em> with <em>GStreamer</em> here.</p> <h4>Ergonomics</h4> <p>The other aspect with parsing data is that it&#8217;s usually a very boring aspect of programming. It should be as painless as possible, as easy as possible to do it in a safe way, and after having written your 100th parser by hand you probably don&#8217;t want to do that again. <a href="" target="_blank">Parser combinator libraries</a> like <a href="" target="_blank">Parsec</a> in <a href="" target="_blank">Haskell</a> provide a nice alternative. You essentially write down something very close to a formal grammar of the format you want to parse, and out of this comes a parser for the format. Other than parser generators like good, old <a href="" target="_blank">yacc</a>, everything is written in target language though, and there is no separate code generation step.</p> <p>Rust, being quite a bit more expressive than C, also made people write parser generator libraries. They are all not as ergonomic (yet?) as in Haskell, but still a big improvement over anything else. There&#8217;s <a href="" target="_blank">nom</a>, <a href="" target="_blank">combine</a> and <a href="" target="_blank">chomp</a>. All having a slightly different approach. Choose your favorite. I decided on nom for the time being.</p> <h3 id="flv-demuxer">A FLV Demuxer in Rust</h3> <p>For implementing a demuxer, I decided on using the <a href="" target="_blank">FLV container format</a>. Mostly because it is super-simple compared to e.g. MP4 and WebM, but also because Geoffroy, the author of nom, wrote a simple <a href="" target="_blank">header parsing library</a> for it already and a <a href="" target="_blank">prototype demuxer</a> using it for <a href="" target="_blank">VLC</a>. I&#8217;ll have to extend that library for various features in the near future though, if the demuxer should ever become feature-equivalent with the existing one in GStreamer.</p> <p>As usual, the code can be found <a href="" target="_blank">here</a>, in the &#8220;demuxer&#8221; branch. The most relevant files are <a href="" target="_blank"></a> and <a href="" target="_blank"></a>.</p> <p>Following the style of the sources and sinks, the first is some kind of base class / trait for writing arbitrary demuxers in Rust. It&#8217;s rather unfinished at this point though, just enough to get something running. All the FLV specific code is in the second file, and it&#8217;s also very minimal for now. All it can do is to play <a href="" target="_blank">one specific file</a> (or hopefully all other files with the same audio/video codec combination).</p> <p>As part of all this, I also wrote bindings for GStreamer&#8217;s <a href="" target="_blank">buffer abstraction</a> and a <a href="" target="_blank">Rust-rewrite of the GstAdapter</a> helper type. Both showed Rust&#8217;s strengths quite well, the buffer bindings by being able to express various concepts of the buffers in a compiler-checked, safe way in Rust (e.g. ownership, reability/writability), the adapter implementation by being so much shorter (it&#8217;s missing features&#8230; but still).</p> <p>So here we are, this can already play one specific file (at least) in any GStreamer based playback application. But some further work is necessary, for which I hopefully have some time in the near future. Various important features are still missing (e.g. other codecs, metadata extraction and seeking), the code is rather proof-of-concept style (stringly-typed media formats, lots of <em>unimplemented!()</em> and <em>.unwrap()</em> calls). But it shows that writing media handling elements in Rust is definitely feasible, and generally seems like a good idea.</p> <p>If only we had Rust already when all this media handling code in GStreamer was written!</p> <h3 id="state-handling">State Handling</h3> <p>Another reason why all this took a bit longer than expected, is that I experimented a bit with expressing the state of the demuxer in a more clever way than what we usually do in C. If you take a look at the <a href="" target="_blank">GstFlvDemux struct definition</a> in C, it contains about 100 lines of field declarations. Most of them are only valid / useful in specific states that the demuxer is in. Doing the same in Rust would of course also be possible (and rather straightforward), but I wanted to try to do something better, especially by making invalid states unrepresentable.</p> <p>Rust has this great concept of enums, also known as tagged unions or sum types in other languages. These are not to be confused with C enums or unions, but instead allow multiple variants (like C enums) with fields of various types (like C unions). But all of that in a type-safe way. This seems like the perfect tool for representing complicated state and building a state machine around it.</p> <p>So much for the theory. Unfortunately, I&#8217;m not too happy with the current state of things. It is like this mostly because of Rust&#8217;s ownership system getting into my way (rightfully, how would it have known additional constraints I didn&#8217;t know how to express?).</p> <h4>Common Parts</h4> <p>The first problem I ran into, was that many of the states have common fields, e.g.</p><pre class="crayon-plain-tag">enum State { ... NeedHeader, HaveHeader {header: Header, to_skip: usize }, Streaming {header: Header, audio: ... }, ... }</pre><p></p> <p>When writing code that matches on this, and that tries to move from one state to another, these common fields would have to be moved. But unfortunately they are (usually) borrowed by the code already and thus can&#8217;t be moved to the new variant. E.g. the following fails to compile</p> <p></p><pre class="crayon-plain-tag">match self.state { ... State::HaveHeader {header, to_skip: 0 } => { self.state = State::Streaming {header: header, ...}; }, }</pre><p></p> <h4>A Tree of States</h4> <p>Repeating the common parts is not nice anyway, so I went with a different solution by creating a tree of states:</p> <p></p><pre class="crayon-plain-tag">enum State { ... NeedHeader, HaveHeader {header: Header, have_header_state: HaveHeaderState }, ... } enum HaveHeaderState { Skipping {to_skip: usize }, Streaming {audio: ... }, }</pre><p></p> <p>Apart from making it difficult to find names for all of these, and having relatively deeply nested code, this works</p> <p></p><pre class="crayon-plain-tag">match self.state { ... State::HaveHeader {ref header, ref mut have_header_state } => { match *have_header_state { HaveHeaderState::Skipping { to_skip: 0 } => { *have_header = HaveHeaderState::Streaming { audio: ...}; } }, }</pre><p></p> <p>If you look at the code however, this causes the code to be much bigger than needed and I&#8217;m also not sure yet how it will be possible nicely to move &#8220;backwards&#8221; one state if that situation ever appears. Also there is still the previous problem, although less often: if I would match on <em>to_skip</em> here by reference (or it was no <em>Copy</em> type), the compiler would prevent me from overwriting <em>have_header</em> for the same reasons as before.</p> <p>So my question for the end: How are others solving this problem? How do you express your states and write the functions around them to modify the states?</p> <h4>Update</h4> <p>I actually implemented the state handling as a <em>State -> State</em> function before (and forgot about that), which seems conceptually the right thing to do. It however has a couple of other <a href="">problems</a>. Thanks for the suggestions so far, it seems like I&#8217;m not alone with this problem at least.</p> <h4>Update 2</h4> <p>I&#8217;ve went a bit closer to the C-style struct definition now, as it makes the code less convoluted and allows me to just get forwards with the code. The current status <a href="" target="_blank">can be seen here now</a>, which also involves further refactoring (and e.g. some support for metadata).</p> Thu, 24 Nov 2016 23:10:52 +0000 GStreamer: GStreamer 1.10.1 stable release (binaries) /news/#2016-11-22T12:10:00Z <p> Pre-built binary images of the 1.10.1 stable release of GStreamer are now available for Windows 32/64-bit, iOS and Mac OS X and Android. </p><p> See <a href="">/releases/1.10/</a> for the full list of changes. </p><p> The builds are available for download from: <a href="">Android</a>, <a href="">iOS</a>, <a href="">Mac OS X</a> and <a href="">Windows</a>. </p> Tue, 22 Nov 2016 12:10:00 +0000 Víctor Jáquez: GStreamer-VAAPI 1.10 (now 1.10.1) <p>A lot of things have happened last month and I have neglected this report. Let me try to fix it.</p> <p>First, the autumn GStreamer Hackfest and the GStreamer Conference 2016 in Berlin.</p> <p>The hackfest&#8217;s venue was the famous <a href="">C-Base</a>, where we talked with friends and colleagues. Most of the discussions were leaned towards the release 1.10. Still, Julien Isorce and I talked about our approaches for DMABuf sharing with downstream; we agreed on to explore how to implement an allocator based on GstDmaBufAllocator. We talked with Josep Torra about the performance he was getting with gstreamer-vaapi encoders (and we already did good progress on this). Also we talked with Nicolas Dufresne about v4l2 and kmssink. And many other informal chats along with beer and pizza.</p> <a href=""><img class="size-medium wp-image-575" src="" alt="GStreamer Hackfest a the C-Base (from @calvaris tweet)" width="580" height="435" /></a>GStreamer Hackfest a the C-Base (from @calvaris <a href="">tweet</a>) <p>Afterwards, in the GStreamer Conference, I talked a bit about the current status of GStreamer-VAAPI and what we can expect from release 1.10. Here&#8217;s the video that the folks from <a href="">Ubicast</a> recorded. You can see <a href="">the slides here</a> too.</p> <p></p> <p>I spent most of the conference with Scott, Sree and Josep tracing the bottlenecks in the frame uploading to VA Intel driver, and trying to answer the questions of the folks who gently approached me. I hope my answers were any helpful.</p> <p>And finally, the first of November, the <a href="">release 1.10 hit the streets</a>!</p> <p>As it is already mentioned in the release notes, these are the main new features you can find in GStreamer-VAAPI 1.10</p> <ul> <li>All decoders have been split, one plugin feature per codec. So far, the available ones, depending on the driver, are: <tt>vaapimpeg2dec</tt>, <tt>vaapih264dec</tt>, <tt>vaapih265dec</tt>, <tt>vaapivc1dec</tt>, <tt>vaapivp8dec</tt>, <tt>vaapivp9dec</tt> and <tt>vaapijpegdec</tt> (which already was split).</li> <li>Improvements when mapping VA surfaces into memory, differenciating from negotiation caps and allocations caps, since the allocation memory for surfaces may be bigger than one that is going to be mapped.</li> <li><tt>vaapih265enc</tt> got support to constant bitrate mode (CBR).</li> <li>Since several VA drivers are unmaintained, we decide to keep a white list with the va drivers we actually test, which is mostly the i915 and, in some degree, gallium from mesa project. Exporting the environment variable <tt>GST_VAAPI_ALL_DRIVERS</tt> disable the white list.</li> <li>The plugin features are registered, in run-time, according to its support by the loaded VA driver. So only the decoders and encoder supported by the system are registered. Since the driver can change, some dependencies are tracked to invalidate the GStreamer registry and reload the plugin.</li> <li>DMA-Buf importation from upstream has been improved, gaining performance.</li> <li><tt>vaapipostproc</tt> now can negotiate through caps the buffer transformations.</li> <li>Decoders now can do reverse playback! But they only shows I frames, because the surface pool is smaller than the required by the GOP to show all the frames.</li> <li>The upload of frames onto native GL textures has been optimized too, keeping a cache of the internal structures for the offered textures by the sink, improving a lot the performance with <tt>glimagesink</tt>.</li> </ul> <p>This is the short log summary since branch 1.8:</p> <pre> 1 Allen Zhang 19 Hyunjun Ko 1 Jagyum Koo 2 Jan Schmidt 7 Julien Isorce 1 Matthew Waters 1 Michael Olbrich 1 Nicolas Dufresne 13 Scott D Phillips 9 Sebastian Dröge 30 Sreerenj Balachandran 1 Stefan Sauer 1 Stephen 1 Thiago Santos 1 Tim-Philipp Müller 1 Vineeth TM 154 Víctor Manuel Jáquez Leal </pre> <p>Thanks to <a href="">Igalia</a>, <a href="">Intel Open Source Technology Center </a>and many others for making GStreamer-VAAPI possible.</p> Sat, 19 Nov 2016 11:31:47 +0000 GStreamer: GStreamer 1.10.1 stable release /news/#2016-11-17T15:00:00Z <p> The GStreamer team is pleased to announce the first bugfix release in the stable 1.10 release series of your favourite cross-platform multimedia framework! </p><p> This release only contains bugfixes and it should be safe to update from 1.10.0. For a full list of bugfixes see <a href=" form&limit=0&list_id=168172&order=bug_id&product=GStreamer&query_format=advanced&resolution=FIXED&target_milestone=1.10.1">Bugzilla</a>. </p><p> See <a href="">/releases/1.10/</a> for the full release notes. </p><p> Binaries for Android, iOS, Mac OS X and Windows will be available shortly. </p><p> Check out the release notes for <a href="">GStreamer core</a>, <a href="">gst-plugins-base</a>, <a href="">gst-plugins-good</a>, <a href="">gst-plugins-ugly</a>, <a href="">gst-plugins-bad</a>, <a href="">gst-libav</a>, <a href="">gst-rtsp-server</a>, <a href="">gst-python</a>, <a href="">gst-editing-services</a>, <a href="">gst-validate</a>, <a href="">gstreamer-vaapi</a>, or <a href="">gst-omx</a>, or download tarballs for <a href="">gstreamer</a>, <a href="">gst-plugins-base</a>, <a href="">gst-plugins-good</a>, <a href="">gst-plugins-ugly</a>, <a href="">gst-plugins-bad</a>, <a href="">gst-libav</a>, <a href="">gst-rtsp-server</a>, <a href="">gst-python</a>, <a href="">gst-editing-services</a>, <a href="">gst-validate</a>, <a href="">gstreamer-vaapi</a>, or <a href="">gst-omx</a>. </p> Thu, 17 Nov 2016 15:00:00 +0000 Jean-François Fortin Tam: Helping Purism structure its messaging <p>I meant to finish writing and posting this a month or two ago, but urgent tasks and life kept getting in the way. I don&#8217;t often talk about client work here, but since this is public-facing ongoing work for a company that is insanely pro-<a href="">Free-Software</a> (not just &#8220;open source&#8221;), a company that ships GNOME3 by default on their laptops (something I have awaited for years), I guess it makes sense to talk about what I&#8217;ve been up to recently.</p> <p>So, for <del>a few weeks</del> three months now, I have been helping <a href="">Purism</a> structure its messaging and get its business in a better shape. Purism is, in itself, a hugely interesting endeavour. Heck, I could go out on a limb and say this venture, alongside the work Endless is doing, is quite possibly one of the most exciting things that has happened to the Free desktop for the past decade—and yet almost nobody heard of it.</p> <p><span id="more-3364"></span></p> <p><a href=""><img class="alignnone size-medium wp-image-3423" src="" alt="2016-06-01" width="500" height="403" /></a></p> <p>Before I can even consider visual branding work (maybe someday—when I get to that point, that would mean things are going <em>really</em> well), there was a fundamental need to fill various gaps in the strategy and daily operations, and to address messaging in a way that simultaneously resonates with:</p> <ul> <li>hardcore Free Software enthusiasts;</li> <li>&#8220;Linux&#8221; (GNU/Linux) users and developers just looking for great ultraportable workhorses;</li> <li>the privacy/security-conscious crowd;</li> <li>the public at large (hopefully).</li> </ul> <p>Purism&#8217;s website and previous campaigns were full of relevant content, but it was disjointed, sometimes outdated, and sometimes a bit imprecise, confusing, or just plain boring. You really had to be abnormally motivated to dig deep and digest a ton of fairly arid content. I set out to fix all that, so I looked at every single page on the website and external campaigns to rearchitect everything back into a solid foundation, something insightful and pleasant to read.</p> <p>There was so much buried and scattered information to analyze, I coudn&#8217;t have done this with computer screens, no matter how large. I decided to print out a summary for easy annotation. Even with super compacted text (small fonts, tight paragraphs, fewer images, etc.) and redundant text passages removed, my &#8220;content review summary&#8221; document of the core content was still a whopping 28 pages in &#8220;US legal&#8221; size (8.5&#215;14 inches). It easily occupies all my office&#8217;s desk space:</p> <p><a href=""><img class="alignnone size-full wp-image-3418" src="" alt="2016-11-16--18.08.06" width="800" height="847" /></a> <a href=""><img class="alignnone wp-image-3425 size-full" src="" width="800" height="929" /></a> <a href=""><img class="alignnone size-full wp-image-3421" src="" alt="2016-11-16--18.14.02" width="1500" height="1010" /></a><a href=""><img class="alignnone size-full wp-image-3426" src="" alt="2016-11-16--18.11.35" width="800" height="600" /></a> <a href=""><img class="alignnone size-full wp-image-3417" src="" alt="2016-11-16--18.08.40" width="1000" height="729" /></a></p> <p>Serious business.</p> <p>Having no time to waste, and having made a very clear list of issues to fix, I applied my proposed changes directly after completing the semantic and structural analysis.</p> <p><img class="alignnone size-full wp-image-3393" src="" alt="Nike Just do it basket" width="480" height="296" /></p> <p>The taxonomy and ontology in the blog section was all wrong, too. So I rethought and reorganized all the tags and categories, to create a reliable system for visitors to browse through. For example, I made the tags go from what a noisy mess (pictured on the left) to something much more meaningful (on the right):</p> <p><a href=""><img class="alignnone size-thumbnail wp-image-3391" src="" alt="before" width="150" height="150" /></a><a href=""><img class="alignnone size-thumbnail wp-image-3390" src="" alt="after" width="150" height="150" /></a></p> <p>The website overhaul led to the new &#8220;Why Purism?&#8221; section, the new Global Shipping Status page, a much nicer stylesheet (my eyes were bleeding trying to read anything on the website and its blog), URLs that actually make sense, and a rethought product discovery and purchase navigation flow. Some more details on the overall changes can be found <a href="">here</a>. Hopefully, the current state of the website is much more enticing now. It may not be perfect (feedback welcome) but it&#8217;s a good start.</p> <p>I also paid attention to a bunch of papercuts when it comes to interacting with the community. For example, blog posts lacked a relatable face, so I added avatars for everybody. Forums had an absolutely horrible mail notification system, so I fixed that. People were unsure about warranties, delivery dates, etc., so I made sure to help clarify those as best as possible throughout the website. That sort of thing.</p> <p>The next big steps, in my view:</p> <ul> <li>Helping ensure the business&#8217; viability, beyond just &#8220;survival&#8221;. Running a &#8220;normal&#8221; hardware manufacturing business is already fairly complicated from the get go—so making an independent &#8220;organically grown&#8221; one <em>profitable</em>, that is <em>much</em> harder. Purism is at a turning point right now, and the coming months will be a real test of the feasibility of reconciling Free Software with modern hardware. Most of us have been waiting so long for this, failure is not an option.</li> <li>Improving the customer experience.</li> <li>Improving the product design.</li> </ul> <p>By the way, I have been testing out a &#8220;Librem 15&#8221; laptop extensively for about two months, and I am hoping to publish a <em>very</em> detailed review and feedback video soon (I don&#8217;t do superficial/half-assed reviews, hence the delay). Stay tuned!</p> <a href=""><img class="size-large wp-image-3379" src="" alt="The test unit I have transplanted my testing drive into" width="584" height="374" /></a>The review unit I have transplanted my (ThinkPad&#8217;s) testing drive into for further testing (this is why you can see the <a href="">dash to dock</a> extension active) Thu, 17 Nov 2016 05:52:50 +0000 Bastien Nocera: Lyon GNOME Bug day #1, Last Friday, both a <a href="">GNOME bug day</a> and a <a href="">bank holiday</a>, a few of us got together to squash some bugs, and discuss GNOME and GNOME technologies.<br /><br /><a href="">Guillaume</a>, a new comer in our group, tested the <a href="">captive portal support for NetworkManager and GNOME</a> in Gentoo, and <a href="">added instructions on how to enable it to their Wiki</a>. He also tested a <a href="">gateway related configuration problem</a>, the patch for which I merged after a code review. Near the end of the session, he also rebuilt WebKitGTK+ to test why Google Docs was not working for him anymore in <a href="">Web</a>. And nobody believed that he could build it that quickly. Looks like opinions based on past experiences are quite hard to change.<br /><br /><a href="">Mathieu</a> worked on <a href="">removing jhbuild's .desktop file</a> as nobody seems to use it, and it was creating the Sundry category for him, in gnome-shell. He also spent time looking into the tracker blocker that is <a href="">Mozilla's Focus</a>, based on <a href="">disconnectme's block lists</a>. It's not as effective as <a href="">uBlock</a> when it comes to blocking adverts, but the memory and performance improvements, and the slow churn rate, could make it a good default blocker to have in <a href="">Web</a>.<br /><br />Haïkel looked into using <a href="">Emeus</a>, potentially the new GTK+ 4.0 layout manager, to implement the <a href="">series properties page</a>&nbsp;for Videos.<br /><br />Finally, I added <a href="">Bolso</a> to jhbuild, and struggled to get gnome-online-accounts/gnome-keyring to behave correctly in my installation, as the application just did not want to log in properly to the service. I also discussed <a href="">Fedora's privacy policy</a> (inappropriate for Fedora Workstation, as it doesn't cover the services used in the default installation), a potential design for Flatpak support of joypads and removable devices in general, as well as the <a href="">future design of the Network panel</a>. Tue, 15 Nov 2016 09:48:56 +0000 (Bastien Nocera) Christian Schaller: Mp3 support now coming to Fedora Workstation 25 <p>So, in Fedora Workstation 24 we added H264 support through OpenH264. In Fedora Workstation 25 I am happy to tell you all that we are taking another step in improving our codec support by adding support for mp3 playback. I know this has been a big wishlist item for a long time for a lot of people so I am really happy that we are finally in a position to fulfil that wish. You should be able to download the mp3 plugin on day 1 through GNOME Software or through the missing codec installer in various GStreamer applications. For Fedora Workstation 26 I would not be surprised if we decide to ship it on the install media. </p> <p>Fo the technically inclined out there, our initial enablement is through the <a href="">mpeg123</a> library and corresponding <a href="">GStreamer</a> plugin. The main reason we choose this library over all the others available out there was a combination of using the same license as GStreamer (lgpl v2) and being a well established library used by a lot of different applications already. There might be other mp3 decoders added in the future depending on interest in and effort by the community. So get ready to install Fedora Workstation 25 when its released soon and play some tunes <img src="" alt=":)" class="wp-smiley" /></p> <p>P.S. To be 110% clear we will not be adding encoding support at this time.</p> Thu, 10 Nov 2016 19:34:39 +0000 Arun Raghavan: GStreamer and Synchronisation Made Easy <p>A lesser known, but particularly powerful feature of GStreamer is our ability to play media synchronised across devices with fairly good accuracy.</p> <p>The way things stand right now, though, achieving this requires some amount of fiddling and a reasonably thorough knowledge of how GStreamer&#8217;s synchronisation mechanisms work. While we have had <a href="">some</a> <a href="">excellent</a> <a href="">talks</a> about these at previous GStreamer conferences, getting things to work is still a fair amount of effort for someone not well-versed with GStreamer.</p> <p>As part of my work with the <a href="">Samsung <acronym title="Open Source Group">OSG</acronym></a>, I&#8217;ve been working on addressing this problem, by wrapping all the complexity in a library. The intention is that anybody who wants to implement the ability for different devices on a network to play the same stream and have them all synchronised should be able to do so with a few lines of code, and the basic know-how for writing GStreamer-based applications.</p> <p>I&#8217;ve started work on this already, and you can find the code in the creatively named <a href="">gst-sync-server repo</a>.</p> <h2>Design and API</h2> <p>Let&#8217;s make this easier by starting with a picture …</p> <p><a href=""><img /></a></p> <p>Let&#8217;s say you&#8217;re writing a simple application where you have two ore more devices that need to play the same video stream, in sync. Your system would consist of two entities:</p> <ul> <li><p>A server: this is where you configure what needs to be played. It instantiates a <code>GstSyncServer</code> object on which it can set a URI that needs to be played. There are other controls available here that I&#8217;ll get to in a moment.</p></li> <li><p>A client: each device would be running a copy of the client, and would get information from the server telling it what to play, and what clock to use to make sure playback is synchronised. In practical terms, you do this by creating a <code>GstSyncClient</code> object, and giving it a <code>playbin</code> element which you&#8217;ve configured appropriately (this usually involves at least setting the appropriate video sink that integrates with your UI).</p></li> </ul> <p>That&#8217;s pretty much it. Your application instantiates these two objects, starts them up, and as long as the clients can access the media URI, you magically have two synchronised streams on your devices.</p> <h3>Control</h3> <p>The keen observers among you would have noticed that there is a control entity in the above diagram that deals with communicating information from the server to clients over the network. While I have currently implemented a simple TCP protocol for this, my goal is to abstract out the control transport interface so that it is easy to drop in a custom transport (Websockets, a REST API, whatever).</p> <p>The actual sync information is merely a structure marshalled into a JSON string and sent to clients every time something happens. Once your application has some media playing, the next thing you&#8217;ll want to do from your server is control playback. This can include</p> <ul> <li>Changing what media is playing (like after the current media ends)</li> <li>Pausing/resuming the media</li> <li>Seeking</li> <li>“Trick modes” such as fast forward or reverse playback</li> </ul> <p>The first two of these work already, and seeking is on my short-term to-do list. Trick modes, as the name suggets, can be a bit more tricky, so I&#8217;ll likely get to them after other things are done.</p> <h2>Getting fancy</h2> <p>My hope is to see this library being used in a few other interesting use cases:</p> <ul> <li><p>Video walls: having a number of displays stacked together so you have one giant display &#8212; these are all effectively playing different rectangles from the same video</p></li> <li><p>Multiroom audio: you can play the same music across different speakers in a single room, or multiple rooms, or even group sets of speakers and play different media on different groups</p></li> <li><p>Media sharing: being able to play music or videos on your phone and have your friends be able to listen/watch at the same time (a silent disco app?)</p></li> </ul> <h2>What next</h2> <p>At this point, the outline of what I think the API should look like is done. I still need to create the transport abstraction, but that&#8217;s pretty much a matter of extracting out the properties and signals that are part of the existing TCP transport.</p> <p>What I would like is to hear from you, my dear readers who are interested in using this library &#8212; does the API look like it would work for you? Does the transport mechanism I describe above cover what you might need? There is <a href="">example code</a> that should make it easier to understand how this library is meant to be used.</p> <p>Depending on the feedback I get, my next steps will be to implement the transport interface, refine the API a bit, fix a bunch of FIXMEs, and then see if this is something we can include in <code>gst-plugins-bad</code>.</p> <p>Feel free to comment either on the Github <a href="">repository</a>, on this blog, or via email.</p> <p>And don&#8217;t forget to watch this space for some videos and measurements of how GStreamer synchronised fares in real life!</p> Fri, 04 Nov 2016 10:16:45 +0000 GStreamer: GStreamer 1.10.0 stable release (binaries) /news/#2016-11-02T16:10:00Z <p> Pre-built binary images of the 1.10.0 stable release of GStreamer are now available for Windows 32/64-bit, iOS and Mac OS X and Android. </p><p> See <a href="">/releases/1.10/</a> for the full list of changes. </p><p> The builds are available for download from: <a href="">Android</a>, <a href="">iOS</a>, <a href="">Mac OS X</a> and <a href="">Windows</a>. </p> Wed, 02 Nov 2016 16:10:00 +0000 GStreamer: GStreamer 1.10.0 stable release /news/#2016-11-01T16:30:00Z <p> The GStreamer team is proud to announce a new major feature release in the stable 1.x API series of your favourite cross-platform multimedia framework! </p><p> As always, this release is again packed with new features, bug fixes and other improvements. </p><p> See <a href="">/releases/1.10/</a> for the full list of changes. </p><p> Binaries for Android, iOS, Mac OS X and Windows will be provided shortly after the source release by the GStreamer project during the stable 1.10 release series. </p><p> Check out the release notes for <a href="">GStreamer core</a>, <a href="">gst-plugins-base</a>, <a href="">gst-plugins-good</a>, <a href="">gst-plugins-ugly</a>, <a href="">gst-plugins-bad</a>, <a href="">gst-libav</a>, <a href="">gst-rtsp-server</a>, <a href="">gst-python</a>, <a href="">gst-editing-services</a>, <a href="">gst-validate</a>, <a href="">gstreamer-vaapi</a>, or <a href="">gst-omx</a>, or download tarballs for <a href="">gstreamer</a>, <a href="">gst-plugins-base</a>, <a href="">gst-plugins-good</a>, <a href="">gst-plugins-ugly</a>, <a href="">gst-plugins-bad</a>, <a href="">gst-libav</a>, <a href="">gst-rtsp-server</a>, <a href="">gst-python</a>, <a href="">gst-editing-services</a>, <a href="">gst-validate</a>, <a href="">gstreamer-vaapi</a>, or <a href="">gst-omx</a>. </p> Tue, 01 Nov 2016 16:30:00 +0000 Christian Schaller: Hybrid Graphics and Fedora Workstation 25 <p>When we started the Fedora Workstation effort one thing we wanted to do was to drain the proverbial swamp of make sure that running Linux on a laptop is a first rate experience. As you see from my last blog entry we have been working on building a team dedicated to that task. There are many facets to this effort, but one that we kept getting asked about was sorting out hybrid graphics. So be aware that some of this has been covered in previous blog entries, but I want to be thorough here. So this blog will cover the big investments of time and effort we are putting into Wayland and X Windows, GNOME Shell and Nouveau (the open source driver for NVidia GPU hardware). </p> <p>A big part of this story is also that we are trying to make it easier to run the NVidia binary driver on Fedora in general which is actually a fairly large undertaking as there are a lot challenges with trying to do that without requiring any manual configuration from the user and making sure it works fully in the hybrid graphics usecase. This is one of the most requested areas of improvement for Fedora. Many users are trying to run Fedora on their existing laptops or other hardware. Rather than allow this gap to be a reason for people to not run Fedora, we want to provide a better experience. Of course users will have the freedom to make their own choice about installing these drivers, using information provided by Software.</p> <p>Hybrid graphics is the term used for when you have a laptop with two GPUs, usually one Intel and one NVidia GPU, but there are also some laptops available which comes with Intel/AMD CPU + AMD GPU. The purpose of hybrid graphics is to have your (Intel) Integrated GPU be your &#8216;day to day GPU&#8217; for running your desktop yet not drawing to much power, but then if you want to for instance play a game on your system you can activate your secondary GPU which got a lot more power, but which also draws more electricity. </p> <p>What complicated this even more was the fact that most users who wanted to use this setup wanted to use it in combination with the binary NVidia driver, in order to get top performance from their second GPU, which is not surprising since that is the whole point of switching to it. The main challenge with this was that the Mesa and binary NVidia driver both provided an OpenGL implementation that due to the way things works in the X Window System ended up overwriting each other, basically breaking the overwritten driver. The solution for this was a system called <a href="">Bumblebee</a> which employed some clever hacks to work around this issue, but Bumblebee is a solution to a problem we shouldn&#8217;t have to begin with. </p> <p>Of course dealing with the OpenGL stack wasn&#8217;t the only challenge here. For instance different display outputs ended up being connected to different GPUs, with one of the most common setups being that the internal screen is connected to your Intel GPU and your external HDMI and DisplayPort connections are connected to your NVidia or AMD GPU. One some systems there where hardware connectors allowing you to display using either GPU to any screen, but a setup we see becoming more common is that the drivers for both card needs to be initialized in all cases, allowing the display connected GPU to slave itself to the other GPU to allow rendering on one GPU and displaying on the other. Within Mesa this is fairly straightforward to handle, so if both Intel and NVidia as rendering using Mesa things are fairly straightforward. What makes this a lot more challenging here is the NVidia binary driver, so once you install the binary driver we need to find a way to bridge and interoperate between these two stacks.</p> <p>And of course that is just the highlights, like anything complex like this there are a long laundry list of items from the point where you can checkbox having a feature to it working really well and seamless. </p> <p><strong>What to expect in Fedora Workstation 25</strong><br /> Lets start with a word of caution here, we are working on a tight deadline here to get all the pieces lined up, so I can not be 100% sure what will make it for the day of release. What we are confident about though is that we will have all the low level plumbing sorted so that we can improve this over the course of the Fedora 25 lifecycle. As always I hope the community will join us in testing and developing this, to ensure we have even the corner cases worked out for Fedora Workstation 26.</p> <p>The initial step we took and this is quite some time ago was that <a href="">Dave Airlie</a> worked on making sure we could handle two GPUs within the Mesa stack. As a bit of wordplay on the NVidia solution being called Optimus the open source Mesa solution was named Prime. This work basically allowed you to use Mesa for your Intel card and use Mesa (Nouveau) for your NVidia card and launch applications on the NVidia card while the screen was connected to the Intel card. You can choose which one is used by setting the DRI_PRIME=1 environment variable.</p> <p>The second step was Adam Jackson collaborating with NVidia on something called <a href="">libglvnd</a>. Libglvnd stands for GL Vendor Neutral Dispatch and it is basically a dispatch layer that allows your OpenGL calls to be dispatched to more than one OpenGL implementation. Nvidia created this specification and have been steadily working on supporting it in their binary driver, but Adam Jackson stepped up to help review their patches and work on supporting glvnd in the Mesa drivers. This means that for Fedora Workstation 25 you should for the first time ever be able to have both the binary NVidia driver and Mesa installed without any file conflicts. So the X server now has the infrastructure to route your OpenGL call to the correct stack when that DRI_PRIME variable is set. That said there is a major catch here, due to the way things currently work once the NVidia binary driver is installed it expects to be rendering the screen all the time. So we need for the short term to figure out a way to allow it to do that, and in the long run we need to work with NVidia to figure out how the Intel open source driver can collaborate with the Nvidia driver to allow us to only use the Intel driver at times (to save on power for instance). We are still pushing hard on trying to have the short term fix in place, but as I write this we haven&#8217;t fully nailed this down yet.</p> <p>The third step is work that Ben Skeggs has been doing on dealing with the monitor handling here, which includes adding MST support to Nouveau, because a lot of external ports and docking stations have not been working with these hybrid graphics setups due to the external screens all being routed through the NVidia chip. These patches just got accepted upstream and we will be including them in Fedora Workstation 25.</p> <p>The fourth step has been <a href="">work that Hans de Goede has been doing around Prime</a>, fixing the modesetting driver and fixing cursor handling with hybrid graphics. In some sense the work Hans has been doing, and check his blog entry linked, is part of that washlist I talked about, the many smaller items that needs to work smoothly for this to go from &#8216;checkbox works&#8217; to &#8216;actually works&#8217;. He is also working with the DNF team to allow us to control the kernel updates if you use the binary NVidia driver, meaning that we will only bump up your kernel driver when we have the binary NVidia driver module ready too. If for any reason the binary Nvidia driver doesn&#8217;t work we want to have graceful fallback to Nouveau.</p> <p>Fifth step has been Jonas Ådahls work on enabling the binary NVidia driver for Wayland. He has put together a set of patches to be able to support NVidias EGLStreams interface, which means that starting from Fedora Workstation 25 you will be able to use Wayland also with NVidias binary driver.<br /> You can see his work in progress <a href="">patches here</a>. Jonas will also be looking at implementing hybrid graphics support in Wayland, so we ensure that also for this usecase Wayland is on par with X for Fedora Workstation 26.</p> <p>The sixth step has been work done by Bastien Nocera to ensure we expose this functionality in the user interface. The idea is that you should be able to configure on a per application basis which GPU they are being launched on. It also means that you can now get all your GPU information in the GNOME about screen also when you have dual-GPU systems. More details in his <a href="">blog post here</a>.<br /> <a href=""><img src="" alt="gnome-shell-discrete-gpu-menu" width="320" height="165" class="aligncenter size-full wp-image-2614" /></a>Choosing discete GPU in GNOME Shell</p> <p>The seventh step is the work that Simone Caronni from Negativo17 has been doing on working with us on packaging the binary NVidia driver in a way that makes all this work. You can find his repo <a href="">here on Negativo17</a>, but Simone has also been working with Kalev Lember and Richard Hughes to ensure the driver show up correctly in GNOME Software once you have the repository enabled.<br /> <img src="" width="768" height="537" alt="NVidia driver in GNOME Software" class="size-full" />NVidia driver in GNOME Software<br /> The plan is to offer the driver in Fedora Workstation 25 as third party software, but we haven&#8217;t yet made the formal proposal due to wanting to be sure we ironed out all important bugs first, both on our side and on NVidia side. </p> <p>So as you can see from all of this there are a lot of components involved and since we are trying to support both X and Wayland with all of this the job hasn&#8217;t exactly been easy. There are for sure some things that will not be ready in time for Fedora Workstation 25, for instance if you want to use the binary NVidia driver we will not be able to make that work with XWayland. So native Wayland applications will be fine, including games using SDL on top of Wayland, but due to how the stack is architected we haven&#8217;t gotten around to implementing bridging OpenGL from Xwayland down to the binary NVidia driver. With Nouveau it should work however. This corner case we will have to figure out for Fedora Workstation 26, but for now we have decided that if we detect a Optimus system we will default you to X instead of Wayland.</p> <p><b>Get involved</b><br /> As always we love for more people to join us in this effort and one good way to get started with that is to join our <a href="">Hybrid graphics test day on Thursday the 3rd of November</a>!</p> Tue, 01 Nov 2016 14:05:00 +0000 Bastien Nocera: Flatpak cross-compilation support: Epilogue, You might remember my <a href="">attempts at getting an easy to use cross-compilation for ARM applications</a> on my x86-64 desktop machine.<br /><br />With Fedora 25 approaching, I'm happy to say that the necessary changes to integrate the feature have now rolled into Fedora 25.<br /><br />For example, to compile the <a href="">GNU Hello Flatpak</a> for ARM, you would run:<br /><br /><div><pre><span>$ flatpak install gnome org.freedesktop.Platform/arm org.freedesktop.Sdk/arm</span><br /><span>Installing: org.freedesktop.Platform/arm/1.4 from gnome</span><br /><span>[...]</span><br /><span>$ sudo dnf install -y qemu-user-static</span><br /><span>[...]</span><br /><span>$ TARGET=arm ./</span><br /></pre></div><div><br />For other applications, add the <span>--arch=arm</span> argument to the <span>flatpak-builder</span> command-line.</div><div><br /></div><div>This example also works for 64-bit ARM with the architecture name <span>aarch64</span>.</div> Mon, 31 Oct 2016 12:00:03 +0000 (Bastien Nocera) Reynaldo Verdejo: Wayland uninstalled. The easy way, <div><br /></div><div><span>This article is an adaptation of another one first posted </span><span>on the&nbsp;</span><a href="">Samsung Open Source Group blog</a><span>.</span></div><div><span><br /></span></div><div><a href=""><img border="0" src="" /></a>I recently started looking at some GStreamer &amp; Wayland integration issues and, as everyone would, commenced by trying to setup a Wayland development environment.</div><div><br /></div><div class="separator"><a href=""><img border="0" height="200" src="" width="200" /></a></div><div>Before getting my feet wet though, I decided to have a chat about&nbsp;this&nbsp;with <a href="" target="_blank">Derek Forman</a>, OSG's&nbsp;resident Wayland expert. This isn't surprising because on our team, pretty much every task starts by having a conversation with one of our field specialists. The idea is to save time, as you might have guessed.</div><div><br /></div><div>This time around I was looking for a fairly trivial piece of info:</div><div><strong><br /></strong></div><div><strong>Me -&nbsp;</strong>"Hey Derek, I have Wayland installed on my distro for some reason - I don't really want to take a look at now - and I would like to setup an upstream Wayland environment without messing it up. Do you have some script like GStreamer's&nbsp;<a href="">gst-uninstalled</a>&nbsp;so I can perform this feat without messing with my local install?</div><div><br /></div><div><strong>Derek&nbsp;</strong>- "Hey Reynaldo, No."</div><div><br /></div><div>He said something else I forgot already. Guess he was ranting as usual. Anyhow, the conversation continued with me taking care of the usual I-forgot-to-ask-how-you-were-doings, and later, I'm not sure who's idea it was but we decided to try upstreaming a </div>wl_uninstalled script --The name had to match existing infrastructure, but that's a boring story you don't want to know about-- that, like gst-uninstalled, would enable developers to quickly setup and use a build/run-time environment consisting of an uninstalled set of interdependent Wayland repos. The idea is not new, but it's still extremely useful. Sure, you can perform similar feats by littering your local disk with custom-prefix installs of pretty much anything, but hey, it is bad taste OK? and you will waste a lot of time, at the very least once.<br /><div><br /></div><div>As if the foreword wasn't boring enough and to continue trying your resilience, here are the instructions to use the thing we come up with:</div><h2>First the What</h2><div>Essentially, wl_uninstalled is a helper script that automates the setup needed to build and work with an uninstalled Wayland/Weston environment comprised at least of the&nbsp;<a href="">wayland, wayland-protocols, libinput, and weston repositories</a>. The provides a shell environment where all build and run-time dependencies are resolved so the uninstalled versions of these modules take precedence.</div><h2>Then the How</h2><div>I'll use Weston as an example although other Wayland-based projects should work as well.</div><div>Edit a local copy of the script to make $WLD point to the base directory where the repositories are located, make sure to use the absolute path. Then, after executing the script, issue the following commands to get everything built and Weston running from the uninstalled environment:</div><pre class="">cd $WLD<br />for i in wayland wayland-protocols libinput weston; do cd $i &amp;&amp; ./ &amp;&amp; make &amp;&amp; cd ..; done<br />weston &amp;</pre><div><br /></div><div>This work is now upstream, you can see the discussions by searching for 'uninstalled' in the&nbsp;<a href="">Wayland mailing lists archive</a>. As an added bonus, it turned out that fiddling with this uninstalled environment mambo-jambo lead to the uncovering of a few additional broken pieces of buildsystem/pkg-config cruft that we fixed too. 'Cause we are good as that see ;). Not everything has been merged though. A word on that at the end.</div><div><br /></div><div>Back to my own use case --GStreamer / Wayland integration work-- now I can simply call the wl_uninstalled script before gst-uninstalled and I'm all set. My uninstalled GStreamer environment will use my (in-turn) uninstalled Wayland/Weston environment making debugging, fixing and integrating against the upstream version of both projects a breeze --Not really but still--, and my filesystem more FHS-compliant. But then again, who cares about that; you probably just want all this to be quick and painless. Hopeless.</div><div><br /></div><div><strong>Note:</strong></div><div><br /></div><div>While all patches for this are on Wayland's public mailing list, some have not been merged yet. If you want to try the procedure described here you will need to manually apply the missing ones till they find their way into the official repository:</div><ul><li><a href="">wayland</a></li><li><a href="">weston</a></li></ul><div>The actual script is in the <a href="" target="_blank">wayland-build-tools</a> repository now.</div> Fri, 28 Oct 2016 17:14:39 +0000 (reynaldo) Bastien Nocera: Dual-GPU integration in GNOME, <div class="separator"></div>Thanks to the work of <a href="">Hans de Goede</a> and many others, dual-GPU (aka <a href="">NVidia Optimus</a> or <a href="">AMD Hybrid Graphics</a>) support works better than ever in Fedora 25.<br /><br />On my side, I picked up some work I originally did for Fedora 24, but ended up being blocked by hardware support. This brings better integration into GNOME.<br /><br />The <i>Details</i> Settings panel now shows which video cards you have in your (most likely) laptop.<br /><br /><div class="separator"><a href=""><img border="0" height="261" src="" width="400" /></a></div><div class="separator"><i>dual-GPU Graphics</i></div><div class="separator"><br /></div><div class="separator">The second feature is what Blender and 3D video games users have been waiting for: a contextual menu item to launch the application on the more powerful GPU in your machine.</div><div class="separator"><br /></div><div class="separator"><a href=""><img border="0" height="165" src="" width="320" /></a></div><div class="separator"><i>Mooo Powaa!</i></div><div class="separator"><br /></div><div class="separator">This demonstration uses a&nbsp;<a href="">slightly modified</a> <a href="">GtkGLArea</a>&nbsp;example, which shows which of the GPUs is used to render the application in the title bar.</div><div class="separator"><br /></div><div class="separator"></div><div class="separator"></div><div class="separator"><a href=""><img border="0" height="320" src="" width="200" /></a></div><div class="separator"><i>on the integrated GPU</i></div><div class="separator"><br /></div><div class="separator"><a href=""><img border="0" height="320" src="" width="200" /></a></div><div class="separator"><i>on the discrete GPU</i></div><div class="separator"><br /></div><div class="separator"><b>Behind the curtain</b></div><div class="separator"><br /></div><div class="separator">Behind those 2 features, we have a <a href="">simple D-Bus service</a>, which runs automatically on boot, and stays running to offer a single property (<span>HasDualGpu</span>) that system components can use to detect what UI to present. This requires the "<a href="">switcheroo</a>" driver to work on the machine in question.</div><div class="separator"><br /></div><div class="separator">Because of the way applications are launched on the discrete GPU, we cannot currently support D-Bus activated applications, but GPU-heavy D-Bus-integrated applications are few and far between right now.</div><div class="separator"><br /></div><div class="separator"><b>Future plans</b></div><div class="separator"><br /></div><div class="separator">There's plenty more to do in this area, to polish the integration. We might want applications to tell us whether they'd prefer being run on the integrated or discrete GPU, as live switching between renderers is still something that's out of the question on Linux.</div><div class="separator"><br /></div><div class="separator">Wayland dual-GPU support, as well as support for the proprietary NVidia drivers are also things that will be worked on, probably by my colleagues though, as the graphics stack really isn't my field.</div><div class="separator"><br /></div><div class="separator">And if the hardware becomes more widely available, we'll most certainly want to support hardware with hotpluggable graphics support (whether gaming laptop "power-ups" or workstation docks).</div><div class="separator"><br /></div><div class="separator"><b>Availability</b></div><div class="separator"><br /></div><div class="separator">All the patches necessary to make this work are now available in GNOME git (targeted at GNOME 3.24), and backports are integrated in Fedora 25, due to be released shortly.</div> Wed, 26 Oct 2016 14:37:50 +0000 (Bastien Nocera) Jean-François Fortin Tam: Pour en finir avec la congestion nasale: comment effectuer le rinçage nasal correctement <p>Ce billet pourrait aussi s&#8217;intituler <em>&#8220;Comment en finir avec les allergies et court-circuiter le rhume, la grippe ou la sinusite&#8221;</em>, car selon mon expérience la congestion nasale est un symptôme qui empire tout le reste au point de prolonger la maladie. Quand on élimine la congestion nasale et la sinusite, on récupère beaucoup plus rapidement (ou, du moins, beaucoup moins péniblement). Regardez comme cet homme semble heureux:</p> <p><img class="alignnone size-full wp-image-3359" src="" alt="neti pot sinus rinsing epic dude" width="800" height="534" /></p> <p><span id="more-3358"></span></p> <p>La dernière fois que j&#8217;ai dû faire appel à cette technique, c&#8217;était il y a trois ans lors d&#8217;une grippe qui m&#8217;avait jeté par terre pendant un mois. Mon ami médecin a alors insisté—à plusieurs reprises—pour que j&#8217;utilise du &#8220;Sinus Rinse&#8221;. Ce truc là avec la boîte chelou:</p> <img class="wp-image-3360 size-full" src="" alt="neilmed sinus rinse" width="580" height="663" />Ça vaut la peine d&#8217;acheter le kit d&#8217;origine pour avoir la bouteille en même temps que les sachets de solution saline &#8220;de grade pharmaceutique&#8221;, mais le &#8220;jetez la bouteille tous les trois mois&#8221; et &#8220;n&#8217;achetez que les sachets de NeilMed&#8221; c&#8217;est de la bullshit commerciale. On trouve facilement des sachets de solution saline &#8220;grade pharmaceutique&#8221; génériques pour moins cher. Pour ce qui est de la bouteille, pas de raison de remplacer ça, surtout que je fais bouillir l&#8217;eau à la casserole avant de la refroidir et la mettre dans la bouteille de rinçage—je ne mets pas la bouteille de rinçage au four à micro-ondes. <p>C&#8217;est donc à ce moment-là que j&#8217;ai découvert le principe du rinçage nasal avec des produits tels que Sinus Rinse, ce qui m&#8217;a permis de me rétablir en quelques jours. Leçon apprise: les médecins ont parfois raison.</p> <p>Trois ans plus tard, cet automne, j&#8217;ai été en proie à des allergies persistantes qui ont résisté à mon rinçage nasal. Après plus d&#8217;un mois de &#8220;comment ça se fait que ça persiste?&#8221;, je découvre que j&#8217;avais oublié les détails de la technique secrète. Ce billet de blog vous partage ma redécouverte.</p> <p>Tout d&#8217;abord pour comprendre le principe général du rinçage nasal, regardez cette vidéo avec cette charmante demoiselle qui vous le présente avec le sourire:</p> <p></p> <p>Ah, quelle femme! Et comme le disait Panoramix en Égypte, &#8220;quel nez!&#8221;</p> <p>Maintenant, voici ce qu&#8217;il faut savoir de plus, concrètement :</p> <ul> <li>Il ne faut pas lésiner sur l&#8217;eau et le sel utilisé : le quart de la bouteille y passe à chaque fois dans chaque narine. Le flot d&#8217;eau doit être continu, il ne s&#8217;agit pas ici de faire des &#8220;petites shot à pression&#8221; comme j&#8217;avais tendance à faire.</li> <li>Il est normal que vous sentiez qu&#8217;une partie se rend dans votre gorge. Si ça vous sort par la bouche, toutefois, c&#8217;est que vous n&#8217;êtes pas assez penché.</li> <li>Dans le cas d&#8217;un rinçage &#8220;d&#8217;urgence&#8221; (ex: après des jours/semaines de congestion), j&#8217;ai quelques astuces plus radicales. Identifions &#8220;A&#8221; la narine où on introduit la bouteille, et &#8220;B&#8221; la narine où la merde sort. <ul> <li>Après avoir rincé en injectant dans la narine &#8220;A&#8221;, attendez quelques secondes, puis plantez votre doigt mouillé (dégueulasse? On s&#8217;en fout, vous êtes tout seul en pantoufles dans la salle de bain) dans la narine &#8220;A&#8221; afin de créer une obstruction partielle, et expirez profondément pour expulser le mucus liquéfié par la narine &#8220;B&#8221;.</li> <li>Utilisez également votre doigt mouillé pour enlever l&#8217;excédent de mucus pendouillant dans la narine B. Hey, j&#8217;ai pas dit que c&#8217;était pour les âmes sensibles.</li> <li>Faire la même chose en inversant A et B.</li> <li>Après avoir ainsi rincé est purgé les deux narines, attendre un moment (quelques minutes, une demie, heure, etc.) pour laisser le mucus se liquéfier davantage (vous le sentirez) puis se moucher de nouveau &#8220;avec la technique du doigt magique&#8221; et répéter le tout depuis le début.</li> </ul> </li> </ul> <p>Si vous faites un grand débloquage d&#8217;un coup, faites-le en matinée et durant la journée, pas avant d&#8217;aller dormir. À savoir, lorsqu&#8217;on dort couché sur le dos le nez a tendance à se boucher, et quand on dort à plat ventre il a tendance à couler.</p> <p>Bref, aux grands mucus les grands moyens.</p> <p><img class="alignnone size-full wp-image-3361" src="" alt="puking rainbows" width="522" height="399" /></p> Wed, 26 Oct 2016 13:29:05 +0000 Jean-François Fortin Tam: 2015 Annual Reports mailed out <p>While the latest GNOME annual reports sold like hotcakes at GUADEC, there is still a need to send some of them by snailmail, like I did <a href="">last year</a>. I was under a pretty big rush from July to mid-October, and since nobody was available to help me determine the list of recipients, I had to wait for the end of the rush to allow myself to spend time devising that list. And so I did, this month.</p> <img class="wp-image-3370 size-full" src="" alt="2016-10-24--15.22.19" width="1000" height="529" />Packaged reports just before I shipped them out. <p><span id="more-3369"></span></p> <p>The shipping cost will be roughly half of last year&#8217;s, but it still turned out to be a surprising amount of copies to be sent out. Thanks to Adelia, Cassandra and Nuritzi (from the Engagement team) for writing postcards to accompany the reports, that saved me quite a bit of time!</p> <img class="size-full wp-image-3373" src="" alt="You thought the annual reports weighted 3 MB? You were wrong. They weigh 3.8 kg." width="1000" height="537" />You thought the annual reports weighted 3 MB?<br />You were wrong. They weigh 3.8 kg. Tue, 25 Oct 2016 15:00:17 +0000 Andy Wingo: An incomplete history of language facilities for concurrency <div><p>I have lately been in the market for better concurrency facilities in <a href="">Guile</a>. I want to be able to write network servers and peers that can gracefully, elegantly, and efficiently handle many tens of thousands of clients and other connections, but without blowing the complexity budget. It's a hard nut to crack.</p><p>Part of the problem is implementation, but a large part is just figuring out what to do. I have often thought that modern musicians must be crushed under the weight of recorded music history, but it turns out in our humble field that's also the case; there are as many concurrency designs as languages, just about. In this regard, what follows is an incomplete, nuanced, somewhat opinionated history of concurrency facilities in programming languages, with an eye towards what I should "buy" for the <a href="">Fibers</a> library I have been tinkering on for Guile.</p><p><center>* * *</center></p><p>Modern machines have the raw capability to serve hundreds of thousands of simultaneous long-lived connections, but it’s often hard to manage this at the software level. Fibers tries to solve this problem in a nice way. Before discussing the approach taken in Fibers, it’s worth spending some time on history to see how we got here.</p><p>One of the most dominant patterns for concurrency these days is “callbacks”, notably in the <a href="">Twisted</a> library for Python and the <a href="">Node.js</a> run-time for JavaScript. The basic observation in the callback approach to concurrency is that the efficient way to handle tens of thousands of connections at once is with low-level operating system facilities like <code>poll</code> or <code><a href="">epoll</a></code>. You add all of the file descriptors that you are interested in to a “poll set” and then ask the operating system which ones are readable or writable, as appropriate. Once the operating system says “yes, file descriptor 7145 is readable”, you can do something with that socket; but what? With callbacks, the answer is “call a user-supplied closure”: a callback, representing the continuation of the computation on that socket.</p><p>Building a network service with a callback-oriented concurrency system means breaking the program into little chunks that can run without blocking. Whereever a program could block, instead of just continuing the program, you register a callback. Unfortunately this requirement permeates the program, from top to bottom: you always pay the mental cost of inverting your program’s control flow by turning it into callbacks, and you always incur run-time cost of closure creation, even when the particular I/O could proceed without blocking. It’s a somewhat galling requirement, given that this contortion is required of the programmer, but could be done by the compiler. We Schemers demand better abstractions than manual, obligatory continuation-passing-style conversion.</p><p>Callback-based systems also encourage unstructured concurrency, as in practice callbacks are not the only path for data and control flow in a system: usually there is mutable global state as well. Without strong patterns and conventions, callback-based systems often exhibit bugs caused by concurrent reads and writes to global state.</p><p>Some of the problems of callbacks can be mitigated by using <a href="">“promises”</a> or other library-level abstractions; if you’re a Haskell person, you can think of this as lifting all possibly-blocking operations into a monad. If you’re not a Haskeller, that’s cool, neither am I! But if your typey spidey senses are tingling, it’s for good reason: with promises, your whole program has to be transformed to return promises-for-values instead of values anywhere it would block.</p><p>An obvious solution to the control-flow problem of callbacks is to use threads. In the most generic sense, a thread is a language feature which denotes an independent computation. Threads are created by other threads, but fork off and run independently instead of returning to their caller. In a system with threads, there is implicitly a scheduler somewhere that multiplexes the threads so that when one suspends, another can run.</p><p>In practice, the concept of threads is often conflated with a particular implementation, <em>kernel threads</em>. Kernel threads are very low-level abstractions that are provided by the operating system. The nice thing about kernel threads is that they can use any CPU that is the kernel knows about. That’s an important factor in today’s computing landscape, where Moore’s law seems to be giving us more cores instead of more gigahertz.</p><p>However, as a building block for a highly concurrent system, kernel threads have a few important problems.</p><p>One is that kernel threads simply aren’t designed to be allocated in huge numbers, and instead are more optimized to run in a one-per-CPU-core fashion. Their memory usage is relatively high for what should be a lightweight abstraction: some 10 kilobytes at least and often some megabytes, in the form of the thread’s stack. There are ongoing efforts to reduce this for some systems but we cannot expect wide deployment in the next 5 years, if ever. Even in the best case, a hundred thousand kernel threads will take at least a gigabyte of memory, which seems a bit excessive for book-keeping overhead.</p><p>Kernel threads can be a bit irritating to schedule, too: when one thread suspends, it’s for a reason, and it can be that user-space knows a good next thread that should run. However because kernel threads are scheduled in the kernel, it’s rarely possible for the kernel to make informed decisions. There are some “user-mode scheduling” facilities that are in development for some systems, but again only for some systems.</p><p>The other significant problem is that building non-crashy systems on top of kernel threads is hard to do, not to mention “correct” systems. It’s an embarrassing situation. For one thing, the low-level synchronization primitives that are typically provided with kernel threads, mutexes and condition variables, are not composable. Also, as with callback-oriented concurrency, one thread can silently corrupt another via unstructured mutation of shared state. It’s worse with kernel threads, though: a kernel thread can be interrupted at any point, not just at I/O. And though callback-oriented systems can theoretically operate on multiple CPUs at once, in practice they don’t. This restriction is <a href="">sometimes touted as a benefit</a> by proponents of callback-oriented systems, because in such a system, the callback invocations have a single, sequential order. With multiple CPUs, this is not the case, as multiple threads can run at the same time, in parallel.</p><p>Kernel threads can work. The Java virtual machine does at least manage to prevent low-level memory corruption and to do so with high performance, but still, even Java-based systems that aim for maximum concurrency avoid using a thread per connection because threads use too much memory.</p><p>In this context it’s no wonder that there’s a third strain of concurrency: shared-nothing message-passing systems like Erlang. Erlang isolates each thread (called <em>processes</em> in the Erlang world), giving each it its own heap and “mailbox”. Processes can spawn other processes, and the concurrency primitive is message-passing. A process that tries receive a message from an empty mailbox will “block”, from its perspective. In the meantime the system will run other processes. Message sends never block, oddly; instead, sending to a process with many messages pending makes it more likely that Erlang will pre-empt the sending process. It’s a strange tradeoff, but it makes sense when you realize that Erlang was designed for network transparency: the same message send/receive interface can be used to send messages to processes on remote machines as well.</p><p>No network is truly transparent, however. At the most basic level, the performance of network sends should be much slower than local sends. Whereas a message sent to a remote process has to be written out byte-by-byte over the network, there is no need to copy immutable data within the same address space. The complexity of a remote message send is O(n) in the size of the message, whereas a local immutable send is O(1). This suggests that hiding the different complexities behind one operator is the wrong thing to do. And indeed, given byte read and write operators over sockets, it’s possible to implement remote message send and receive as a process that serializes and parses messages between a channel and a byte sink or source. In this way we get cheap local channels, and network shims are under the programmer’s control. This is the approach that the Go language takes, and is the one we use in Fibers.</p><p>Structuring a concurrent program as separate threads that communicate over channels is an old idea that goes back to Tony Hoare’s work on <a href="">“Communicating Sequential Processes”</a> (CSP). CSP is an elegant tower of mathematical abstraction whose layers form a pattern language for building concurrent systems that you can still reason about. Interestingly, it does so without any concept of time at all, instead representing a thread’s behavior as a <em>trace</em> of instantaneous events. Threads themselves are like functions that unfold over the possible events to produce the actual event trace seen at run-time.</p><p>This view of events as instantaneous happenings extends to communication as well. In CSP, one communication between two threads is modelled as an instantaneous event, partitioning the traces of the two threads into “before” and “after” segments.</p><p>Practically speaking, this has ramifications in the Go language, which was heavily inspired by CSP. You might think that a channel is just a an <a href="">asynchronous queue</a> that blocks when writing to a full queue, or when reading from an empty queue. That’s a bit closer to the Erlang conception of how things should work, though as we mentioned, Erlang simply slows down writes to full mailboxes rather than blocking them entirely. However, that’s not what Go and other systems in the CSP family do; sending a message on a channel will block until there is a receiver available, and vice versa. The threads are said to “rendezvous” at the event.</p><p>Unbuffered channels have the interesting property that you can <a href=""><code>select</code></a> between sending a message on channel <var>a</var> or channel <var>b</var>, and in the end only one message will be sent; nothing happens until there is a receiver ready to take the message. In this way messages are really owned by threads and never by the channels themselves. You can of course add buffering if you like, simply by making a thread that waits on either sends or receives on a channel, and which buffers sends and makes them available to receives. It’s also possible to add explicit support for buffered channels, as Go, <tt>core.async</tt>, and many other systems do, which can reduce the number of context switches as there is no explicit buffer thread.</p><p>Whether to buffer or not to buffer is a tricky choice. It’s possible to implement singly-buffered channels in a system like Erlang via an explicit send/acknowlege protocol, though it seems difficult to implement completely unbuffered channels. As we mentioned, it’s possible to add buffering to an unbuffered system by the introduction of explicit buffer threads. In the end though in Fibers we follow CSP’s lead so that we can implement the nice <code>select</code> behavior that we mentioned above.</p><p>As a final point, <code>select</code> is OK but is not a great language abstraction. Say you call a function and it returns some kind of asynchronous result which you then have to <code>select</code> on. It could return this result as a channel, and that would be fine: you can add that channel to the other channels in your <code>select</code> set and you are good. However, what if what the function does is receive a message on a channel, then do something with the message? In that case the function should return a channel, plus a continuation (as a closure or something). If <code>select</code> results in a message being received over that channel, then we call the continuation on the message. Fine. But, what if the function itself wanted to <code>select</code> over some channels? It could return multiple channels and continuations, but that becomes unwieldy.</p><p>What we need is an abstraction over asynchronous operations, and that is the main idea of a CSP-derived system called <a href="">“Concurrent ML”</a> (CML). Originally implemented as a library on top of Standard ML of New Jersey by John Reppy, CML provides this abstraction, which in Fibers is called an <em>operation</em><a href="" name="DOCF1"><sup>1</sup></a>. Calling <code>send-operation</code> on a channel returns an operation, which is just a value. Operations are like closures in a way; a closure wraps up code in its environment, which can be later called many times or not at all. Operations likewise can be <em>performed</em><a href="" name="DOCF2"><sup>2</sup></a> many times or not at all; performing an operation is like calling a function. The interesting part is that you can compose operations via the <code>wrap-operation</code> and <code>choice-operation</code> combinators. The former lets you bundle up an operation and a continuation. The latter lets you construct an operation that chooses over a number of operations. Calling <code>perform-operation</code> on a choice operation will perform one and only one of the choices. Performing an operation will call its <code>wrap-operation</code> continuation on the resulting values.</p><p>While it’s possible to implement Concurrent ML in terms of Go’s channels and baked-in <code>select</code> statement, it’s more expressive to do it the other way around, as that also lets us implement other operations types besides channel send and receive, for example timeouts and condition variables.</p><p><hr /><a href="" name="FOOT1"><sup>1</sup></a> CML uses the term <em>event</em>, but I find this to be a confusing name. In this isolated article my terminology probably looks confusing, but in the context of the library I think it can be OK. The jury is out though.</p><p><a href="" name="FOOT2"><sup>2</sup></a> In CML, <em>synchronized</em>.</p><p><center>* * *</center></p><p>Well, that's my limited understanding of the crushing weight of history. Note that part of this article is now in the <a href="">Fibers manual</a>.</p><p>Thanks very much to Matthew Flatt, Matthias Felleisen, and Michael Sperber for pushing me towards CML. In the beginning I thought its benefits were small and complication large, but now I see it as being the reverse. Happy hacking :)</p></div> Wed, 12 Oct 2016 13:45:12 +0000 Thomas Vander Stichele: Puppet/puppetdb/storeconfigs validation issues <p>Over the past year I&#8217;ve chipped away at setting up new servers for <a href="">apestaart</a> and managing the deployment in puppet as opposed to a by now years old manual single server configuration that would be hard to replicate if the drives fail (one of which did recently, making this more urgent).</p> <p>It&#8217;s been a while since I felt like I was good enough at puppet to love and hate it in equal parts, but mostly manage to control a deployment of around ten servers at a previous job.</p> <p>Things were progressing an hour or two here and there at a time, and accelerated when a friend in our collective was launching a new business for which I wanted to make sure he had a decent redundancy setup.</p> <p>I was saving the hardest part for last &#8211; setting up Nagios monitoring with <a href="">Matthias Saou&#8217;s</a> <a href="">puppet-nagios</a> module, which needs External Resources and storeconfigs working.</p> <p>Even on the previous server setup based on CentOS 6, that was a pain to set up &#8211; needing MySQL and ruby&#8217;s ActiveRecord. But it sorta worked.</p> <p>It seems that for newer puppet setups, you&#8217;re now supposed to use something called PuppetDB, which is not in fact a database on its own as the name suggests, but requires another database. Of course, it chose to need a different one &#8211; Postgres. Oh, and PuppetDB itself is in Java &#8211; now you get the cost of two runtimes when you use puppet!</p> <p>So, to add useful Nagios monitoring to my puppet deploys, which without it are quite happy to be simple puppet apply runs from a local git checkout on each server, I now need storedconfigs which needs puppetdb which pulls in Java and Postgres. And that&#8217;s just so a system that handles distributed configuration can actually be told about the results of that distributed configuration and create a useful feedback cycle allowing it to do useful things to the observed result.</p> <p>Since I test these deployments on local vagrant/VirtualBox machines, I had to double their RAM because of this &#8211; even just the puppetdb java server by default starts with 192MB reserved out of the box.</p> <p>But enough complaining about these expensive changes &#8211; at least there was a working <a href="">puppetdb</a> module that managed to set things up well enough.</p> <p>It was easy enough to get the first host monitored, and apart from some minor changes (like updating the default Nagios config template from 3.x to 4.x), I had a familiar Nagios view working showing results from the server running Nagios itself. Success!</p> <p>But all runs from the other vm&#8217;s did not trigger adding any exported resources, and I couldn&#8217;t find anything wrong in the logs. In fact, I could not find /var/log/puppetdb/puppetdb.log at all&#8230;</p> <p><em><strong>fun with utf-8</strong><br /> </em><br /> After a long night of experimenting and head scratching, I chased down a first clue in /var/log/messages saying puppet-master[17702]: Ignoring invalid UTF-8 byte sequences in data to be sent to PuppetDB</p> <p>I traced that down to puppetdb/char_encoding.rb, and with my limited ruby skills, I got a dump of the offending byte sequence by adding this code:</p> <p><code> Puppet.warning "Ignoring invalid UTF-8 byte sequences in data to be sent to PuppetDB"<br />'/tmp/ruby', 'w') { |file| file.write(str) }<br /> Puppet.warning "THOMAS: is here"</code></p> <p>(I tend to use my name in debugging to have something easy to grep for, and I wanted some verification that the File dump wasn&#8217;t triggering any errors)<br /> It took a little time at 3AM to remember where these /tmp files end up thanks to systemd, but once found, I saw it was a json blob with a command to &#8220;replace catalog&#8221;. That could explain why my puppetdb didn&#8217;t have any catalogs for other hosts. But file told me this was a plain ASCII file, so that didn&#8217;t help me narrow it down.</p> <p>I brute forced it by just checking my whole puppet tree:</p> <p><code><br /> find . -type f -exec file {} \; &gt; /tmp/puppetfile<br /> grep -v ASCII /tmp/puppetfile | grep -v git</code></p> <p>This turned up a few UTF-8 candidates. Googling around, I was reminded about how terrible utf-8 handling was in ruby 1.8, and saw information that <a href="">puppet recommended using ASCII only</a> in most of the manifests and files to avoid issues.</p> <p>It turned out to be a config from a webalizer module:</p> <p><code> webalizer/templates/webalizer.conf.erb: UTF-8 Unicode text</code></p> <p>While it was written by a Jesús with a unicode name, the file itself didn&#8217;t have his name in it, and I couldn&#8217;t obviously find where the UTF-8 chars were hiding. <a href="">One StackOverflow post</a> later, I had nailed it down &#8211; UTF-8 spaces!</p> <p><code> 00004ba0 2e 0a 23 c2 a0 4e 6f 74 65 20 66 6f 72 20 74 68 |..#..Note for th|<br /> 00004bb0 69 73 20 74 6f 20 77 6f 72 6b 20 79 6f 75 20 6e |is to work you n|</code></p> <p>The offending character is c2 a0 &#8211; <a href="">the non-breaking space</a></p> <p>I have no idea how that slipped into a comment in a config file, but I changed the spaces and got rid of the error.</p> <p><strong>Puppet&#8217;s error was vague, did not provide any context whatsoever (Where do the bytes come from? Dump the part that is parseable? Dump the hex representation? Tell me the position in it where the problem is?), did not give any indication of the potential impact, and in a sea of spurious puppet warnings that you simply have to live with, is easy to miss</strong>. One down.</p> <p>However, still no catalogs on the server, so still only one host being monitored. What next?</p> <p><em><strong>users, groups, and permissions</strong><br /> </em><br /> Chasing my next lead turned out to be my own fault. After turning off SELinux temporarily, checking all permissions on all puppetdb files to make sure that they were group-owned by puppetdb and writable for puppet, I took the last step of switching to that user role and trying to write the log file myself. And it failed. Huh? And then id told me why &#8211; while /var/log/puppetdb/ was group-writeable and owned by puppetdb group, my puppetdb user was actually in the www-data group.</p> <p>It turns out that I had tried to move some uids and gids around after the automatic assignment puppet does gave different results on two hosts (a problem I still don&#8217;t have a satisfying answer for, as I don&#8217;t want to hard-code uids/gids for system accounts in other people&#8217;s modules), and clearly I did one of them wrong.</p> <p>I think a server that for whatever reason cannot log should simply not start, as this is a critical error if you want a defensive system.</p> <p>After fixing that properly, I now had a puppetdb log file.</p> <p><em><strong>resource titles</strong><br /> </em><br /> Now I was staring at an actual exception:</p> <p><code>2016-10-09 14:39:33,957 ERROR [c.p.p.command] [85bae55f-671c-43cf-9a54-c149cede<br /> c659] [replace catalog] Fatal error on attempt 0<br /> java.lang.IllegalArgumentException: Resource '{:type "File", :title "/var/lib/p<br /> uppet/concat/thomas_vimrc/fragments/75_thomas_vimrc-\" allow adding additional<br /> config through .vimrc.local_if filereadable(glob(\"~_.vimrc.local\"))_\tsource<br /> ~_.vimrc.local_endif_"}' has an invalid tag 'thomas:vimrc-" allow adding additi<br /> onal config through .vimrc.local<br /> if filereadable(glob("~/.vimrc.local"))<br /> source ~/.vimrc.local<br /> endif<br /> '. Tags must match the pattern /\A[a-z0-9_][a-z0-9_:\-.]*\Z/.<br /> at com.puppetlabs.puppetdb.catalogs$validate_resources.invoke(catalogs.<br /> clj:331) ~[na:na]</code></p> <p>Given the name of the command (replace catalog), I felt certain this was going to be the problem standing between me and multiple hosts being monitored.</p> <p>The problem was a few levels deep, but essentially I had code creating fragments of vimrc files using the concat module, and was naming the resources with file content as part of the title. That&#8217;s not a great idea, admittedly, but no other part of puppet had ever complained about it before. Even the files on my file system that store the fragments, which get their filename from these titles, happily stored with a double quote in its name.</p> <p><strong>So yet again, puppet&#8217;s lax approach to specifying types of variables at any of its layers (hiera, puppet code, ruby code, ruby templates, puppetdb) in any of its data formats (yaml, json, bytes for strings without encoding information) triggers errors somewhere in the stack</strong> without informing whatever triggered that error (ie, the agent run on the client didn&#8217;t complain or fail).</p> <p>Once again, puppet has given me plenty of reasons to hate it with a passion, tipping the balance.</p> <p>I couldn&#8217;t imagine doing server management without a tool like puppet. But you love it when you don&#8217;t have to tweak it much, and you hate it when you&#8217;re actually making extensive changes. Hopefully after today I can get back to the loving it part.</p> <p><a href="" title="Flattr" target="_blank"><img src="" alt="flattr this!" /></a></p> Sun, 09 Oct 2016 20:31:08 +0000 Jean-François Fortin Tam: Back from the GNOME Summit <p>In my previous blog post, where I was providing an update on the 2016 GNOME Summit I was organizing in Montréal, I wrote,</p> <blockquote><p>With a change of attendees comes a change of the nature of the event: instead of being an extremely technical, “deep end of the pool” event as it has been in the past, this edition will be focused on newcomers and potential contributors, with presentations and workshops targetted for this purpose.</p></blockquote> <p><span id="more-3350"></span></p> <p>In practice, it turned out to be a great event. It was halfway between the traditional highly technical gathering and the &#8220;event aimed at new contributors&#8221;, with 13 attendees including myself.</p> <p><a href=""><img class="alignnone size-large wp-image-3351" src="" alt="ets-maison-du-logiciel-libre-2015" width="584" height="314" /></a></p> <p>After welcoming participants and opening the event with a rough action plan, I gave a short presentation to set the stage and provide an overview of what GNOME is (a collection of software, a community, and the GNOME <a href="">Foundation</a>). The point was to have an icebreaker talk that would ensure all participants are on the same level of understanding as to who they were dealing with. This eventually morphed into a joint talk with Hubert that looked more like a Saturday <del>Night</del> Live show with two hosts sitting in fancy chairs in front of the audience (wish I had pictures of the two of us taken from the outside to illustrate the &#8220;TV show&#8221; feel).</p> <a href=""><img class="wp-image-3353 size-large" src="" alt="2016-10-08--13.58.13" width="584" height="322" /></a>My co-animator <p>We gradually got people to open up and get involved by probing participants with questions. After all, the chief goal is to motivate people into contributing to the GNOME, showing them the various ways they can <a href="">get involved</a>, how the GNOME Project&#8217;s release cycle works (and where the timing of their contributions comes in) and what <a href="">technologies</a> are involved. We sang the praise of <a href="">Flatpak</a> and <a href="">GNOME Builder</a> (even though he uses Emacs and I use Gedit ;)</p> <p>Our introductory discussion was then followed by a presentation by Guillaume Poirier-Morency on Vala, Meson, pollcore and Valum.</p> <a href=""><img class="wp-image-3354 size-large" src="" alt="2016-10-08--14.50.45" width="584" height="293" /></a>Guillaume presenting <p>David Mirza Ahmad also presented <a href="">Subgraph</a> OS, the problem it aims to solve and how GNOME fits into the picture. Their distro runs a hardened GNOME3 and it&#8217;s quite relevant to our interests. One could have a fun week-end in Manchester with these folks!</p> <a href=""><img class="wp-image-3352 size-large" src="" alt="2016-10-08--15.19.03" width="584" height="328" /></a>David talking about Subgraph, security, and GNOME <p>It seems the Subgraph talk captured attendees&#8217; interest and ended up morphing into a security discussion that took the rest of the afternoon.</p> <p>While I initially thought we wouldn&#8217;t have enough content to fill more than a day, we ended up not having time to do newcomers workshops (short &amp; sweet, as I would say). There was a shared interest among participants to try creating a GNOME community in Montréal to have such meetings more often. So, for starters, I have now created the #gnome-montreal IRC channel on Get your butt over there if you&#8217;re in the area.</p> <p>I would like to thank <a href="">ÉTS</a> and <a href="">ML2</a> for providing us with a nice meeting space for the event, and the <a href="">Subgraph folks</a> for sponsoring dinner on Saturday night (a very nice touch), where lively debates were facilitated by beer and food.</p> Sun, 09 Oct 2016 12:53:03 +0000 GStreamer: GStreamer Core, Plugins, RTSP Server, Editing Services, Python, Validate, VAAPI, OMX 1.10.0 release candidate 1 (1.9.90) (binaries) /news/#2016-10-03T9:40:00Z <p> Pre-built binary images of the 1.9.90 release candidate of GStreamer are now available for Windows 32/64-bit, iOS and Mac OS X and Android. </p><p> The builds are available for download from: <a href="">Android</a>, <a href="">iOS</a>, <a href="">Mac OS X</a> and <a href="">Windows</a>. </p> Mon, 03 Oct 2016 00:00:00 +0000 GStreamer: GStreamer Core, Plugins, RTSP Server, Editing Services, Python, Validate, VAAPI, OMX 1.10.0 release candidate 1 (1.9.90) /news/#2016-09-30T10:00:00Z <p> The GStreamer team is pleased to announce the first release candidate of the stable 1.10 release series. The 1.10 release series is adding new features on top of the 1.0, 1.2, 1.4, 1.6 and 1.8 series and is part of the API and ABI-stable 1.x release series of the GStreamer multimedia framework. </p><p> Binaries for Android, iOS, Mac OS X and Windows will be provided in the next days. </p><p> Check out the release notes for <a href="">GStreamer core</a>, <a href="">gst-plugins-base</a>, <a href="">gst-plugins-good</a>, <a href="">gst-plugins-ugly</a>, <a href="">gst-plugins-bad</a>, <a href="">gst-libav</a>, <a href="">gst-rtsp-server</a>, <a href="">gst-python</a>, <a href="">gst-editing-services</a>, <a href="">gst-validate</a>, <a href="">gstreamer-vaapi</a>, or <a href="">gst-omx</a>, or download tarballs for <a href="">gstreamer</a>, <a href="">gst-plugins-base</a>, <a href="">gst-plugins-good</a>, <a href="">gst-plugins-ugly</a>, <a href="">gst-plugins-bad</a>, <a href="">gst-libav</a>, <a href="">gst-rtsp-server</a>, <a href="">gst-python</a>, <a href="">gst-editing-services</a>, <a href="">gst-validate</a>, <a href="">gstreamer-vaapi</a>, or <a href="">gst-omx</a>. </p> Fri, 30 Sep 2016 10:00:00 +0000 Andy Wingo: is go an acceptable cml? <div><p>Yesterday I tried to summarize the things I know about Concurrent ML, and I came to the <a href="">tentative conclusion</a> that Go (and any Go-like system) was an acceptable CML. Turns out I was both wrong and right.</p><p><b>you were wrong when you said everything's gonna be all right</b></p><p>I was wrong, in the sense that programming against the CML abstractions lets you do more things than programming against channels-and-goroutines. Thanks to Sam Tobin-Hochstadt to pointing this out. As an example, consider a little process that tries to receive a message off a channel, and times out otherwise:</p><pre> func withTimeout(ch chan int, timeout int) (result int) { var timeoutChannel chan int; var msg int; go func() { sleep(timeout) timeoutChannel &lt;- 0 }() select { case msg = &lt;-ch: return msg; case msg = &lt;-timeoutChannel: return 0; } } </pre><p>I think that's the first Go I've ever written. I don't even know if it's syntactically valid. Anyway, I think we see how it should work. We return the message from the channel, unless the timeout happens before.</p><p>But, what if the message is itself a composite message somehow? For example, say we have a transformer that reads a value from a channel and adds 1 to it:</p><pre> func onePlus(in chan int) (result chan int) { var out chan int go func () { out &lt;- 1 + &lt;-in }() return out } </pre><p>What if we do a <tt>withTimeout(onePlus(numbers), 0)</tt>? Assume the timeout fires first and that's the result that <tt>select</tt> chooses. There's still that <tt>onePlus</tt> goroutine out there trying to read from <tt>in</tt> and at some point probably it will succeed, but nobody will read its value. At that point the number just vanishes into the ether. Maybe that's OK in certain domains, but certainly not in general!</p><p>What CML gives you is the ability to express an <i>event</i> (which is kinda like a possibility of sending or receiving a message on a channel) in such a way that we don't run into this situation. Specifically with the <tt>wrap</tt> combinator, we would make an event such that receiving on <tt>numbers</tt> would run a function on the received message and return that as the message value -- which is of course the same as what we have, except that in CML the <tt>select</tt> wouldn't actually read the message off unless it <tt>select</tt>'d that channel for input.</p><p>Of course in Go you could just rewrite your program, so that the <tt>select</tt> statement looks like this:</p><pre> select { case msg = &lt;-ch: return msg + 1; case msg = &lt;-timeoutChannel: return 0; } </pre><p>But here we're operating at a lower level of abstraction; we were forced to intertwingle our concerns of adding 1 and our concerns of timeout. CML is more expressive than Go.</p><p><b>you were right when you said we're all just bricks in the wall</b></p><p>However! I was right in the sense that you can build a CML system on top of Go-like systems (though possibly not Go in particular). Thanks to Vesa Karvonen for <a href="">this comment</a> and the link to their <a href="">proof-of-concept CML implementation in Clojure's <tt>core.async</tt></a>. I understand Vesa also has an implementation in F# as well.</p><p>Folks should read Vesa's code, after reading the Reppy papers of course; it's delightfully short and expressive. The basic idea is that event composition operators like <tt>choose</tt> and <tt>wrap</tt> build up data structures instead of doing things. The <tt>sync</tt> operation then grovels through those data structures to collect a list of channels to pass on to <tt>core.async</tt>'s equivalent of <tt>select</tt>. When <tt>select</tt> returns, <tt>sync</tt> determines which event that chosen channel and message corresponds to, and proceeds to "activate" the event (and, as a side effect, possibly issue NACK messages to other channels).</p><p>Provided you can map from the chosen <tt>select</tt> channel/message back to the event, (something that <tt>core.async</tt> can mostly do, with a caveat; see the code), then you can build CML on top of channels and goroutines.</p><p><b>o/~ yeah you were wrong o/~</b></p><p>On the other hand! One advantage of CML is that its events are not limited to channel sends and receives. I understand that timeouts, thread joins, and maybe some other event types are first-class event kinds in many CML systems. Michael Sperber, current Scheme48 maintainer and functional programmer, tells me that simply wrapping events in channels+goroutines works but can incur a big performance overhead relative to supporting those event types natively, due to the need to make the new goroutine and channel and the scheduling costs. He quotes 10X as the overhead!</p><p>So although CML and Go appear to be inter-expressible, maybe a proper solution will base the simple channel send/receive interface on CML rather than the other way around.</p><p>Also, since these events are now second-class, it must be OK to lose these events, for the same reason that the naïve <tt>withTimeout</tt> could lose a message from <tt>numbers</tt>. This is the case for timeouts usually but maybe you have to think about this more, and possibly provide an infinite stream of the message. (Of course the wrapper goroutine would be collected if the channel becomes unreachable.)</p><p><b>you were right when you said this is the end</b></p><p>I've long wondered how contemporary musicians deal with the enormous, crushing weight of recorded music. I don't really pick any more but hoo am I feeling this now. I think for Guile, I will continue hacking on fibers in a separate library, and I think that things will remain that way for the next couple years and possibly more. We need more experience and more mistakes before blessing and supporting any particular formulation of highly concurrent programming. I will say though that I am delighted that we are able to actually do this experimentation on a library level and I look forward to seeing what works out :)</p><p>Thanks again to Vesa, Michael, and Sam for sharing their time and knowledge; all errors are of course mine. Happy hacking!</p></div> Wed, 21 Sep 2016 21:29:15 +0000 Andy Wingo: concurrent ml versus go <div><p>Peoples! Lately I've been navigating the guile-ship through waters unknown. This post is something of an echolocation to figure out where the hell this ship is and where it should go.</p><p>Concretely, I have been working on getting a nice <a href="">lightweight concurrency system</a> rolling for Guile. I'll write more about that later, but you can think of it as being modelled on Go, though built as a library. (I had previously described it as "Erlang-like", but that's just not accurate.)</p><p>Earlier this year at <a href="">Curry On</a> this topic was burning in my mind and of course when I saw the language-hacker fam there I had to bend their ears. My targets: Matthew Flatt, the amazing boundary-crossing engineer, hacker, teacher, researcher, and implementor of <a href="">Racket</a>, and Matthias Felleisen, the godfather of the PLT research family. I saw them sitting together and I thought, you know what, what can they have to say to each other? These people have been talking together for 30 years right? Surely they are actually waiting for some ignorant dude to saunter up to the PL genius bar, right?</p><p>So saunter I do, saying, "if someone says to you that they want to build a server that will handle 100K or so simultaneous connections on Racket, what abstraction do you tell them to use? Racket threads?" Apparently: yes. A definitive yes, in the case of Matthias, with a pointer to <a href="">Robby Findler's paper on kill-safe abstractions</a>; and still a yes from Matthew with the caveat that for the concrete level of concurrency that I described, you'd have to run tests. More fundamentally, I was advised to look at Concurrent ML (on which Racket's concurrency facilities were based), that CML was much better put together than many modern variants like Go.</p><p>This was very interesting and new to me. As y'all probably know, I don't have a formal background in programming languages, and although I've read a lot of literature, reading things only makes you aware of the growing dimension of the not-yet-read. Concurrent ML was even beyond my not-yet-read horizon.</p><p>So I went back and read a bunch of papers. Turns out Concurrent ML is like Lisp in that it has a tribe and a tightly-clutched history and a diaspora that reimplements it in whatever language they happen to be working in at the moment. Kinda cool, and, um... a bit hard to appreciate in the current-day context when the only good references are papers from 10 or 20 years ago.</p><p>However, after reading a bunch of <a href="">John Reppy papers</a>, here is my understanding of what Concurrent ML is. I welcome corrections; surely I am getting this wrong.</p><p>1. CML is like Go, composed of channels and goroutines. (Forgive the modern referent; I assume most folks know Go at this point.)</p><p>2. Unlike Go, in CML a channel is never buffered. To make a buffered channel in CML, you spawn a thread that manages a buffer between two channels.</p><p>3. Message send and receive operations in CML are built on a lower-level primitive called "events". <tt>(send ch x)</tt> is instead euivalent to <tt>(sync (send-event ch x))</tt>. It's like an event is the derivative of a message send with respect to time, or something.</p><p>4. Events can be combined and transformed using the <tt>choose</tt> and <tt>wrap</tt> combinators.</p><p>5. Doing a <tt>sync</tt> on an event created by <tt>choose</tt> allows a user to build <tt>select</tt> in "user-space", as a library. Cool stuff. So this is what events are for.</p><p>6. There are separate event type implementations for timeouts, channel send/recv blocking operations, file descriptor blocking operations, syscalls, thread joins, and the like. These are supported by the CML implementation.</p><p>7. The early implementations of Concurrent ML were concurrent but not parallel; they did not run multiple "goroutines" on separate CPU cores at the same time. It was only in like 2009 that people started to do CML in parallel. I do not know if this late parallelism has a practical impact on the viability of CML.</p><p><b>ok go</b></p><p>What is the relationship of CML to Go? Specifically, is CML more expressive than Go? (I assume the reverse is not the case, but that would also be an interesting result!)</p><p>There are a few languages that only allow you to <tt>select</tt> over message receives (not sends), but <a href="">Go's <tt>select</tt> doesn't have this limitation</a>, so that's not a differentiator.</p><p>Some people say that it's nice to have events as the common denominator, but I don't get this argument. If the only event under consideration is message send or receive over a channel, events + <tt>choose</tt> + <tt>sync</tt> is the same in expressive power as a built-in <tt>select</tt>, as far as I can see. If there are other events, then your runtime already has to support them either way, and something like <tt>(let ((ch (make-channel))) (spawn-fiber (lambda () (put-message ch <i>exp</i>))) (get-message ch))</tt> should be sufficient for any runtime-supported event in <i>exp</i>, like sleeps or timeouts or thread joins or whatever.</p><p>To me it seems like Go has made the right choices here. I do not see the difference, and that's why I wrote all this, is to be shown the error of my ways. Choosing channels, send, receive, and <tt>select</tt> as the primitives seems to have the same power as SML events.</p><p>Let this post be a pentagram on the floor, then, to summon the CML cognoscenti. Well-actuallies are very welcome; hit me up in the comments!</p><p><i>[edit: Sam Tobin-Hochstadt tells me I got it wrong and I believe him :) In the meantime while I work out how I was wrong, examples are welcome!]</i></p></div> Tue, 20 Sep 2016 21:33:41 +0000 GStreamer: GStreamer Conference 2016: Last chance for early-bird discount on tickets /news/#2016-09-15T10:00:00Z <p> This is a quick reminder that registration for the GStreamer conference 2016 is open, and <b>if you register today you can still benefit from the discounted early-bird registration fee</b>, which is only available until Thursday 15 September 2016 (inclusive). After that the registration fee for professional tickets will rise to 340 EUR. </p><p> <b><a href="">Register now for the GStreamer Conference!</a></b> </p><p> <a href=""><img src="" alt="GStreamer Conference 2016 Berlin" height="150" /></a> </p><p><i>About the GStreamer Conference</i></p><p> The GStreamer Conference 2016 will take place on 10-11 October 2016 in Berlin (Germany), and will take place in the same week as the Embedded Linux Conference Europe. More information and details how to register can be found on the <a href="">conference website</a>. </p> Thu, 15 Sep 2016 10:00:00 +0000 GStreamer: GStreamer Conference 2016: Collabora Platinum Sponsor /news/#2016-09-14T13:00:00Z <p> The GStreamer project is pleased to welcome back Collabora as Platinum level sponsor at this year's GStreamer Conference in Berlin. </p><p> Collabora (<a href=""></a>) is a consultancy with more than 10 years of experience in open source technologies. As well as employing several core contributors, they have been sponsoring the GStreamer conference for every year since the very first conference. </p><p> Thanks Collabora! </p><p> <a href=""><img src="" alt="Collabora" height="50" width="305" /></a> </p><p><i>About the GStreamer Conference</i></p><p> The GStreamer Conference 2016 will take place on 10-11 October 2016 in Berlin (Germany), and will take place in the same week as the Embedded Linux Conference Europe. More information and details how to register can be found on the <a href="">conference website</a>. </p> Wed, 14 Sep 2016 13:00:00 +0000 Jean-François Fortin Tam: Taking down my online portfolio <p>Cleaning up my apartment today, hoping to get rid of a pile of draft papers that has been cluttering my space for six months, I&#8217;m taking the opportunity to write about what this particular pile of paper means (yes, my blogging backlog goes that far—I am draining the swamp one post at a time!)</p> <a href=""><img class="wp-image-3334 size-large" src="" alt="2016-09-11--10.40.19" width="584" height="363" /></a>Pictured: my floor, littered with intermediate drafts of my new portfolio, along with a few printer calibration photos (upper-left) and a copy of the new Annual Report sitting around <p><span id="more-3331"></span></p> <p>For the longest of times, as a designer flying under the radar (my management and marketing work is more &#8220;visible&#8221;, ironically), I did not have a proper portfolio. In fact, I had made a portfolio only once in my life, assembled in a few days in 2011. While my work as <a href="">idéemarque</a> in the past few years has led me to do a fair amount of design work, I was so busy with business work in general that I never took the time to do the massive amount of research and planning required to make a new one.</p> <p>In early 2016, I finally sat down and made a new portfolio. Those who have worked with me know I&#8217;m a perfectionist, and since I had finally set my mind on doing this, I would be <em>doing it right</em>. It indeed took me 2-3 months of research, writing, designing, revising, redesigning, print testing, revising again and fixing issues, until I ran out of issues to fix.</p> <p>From the start, I decided that it would be a top-quality print-only edition. I used beautiful, legible and readable fonts to make reading on paper a delightful experience. I invested in premium quality paper (including some semi-transparent sheets) and used a high-end printer, but not a commercial printing press as that would have been ridiculously expensive and inflexible for printing only a handful of custom-made copies. A print shop couldn&#8217;t have provided all the fancy features I required anyway (fancy binding, specialty paper, embossing, etc.). Therefore each of my portfolios is <strong>a hand-made craft</strong>—even if it might not seem that way as I made it <a href="">full bleed</a> (edge to edge). This whole project made me learn way too much about troubleshooting printer color problems, printing on unconventional paper types, faking bleeds, and the traditional bookbinding techniques of my ancestors.</p> <a href=""><img class="size-full wp-image-3337" src="" alt="Pictured: some really thick paper used for the portfolio" width="1000" height="562" /></a>Pictured: some really thick and heavy paper used for the portfolio <p>In 2016, a printed portfolio might seem outdated compared to an online portfolio. Yet, mine is <strong>designed for print,</strong> precisely because the web and cheap emails have become the norm in this era of constant noise, and because I prefer to control everything in the reader&#8217;s experience—including the delivery, the typography, the texture, size, depth, resolution, and the physical (and mental) weight of the document. And not having to &#8220;/%$?%*)&amp;!?&#8221; around with CSS and &#8220;modern&#8221; web development is a huge plus.</p> <a href=""><img class="wp-image-3332 size-large" src="" alt="CSS is awesome mug" width="584" height="314" /></a>Pictured: not my mug <p>Some said, &#8220;How can this be such an involved process, taking you months?&#8221; to which I replied that, among other things, the document was roughly thirty pages. This got me some expressions of bewilderment, until I explained, &#8220;It&#8217;s not a graphic design portfolio, it&#8217;s a <strong>human-computer interaction design</strong> research portfolio.&#8221;</p> <p>While it took a long time, the project was worth it, both as a personal challenge and from the reactions I got afterwards.</p> <p>I have now decided to remove any traces of my portfolio from my website (except the photo and illustrations gallery). A website is not only a pain in the ass to maintain regularly, it&#8217;s typically ignored or skimmed anyway (basically, Schrödinger&#8217;s Visitor). For those who requested an electronic version of my portfolio over the past few months, I have insisted on shipping them the print version instead—tailored for them, of course.</p> Sun, 11 Sep 2016 18:38:09 +0000 Sebastian Dröge: Writing GStreamer Elements in Rust (Part 2): Don’t panic, we have better assertions now – and other updates <p><strong>This is part 2, the other parts can be found here: <a href="">part 1</a> and <a href="">part 3</a></strong></p> <p>It&#8217;s a while since <a href="" target="_blank">last article</a> about writing <a href="" target="_blank">GStreamer</a> plugins in <a href="" target="_blank">Rust</a>, so here is a short (or not so short?) update of the changes since back then. You might also want to attend my talk at the <a href="" target="_blank">GStreamer Conference</a> on 10-11 October 2016 in Berlin about this topic.</p> <p>At this point it&#8217;s still only with the same elements as before (HTTP source, file sink and source), but the next step is going to be something more useful (an element processing actual data, parser or demuxer is not decided yet) now that I&#8217;m happy with the general infrastructure. You can still find the code in the same place as before on <a href="" target="_blank">GitHub here</a>, and that&#8217;s where also all updates are going to be.</p> <p>The main sections here will be <a href="">Error Handling</a>, <a href="">Threading</a> and <a href="">Asynchonous IO</a>.</p> <h3 id="error-handling">Error Handling &#038; Panics</h3> <p>First of all let&#8217;s get started with a rather big change that shows some benefits of Rust over C. There are two types of errors we care about here: <a href="">expected errors</a> and <a href="">unexpected errors</a>.</p> <h4 id="expected-errors">Expected Errors</h4> <p>In <a href="" target="_blank">GLib</a> based libraries we usually report errors with some kind of boolean return value plus an optional <a href="" target="_blank">GError</a> that allows to propagate further information about what exactly went wrong to the caller but also to the user. Bindings sometimes convert these directly into exceptions of the target language or whatever construct exists there.</p> <p>Unfortunately, in GStreamer we use GErrors not very often. Consider for example GstBaseSink (in pseudo-C++/Java/&#8230; for simplicity):</p><pre class="crayon-plain-tag">class BaseSrc { ... virtual gboolean start(); virtual gboolean stop(); virtual GstFlowReturn create(GstBuffer ** buffer); ... }</pre><p></p> <p>For start()/stop() there is just a boolean, for render() there is at least an enum with a few variants. This is for from ideal, so what is additionally required by implementors of those virtual methods is that they post <a href="" target="_blank">error messages</a> if something goes wrong with further details. Those are propagated out of the normal control flow via the GstBus to the surrounding bins and in the end the application. It would be much nicer if instead we would have GErrors there and make it mandatory for implementors to return one if something goes wrong. These could still be converted to error messages but at a central place then. Something to think about for the next major version of GStreamer.</p> <p>This is of course only for expected errors, that is, for things where we know that something can go wrong and want to report that.</p> <h5>Rust</h5> <p>In Rust this problem is solved in a similar way, see the huge chapter about error handling in the <a href="" target="_blank">documentation</a>. You basically return either the successful result, or something very similar to a GError:</p> <p></p><pre class="crayon-plain-tag">trait Src { ... start(&amp;mut self) -> Result(), ErrorMessage>; stop(&amp;mut self) -> Result(), ErrorMessage>; create(&amp;mut self, &amp;[u8] buffer) -> Result; ... }</pre><p></p> <p><a href="" target="_blank">Result</a> is the type behind that, and it comes with convenient macros for propagating errors upwards (try!()), chaining multiple failing calls and/or converting errors (map(), and_then(), map_err(), or_else(), etc) and <a href="" target="_blank">libraries</a> that make defining errors with all the glue code required for combining different errors types from different parts of the code easier.</p> <p>Similar to Result, there is also <a href="" target="_blank">Option</a>, which can be Some(x) or None, to signal the possible absence of a value. It works similarly, has similar API, but is generally not meant for error handling. It&#8217;s now used instead of GST_CLOCK_TIME_NONE (aka u64::MAX) to signal the absence of e.g. a stop position of a seek, or the absence of a known size of the stream. It&#8217;s more explicit then giving a single integer value of all of them a completely different meaning.</p> <h5>How is the different?</h5> <p>The most important difference from my point of view here is, that you <strong>must</strong> handle errors in one way or another. Otherwise the compiler won&#8217;t accept your code. If something can fail, you must explicitly handle this and can&#8217;t just silently ignore the possibility of failure. While in C people tend to just ignore error return values and assume that things just went fine.</p> <h5>What&#8217;s ErrorMessage and FlowError, what else?</h5> <p>As you probably expect, ErrorMessage maps to the GStreamer concept of error messages and contains exactly the same kind of information. In Rust this is implemented slightly different but in the end results in the same thing. The main difference here is that whenever e.g. start fails, you <strong>must</strong> provide an error message and can&#8217;t just fail silently. That error message can then be used by the caller, and e.g. be posted on the bus (and that&#8217;s exactly what happens).</p> <p>FlowError is basically the negative part (the errors or otherwise non-successful results) of GstFlowReturn:</p> <p></p><pre class="crayon-plain-tag">pub enum FlowError { NotLinked, Flushing, Eos, NotNegotiated(ErrorMessage), Error(ErrorMessage), }</pre><p></p> <p>Similarly, for the actual errors (NotNegotiated and Error), an actual error message must be provided and that then gets used by the caller (and is posted on the bus).</p> <p>And in the same way, if setting an URI fails we now return a Result(), UriError>, which then reports the error properly to GStreamer.</p> <p>In summary, if something goes wrong, we know about that, have to handle/report that and have an error message to post on the bus.</p> <h5>Macros are awesome</h5> <p>As a side-note, creating error messages for GStreamer is not too convenient and they want information like the current source file, line number, function, etc. Like in C, I&#8217;ve created a macro to make such an error message. Different to C, macros in Rust are actually awesome though and not just arbitrarily substituting text. Instead they work via pattern matching and allow you to distinguish all kinds of different cases, can be recursive and are somewhat typed (expression vs. statement vs. block of code vs. type name, &#8230;).</p> <h4 id="unexpected-errors">Unexpected Errors</h4> <p>So this was about expected errors so far, which have to be handled explicitly in Rust but not in C, and for which we have some kind of data structure to pass around. What about the other cases, the errors that will never happen (but usually do sooner or later) because your program would just be completely broken then and all your assumptions wrong, and you wouldn&#8217;t know what to do in those cases anyway.</p> <p>In C with GLib we usually have 3 ways of handling these. 1) Not at all (and crashing, doing something wrong, deadlocking, deleting all your files, &#8230;), 2) explicitly <a href="" target="_blank">asserting</a> what the assumptions in the code are and crashing cleanly otherwise (SIGABRT), or 3) <a href="" target="_blank">returning some default value</a> from the function but just returning immediately and printing a warning instead of going on.</p> <p>None of these 3 cases are handleable in any case, which seems fair because they should never happen and if they do we wouldn&#8217;t know what to do anyway. 1) is obviously least desirable but the most common, 3) is only slightly better (you get a warning, but usually sooner or later something will crash anyway because you&#8217;re in an inconsistent state) and 2) is cleanest. However 2) is nothing you really want either, your application should somehow be able to return back to a clean state if it can (by e.g. storing the current user data, stopping everything and loading up a new UI with the stored user data and some dialog).</p> <h5>Rust</h5> <p>Of course no Rust code should ever run into case 1) above and randomly crash, cause memory corruptions or similar. But of course this will also happen due to bugs in Rust itself, using unsafe code, or code wrapping e.g. a C library. There&#8217;s not really anything that can be done about this.</p> <p>For the other two cases there is however: catching panics. Whenever something goes wrong in unexpected ways, the corresponding Rust code can call the panic!() macro in one way or another. Like via assertions, or when &#8220;asserting&#8221; that a Result is never the error case by calling unwrap() on it (you don&#8217;t <strong>have</strong> to handle errors but you have to explicitly opt-in to ignore them by calling unwrap()). </p> <p>What happens from there on is similar to exception handling in other languages (unless you compiled your code so that <a href="" target="_blank">panics automatically kill the application</a>). <a href="" target="_blank">The stack gets unwound</a>, everything gets cleaned up on the way, and at some point either everything stops or <a href="" target="_blank">someone catches that</a>. The boundary for the unwinding is either your main() in Rust, or if the code is called from C, then at that exact point (i.e. for the GStreamer plugins at the point where functions are called from GStreamer).</p> <h5>So what?</h5> <p>At the point where GStreamer calls into the Rust code, we now catch all unwinds that might happen and remember that one happened. This is then converted into a GStreamer error message (so that the application can handle that in a meaningful way) and by remembering that we prevent any further calls into the Rust code and immediately make them error messages too and return.</p> <p>This allows to keep the inconsistent state inside the element and to allow the application to e.g. remove the element and replace it with something else, restart the pipeline, or do whatever else it wants to do. Assertions are <strong>always local</strong> to the element and not going to take down the whole application!</p> <h3 id="threading">Threading</h3> <p>The other major change that happened is that Sink and Source are now single-threaded. There is no reason why the code would have to worry about threading as everything happens in exactly one thread (the streaming thread), except for the setting/getting of the URI (and possibly other &#8220;one-time&#8221; settings in the future).</p> <p>To solve that, at the translation layer between C and Rust there is now a (Rust!) wrapper object that handles all the threading (in Rust with <a href="" target="_blank">Mutexes</a>, which work like the ones in C++, or atomic booleans/integers), stores the URI separately from the Source/Sink and just passes the URI to the start() function. This made the code much cleaner and made it even simpler to write new sources or sinks. No more multi-threading headaches.</p> <p>I think that we should in general move to such a simpler model in GStreamer and not require a full-fledged, multi-threaded GstElement subclass to be implemented, but instead something more use-case oriented (Source, sink, encoder, decoder, &#8230;) that has a single threaded API and hides all the gory details of GstElement. You don&#8217;t have to know these in most cases, so you shouldn&#8217;t have to know them as is required right now.</p> <h3>Simpler Source/Sink Traits</h3> <p>Overall the two traits look like this now, and that&#8217;s all you have to implement for a new source or sink:</p> <p></p><pre class="crayon-plain-tag">pub type UriValidator = Fn(&amp;Url) -> Result(), UriError>; pub trait Source { fn uri_validator(&amp;self) -> Box; fn is_seekable(&amp;self) -> bool; fn get_size(&amp;self) -> Option; fn start(&amp;mut self, uri: Url) -> Result(), ErrorMessage>; fn stop(&amp;mut self) -> Result(), ErrorMessage>; fn fill(&amp;mut self, offset: u64, data: &amp;mut [u8]) -> Result; fn seek(&amp;mut self, start: u64, stop: Option) -> Result(), ErrorMessage>; } pub trait Sink { fn uri_validator(&amp;self) -> Box; fn start(&amp;mut self, uri: Url) -> Result(), ErrorMessage>; fn stop(&amp;mut self) -> Result(), ErrorMessage>; fn render(&amp;mut self, data: &amp;[u8]) -> Result(), FlowError>; }</pre><p></p> <h3 id="async-io">Asynchronous IO</h3> <p>The last time I mentioned that a huge missing feature was asynchronous IO, in a composeable way. This has some news now, there&#8217;s an abstract implementation for <a href="" target="_blank">futures</a> and a set of higher-level APIs around <a href="" target="_blank">mio</a> for doing actual IO, called <a href="" target="_blank">tokio</a>. Independent of that there&#8217;s also <a href="" target="_blank">futures-cpupool</a>, which allows to call arbitrary calculations as futures on threads of a thread pool.</p> <p>Recently also the HTTP library <a href="" target="_blank">Hyper</a>, as used by the HTTP source (and Servo), also <a href="" target="_blank">got a branch that moves it to tokio</a> for allowing asynchronous IO. Once that is landed, it can relatively easily be used inside the HTTP source for allowing to interrupt HTTP requests at any time.</p> <p>It seems like this area moves into a very promising direction now, solving my biggest technical concern in a very pleasant way.</p> Wed, 07 Sep 2016 12:37:46 +0000 Arun Raghavan: GStreamer on Android and universal builds <p>This is a quick <acronym title="Public Safety Announcement">PSA</acronym> for those of you using the GStreamer binary builds for Android.</p> <p>With the Android NDK r12, the default behaviour while building native code changed from building for <code>armeabi</code> to building for all ABIs. So if your app doesn&#8217;t specify <code>APP_ABI</code> in its <code></code>, you will now get an error about unsupported architectures. This was tracked as <a href="">bug 770631</a>.</p> <p>The idea behind this change is that your Android app should ship versions of your native code for all supported architectures as a &#8220;universal&#8221; build, so it is accessible to as many devices as possible.</p> <p>To deal with this, we now provide a <a href="">universal tarball</a> which contains binaries for all archiectures that we support. This is currently ARM, ARMv7-A, ARMv8-A (64-bit), x86, and x86-64. That leaves MIPS and MIPS64 that are not currently supported.</p> <p>If you&#8217;ve been using the GStreamer Android binaries before GStreamer 1.9.2, then you should start using the universal tarball rather than the architecture-specific tarball. You will need minor updates to your native build, like we made to the <a href="">player example</a>. You probably want to put the <code>gstAndroidRoot</code> variable in <code>~/.gradle/</code> instead, though.</p> <p>As Sebastian <a href="">announced</a>, assuming all goes well with the universal tarballs, we will stop shipping the per-arch tarballs &#8212; they are redundant, and just take up <acronym title="Continuous Integration">CI</acronym> and disk resources.</p> <p>There are some things that I&#8217;d like for us to be able to do better. The first is that Android Studio doesn&#8217;t pick up native code with our current build approach. This is a limitation of the Android Gradle NDK plugin, which doesn&#8217;t support a custom build. This should change with Android Studio 2.2.</p> <p>I would also like to integrate better with Android Studio &#8212; either be able to specify the GStreamer Android binary path in the UI (like you do for the SDK/NDK), or better yet, have it be possible to specify the dependency in Gradle, and have it be automatically pulled from the Internet. If any of you are familiar with how we can do this, please shout out!</p> Tue, 06 Sep 2016 03:34:20 +0000 GStreamer: GStreamer Conference 2016: Registration now open /news/#2016-09-05T09:00:00Z <p><i>About the GStreamer Conference</i></p><p> The GStreamer Conference 2016 will take place on 10-11 October 2016 in Berlin (Germany), and will take place in the same week as the Embedded Linux Conference Europe. </p><p> It is a conference for developers, decision-makers, and anyone else interested in the GStreamer multimedia framework and open source multimedia. </p><p><i>Registration now open</i></p><p> You can now register for the GStreamer Conference 2016 via the <a href="">conference website</a>. </p> Mon, 05 Sep 2016 09:00:00 +0000 GStreamer: GStreamer Core, Plugins, RTSP Server, Editing Services, Python, Validate, VAAPI, OMX 1.9.2 unstable release (binaries) /news/#2016-09-05T9:00:00Z <p> Pre-built binary images of the 1.9.2 unstable release of GStreamer are now available for Windows 32/64-bit, iOS and Mac OS X and Android. </p><p> For Android, there is a new variant this time: an universal release that contains binaries for all supported architectures. Starting with the next release, this will be the only one that is provided. Updating your application should be straightforward and similar to <a href="">this</a> </p><p> The builds are available for download from: <a href="">Android</a>, <a href="">iOS</a>, <a href="">Mac OS X</a> and <a href="">Windows</a>. </p> Mon, 05 Sep 2016 00:00:00 +0000 GStreamer: Orc 0.4.26 bug-fix release /news/#2016-09-04T10:00:00Z <p> The GStreamer team announces another maintenance bug-fix release of liborc, the Optimized Inner Loop Runtime Compiler. Main changes since the previous release: </p><p> <ul> <li>Use 64 bit arithmetic to increment the stride if needed (fixing crashes in certain libgstvideo functions on OS X)</li> <li>Fix generation of ModR/M / SIB bytes for the EBP, R12, R13 registers on X86/X86-64 (fixing crashes in compositor on Windows)</li> <li>Fix test_parse unit test if no executable backend is available</li> <li>Add orc-test path to the -uninstalled .pc file</li> <li>Fix compiler warnings in the tests on OS X</li> </ul> </p><p> Direct tarball download: <a href="">orc-0.4.26</a>. </p> Sun, 04 Sep 2016 10:00:00 +0000 GStreamer: GStreamer Core, Plugins, RTSP Server, Editing Services, Python, Validate, VAAPI, OMX 1.9.2 unstable release /news/#2016-09-01T09:00:00Z <p> The GStreamer team is pleased to announce the second release of the unstable 1.9 release series, which marks the feature freeze for 1.10. The 1.9 release series is adding new features on top of the 1.0, 1.2, 1.4, 1.6 and 1.8 series and is part of the API and ABI-stable 1.x release series of the GStreamer multimedia framework. The unstable 1.9 release series will lead to the stable 1.10 release series in the next weeks. Any newly added API can still change until that point. </p><p> Binaries for Android, iOS, Mac OS X and Windows will be provided in the next days. </p><p> Check out the release notes for <a href="">GStreamer core</a>, <a href="">gst-plugins-base</a>, <a href="">gst-plugins-good</a>, <a href="">gst-plugins-ugly</a>, <a href="">gst-plugins-bad</a>, <a href="">gst-libav</a>, <a href="">gst-rtsp-server</a>, <a href="">gst-python</a>, <a href="">gst-editing-services</a>, <a href="">gst-validate</a>, <a href="">gstreamer-vaapi</a>, or <a href="">gst-omx</a>, or download tarballs for <a href="">gstreamer</a>, <a href="">gst-plugins-base</a>, <a href="">gst-plugins-good</a>, <a href="">gst-plugins-ugly</a>, <a href="">gst-plugins-bad</a>, <a href="">gst-libav</a>, <a href="">gst-rtsp-server</a>, <a href="">gst-python</a>, <a href="">gst-editing-services</a>, <a href="">gst-validate</a>, <a href="">gstreamer-vaapi</a>, or <a href="">gst-omx</a>. </p> Thu, 01 Sep 2016 09:00:00 +0000 Sebastian Dröge: Getting RSS feeds for news websites that don’t provide them <p>This is about a fun little project I already wrote a few months ago, completely unrelated to other things I&#8217;m usually writing about, but I thought it might be useful for others too.</p> <p>When I moved to Greece last year, I had the problem that there were not many news websites that provided local news in English and actually had an <a href="" target="_blank">RSS feed</a>. And having local news, next to global ones about what happens all over the world, seems like a good idea to know what happens close around you.<br /> The only useful website I found was <a href="" target="_blank">Ekathimerini</a>. There were two others that seemed useful to have, <a href="" target="_blank">The Press Project</a> (a crowd-funded project) and <a href="" target="_blank">To Vima</a>, both of which don&#8217;t have an RSS feed (or not for their English version). Of course the real solution to this problem is to learn Greek, which I&#8217;m doing now but that&#8217;s going to take a while until I&#8217;m able to understand news articles without too much effort.</p> <p>So what did I do? I wrote a small web service that parses the HTML of those websites and returns an RSS feed based on that, together with having it update regularly in the background and keeping some history of items. You can find it here: <a href="" target="_blank">html-rss-proxy</a>. The resulting RSS feeds seem to work very well in <a href="" target="_blank">Liferea</a> and <a href="" target="_blank">Newsblur</a> at least.</p> <p>Since then it was also extended by another news website, and generally it&#8217;s rather simple to add new ones to the code. You just need to figure out how to extract the relevant information from the HTML of some website and then convert that into code like <a href="" target="_blank">here</a> and wrap it up in the general interface that the other parts of the code expect, like <a href="" target="_blank">here</a>.</p> <p>If you add some website yourself, feel free to send me a pull request and I&#8217;ll merge it!</p> <h3>Technical bits</h3> <p>On the technical side, this seems to be one of the most stable pieces of software I ever wrote. It never crashed or otherwise failed since I started running it, and fortunately I also didn&#8217;t have to update the HTML parsing code yet because of website changes.</p> <p>It&#8217;s written in <a href="" target="_blank">Haskell</a>, using the <a href="" target="_blank">Scotty</a> web framework, <a href="" target="_blank">Cereal</a> serialization library for storing the history of the past articles, <a href="" target="_blank">http-conduit</a> for fetching the websites, and <a href="" target="_blank">html-conduit</a> for parsing the HTML. Overall a very pleasant experience, thanks to the language being very convenient to write and preventing most silly mistakes at compile-time, and the high quality of the libraries.</p> <p>The only part I&#8217;m not yet too happy about is the actual HTML parsing, it seems to verbose and redudant. I might port it to another library at a later time, maybe <a href="" target="_blank">xml-html-conduit-lens</a>.</p> <h4>Update</h4> <p>After saying that I don&#8217;t like the HTML parsing, I actually reimplemented it around <a href="" target="_blank">xml-html-conduit-lens</a> now. The result is much shorter code and it resembles the structure of the HTML, as you can see <a href="" target="_blank">here</a> for example.</p> <p>Considering that people always say that lens is so <a href="" target="_blank">complicated</a>, and this is more than simple getters, I have to say it went rather painless. Only the compiler errors if the types don&#8217;t match are a bit tricky to understand at first.</p> Sat, 27 Aug 2016 15:02:44 +0000 GStreamer: GStreamer Core, Plugins, RTSP Server, Editing Services, Python, Validate, VAAPI 1.8.3 stable release (binaries) /news/#2016-08-22T10:00:00Z <p> Pre-built binary images of the 1.8.3 stable release of GStreamer are now available for Windows 32/64-bit, iOS and Mac OS X and Android. </p><p> The builds are available for download from: <a href="">Android</a>, <a href="">iOS</a>, <a href="">Mac OS X</a> and <a href="">Windows</a>. </p> Mon, 22 Aug 2016 10:00:00 +0000 Jean-François Fortin Tam: GUADEC 2016, laptops and tablets made to run GNOME, surprise Pitivi meeting <p>I went there for the <a href="">2016 edition of GUADEC</a>:</p> <p><a href=""><img class="alignnone wp-image-3303 size-full" src="" alt="2016-08-17--21.35.31" width="1000" height="418" /></a></p> <p>I arrived a couple of days early to attend my last GNOME Foundation board meeting, in one of the KIT&#8217;s libraries. The building&#8217;s uncanny brutalist architecture only added to the nostalgia of <a href="">a two years adventure</a> coming to an end:</p> <p><span id="more-3296"></span></p> <p><a href=""><img class="alignnone wp-image-3302 size-medium" src="" alt="2016-08-16--17.44.34" width="375" height="500" /></a></p> <p>Then&#8230;</p> <blockquote class="twitter-tweet"> <p lang="en" dir="ltr">A wild Kat strong-armed me into preparing an &quot;unconference&quot; talk for this year's <a href="">#GUADEC</a> after all&#8230;</p> <p>&mdash; Jeff (@nekohayo) <a href="">August 10, 2016</a></p></blockquote> <p></p> <p>And so I made a new talk proposal at the last minute, which was upvoted fairly quickly by attendees:</p> <p><a href=""><img class="alignnone wp-image-3297 size-full" src="" alt="2016-08-12--15.03.12" width="1000" height="561" /></a></p> <p>The conference organizers counter-trolled me by inscribing it exactly like this onto the giant public schedule in the venue&#8217;s lobby:</p> <p><img class="alignnone size-large wp-image-3298" src="" alt="2016-08-12--16.08.16" width="584" height="328" /></p> <p><a href=""><img class="alignright wp-image-3310" src="" alt="" width="263" height="226" /></a>The result was this talk: <strong><a href="">Laptops &amp; Tablets Manufactured to Run a Pure GNOME</a></strong>. Go watch it now if you missed it. Note: during the talk&#8217;s Q&amp;A session, I mistakenly thought that <a href="">Purism</a>&#8216;s tablets were using an ARM architecture; they&#8217;re actually planned to be Intel-based. And to make things clear, for laptop keyboard layouts, Purism is currently offering US/UK, which are different <em>physical</em> layouts (different cutting etc.).</p> <p>Also relevant to your interests if you&#8217;re into that whole privacy thing:</p> <ul> <li>Cosimo&#8217;s <a href="">unconference session about the earmarked privacy funds</a></li> <li>Federico&#8217;s talk about our, ahem, suboptimal <a href="">key management UX in GNOME</a>. <ul> <li>Stef Walters pointed out during this talk&#8217;s Q&amp;A (and to me personally when I discussed with him on the previous day) that he&#8217;d be glad to review &amp; merge patches, but does not currently have enough time to maintain <a href="">Seahorse</a> otherwise. Interested in improving security for our PGP, SSH and geeral keys management in GNOME? Check out Seahorse and start polishing that code!</li> </ul> </li> <li>Werner Koch&#8217;s keynote: <a href="">We want more centralization, do we?</a></li> <li>Alexander Larsson&#8217;s <a href="">Flatpak status update and future plans</a></li> </ul> <p>In my luggage, I carried ~20 kilograms of the Foundation&#8217;s annual reports. Some folks were skeptical and thought I should &#8220;only bring a few of them, people aren&#8217;t usually interested in the annual report.&#8221; Well, they were <em>dead wrong:</em> within one afternoon, the annual reports &#8220;sold out&#8221; like hotcakes. See also <a href="">my tandem lightning talk</a> (at the 29mins mark) to get a glimpse of how much work we put into designing the new annual report this year. Also, if you took one of them at the conference, remember: they&#8217;re precious little works of art <em>and</em> <strong>a powerful tool to convince people to become contributors or sponsors to support GNOME</strong>, so make sure to use them towards that goal!</p> <p>I was very happy to see Mathieu and Alexandru from the <a href="">Pitivi</a> team deciding to attend at the last minute, even if it was just for one day. We spent time with Jakub Steiner discussing his workflow and wishlist for a &#8220;perfect&#8221; video editor:</p> <a href=""><img class="wp-image-3300 size-large" src="" alt="2016-08-15--17.05.53" width="584" height="412" /></a>Pitivi makes people smile!<br /> Left to right: Alexandru Băluț, Jakub &#8220;Skywalker&#8221; Steiner, Mathieu Duponchelle. <p>I took a handful of photos of the BoFs and uploaded them to <a href="">my gallery for GUADEC 2016</a>. Licensed under the Creative Commons &#8220;by attribution&#8221; 4.0 as usual, so that the GNOME Engagement team can use them for GNOME promotional materials if needed. Make sure you do the same and list yours <a href="">in here</a>.</p> <a href=""><img class="size-large wp-image-3299" src="" alt="The Flatpak BoF session" width="584" height="267" /></a>The Flatpak BoF session <p>Overall, this GUADEC was one of the most well-organized ones I&#8217;ve seen in years. I was floored by the amount of efforts and planning the local team put into this. They really deserve some kudos! Everything ran smoothly on the surface (I know that in such events there will always be odd situations happening here and there, but they dealt with them so efficiently that they were invisible). The team had a professional-grade two-way radio system to coordinate, a car and trailer to carry stuff around every day, made and reused food (pro-grade cafeteria counter metal containers = genius), a lifetime supply of IKEA mugs that got washed and reused frequently, tons of snacks, managed to pull in great sponsors even at the last minute, put signage in various parts of the city to guide people to the venue, had huge quantities of tasty dead animals (and plants) to eat at a very successful barbecue event, got an icecream vendor to come to the venue, and even filled up a pool and beanbags, for pete&#8217;s sake!</p> <blockquote class="twitter-tweet"> <p lang="en" dir="ltr">Spotted at <a href="">#GUADEC2016</a>: <a href="">@GNOME</a> developers hard at work! Only missing some piñas coladas <img src="" alt="😜" class="wp-smiley" /> <a href=""></a></p> <p>&mdash; Jeff (@nekohayo) <a href="">August 13, 2016</a></p></blockquote> <p></p> <p>Thanks to the the Chaos Computer Club for providing a flawless live video streaming, recording and publishing service. Very rarely did we have GUADEC videos published in a timely fashion in the past, let alone streamed live and with proper laptop video output capture as well as proper sound mixing. This is fantastic.</p> <p>I should mention the closing night&#8217;s event in the biergarten, with sponsored drinks and food by our friendly GStreamer experts <a href="">Centricular</a>, a very nice gesture that was well appreciated by everyone.</p> <a href=""><img class="wp-image-3301 size-large" src="" alt="2016-08-15--19.23.28" width="584" height="332" /></a>Survivors at rest in the snacks bunker on the last day of the BoFs <p>Thanks to everybody involved to make this event a success, and thanks to the GNOME Foundation for making it possible for me to attend.</p> <p><img class="alignnone wp-image-1641 size-full" src="" alt="gnome sponsored badge shadow" width="230" height="230" /></p> Sun, 21 Aug 2016 11:45:17 +0000 GStreamer: GStreamer Core, Plugins, RTSP Server, Editing Services, Python, Validate, VAAPI 1.8.3 stable release /news/#2016-08-19T10:00:00Z <p> The GStreamer team is pleased to announce the third bugfix release in the stable 1.8 release series of your favourite cross-platform multimedia framework! </p><p> This release only contains bugfixes and it should be safe to update from 1.8.x. For a full list of bugfixes see <a href="">Bugzilla</a>. </p><p> See <a href="">/releases/1.8/</a> for the full release notes. </p><p> Binaries for Android, iOS, Mac OS X and Windows will be available shortly. </p><p> Check out the release notes for <a href="">GStreamer core</a>, <a href="">gst-plugins-base</a>, <a href="">gst-plugins-good</a>, <a href="">gst-plugins-ugly</a>, <a href="">gst-plugins-bad</a>, <a href="">gst-libav</a>, <a href="">gst-rtsp-server</a>, <a href="">gst-python</a>, <a href="">gst-editing-services</a>, <a href="">gst-validate</a>, or <a href="">gstreamer-vaapi</a>, or download tarballs for <a href="">gstreamer</a>, <a href="">gst-plugins-base</a>, <a href="">gst-plugins-good</a>, <a href="">gst-plugins-ugly</a>, <a href="">gst-plugins-bad</a>, <a href="">gst-libav</a>, <a href="">gst-rtsp-server</a>, <a href="">gst-python</a>, <a href="">gst-editing-services</a>, <a href="">gst-validate</a>, or <a href="">gstreamer-vaapi</a>. </p> Fri, 19 Aug 2016 10:00:00 +0000 Christian Schaller: Want make Linux run better on laptops? <p>So we have two jobs openings in the Red Hat desktop team. What we are looking for is people to help us ensure that Fedora and RHEL runs great on various desktop hardware, with a focus on laptops. Since these jobs require continuous access to a lot of new and different hardware we can not accept applications this time for remotees, but require you to work out of out office in Munich, Germany. We are looking for people with people not afraid to jump into a lot of different code and who likes tinkering with new hardware. The hardware enablement here might include some kernel level work, but will more likely involve improving higher level stacks. So for example if we have a new laptop where bluetooth doesn&#8217;t work you would need to investigate and figure out if the problem is in the kernel, in the bluez stack or in our Bluetooth desktop parts. </p> <p>This will be quite varied work and we expect you to be part of a team which will be looking at anything from driver bugs, battery life issues, implementing new stacks, biometric login and enabling existing features in the kernel or in low level libraries in the user interface.</p> <p>You can read more about the jobs at the <a href=""></a>. That link lists a Senior Engineer, but we also got a Principal Engineer position open with id 53653, but that one is not on the website as I post this, but should hopefully be very soon.</p> <p>Also if you happen to be in the Karlsruhe area or at GUADEC this year I will be here until Sunday, so you could come over for a chat. Feel free to email me on if you are interested in meeting up.</p> Fri, 12 Aug 2016 09:07:47 +0000 Bastien Nocera: Flatpak cross-compilation support, A couple of weeks ago, I <a href="">hinted at a presentation</a> that I wanted to do during this year's GUADEC, as a Lightning talk.<br /><br />Unfortunately, I didn't get a chance to finish the work that I set out to do, encountering a couple of bugs that set me back. Hopefully this will get resolved post-GUADEC, so you can expect some announcements later on in the year.<br /><br />At least one of the tasks I set to do worked out, and was promptly obsoleted by a nicer solution. Let's dive in.<br /><br /><b>How to compile for a different architecture</b><br /><br />There are four possible solutions to compile programs for a different architecture:<br /><br /><ul><li><i>Native compilation</i>: get a machine of that architecture, install your development packages, and compile. This is nice when you have fast machines with plenty of RAM to compile on, usually developer boards, not so good when you target low-power devices.</li><li><i>Cross-compilation</i>: install a version of GCC and friends that runs on your machine's architecture, but produces binaries for your target one. This is usually fast, but you won't be able to run the binaries created, so might end up with some data created from a different set of options, and won't be able to run the generated test suite.</li><li><i>Virtual Machine</i>: you'd run a virtual machine for the target architecture, install an OS, and build everything. This is slower than cross-compilation, but avoids the problems you'd see in cross-compilation.</li></ul>The final option is one that's used more and more, mixing the last 2 solutions: the <a href="">QEmu user-space emulator</a>.<br /><br /><b>Using the QEMU user-space emulator</b><br /><br />If you want to run just the one command, you'd do something like:<br /><!-- HTML generated using --><br /><div><pre><span>qemu-static-arm myarmbinary</span><br /></pre></div><br />Easy enough, but hardly something you want to try when compiling a whole application, with library dependencies. This is where <a href="">binfmt support in Linux</a>&nbsp;comes into play. Register the ELF format for your target with that user-space emulator, and you can run <span>myarmbinary</span> without any commands before it.<br /><pre class="example"><span><br /></span></pre>One thing to note though, is that this won't work as easily if the qemu user-space emulator and the target executable are built as a dynamic executables: QEmu will need to find the libraries for your architecture, usually x86-64, to launch itself, and the emulated binary will also need to find its libraries.<br /><br />To solve that first problem, there are QEmu static binaries available in a number of distributions (<a href="">Fedora support is coming</a>). For the second one, the easiest would be if we didn't have to mix native and target libraries on the filesystem, in a chroot, or container for example. Hmm, container you say.<br /><div><br /></div><div><b>Running QEmu user-space emulator in a container</b><br /><br />We have our statically compiled QEmu, and a filesystem with our target binaries, and switched the root filesystem. Well, you try to run anything, and you get a bunch of errors. The problem is that there is a single binfmt configuration for the kernel, whether it's the normal OS, or inside a container or chroot.<br /><br /><b>The Flatpak hack</b><br /><br /><a href="">This commit for Flatpak</a> works-around the problem. The binary for the emulator needs to have the right path, so it can be found within the chroot'ed environment, and it will need to be copied there so it is accessible too, which is what this patch will do for you.<br /><br />Follow the instructions in the commit, and test it out with this <a href="">Flatpak script for GNU Hello</a>.<br /><!-- HTML generated using --><br /><div><pre><span>$ TARGET=arm ./</span><br /><span>[...]</span><br /><span>$ ls org.gnu.hello.arm.xdgapp </span><br /><span>918k org.gnu.hello.arm.xdgapp</span><br /></pre></div><br />Ready to install on your device!<br /><br /><b>The proper way</b><br /><br />The above solution was built before it looked like <a href="">the "proper way"</a> was going to find its way in the upstream kernel. This should hopefully land in the upcoming 4.8 kernel.<br /><br />Instead of launching a separate binary for each non-native invocation, this patchset allows the kernel to keep the binary opened, so it doesn't need to be copied to the container.<br /><br /><b>In short</b><br /><br />With the work being done on Fedora's static QEmu user-space emulators, and the kernel feature that will land, we should be able to have a nice tickbox in Builder to build for any of the targets supported by QEmu.<br /><br />Get cross-compiling!</div> Thu, 11 Aug 2016 15:00:37 +0000 (Bastien Nocera) Zeeshan Ali: Life is change <div>Quite a few major life events happened/happening this summer so I thought I blog about them and some of the experiences I had.<br /><h3>New job &amp; new city/country</h3>Yes, I found it hard to believe too that I'll ever be leaving Red Hat and the best manager I ever had (no offence to others but competing with Matthias is just impossible) but I'll be moving to Gothenburg to join Pelagicore folks as a Software Architect in just 2 weeks. I have always found Swedish language to be a very cute language so looking forward to my attempt of learning Swedish. If only I had learnt Swedish rather than Finnish when I was in Finland.<br /><br />BTW, I'm selling all my furniture so if you're in London and need some furniture, get in touch!<br /><br /><h3>Fresh helicopter pilot</h3><div>So after two years of hard work and getting myself sinking in bank loans, I finally did it! Last week, I passed the skills test for Private Pilot License (Helicopters) and currently awaiting anxiously for my license to come through (it usually takes at least two weeks). Once I have that, I can rent Helicopters and take passengers with me. I'll be able to share the costs with passengers but I'm not allowed to make money out of it. The test was very tough and I came very close to failing at one particular point. The good news is that despite me being very tense and very windy conditions on test day, the biggest negative point from my examiner was that I was being over-cautious and hence very slow. So I think it wasn't so bad.</div><div><br /></div><div><a href=""><img border="0" height="320" src="" width="240" /></a></div><div><br /><br />There are a few differences to a driving test. A minor one is is that in driving test, you are not expected to explain your steps but simply execute, where as in skills test for flying, you're expected to think everything out loud. But the most major difference is that in driving test, you are not expected to drive on your own until you pass the test, where as in flying test, you are required to have flown solo for at least 10 hours, which needs to include a solo cross country flight of at least a 100 nautical miles (185 KM) involving 3 major aeorodromes.  Mine involved Estree, Cranfield and Duxford. I've been GPS logging while flying so I can show you log of my qualifying solo cross country flight (click <a href="">here</a> to see details and notes):<br /><br /></div><div><img border="0" height="218" src="" width="400" /></div><div><br /></div><div><br /></div><div>I still got a long way towards Commercial License but at least now I can share the price with friends so building hours towards commercial license, won't be so expensive (I hope). I've found a <a href="">nice company in Gothenburg</a> that trains in and rents helicopters so I'm very much looking forward to flying over the costs in there. Wanna join? Let me know. :)</div></div> Wed, 10 Aug 2016 14:10:38 +0000 Bastien Nocera: Blog backlog, Post 4, Headset fixes for Dell machines, At the <a href="">bottom of the release notes for GNOME 3.20</a>, you might have seen the line:<br /><blockquote class="tr_bq">If you plug in an audio device (such as a headset, headphones or microphone) and it cannot be identified, you will now be asked what kind of device it is. This addresses an issue that prevented headsets and microphones being used on many Dell computers.</blockquote>Before I start explaining what this does, as a picture is worth a thousand words:<br /><br /><div class="separator"><a href=""><img border="0" height="225" src="" width="400" /></a></div><br />This selection dialogue is one you will get <a href="">on some laptops and desktop machines</a> when the hardware is not able to detect whether the plugged in device is headphones, a microphone, or a combination of both, probably because it doesn't have an impedance detection circuit to figure that out.<br /><br />This functionality was <a href="">integrated into Unity's gnome-settings-daemon</a> version a couple of years ago, written by David Henningsson.<br /><br />The code that existed for this functionality was completely independent, not using any of the facilities available in the media-keys plugin to volume keys, and it could probably have been split as an external binary with very little effort.<br /><br />After a bit of to and fro, most of the sound backend functionality was <a href="">merged into libgnome-volume-control</a>, leaving just 2 entry points, one to signal that something was plugged into the jack, and another to select which type of device was plugged in, in response to the user selection. This means that the functionality should be easily implementable in other desktop environments that use libgnome-volume-control to interact with PulseAudio.<br /><br />Many thanks to David Henningsson for the original code, and his help integrating the functionality into GNOME, <a href="">Bednet</a> for providing hardware to test and maintain this functionality, and Allan, Florian and Rui for working on the UI notification part of the functionality, and wiring it all up after I abandoned them to go on holidays ;) Tue, 09 Aug 2016 10:49:38 +0000 (Bastien Nocera) Jean-François Fortin Tam: Vice-President’s Report — The State of the GNOME Foundation <p>Hi! Long time no see. My blog has been pretty quiet in recent months, in the big part due to my extended commitment on the <a href="">GNOME Foundation</a>&#8216;s Board of Directors (for a second year without an executive director present to take some of the load) and the various business engagements I&#8217;ve had.</p> <p>Generally speaking, this year was a bit less intense than the one before it (we didn&#8217;t have to worry about a legal battle with a giant corporation this time around!) although we did end up touching a fair amount of legal matters, such as trademark agreements. One big item we got cleared was the <a href="">Ubuntu GNOME</a> trademark agreement. We also welcomed businesses that wanted to sell GNOME-related merchandise, you can find them <a href="">listed here</a>—supporting them by purchasing GNOME-related items also supports the Foundation with a small percentage shared as royalties.</p> <p><span id="more-3284"></span></p> <p><img class="alignnone size-full wp-image-3287" src="" alt="gnome 2016 merchandize - 1" width="615" height="265" /> <img class="alignnone size-large wp-image-3289" src="" alt="gnome 2016 merchandize - 3" width="584" height="438" /> <img class="alignnone size-large wp-image-3290" src="" alt="gnome 2016 merchandize - 4" width="584" height="584" /> <img class="alignnone size-large wp-image-3288" src="" alt="gnome 2016 merchandize - 2" width="584" height="438" /></p> <p>In the summer of 2015, I thought I&#8217;d take a break from my presidency from the year before, so I was pretty happy to have a new president and vice-president starting at GUADEC 2015 and me just being a regular board member. Some months later, Christian Hergert had to step down from his role as vice-president because he joined Red Hat, and the GNOME Foundation has a rule where the board of the directors cannot have more than two members (out of seven) from the same company/employer. I took over his role as vice-president then.</p> <blockquote class="twitter-tweet"> <p lang="en" dir="ltr"><a href="">@nekohayo</a> Together, we shall rule the galaxy.</p> <p>&mdash; Shaun McCance (@shaunm) <a href="">November 18, 2015</a></p></blockquote> <p></p> <p>And so it went:</p> <img class="wp-image-3285 size-full" src="" alt="tails flying and supporting sonic" width="320" height="181" />Pictured: me, helping Shaun tackle the big things. <a href="">Sonic boom</a>! <p>The board has done a lot of work in recent months. In addition to the legal agreements mentioned above, since <a href="">my last report</a> we&#8217;ve <strong>held <a href="">50 meetings</a></strong> (double the amount from last year; it seems bi-weekly meetings were not enough to cover all that we had to discuss!), <strong>over 2400 emails</strong> were exchanged on the board mailing list, and we wrote <strong>over 24,800 lines of discussion on our IRC channel</strong>.</p> <p>When the 2016 elections came up I thought it was time to let new blood come in and participate. I needed to move on and focus on growing my own <a href="">business</a> that I have been neglecting for two years, anyway. As the new board came in, I have been gradually winding down (my role after the election and until GUADEC is mostly advisory, as I do not hold voting powers).</p> <p>I am excited about the team that composes the new Board of Directors and I trust that they will do a great job. The GNOME Foundation always needs a team of experimented, positive and energetic people to come together to think, discuss and make decisions regarding the various challenges its faces. As I <a href="">wrote</a> during last year&#8217;s elections period:</p> <blockquote><p>For this to work, we need people that are what I call &#8220;powerhouses&#8221;, because the GNOME Foundation Board is an &#8220;active&#8221; board. This means great thinkers and proactive doers ready to deal with anything while being very capable in the board room.</p> <p>The best metaphor I have for a healthy GNOME board is taken from role-playing games: a well-coordinated &#8220;level 45-70&#8221; party that will not be afraid to crawl dungeons together for a year. You need polyvalent classes just like you need specialists (analytic mages, &#8220;massive damage&#8221; knights, resourceful healers, quick &amp; agile rangers, etc.).</p> <p>So if this makes sense to anyone, I&#8217;m <a href="">a hybrid mage-knight with a ton of HP/MP potions and phoenix feathers</a> ;)</p></blockquote> <img class="size-full wp-image-3286" src="" alt="Party line-up, by Laikkuseia" width="800" height="455" /><a href="">Party line-up</a>, by Laikkuseia <p>During the 2016 winter and spring, I have also been heavily involved in the engagement team to redesign the Annual Report and get it ready in time for GUADEC. I&#8217;ll bring it along with me in my luggage. Do you want a copy? Let me know by tomorrow (Sunday):</p> <blockquote class="twitter-tweet"> <p lang="en" dir="ltr">Printed <a href="">@GNOME</a> reports are ready! Available at <a href="">@GUADEC</a>. Want one? Let us know so we can forecast <a href="">@nekohayo</a>'s luggage <a href=""></a></p> <p>&mdash; idéemarque (@ideemarque) <a href="">August 1, 2016</a></p></blockquote> <p></p> <p><span>(note that I can also bring copies of last year&#8217;s report if you&#8217;d like to have them for your vintage collection)</span></p> <p><a href=""><img class="alignright" src="" width="256" height="256" /></a> I am looking forward to seeing all of you at <a href="">GUADEC</a> next week; don&#8217;t forget to register! And look forward to some very interesting news coming up from my side in the near future.</p> Sat, 06 Aug 2016 13:06:10 +0000 Nirbheek Chauhan: GStreamer and Meson: A New Hope, <div dir="ltr">Anyone who has written a non-trivial project using Autotools has realized that (and wondered why) it requires you to be aware of <abbr title="m4, autoconf, automake, shell, make—add Perl for many projects">5 different languages</abbr>. Once you spend enough time with the innards of the system, you begin to realize that it is nothing short of an astonishing feat of engineering. Engineering that belongs in a museum. Not as part of critical infrastructure.<br /><br />Autotools was created in the 1980s and caters to the needs of an entirely different world of software from what we have at present. Worse yet, it carries over accumulated cruft from the past 40 years — ostensibly for better “cross-platform support” but that “support” is mostly for <span id="goog_326547089"></span>extinct platforms<span id="goog_326547090"></span> that five people in the whole world remember.<br /><br />We've learned how to make it work for most cases that concern FOSS developers on Linux, and it can be made to limp along on other platforms that the majority of people use, but it <a href="">does not inspire confidence</a> or really anything except frustration. People will not like your project or contribute to it if the build system takes 10x longer to compile on their platform of choice, does not integrate with the preferred IDE, and requires knowledge arcane enough to be indistinguishable from cargo-cult programming.<br /><br />As a result there have been several (terrible) efforts at replacing it and each has been either incomplete, short-sighted, slow, or just plain ugly. During my time as a Gentoo developer in another life, I came in close contact with and developed a keen hatred for each of these alternative build systems. And so I mutely went back to Autotools and learned that I hated it the least of them all.<br /><br />Sometime last year, <a href="">Tim</a> heard about this new build system called ‘<a href="">Meson</a>’ whose author had created an experimental port of <a href="">GStreamer</a> that built it in record time.<br /><br />Intrigued, he tried it out and found that it finished suspiciously quickly. His first instinct was that it was broken and hadn’t actually built everything! Turns out this build system written in Python 3 with <a href="">Ninja</a> as the backend actually <b><i>was</i></b> that fast. About 2.5x faster on Linux and 10x faster on Windows for building the <a href="">core GStreamer repository</a>.<br /><br />Upon further investigation, Tim and I found that Meson also has really clean generic cross-compilation support (including iOS and Android), runs natively (and just as quickly) on OS X and Windows, supports GNU, Clang, and MSVC toolchains, and can even (configure and) generate XCode and Visual Studio project files!<br /><br />But the critical thing that convinced me was that the creator <a href="">Jussi Pakkanen</a> was genuinely interested in the use-cases of widely-used software such as Qt, GNOME, and GStreamer and had already added support for several tools and idioms that we use — pkg-config, gtk-doc, gobject-introspection, gdbus-codegen, and <a href="">so on</a>. The project places strong emphasis on both speed and ease of use and is quite friendly to contributions.<br /><br />Over the past few months, Tim and I at Centricular <a href="">have been working</a> on creating Meson ports for <a href="">most</a> <a href="">of</a> <a href="">the</a> <a href="">GStreamer</a> <a href="">repositories</a> and the fundamental dependencies (<a href="">libffi</a>, <a href="">glib</a>, <a href="">orc</a>) and improving the MSVC toolchain support in Meson.<br /><br />We are proud to report that you can now build GStreamer on Linux using the GNU toolchain and on Windows with either MinGW or MSVC 2015 using Meson build files that ship with the source (building upon Jussi's initial ports).<br /><br />Other toolchain/platform combinations haven't been tested yet, but they should work in theory (minus bugs!), and we intend to test and bugfix all the configurations supported by GStreamer (Linux, OS X, Windows, iOS, Android) before proposing it for inclusion as an alternative build system for the GStreamer project.<br /><br />You can either grab the source yourself and <a href="">build everything</a>, or use our (with luck, temporary) fork of GStreamer's cross-platform build aggregator <a href="">Cerbero</a>.<br /><br /><b>Update:</b> I wrote a new post with detailed steps on how to <a href="">build using Cerbero and generate Visual Studio project files</a>.<br /><br />Personally, I really hope that Meson gains widespread adoption. Calling Autotools the Xorg of build systems is flattery. It really is just a terrible system. We really need to invest in something that works for us rather than against us.<br /><br />PS: If you just want a quick look at what the build system syntax looks like, <a href="">take a look at this</a> or <a href="">the basic tutorial</a>.</div> Sun, 31 Jul 2016 02:22:07 +0000 (Nirbheek) Nirbheek Chauhan: Building and Developing GStreamer using Visual Studio, <div dir="ltr">Two months ago, I talked about how we at <a href="">Centricular</a> have been working on a Meson port of GStreamer and its basic dependencies (glib, libffi, and orc) for various reasons — faster builds, better cross-platform support (particularly Windows), better toolchain support, ease of use, and for a better build system future in general.<br /><br />Meson also has built-in support for things like gtk-doc, gobject-introspection, translations, etc. It can even generate Visual Studio project files at build time so projects don't have to expend resources maintaining those separately.<br /><br />Today I'm here to share instructions on how to use <a href="">Cerbero</a> (our “aggregating” build system) to build all of GStreamer on Windows using MSVC 2015 (wherever possible). Note that this means you won't see any Meson invocations at all because Cerbero does all that work for you.<br /><br />Note that this is still all unofficial and has not been proposed for inclusion upstream. We still have a few issues that need to be ironed out before we can do that¹.<br /><br />First, you need to setup the environment on Windows by installing a bunch of external tools: Python 2, Python3, Git, etc. You can find the instructions for that here:<br /><br /><a href=""></a><br /><br />This is very similar to the old Cerbero instructions, but some new tools are needed. Once you've done everything there (Visual Studio especially takes a while to fetch and install itself), the next step is fetching Cerbero:<br /><br /><code>$ git clone</code><br /><br />This will clone and checkout the <code>meson-1.8</code> branch that will build GStreamer 1.8.x. Next, we bootstrap it:<br /><br /><a href=""></a><br /><br />Now we're (finally) ready to build GStreamer. Just invoke the package command:<br /><br /><code>python2 cerbero-uninstalled -c config/win32-mixed-msvc.cbc package gstreamer-1.0</code><br /><br />This will build all the `recipes` that constitute GStreamer, including the core libraries and all the plugins including their external dependencies. This comes to about 76 recipes. Out of all these recipes, only the following are ported to Meson and are built with MSVC:<br /><br />bzip2.recipe<br />orc.recipe<br />libffi.recipe (only 32-bit)<br />glib.recipe<br />gstreamer-1.0.recipe<br />gst-plugins-base-1.0.recipe<br />gst-plugins-good-1.0.recipe<br />gst-plugins-bad-1.0.recipe<br />gst-plugins-ugly-1.0.recipe<br /><br />The rest still mostly use Autotools, plain GNU make or cmake. Almost all of these are still built with MinGW. The only exception is libvpx, which uses its custom make-based build system but is built with MSVC.<br /><br />Eventually we want to build everything including all external dependencies with MSVC by porting everything to Meson, but as you can imagine it's not an easy task. :-)<br /><br />However, even with just these recipes, there is a large improvement in how quickly you can build all of GStreamer inside Cerbero on Windows. For instance, the time required for building <code>gstreamer-1.0.recipe</code> which builds <code>gstreamer.git</code> went from 10 minutes to 45 seconds. It is now easier to do <a href="">GStreamer development on Windows</a> since rebuilding doesn't take an inordinate amount of time!<br /><br />As a further improvement for doing GStreamer development on Windows, for all these recipes (except libffi because of complicated reasons), you can also <a href="">generate Visual Studio 2015 project files</a> and use them from within Visual Studio for editing, building, and so on.<br /><br />Go ahead, try it out and tell me if it works for you!<br /><br />As an aside, I've also been working on some proper in-depth documentation of Cerbero that explains how the tool works, the recipe format, supported configurations, and so on. You can see the <a href="">work-in-progress</a> if you wish to.<br /><br /><span>1. Most importantly, the tests cannot be built yet because GStreamer bundles a very old version of libcheck. I'm currently working on fixing that.</span> </div> Wed, 27 Jul 2016 15:58:30 +0000 (Nirbheek) Bastien Nocera: GUADEC Flatpak contest, I will be presenting a lightning talk during <a href="">this year's GUADEC</a>, and running a contest related to what I will be presenting.<br /><br /><b>Contest</b><br /><br />To enter the contest, you will need to create a <a href="">Flatpak</a> for a piece of software that hasn't been flatpak'ed up to now (application, runtime or extension), hosted in a public repository.<br /><br />You will have to send me an email about the location of that repository.<br /><br />I will choose a winner amongst the participants, <a href="">on the eve of the lightning talks</a>, depending on, but not limited to, the difficulty of packaging, the popularity of the software packaged and its redistributability potential.<br /><br />You can find plenty of examples (and a list of already packaged applications and runtimes) <a href="">on this Wiki page</a>.<br /><br /><b>Prize</b><br /><br />A piece of hardware that you can use to replicate my presentation (or to replicate my attempts at a presentation, depending ;). You will need to be present during my presentation at GUADEC to claim your prize.<br /><br />Good luck to one and all! Tue, 19 Jul 2016 15:39:35 +0000 (Bastien Nocera) GStreamer: GStreamer Core, Plugins, RTSP Server, Editing Services, Python, Validate 1.9.1 unstable release (binaries) /news/#2016-07-12T7:00:00Z <p> Pre-built binary images of the 1.9.1 unstable release of GStreamer are now available for Windows 32/64-bit, iOS and Mac OS X and Android. </p><p> The builds are available for download from: <a href="">Android</a>, <a href="">iOS</a>, <a href="">Mac OS X</a> and <a href="">Windows</a>. </p> Tue, 12 Jul 2016 00:00:00 +0000 GStreamer: GStreamer Core, Plugins, RTSP Server, Editing Services, Python, Validate, VAAPI, OMX 1.9.1 unstable release /news/#2016-07-06T12:00:00Z <p> The GStreamer team is pleased to announce the first release of the unstable 1.9 release series. The 1.9 release series is adding new features on top of the 1.0, 1.2, 1.4, 1.6 and 1.8 series and is part of the API and ABI-stable 1.x release series of the GStreamer multimedia framework. The unstable 1.9 release series will lead to the stable 1.10 release series in the next weeks. Any newly added API can still change until that point. </p><p> Binaries for Android, iOS, Mac OS X and Windows will be provided in the next days. </p><p> Check out the release notes for <a href="">GStreamer core</a>, <a href="">gst-plugins-base</a>, <a href="">gst-plugins-good</a>, <a href="">gst-plugins-ugly</a>, <a href="">gst-plugins-bad</a>, <a href="">gst-libav</a>, <a href="">gst-rtsp-server</a>, <a href="">gst-python</a>, <a href="">gst-editing-services</a>, <a href="">gst-validate</a>, <a href="">gstreamer-vaapi</a>, or <a href="">gst-omx</a>, or download tarballs for <a href="">gstreamer</a>, <a href="">gst-plugins-base</a>, <a href="">gst-plugins-good</a>, <a href="">gst-plugins-ugly</a>, <a href="">gst-plugins-bad</a>, <a href="">gst-libav</a>, <a href="">gst-rtsp-server</a>, <a href="">gst-python</a>, <a href="">gst-editing-services</a>, <a href="">gst-validate</a>, <a href="">gstreamer-vaapi</a>, or <a href="">gst-omx</a>. </p> Wed, 06 Jul 2016 12:00:00 +0000 Víctor Jáquez: VA-API and DRM/KMS in MinnowBoard <p>In 2012 I started to work on a video renderer for GStreamer which uses directly the <a href="">DRM/KMS kernel subsystem</a> to display images. I even <a href="">blogged about it</a>, but I didn&#8217;t finished it.</p> <p>Nonetheless, in December last year, a customer asked us to finish the element, but this time focusing on i.MX6, rather than OMAP4. Consequently, early this year, <tt>kmssink</tt> got <a href="">merged into upstream</a>.</p> <p>The video sink has some nice features, such as <a href="">DMA-buf</a> importing through the PRIME kernel interface. This feature makes possible a zero-copy path when the video decoder delivers <a href="">dmabuf based frames</a>. For this particular project, the kernel we used supports the <a href="">CODA media subsystem</a>, and through the GStreamer element <tt>v4l2videodec</tt> we could link pipelines negotiating dmabuf sharing, and thus we played videos very efficiently.</p> <p>During the last GStreamer Hackfest, Nicolas Dufresne got working the <a href="">MFC media subsystem</a>, for Exynos, with <tt class="docutils literal">v4l2videodec</tt> and <tt class="docutils literal">kmssink</tt>, but I don&#8217;t remember if he also got the dmabuf sharing working.</p> <p>All in all, <tt>kmssink</tt> had only been tested in a couple of ARM devices, so I wonder, as a <em>gstremer-vaapi</em> co-maintainer, if it also works in x86 devices.</p> <p>Some colleagues at Igalia, use a <a href="">Minnowboard</a> to test <a href="">WebKit4Wayland</a>, so I borrowed it, installed a minimal Fedora 23 in it, and, surprisingly, I found out that it has a nice VA-API support:</p> <pre> $ vainfo error: can't connect to X server! libva info: VA-API version 0.38.1 libva info: va_getDriverName() returns 0 libva info: Trying to open /usr/lib64/dri/ libva info: Found init function __vaDriverInit_0_38 libva info: va_openDriver() returns 0 vainfo: VA-API version: 0.38 (libva 1.6.2) vainfo: Driver version: Intel i965 driver for Intel(R) Bay Trail - 1.6.2 vainfo: Supported profile and entrypoints VAProfileMPEG2Simple : VAEntrypointVLD VAProfileMPEG2Simple : VAEntrypointEncSlice VAProfileMPEG2Main : VAEntrypointVLD VAProfileMPEG2Main : VAEntrypointEncSlice VAProfileH264ConstrainedBaseline: VAEntrypointVLD VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice VAProfileH264Main : VAEntrypointVLD VAProfileH264Main : VAEntrypointEncSlice VAProfileH264High : VAEntrypointVLD VAProfileH264High : VAEntrypointEncSlice VAProfileH264StereoHigh : VAEntrypointVLD VAProfileVC1Simple : VAEntrypointVLD VAProfileVC1Main : VAEntrypointVLD VAProfileVC1Advanced : VAEntrypointVLD VAProfileNone : VAEntrypointVideoProc VAProfileJPEGBaseline : VAEntrypointVLD </pre> <p>Afterwards, I compiled GStreamer using <a href="">gst-uninstalled</a> in order to have <tt>kmssink</tt> and the master branch of <em>gstreamer-vaapi</em>. And got it! I had hardware accelerated decoding with VA-API, and video rendering using KMS/DRM. But, sadly, no zero copy, since, in order to have dmabuf sharing, we have to finish and to merge <a href="">bug 755072</a> in <em>gstreamer-vaapi</em>.</p> <p>Running the typical Big Bunny video (1080p, H.264) the playback consumes less than the 50% of the CPU usage, compared with the 100% of CPU usage with software decoders (and a lot of frames dropped).</p> <p>I recorded a small video with the experiment as a proof.</p> <p></p> <p><strong>Update:</strong> Javer Martinez <a href="">told me in twitter</a>, that MFC can do dma-buf export since uses vb2 and Exynos DRM can import, but he couldn&#8217;t get zero copy to work due missing format support.</p> Fri, 01 Jul 2016 17:00:09 +0000 Nicolas Dufresne: GStreamer Echo Canceller <p>For a long time I believed that echo cancellers had no place inside GStreamer. The theory was that GStreamer was too high level and would never be able to provide accurate enough delay information for any canceller to work. With fairly simple test, I could quickly confirm that the reported latency is often off by a period (generally 10ms). This isn&#8217;t strictly GStreamer&#8217;s fault and is not in any ways catastrophic for general playback experience.</p> <p>With the apparition of WebRTC in browsers, it most likely became apparent that to be cross-platform browsers needed to have their own canceller. That&#8217;s exactly what happened in <a href="">libWebRTC</a> (former libjingle, used in both Firefox and Chrome to implement WebRTC). They implemented an echo canceller that accept an approximate delay and this changes everything for GStreamer.</p> <p>At <a href="">Collabora</a>, I recently had the opportunity to implement this <a href="">WebRTC Audio Processing</a> based echo canceller. The main motivation was that the canceller on the hardware DSP we had didn&#8217;t work due to a hardware bug. A lot of those boards had been produced and no rework was possible. To save these boards, we decided to try with a software echo canceller. Even though it was using a fair amount of CPU, the experiment was a success. I have then clean-up the code and the new elements are now available in <a href="">GStreamer Plugins Bad</a>.</p> <p>How does it work ?</p> <p><a href="" rel="attachment wp-att-136"><img class="size-full wp-image-136 aligncenter" src="" alt="Echo" width="493" height="270" /></a></p> <p>The first step is to understand what is the echo. In a phone call with loud speaker, what happens is that your microphone records both your voice and the far end voices. The side effect, is that you are sending to the far end listeners both your voice along with a bad copy of their voices a moment before (the echo). To avoid this echo, you need to monitor the far end stream that you are playing back and &#8220;subtract&#8221; it from the recorded stream. In practice, it&#8217;s much more complex work, since the signal is deteriorated by the speaker and the microphone. You also need to figure-out the delays and hint the canceller, otherwise you may end-up with a terrible startup time or it may simply not work.</p> <p>The implementation was greatly inspired from an experiment <a href="">Olivier Crête did in 2008</a> using <a href="">Speex DSP</a>. I must admit, I never really understood his way of synchronizing the streams and literally ignored pretty much all the code that wasn&#8217;t GStreamer specific. The design works this way, you have a DSP element (webrtcdsp) that process the recorded stream and a probe element (webrtcechoprobe) that analyses the far end stream (before playing it back). Due to WebRTC library limitation, those two elements will transform the input buffer into chunk of 10ms. This is done using GstAdapter help. On the probe side, we push buffers in the adapter with timestamp transformed to running time. This time, plus the pipeline latency, gives us the moment in running time when the buffer should be heard by the microphone. We then synchronize the far end data against the recorded data and then let WebRTC Audio Processing library do it&#8217;s magic. A simple way of testing the element is by using an echo loop.</p> <p><code>  gst-launch-1.0 pulsesrc ! webrtcdsp ! webrtcechoprobe ! pulsesink</code></p> <p>Without the canceller, this pipeline would create a lot of echo, and probably end with loud feedback if your microphone volume is high enough. With the canceller, you should instead ear only one echo. It behaves a bit like a sound monitor but with too high latency and the side effect of fading in and out monotonic frequencies. After all, this is not what the algorithm have been design for. Try it in your real audio call application, that&#8217;s where you will most likely get the best results.</p> <p><a href="" rel="attachment wp-att-137"><img class="alignnone wp-image-143 size-large" src="" width="792" height="236" /></a></p> <p>Before I conclude, there is a good reason why I called the element DSP rather then AEC. WebRTC Audio Processing is much more then just an echo canceller. In fact, it implements a wide variety of filters, noise suppressor, voice activity detection, etc. Currently we enable of subset of it, but I&#8217;m definitely looking forward enabling more (if not all) features from this library. I also encourage contributions. This works was only possible because of the great effort Arun Raghavan have put into extracting the echo canceller form the WebRTC project, create a standalone library usable by all. If you are interested about what cool feature could be added in the future, have a look at <a href="">Arun&#8217;s blog</a> about beamforming. And last one, thanks to my colleague who had to suffer me speaking with my computer listening to my echo for few weeks.</p> Thu, 30 Jun 2016 22:14:06 +0000 Arun Raghavan: Beamforming in PulseAudio <p>In case you missed it &#8212; we got PulseAudio 9.0 <a href="">out the door</a>, with the echo cancellation improvements that <a href="">I wrote about</a>. Now is probably a good time for me to make good on my promise to expand upon the subject of beamforming.</p> <p>As with the last post, I&#8217;d like to shout out to the wonderful folks at <a href="">Aldebaran Robotics</a> who made this work possible!</p> <h2>Beamforming</h2> <p>Beamforming as a concept is used in various aspects of signal processing including radio waves, but I&#8217;m going to be talking about it only as applied to audio. The basic idea is that if you have a number of microphones (a <em>mic array</em>) in some known arrangement, it is possible to &#8220;point&#8221; or <em>steer</em> the array in a particular direction, so sounds coming from that direction are made louder, while sounds from other directions are rendered softer (<em>attenuated</em>).</p> <p>Practically speaking, it should be easy to see the value of this on a laptop, for example, where you might want to focus a mic array to point in front of the laptop, where the user probably is, and suppress sounds that might be coming from other locations. You can see an example of this in the webcam below. Notice the grilles on either side of the camera &#8212; there is a microphone behind each of these.</p> <a href=""><img /></a>Webcam with 2 mics <p>This raises the question of <em>how</em> this effect is achieved. The simplest approach is called &#8220;delay-sum beamforming&#8221;. The key idea in this approach is that if we have an array of microphones that we want to steer the array at a particular angle, the sound we want to steer at will reach each microphone at a different time. This is illustrated below. The image is taken from this <a href="">great article</a> describing the principles and math in a lot more detail.</p> <a href=""><img /></a>Delay-sum beamforming <p>In this figure, you can see that the sound from the source we want to listen to reaches the top-most microphone slightly before the next one, which in turn captures the audio slightly before the bottom-most microphone. If we know the distance between the microphones and the angle to which we want to steer the array, we can calculate the additional distance the sound has to travel to each microphone.</p> <p>The speed of sound in air is roughly 340 m/s, and thus we can also calculate how much of a delay occurs between the same sound reaching each microphone. The signal at the first two microphones is delayed using this information, so that we can line up the signal from all three. Then we take the sum of the signal from all three (actually the average, but that&#8217;s not too important).</p> <p>The signal from the direction we&#8217;re pointing in is going to be strongly correlated, so it will turn out loud and clear. Signals from other directions will end up being attenuated because they will only occur in one of the mics at a given point in time when we&#8217;re summing the signals &#8212; look at the noise wavefront in the illustration above as an example.</p> <h2>Implementation</h2> <p><em>(this section is a bit more technical than the rest of the article, feel free to skim through or skip ahead to the next section if it&#8217;s not your cup of tea!)</em></p> <p>The devil is, of course, in the details. Given the microphone geometry and steering direction, calculating the expected delays is relatively easy. We capture audio at a fixed sample rate &#8212; let&#8217;s assume this is 32000 samples per second, or 32 kHz. That translates to one sample every 31.25 µs. So if we want to delay our signal by 125µs, we can just add a buffer of 4 samples (4 × 31.25 = 125). Sound travels about 4.25 cm in that time, so this is not an unrealistic example.</p> <p>Now, instead, assume the signal needs to be delayed by 80 µs. This translates to 2.56 samples. We&#8217;re working in the digital domain &#8212; the mic has already converted the analog vibrations in the air into digital samples that have been provided to the CPU. This means that our buffer delay can either be 2 samples or 3, not 2.56. We need another way to add a fractional delay (else we&#8217;ll end up with errors in the sum).</p> <p>There is a fair amount of academic work describing methods to perform filtering on a sample to provide a fractional delay. One common way is to apply an <acronym title="Finite Input Response">FIR</acronym> filter. However, to keep things simple, the method I chose was the <a href="">Thiran approximation</a> &#8212; the literature suggests that it performs the task reasonably well<strike>, and has the advantage of not having to spend a whole lot of CPU cycles first transforming to the frequency domain (which an FIR filter requires)</strike>(edit: converting to the frequency domain isn&#8217;t necessary, thanks to the folks who pointed this out).</p> <p>I&#8217;ve implemented all of this as a separate module in PulseAudio as a <a href="">beamformer filter module</a>.</p> <p>Now it&#8217;s time for a confession. I&#8217;m a plumber, not a <acronym title="Digital Signal Processing">DSP</acronym> ninja. My delay-sum beamformer doesn&#8217;t do a very good job. I suspect part of it is the limitation of the delay-sum approach, partly the use of an <acronym title="Infinite Impulse Response">IIR</acronym> filter (which the Thiran approximation is), and it&#8217;s also entirely possible there is a bug in my fractional delay implementation. Reviews and suggestions are welcome!</p> <h2>A Better Implementation</h2> <p>The astute reader has, by now, realised that we are already doing a bunch of processing on incoming audio during voice calls &#8212; I&#8217;ve written in the previous article about how the <code>webrtc-audio-processing</code> engine provides echo cancellation, acoustic gain control, voice activity detection, and a bunch of other features.</p> <p>Another feature that the library provides is &#8212; you guessed it &#8212; beamforming. The engineers at Google (who clearly <em>are</em> DSP ninjas) have a pretty good beamformer implementation, and this is also available via <code>module-echo-cancel</code>. You do need to configure the microphone geometry yourself (which means you have to manually load the module at the moment). Details are <a href="">on our wiki</a> (thanks to Tanu for that!).</p> <p>How well does this work? Let me show you. The image below is me talking to my laptop, which has two microphones about 4cm apart, on either side of the webcam, above the screen. First I move to the right of the laptop (about 60°, assuming straight ahead is 0°). Then I move to the left by about the same amount (the second speech spike). And finally I speak from the center (a couple of times, since I get distracted by my phone).</p> <p>The upper section represents the microphone input &#8212; you&#8217;ll see two channels, one corresponding to each mic. The bottom part is the processed version, with echo cancellation, gain control, noise suppression, etc. <em>and</em> beamforming.</p> <a href=""><img /></a>WebRTC beamforming <p>You can also listen to the actual recordings &#8230;</p> <a href=""></a> <p>&#8230; and the processed output.</p> <a href=""></a> <p>Feels like black magic, doesn&#8217;t it?</p> <h2>Finishing thoughts</h2> <p>The <code>webrtc-audio-processing</code>-based beamforming is already available for you to use. The downside is that you need to load the module manually, rather than have this automatically plugged in when needed (because we don&#8217;t have a way to store and retrieve the mic geometry). At some point, I would really like to implement a configuration framework within PulseAudio to allow users to set configuration from some external UI and have that be picked up as needed.</p> <p>Nicolas Dufresne has done some work to wrap the <code>webrtc-audio-processing</code> library functionality in a <a href="">GStreamer element</a> (and this is in master now). Adding support for beamforming to the element would also be good to have.</p> <p>The <code>module-beamformer</code> bits should be a good starting point for folks who want to wrap their own beamforming library and have it used in PulseAudio. Feel free to get in touch with me if you need help with that.</p> Wed, 29 Jun 2016 05:22:12 +0000