Welcome to a Daily Tech News Show special on how to stream a live video from your home. Many of you are in a situation where you need to stream video to a remote audience for the first time and we are here to help. We've all been doing this from our homes with varying approaches and varying needs for around a decade or so. We're going to share our experiences with cameras, with mics, with getting the video and audio into the computer, with getting it out of the computer onto the internet and hopefully you'll find something in our experience to help you with yours. I'm Tom Merritt. I've been streaming from my home office in various ways for around ten years. Joining me is my DTNS co-host Sarah Lane. Sarah, how long have you been streaming from homes or remote locations? On and off for as long as it's been possible, but pretty much every day since I started DTNS about two and a half years ago. And then Roger Chang, our producer. You've been doing this from home for a while as well. I have been doing it since 2015. Yeah, so five years. Also joining us is Allison Sheridan. You've been streaming your shows from your home. How long now? Well, I'm not even sure how to count that. The podcast has been going for 15 years, but then chit chat started maybe ten years ago. But video, I just keep adding more and more things on to make it as complicated as possible, so somewhere in that range. Fantastic. Amos, sometimes a producer of Daily Tech News Show, great producer of lots of shows. You've been doing Ritual Misery yourself for a while too. Yeah, for six years now. Okay. Bryce Castillo, probably doing this at the highest level of the bunch of us because you not only do it from your own home, but you also do it from Brian Brushwood Studios. Bryce is going to join us back. Bryce, you do this from your own home, but also from Brian Brushwood Studios. Yeah. And I've been doing live streaming mixed with podcasting since about 2012 or so. Yeah. Justin, Robert Young, I feel like you've been doing it forever. Something like that. The dawn of time is me talking into a microphone. When do you start streaming from your own place? Oh, God, it really depends on what we define as my own place. You know, we built this, we moved out of our main bedroom and moved into our living room so we could have a studio plus studio apartment about five years ago. And that's when we really kind of built a set and everything, but I don't know, a billion years ago is when I first started streaming from my house. Shannon Morris, you've been doing this in all kinds of ways, going back to the old hack house days. How long have you been doing this? Yeah, that's right. Back when we were doing hack five live episodes, way back in Virginia. But over the years, I've tried to make it as simple as possible. So that's what I'm hoping to share with everybody today. And of course, Scott Johnson, you've built your career on just being at home doing this. Yeah, I think mine might go back to furthest, I'm not sure. But in 1999, I attempted a MPEG live compression stream from an office. And it didn't work great and only about 10 people could watch it, but it sort of worked and it gave me a taste for it. And then everything that's come along since I've tried to jump on and, you know, make part of my routine. So the point here is that we all know each other, we all help each other out. We've been doing this for a long time, so we're now going to try to combine to help you out. Let's start with the camera. You can use the camera built into your laptop, especially if that's all you have. Prop it up on some books to help frame the shot or whatever you've got, boxes. I've done that a million times in hotels. The trick is to take the ice bucket, turn it upside down, use that as your prop. But there are other options. How many of us here are using a Logitech C920? Me. I am. Me. Three of them. This is the workhorse of streaming. It's USB. It works in almost any situation. It's often paired with Logitech's own camera software on the Mac. There's an app called Webcam Settings, which can help you adjust the picture quality, zoom, etc. I don't use it. I use a camcorder, the Canon G40. So it has a lot more control. But the thing about camcorders is they're really meant to output video to a monitor in most cases. So they can be a little more buggy connected to a computer. I've used a black magic intensity to get my video into a Mac mini using Thunderbolt. But these days I use a Magewell converter to go from HDMI to USB. So the C920 seems to be the one that we would probably recommend to most people. Any other thoughts on that? One thing to look out for right now is price gouging. I went out and looked at the C920, and it was like $300 or $350. I was like, what? It's only, what, $70? Something like that from Logitech. It's better on Amazon now, but watch out. Just pay the right price. $70. Yeah. Just in it comes to video content, obviously you would want that. But in light of the fact that there's going to be a run on them. And by the way, that is a device that does have outages even in regular times that aren't the plague. Just don't be afraid to use your phone. Don't be afraid to use your built-in camera, especially when you're just getting your sea legs. The quality on those have gone up so much that while a C920 will look better, if you can't get it for the right price or you just want to get your feet wet, I wouldn't shy away from that. Yeah. I'd also mention Logitech makes a C922X, what they call a pro stream webcam. They also have a couple other models, including the Brio. Those, it's weird. Amazon or excuse me, Logitech hit pay dirt with the C920. And everything that's supposed to succeed, it has not been as good. There have been issues and bugs and it's not, not as good. So I would try to aim for that camera. If you're looking for something in that range, don't get gouged. Justin's right. Start with something small. But if you're going to do it and you got the cash for it and you can find one, it's still the primo model to get. Yeah. And if you're not going 1080p, you can get the C910, I believe, which is the 720 version. It's essentially the same camera, same lens, everything else, same connectivity. Logitech does have the C920 for sale for 70 bucks. It's just price couters trying to trick you. So it's available. There you go. Go straight to Logitech. Before we move off of camera, Allison, I know you wanted to bring up the importance of lighting. Yeah. So I am no lighting expert by any means. But what I have figured out is if you take a still photo and there's not enough light and somebody moves a little bit, it gets blurry, right? Same exact thing happens with video. If you don't have enough light on your face, on the front, you're going to look all smeary. So you can change a mediocre quality camera into a good-looking camera just by getting lighting on the front. Having a little bit of light in the back, that's kind of cool. But the lighting on the front really, really matters for a crisp picture. Yeah, I ended up just getting these LEDs, Sarah. They're the same ones that you have, I think, right? Yeah. It's just some really, fairly inexpensive panels of LED lights just to point at me. Yeah, it gives you some color balance options if you want something that's a little bit more mimicking outdoor light versus something warmer. But Allison, I agree with you. In fact, the C920, I've been using it for years. It's a great little camera considering the cost and the fact that it's really small. I've got it on top of a tripod right now. However, my current setup is in the same garage studio that I've been in for six months. But until very recently, I was pointed in the other direction. Just my whole desk has flipped 180 degrees. And the light just hits me better this way. Nothing has changed. But lots of people have been like, wow, you're shot so much better. Did you get a new camera? Nah, it's lighting. The camera wants lighting help. And you do have some settings. There's some third-party programs. Logitech has some of its own somewhat buggy software. But it's worth just poking around in there because there'd be some sort of backlight compensation that you don't even realize you have checked. And you're like, gosh, this is terrible. And then maybe you just didn't need it to begin with. So just look through your settings. Also, you don't have to go very expensive. I have these work clamp lights from Home Depot. They are super cheap. Yeah. The cheapest bulbs in there. And try to get the color temperature to match. But you don't need to go get expensive or internet of things lights if you don't need to.  Yeah. I've got these little clamps that I got for $5 a Walmart one on each side of me. And I just recently upgraded from a standard LED bulb to Phillips Hughes. And I can control the color and the temperature and, you know, the brightness and dim them down. But initially with this, I was out with 20 bucks for two lights one on each side of me. And it lights up really nicely. So yeah. And by the way, you don't necessarily need to go. I have Phillips Hughes. And so I can do fun things with like the, the lighting and stuff like that. But in general, there are a lot of lights that are, are adjustable via an app that are a lot cheaper. They don't necessarily do colors, but they can do just a little variabilities on, especially if you're working with two different people or different people are using it. I have a darker complexion. My wife has a much lighter complexion. And so there is just a need for a touch and being able to do that on the app is, is helpful. But again, that's a little bit more advanced as you go forward. I will add real quick, generally stay away from anything that uses an incandescent bulb because based on whether or not the ballast is old or not, you can get some flickering in it. So if you can stay with LED, even LED bulbs that you use. Cooler to screw it will eliminate a lot of those flickering issues. All right, let's move on to the microphone again. It's okay to use the microphone in your laptop. It's going to sound a little echoey or roomy, which can sometimes make it hard for you to understand. So I tend to emphasize if you want to upgrade the first thing you, you upgrade your mic first over getting a camera. You can use headphones, the microphone that's in your headphones, either wired or wireless. They are a step up from the built in mic in your laptop. Wireless buds can sound a little better, but they can be buggy sometimes because they're using Bluetooth.  The next step up is a USB mic that plugs into your laptop. That has the advantage of being easy. You get a nice mic, you plug it in, you don't have to think about it. I personally don't love them because they don't offer a lot of flexibility. If something goes wrong, your whole mic's done, or if you want to grow into a new situation, you have to replace the whole thing. My recommendation is often getting an XLR to USB adapter. Don't let the acronym scare you. It's just an adapter that takes a normal mic input and converts it to USB for the computer. So you can use a regular mic and plug it into your computer.  And if you need multiple mics, I recommend getting a USB mixer. You can get little ones that are 4-channel from Behringer for pretty affordable amounts. You'll want to get XLR cables. And again, they're just called XLR because that's the name of the connector to the mic. It's a cable that fits in your mic on one end and fits into your mixer or the adapter on the other end. You'll want those cables for the mics if you go that way. And mic-wise, the workhorse is the Sure SM58. It's durable. It sounds great. You can throw it in a bag. It's got excellent quality. A step up from that, I think, how many of us here are using Heil PR40s right now? Me. Yeah. It's a higher-end studio quality mic that's more affordable than similar quality mics. It's still expensive. It's a few hundred dollars. But it gets you really good sound. Amos, I know you've got props to kind of show the cost-benefit analysis of all these different microphone models. So the first one that I would recommend, and this is one of the cheapest ones, is the ATR 2100. And the reason I would recommend that, it's got USB and XLR inputs. So if you want to start with USB and then move up to a more professional or more high-end setup later on, you can easily do that. Those run right around $60. You can also get the Samson Q2U, which is essentially the same microphone, same hookups, everything else. I saw price gouging on that yesterday, too, by the way, the ATR 2100. They were like $250. That's good blanket advice. Watch out for the price gouging on all this stuff right now, right? Yeah. And then as you go up, the next one that you're going to hear a lot about, and I believe this is the one that Shannon uses, is the Blue Yeti. This is a great mic. It's about $115, typically, from what I've seen. Yeah, see, the problem with this mic is it can hear an antelope sneeze from 50 miles away. So it's really great. And Amos lives in Alaska. So he knows. Yeah.  If you don't want to worry about mic technique or anything else, it's a great mic for that. Other than that, I wouldn't necessarily recommend it.  Oh, come on, Justin. It looks like you're talking to a ham or something. It's pretty good. It looks like you're Dr. Mario throwing pills. Like it just looks dumb. It's a great piece of equipment, but it looks dumb. It's USB. And I will say with a lot of USB inputs like mics and webcams, if you're using it on something like Linux, a lot of times those are going to be plug-and-play. So you won't have to worry about trying to get an XLR input to work. So one of the reasons that I went into USB in the first place was because I do switch to Linux quite often. So I do like my Yeti. Thank you very much. And it's great that I can just throw it in a bag and get up and go.   I think it looks like a ham. Beauty is in the eye of the beholder. That's fine. We can all agree to disagree. So that's going to be the next lineup. After that, you're going to get to a midline area, which is the road procaster, which I have, the Heil PR40s, which some of you are using. That really is going to depend on the flavor of sound that you want to get out of it, the quality of the sound that you want to get out of it. The hiles are a little bit tinier. The procaster doesn't do high ends as much, but I've got a deeper voice. So it works out for me. That's really the $250 to $400 range. And then after that, you hit the law of diminishing returns. There are some microphones that are really, really nice and sound amazing, but you're going to pay a lot more for very minimal quality adjustment. So stick. If you're, if you're just starting out again, the ATR 2100 is amazing $60. If you're taking it seriously, hit that $250 to $350 to $350, $400 range. That's going to be a really sweet spot. If your voice is really attuned to certain things, you can get Bryce uses with the SM58, I believe. Yeah, I have an SM58 here, which is great. It will work forever. And if you're looking for an ultra budget, the, what I have here is the Behringer Ultra Voice, the XM 850, 8, 8500. This thing is like less than 30 bucks. If you have a thing where you're getting three people in a room, you can buy a bunch of these for cheaper than one of the SM58s. And it sounds about 90% as good. There are options at every budget. You should put a warning out here is that once you get into this madness, you're going to be like all of us and say, well, then I've got this mic and this mic and this mic and this mic. Yeah, absolutely. Let me just also note, Tom, did you mention the icicle, the blue icicle? No, I mentioned the XLR adapters, but I didn't mention any brands by name. The blue icicle is the one I use. Yeah. Yeah. It's kind of the standard. Yeah. This to me, it's still being sold for the normal price, at least that I saw amongst podcasting professionals. To me, it is the standard. It gives you the opportunity to not only just use a mic, if you have a mic laying around and maybe you don't want to necessarily buy another mic, if you have an XLR cord, you are good to go getting it in via USB. And that also works if you get a little mixer. Now, anything that is XLR out, you can now input via USB. It's very reliable. The only thing that I would say is especially if you're traveling with it, which again, this is a little bit more advanced. It's a little fragile in the mini USB input. I've had to buy a few new ones because I threw it in a bag haphazardly. But other than plugging it and unplugging it, sometimes people will pull the ring off the end of the blue icicle. I think that was that too. It is dirt dumb. I moved away from that to the Shure MVI interface. It's more of like a little puck that sits on the table because one of the main things I needed was a mute switch for the coughing and sneezing and dog barking and stuff. And then when you talk about mixers, Bryce, I know you have some good recommendations for that. Yeah. I think the easiest way to get into it, if you want to go one level above the icicle, is Behringer has the XENIX line, X-E-N-Y-X. They have a bunch of different options for how many mics you want input. So if you need more than one and you don't want to go and buy a bunch of icicles, you can go get one of these XENIXs and plug in two, three, four mics into them. That plugs into the computer via USB. And then at that point, you are starting to use mixers. You are getting into using faders and actually having EQ options.  I have a bunch of them here. We use them all the time. They are rock solid. And if you're traveling, I recommend the Elysis brand. They are just durable. If you're having to move things around, maybe to a church or a school or something like that, they are super durable and good for travel. I use a Mackey Pro FX8 here, and that has served me well for a lot of years. Yeah, Mackey is a good brand. Okay, so you have hooked up your hardware. Now you need the software to get everything into the computer. Now assuming you are attached by USB some way, your audio source will likely be something called the USB audio codec. That is just good to know when you are looking through this. But that does vary depending on your system. Video will be your camera if it is USB or your adapter. So if you are using Magewell, it will be whatever the Magewell dropdown is. If it is just one video source, you can skip the rest of this part. But if you need to bring in other people who are also remote, you will need to use something like Skype or Zoom or Discord or Hangouts. We are using Skype right now, but it seems like almost every show I am on uses a different combination of stuff. Scott, I know you are using Discord, for instance. Yeah, actually it is even weirder than that. Over the last year or so, I have been struggling with finding what is the current best way to go. Zoom did not have individual volume control for people coming in, so that sort of drove me nuts. Skype dropped some features, but added some back, and really stayed on top in terms of video quality, but I didn't like their audio quality as much. So here is what I found out. When I use Discord for both video and audio, the audio codec changes and people sound worse when video is going. When there is no video going in Discord, you get a much better sound quality. So what I started doing is this weird Frankenstein hybrid of calling people on Discord for audio and then bringing them in on regular shows anyway, bringing them in on Skype for the video portion of it. So I'm running the two services at once. You just mute Skype and do the other. It syncs up fine. You don't get any lag or anything, but that way I'm getting the best of the audio choices and the best of those two video choices all at once. It seemed crazy. I realize it does. It doesn't stop you from bringing somebody in, like Tom on Wednesdays. I bring him in on the morning show and we do a jury just like that on Tuesdays, bring you in and still do Discord video with some guests, but it maintains the best audio quality. And I should say latency for myself and my co-hosts. I have way less latency on Discord audio, but not when it's video. It gets bad again. So it's not just a bandwidth issue. I actually don't know what it is, but there's something going on there. And I experimented enough to find out that these two worked as a pretty good combo. So that's what I'm doing these days. That's crazy pants. It is crazy. I love it. I know it sounds crazy, but it works so well. It doesn't sound crazy. If there was a lag, I'd say, eh, well, that's the trade-off. But if there isn't, that's a great solution. There's no lag. And the other advantage I forgot to mention is Discord, as they've added new features, one of the features I like is that you can control everybody's audio individually, no matter who's in the call. And so I can go up and down on people as needed and Skype doesn't let you do that anymore. And you could do all kinds of crazy tie-ins with like mixers and taking Skype out from another computer and bring it in and then control it that way. But it's kind of a nightmare. So this turned out to be a pretty fun, if not kind of Mad Max way to get it done. If there is one gift that this plague could give us, it is somebody creating a prosumer level of VOIP, you know, video solution. Because this, if you are just getting into this and you are frustrated by your options, congratulations. You have now reached the double black belt level of this profession because nobody knows the best solution is the best solution. We have all, at one time or another, just dreamed about our own XPRIZE that we could create just for some, you know, off-market solution that we can all share with each other. It is frustrating. The one thing I would say, if you just want to get a bunch of people on, Zoom works very good. Hangouts is effectively a ghost ship. They have not updated it in forever. It increasingly gets buggy and it will get buggy until one day it dies. Skype very unfortunately is just the abusive relationship that many of us cannot leave. It is always frustrating. The problems are always new and novel. And yet, as Scott mentioned, they're the tools that we have worked with for a very, very long time. So at least we know or we think we know where the menus are going to be and we can share our own knowledge base. And it does provide fairly crystal clear video. I don't hate Skype at all. I keep thinking I should, now that I know where the buttons are. Like I said, it's not a problem if you, like if your audio problems are taken care of or if it's just one co-host, it's fine. It's totally fine. The many you start bringing additional people and you just can get a little funky. In fact, I think the only reason this is working out well for us right this second is because we all know how to make it so we're good on our end so that Roger's not pulling his teeth out. So, you know what I mean? Like that's kind of that.  All good. Good. Thank you for the confirmation. You know, another option on this, and I know we're talking about an era of social distancing right now, but you may have sources locally that you pull in. You're all two meters apart. So Bryce, I know you have some options on bringing in HDMI sources. Yeah.  If you have a game console, if you've got another computer that you want to bring a data signal in, you are probably going to want an HDMI card for a single input. If you just need one HDMI in, my recommendation right now is the Elgato HD60 line and the HD60 Pro. That's a PCIe card. So if you have a desktop, you have to plug it in. You have to, you know, plug it into your card, but that is a very solid way to get a single one in. So it looks like Shannon's got one. Is that right, Shannon? Yeah. I have one in my computer right now. I've got the external version and there's a little lag with the external version, but it's super usable. It's a really good recommendation. Yeah. If you find that you have, if you need, if you, for whatever reason, need even more than one, Blackmagic makes a really great device called the Declink Quad HDMI Recorder. That is for HDMI inputs, up to 4K resolution. And so that will cover you in terms of all the things you need to bring in. Once you install the drivers, look in your program for that as a video source or an input and that'll get you going. You know, it's best to go with the internal cards, but the external stuff has gotten way better over the past five years. It used to be really, really bad with external capture cards and they are so much better now. Yeah. And this is the HD 60S.  It's really easy to use. And again, just like Bryce said, it's going to show up in your software and it's just going to be there. Now we could go down a rabbit hole really fast if we start talking about Mix Minus, but Amos, I know you wanted to bring it up. So keeping it as simple as possible, what is Mix Minus and a couple of tips to deal with it? Mix Minus is the ability to send out audio without. So if Shannon and I are in a call, I don't want to send Shannon's audio back to her, especially over Skype, because it's going to have a lag. It's going to interrupt her thought process and everything else. And basically, when you're looking at Mixers, because really, I know there are solutions out there, software solutions for doing this. I use the hardware solution because it's what I understand. And if you're looking for a Mixer to do Mix Minuses, so you can have a little bit cleaner audio, a little less lag, a little less Skype ducking, although it still doesn't eliminate it. You can essentially have one Mix Minus per auxiliary channel that you have on your board. So the board I'm using right now is the Zoom L12. I have five auxiliary channels, essentially. So I can have up to five Mix Minuses. So I can have five people coming in at one time and them not hear themselves when they're talking. Now, if you can't do that, it's understandable. Skype does a decent job at it, although it tends to cut everyone else out while that one person is talking. But if you can do it, it will give you better cleaner audio. It'll help to the conversation. And if you're confused, feel free to reach out to me. Ethan Kane on Twitter, and I'll help you where I can. I'll show you some guides, whatever else. But yeah, it's one of the things that if you understand, it's amazing. If you don't understand, it's Greek. You're just not going to get it. I have a recommendation for people who might be a little more technically in the know on how audio writing goes. There's a great piece of software called Voice Meter, V-O-I-C-E-M-E-E-T-E-R. They have a bunch of different versions for how much power you need. But they also allow you to make auxiliary or buses in software via your computer. So if you don't have a physical mixer, if you're trying to keep it in software, Voice Meter is a really great solution. It's very complicated. It is not for the faint of heart. But if you take a couple of days and really learn it, you could also possibly solve this with software with Voice Meter. Looks like that's Windows only, right? That sounds right. Yeah, but Voice Meter is Windows only, and then I know there are Mac solutions. Yeah, right. The one I'd recommend from Rogue Amoeba, they make audio hijack and some other stuff. They make something called Loopback that'll do what you're talking about. It's very good. And also they're just a side note. They have a sound board called Farago, which I highly recommend. It's very, very good. I live on audio hijack. I use audio hijack with Loopback instead of using a mixer. And I can do some of the mix-minus kind of stuff that Amos was talking about with double left channels, right kind of stuff. And I've written a bunch of tutorials on how to do it over at podfee.com. So definitely audio hijack for the win. All right. Now you need to send your video on the Internet. One of the main reasons we're doing this is because a lot of you suddenly need to have video and send it to a bunch of people who aren't in the same place as you. For that, Google Hangouts on Air is still available in YouTube. We don't know for how long, but it is the easiest way in YouTube to bring in multiple people and then stream right to YouTube. Otherwise, you'll need something like OBS or it's very in Streamlabs OBS to capture your source video and audio and stream it out where people can watch. OBS also works with YouTube. YouTube's a good choice to stream to since a lot of people already use it there. It depends on the savviness of your audience. They may like, oh, I know YouTube, or they may know Facebook. Facebook Live is another place that you can stream to from this type of software. Twitch is a great place to stream to. It isn't just for gamers anymore and it has some excellent community tools. It just depends on what your audience uses, what you're comfortable streaming into and all of that. To start talking about OBS though, Amos, give us and Roger, if you can jump in on this too because I know you do this a lot as well, give us the download on what OBS is and how technically savvy you have to be to use it. OBS stands for Open Broadcaster System Software Open Broadcaster Software It's OBS Whatever So it stands for Open Broadcast Scott. Who cares? It's OBS. Open Broadcast something. And it's multi-platform. You can use it on Mac or Windows. I believe there's Linux versions as well and it's a great piece of software. It does take a little bit of time to learn but once you learn it, it's very simple to get something out there and especially if you're going to broadcast to multiple sources. So you're going to go to YouTube and Twitch and Facebook. There are tools to help you with that. It's very simple. There's not a whole lot of really massive features you have to learn to use the software. There are really awesome features in it but for just basic streaming it's really easy. If you are on Windows and you are streaming, especially if you're streaming to Twitch, I recommend Streamlabs OBS which is basically OBS. It's a little bit older because OBS is open source so the Slavs version is a little bit older but it has a lot of plugins that work specifically with Twitch and it makes the whole process of streaming to Twitch very, very simple. Yeah, Streamlabs used to be just a website that you would use and you would use for like, you know, ways of putting in animated stuff that said, hey, you got some bits from a listener or somebody subscribed to your channel or whatever and they built on that and eventually did their own fork here and it's very, very good for gamers like extremely good. So I just wanted to throw my hat in and say if you're on PC, even if you're not gaming, I still think I would go with Streamlabs and I think they got bought by Logitech so your favorite camera people are now owning your favorite streaming software. And by the way, they're both free. OBS and Streamlabs OBS are free. Yeah, and the idea with OBS is it's taking Skype or Zoom or whatever you have and putting it out into the stream and it can interface with Facebook and YouTube and Twitch and all of that. An easy way to look at it is it's basically virtual it's a virtual video switcher so if you think what an audio mixer does add video on top of that and so instead of being able just to send audio to receive audio from a bunch of sources and send it to one place you can send pick up a bunch of virtual video in this case, Skype not Facebook but Skype, whatever Google Hangouts at some point or whatever you have and send it off to YouTube or send it off to Twitch and the great thing is it's finally integrated NDI which is a standard that NewTek the guys who make the TriCaster developed and it's freely available allows you to essentially connect between all those devices without having a resort to basically screen grabbing an image so you can think of it as a video output from one application into it and then you can send that out without having to have all those weird glitches it's not directly integrated Roger you have to install a plugin because I just did it last week and it was not for the faint of heart it was fairly nerdy it's actually downloading from GitHub and stuff they actually offer if you go to the NDI website it offers a complete download package codec as well as all the adjacent tools and something like Skype as long as you start it after you install all the codecs as well as all the feature sets it will see it and it will allow you to connect directly in Streamlabs OBS, not OBS but Streamlabs OBS will see it out of the box Allison I know you're bringing this up because you use a different one called Boinks tell us a little bit about that Boinks is the company the software is called MIMO Live and so our goal really is to bring you guys all into our madness you're just taking your baby steps in but when you want to take it up a notch MIMO Live is a commercial product for the Mac and it enables you to grab and switch multiple cameras, presentations you can add graphics like weather maps, lower thirds social media feeds, green screens you can scale and crop and convert all kinds of different formats for it again it takes NDI video streams the way we use it in my house is I do my live show from a MacBook Pro and that's a super capable machine but I was doing all of the video production myself in addition to producing the show and doing I mean I was like a one-man band spinning things and my husband has a high-end iMac and we thought well we really want to move the power the hard work over to his machine so we put MIMO Live over there and then I send him my screens and my audio my video all by either NDI or they have a thing called MIMO Call that actually sends it through a web browser window. It's a really really slick piece of software and allows some really nifty effects a lot like OBS but maybe a little bit easier to understand in my opinion I love me some open source don't get me wrong but big fan of BoiX's MIMO Live it's MIMX.com slash MIMO Live MIMO Live in case you're wondering. One tiny tiny thing I wanted to say about this real fast for those who aren't just trying to do podcasts or not trying to be the next game streamer or whatever it is you're just trying to talk to your grandparents or you want to have a presentation let's say you're holding a very small wedding because all the other big plans got changed because of everything and you're trying to broadcast this out to all of those hundreds of people that want to see it I would recommend all the stuff they've talked about so far but using OBS or something like it connect to YouTube Live specifically because in there unlike Twitch you can make a private video link only the people you give the link to can see it I suppose they can spread it out if they want to but if you're trying to keep your event or your broadcast out to many a more private affair do it on YouTube it also archives it so you can simply send that same private link out to other people and they can see it if they missed it like that's totally the way to go for that kind of event If you're lucky it'll even add captions Unpublish It allows captions it also allows you to unpublish And in YouTube you want to look for Unlisted the setting is Unlisted Private Now Bryce while we got you here if you really want to step up your game you want to go to the opposite side from what Scott's talking about you can use something like VMIX the aforementioned TriCaster stuff like that VMIX is kind of a Windows only sort of similar to Mimo Live like Allison was talking about VMIX is a broadcast quality video mixer VMIX video mixer that's what we use in the studio for our podcasts and like Allison was saying it brings in cameras graphics, lower thirds, virtual shots it has NDI support built in and what it helps for us is it gives us that next step in between you know, me what I do at home which uses using OBS and NBC going and using some huge new tech TriCaster and so it lets us have a bunch of shots it lets us mix and composite shots together very well it has its own audio mixing and routing features and it even has a video calling feature on one of the higher tiers of the software so if you're looking for a VoIP solution that might be a thing we've tried it once or twice but it's still kind of a newer feature but if you're looking for a software only solution without having to buy say a TriCaster box I really like VMIX for Windows and there's Wirecast which is similar to VMIX which is out there for Windows and Mac OS as well alright folks that's the basics but we'll finish up with a few items to make things a little better for instance I use Google Sheets to create collaborative documents that way people know we're using one right now this is the point in the show we're at here's who's going to say what it's collaborative everybody's got a web browser so it's really easy for people to get into I use backpack studios iOS app on an iPad plug into my mixer that way I'm able to play audio cues and oh thank you give myself applause stuff like that some of our favorite little extras that you can add Shannon I know you want to talk a little bit about IOT stuff I do yeah so as soon as I moved into this new house I was like I'm going to make everything smart so it's really easy to just plug and play and one of the things that I'm looking into is voice assistance in the studio so one thing I'm planning to introduce in this room which I don't have yet because everything's still in boxes is putting in a Google Assistant and connecting it to a Belkin Wemo as Philips Hughes which will automatically turn it turn on my lights and I can also set different scenes so I could do a different scene for each video or a different show that I'm doing so for example if I wanted to do a hack 5 call in guest hosting then I could have red for that show if I do DTNS I could do backlighting in blue so I can change those depending on different scenes I'm also planning to connect everything with Elgato's stream deck so stream deck has work with Philips Hughes you have to use a plug in to be able to do that which I'm very excited for I see Bryce has one as well as Amos yeah I have one too it's great it's wonderful one thing that I am doing with that as well for lighting is connecting the stream deck to my Elgato key lights and I have two of those right in the left one so I have a fill in a key and I can change the temperatures on those they did just introduce cheaper ones which are called the key light airs I think or key airs something like that but they are cheaper they don't have as big of a temperature range but they will work pretty much as much as you would like them to like whatever you would generally need them for that's what you would be able to use them for so voice assistance Philips Hughes and Elgato's can all basically work together to create a very easy to plug and play easy to turn on a scenario depending on what kind of shows that you're doing and that's the new thing that I'm going to be introducing in this place you think it's more fun to yell at your voice assistant who is supposed to have it all automated and have it not work all the time that's the way I do it that's the way you do it now I will mention that make sure that you're putting everything on a separate if you really want to get geeky and technical put everything on a separate VLAN so that you are not connecting a bunch of IoT devices onto your secure network at home especially if you are logging into a work related thing if you're working from home you want to make sure that you're putting all those IoT devices on a separate network so that in case there are any vulnerabilities they don't end up using that as a pivot point to get onto your secure machine now I'm going to jump in here real quick Allison and Shannon I know you both do this virtual LAN thing to keep your IoT devices on a secure main network where would people go to see a tutorial like that say they were interested in doing such a thing to have that functionality where would be some good resources for people to go to for that Allison and Shannon go first it's funny you asked that because I'm currently writing one right now for my YouTube channel so just keep an eye on my Twitter account to see when I introduce that video so I used to do a real complex thing and stuff but then the way I'm just doing it now is the private side of my Euro network every every device can only see out that's all it can do so by definition just putting it on the guest network anything I don't mean to offend every most everybody here but all Windows machines Android devices and IoT devices that are not home kit or over on the guest network everything in home kit and the Max and the iPhones are on the on the clean network nice another happy happy customer when it comes to the stream deck and the reason I love this thing is it goes back to stuff we said before about OBS and stream labs OBS there is direct built-in integration with those services so if you are someone who wants to do some easy switching between scenes between sources between between entire collections that you use in those in those softwares when you're doing your stream that thing is great it's like having your own little studio in a tiny little box it's amazing Allison we I know you wanted to mention audio hijack and loopback we kind of already mentioned that is there anything else you want to say about those or any other little little things that make the stream go better well one of the big things for me is that when you're working from home and especially if you're just if you're broadcasting getting feedback is harder you're not in a room full of people you can't hear what they're thinking so we do my show live to YouTube live but we also pipe the audio from through Mimo live Mimo live sending it to YouTube live but it sends the audio also over to discord and then we have a live chat room that I don't have any restrictions on we let anybody come into it it's podfeed.com chat and people chat to each other and it's giving people a place to be at a specific time I do my show on Sunday night to 5pm every single week so they know at 5pm they can go in there and even if they aren't listening to me which happens they have somebody to talk to and with everybody working from home now I have Tom you mentioned this on your show is all of a sudden there's a lot more people in the chat rooms but having them be able to talk to each other and get to know each other along with talking back to what you're doing is super valuable to me that makes that makes it a lot less lonely of an enterprise for me okay let's talk a little bit about recording not everybody who wants to stream these things needs to record them if you're using YouTube it usually is going to make a recording for you and that may be good enough but Amos and then Bryce what are some thoughts on recording editing that sort of thing so my first point of advice is if you are recording it and you need to make sure that it's saved always have a backup plan I can record on my mixer and then I record in my doll which I use Adobe audition and I record each track individually that way if someone ducks out has a lot of noise I can adjust that modify that however I need to and if I'm not in my studio I will use this H6 right here it's a beautiful device you can get this is probably the most elite one but there's cheaper ones out there but just make sure that you have a way of definitely recording all of your audio at least two ways because something's going to fail guarantee it the rule is two is one one is none and that's how that's kind of how you have to live if you're trying to record it for posterity and you can't really do the same thing with video but luckily I mean yeah Bryce has back always for video I'm sure yeah so like this this is going to be a very more complicated technique but the way that we record studio video in our studio is that we have three different computers doing stuff we have a computer that shows us our web content and videos and audio we have a computer that is handling all takes all the inputs and switches them and creates the signal for our feed and then we have a computer that just takes that signal and encodes it and pushes it out to our services and so there are multiple points there where you can have redundancy you can record at the streaming computer at the switching computer the second one as well as have a recording on the encoding computer on the third one so that you have redundancies if one breaks the other one is still going if one breaks the stream is still live another important thing in terms of backup when it comes to video I recommend a lot of people are going to want to record to say mp4 which is a very common internet format I do not recommend that at all please try to use a format that does not need to be quote-unquote closed I use MKV at our studio and here at home MKV is not much different than mp4 other than if your computer if you get a power out your file is not corrupted you can you can plug it into OBS you can do something called remixing MUX to turn it into an mp4 it takes no time to make that change and it really will save your butt when you have that power outage not only that but OBS and stream as OBS will both pause that format and they will not pause mp4s if you stop one of those videos it stopped you make a new one if you need to pause it stream you can do that with the format you're talking about and it stitches it and keeps it together now one thing about pausing it thank you for bringing that up Scott because I had this on my home computer and it was a big issue where OBS has a feature called recording quality and you can either set it to be the same as your stream so whatever your stream is setting out your computer is recording that same thing or you could have it be a different profile of recording higher quality lower quality whatever you want the thing that I ended up finding out on live on air was that I accidentally had not had it set to same as stream I had it set to high quality and my home computer was not quite beefy enough to handle doing two encodes to basically doing double the work at once and so if you're using OBS or stream labs OBS look for recording quality and if you have a consumer grade computer you're probably going to need to use the same as stream setting because otherwise it's just it's a lot of extra work for this other file that you may not find any difference with what Amos said back a little bit when he said the H6 he's talking about the zoom H6 if you're looking for it the portable recorder that he was holding up yes I will add one thing on top of what Amos and Amos were mentioning is that you're going to want a beefy computer if you think you can get by with your five year old laptop and you want to do video on top of it not going to work I have a Intel 4790k which at the time was like the stuff to get and it's overclocked to well over 4.5 it will get bogged down with everything including this video that I'm doing which is why I was killing my video when I was talking because right now I'm running at 85% of CPU and that's on top of the fact that I have a 1060 GTX card that's doing the video encoding as well as Skype video decoding so do plan to spend some money on PC hardware if you want to do that level of stuff just to bring it back to earlier if you're using Hangouts on Air to stream directly into YouTube you can do that on your Mac Mini it just depends on how much you're making your computer do and one of the things that I'll bring up is back to Bryce's point I record locally in 1080p I stream out on my live stream at 720p just to save the bandwidth and just make sure we got a good connection but then when I post our videos to YouTube I post them from the 1080p source that I'm recording locally so it's adequate because we're not gaming we don't need a lot of high graphics we're not Scott making magic happen with video games on the internet but when it comes to actually the posterity of it because the Twitch files will eventually go away YouTube kind of sticks around forever unfortunately for some of us so the 1080p file goes to YouTube where it'll be there for posterity whereas the Twitch stream is important to be lag free because you have a live audience to bring that down but I have the horsepower in my computer to do that to have one stream recording and then break it down to a smaller size and send it out to save on the bandwidth so that's not a bad thing One last thing while we're talking about resolutions is to consider your bitrate your bitrate is basically the level of quality of the image that's being generated if it's too low it'll look all blocky and watercoloury and if it's too high their devices are going to have to download a much higher stream and may have to decode a lot more information than a lower tech device would be able to do so all of the streaming services if you're running say OBS or if you're running a video streaming software on yourself all of the streaming services have guides on what they recommend for bitrates, resolutions all of the setup stuff so when you go into OBS and you look at it and you say I don't know what the hell a keyframe interval is you could look up twitch bitrate guide or mixer bitrate guide whatever and they have a very simple to follow formatting for that because it changes a lot there are a lot of different codecs and that stuff can be tough to guess if you don't have guidance and if you're not streaming video games and people are having a hard time seeing your content especially if they're on a mobile device or on a cell connection going from 60 frames a second to 30 frames a second is a massive lifesaver if you're not concerned about high quality graphics even then depending on the game you always 30 is reliable 60 is a freaking nightmare sometimes no matter what you're trying to do because of other factors you can't control so great advice stick to 30 when you can we know we've only touched the surface very basic to nearing advanced so don't be afraid to get on social networks where appropriate where we're available and ask questions or follow us on our shows and email those shows so with that in mind first of all thank you all of you for contributing your expertise to this I cannot thank you enough this has been better than I could have possibly imagined and let's go around the horn Scott Johnson where can they find you at frogpants.com and every show you hear is an example of the stuff we talked about so if you're interested in the results you can have go check it out frogpants.com Shannon Morse what about you youtube.com slash Shannon Morse which is where I'm going to start getting the ball rolling in my new studio as well as twitter.com slash snubs to keep track of my everyday whereabouts in my house since I'm not going anywhere how about you Justin rubber young twitch.tv slash Justin our young Bryce Castillo you can see me streaming a lot on twitch.tv slash night attack if you want to get in touch with me twitter.com slash brykis b-r-y-c-a-s or neshcom.com is my website there's a contact form there n-e-s-h-c-o-m dot com Amos you already slipped your contact info in but give them the pitch again if you need to hold me I'm on the only social media platform that matters I'm on Twitter at Ethan Kane E-T-H-A-N-C-A-I-N-E and they can find me pretty much anywhere else at Ethan Kane or Ethan Kane 77 everything good can be found at pod feet dot com right Alison Sheridan that's right pod feet dot com for all your greater podcasts with an ever so slight Apple bias Roger Chang what about you you can find me if people want to ask you a question oh you can find me at ask me a question you can find me at Roger at Daily Tech News show where you can find me at Twitter at Jolly Roger Sarah Lane well you can find me at dailytechnewshow.com but I'm also on Twitter and Instagram and Facebook all Sarah Lane although I am on Twitch at I am Sarah Lane and there might be more streaming in my future look at that I'm also at dailytechnewshow.com feedback at dailytechnewshow.com Roger and Sarah so you can ask us questions there I may detect on Twitter and thank you thank you thank you everybody for being with us hope this helps people out and have a good day I guess


Soldiers and sailors with USNS Milanochate gave a musical performance at the School and Center for Children with Disabilities on the island of Kiribati as part of Pacific Partnership. Doctors at Walter Reed Medical Center are turning to a centuries-old method to help service members with mental disorders. Yoga combines stretching, breathing and meditation, and doctors are using it to help treat disorders ranging from depression and PTSD to schizophrenia. Spay war celebrated its 75th anniversary by giving a rare glimpse of what the future Navy may look like. 3D imaging, underwater optical communications and virtual reality simulators were just some of the technologies on display. And finally, each year women from around the fleet be with senior Navy leaders to discuss issues related to women in the Navy. This year's Joint Women's Leadership Symposium encouraged women to embrace their future now. The Symposium wrapped up June 12. For these stories and more, visit navy.mil.


Alec Baldwin's in the news again. Let's do a really quick review on what we know so far My favorite is the what are you doing up staging me moved by Alec Baldwin more than one time It was she steps excuse me. We were a very very excuse me I believe the excuse me is don't upstage me This is an actor and somebody walking in front of him on the camera very excuse me But there are a couple of things that I think you can see good are going on here my assumption based on what I see is that yes They've been followed by these reporters and yes, they're getting frustrated I'll bet there's a bunch of going on in the car right now over that very thing Look what we've gotten into and they're bickering and going back and forth because you can sense some Stress between the two of them as they get out. You can't miss it They they literally take their mark at the start of the mark is the little cross on the on the ground Where you stand because you know you're gonna be in the right place for the camera and so yes, I think some of it is is is is push away of Of is it is it hilarious hilarious something her birth name isn't even a laria. It's all American Hillary Anyway, I think part of the push away is is number one, you know, don't crowd my lines and don't crowd my my space I've got this handle. I think Alec turns to face Hilaria, it's all American Hillary Alec turns to face her with shoulders and feet Pointed at her which I think indicates a little aggression Territory and intent and I think there's a possibility that it might not be don't steal the spotlight, but don't embarrass me Because he has a clear narrative that he's trying to do and she's clearly doing the opposite of what he's trying to do And be friendly and show some candor We're seeing everything that shows he's he's in a panic and he's worried It's almost like somebody's jumped out the car and said all right, man.  His his respiration is up. He's talking really loud He's looking back and forth his eyes are wide and they're all baggy. You can tell you and slap a whole lot He's tired. I'm sure there's got his wife out there run around with a camera Excuse me, excuse me once she gets close to him. Wow And when he starts talking about the an act he can't talk about an active investigation I think he's he's didn't want to talk about how much trouble he's in. I'm not allowed to make any comments Click right here and watch the entire episode


This is stock number 10534. We are here at Summit Automotive in Fond du Lac, Wisconsin. You're new and used SUV headquarters. Today, we are checking out this super clean 2015 Chevy Tahoe. This vehicle has the 5.3 liter V8 motor. It has been fully-safetyed and inspected by our service shop. It has a fresh oil and filter change. All the fluids have been checked and topped off. The vehicle has been gone through mechanically 100%. And this Tahoe is 100% ready to go. Black clear coat is the color we shoot all of our videos in 1080p, 60 frames per second. So if you have HD capabilities on your computer, tablet, or smartphone device, turn them on right now because it's like you're right here looking at the vehicle with me. And definitely your best way to ensure the quality and condition of the vehicle before seeing it in person. If you want to check out all the photos on this Tahoe in the upper right-hand part of your screen, there's a link right to that website. It has gear, 20-inch alloy rims, painted alloy rims. And it has Nitto Ridge Grappler LT295 65R20 tires on here. These tires have probably about 50% to 60% of the tread left. I'm not really sure if it has a leveling kit or an actual lift kit, but this vehicle you can see right there if you're curious. Definitely sits up a little bit higher than your normal Tahoe. The ride's great though. I've driven it up and down the highway. Pretty good. Front bumper and grille is a nice shape. Pastner side front fender, no dents or dings on that. And the passenger side rim, no scuffs or scrapes on there. As you go down this side of the vehicle, you can see just how clean the body is, how reflective and mirror-like that paint is. We take these HD videos so if you are far away, or even if you're close by and just cannot make the trip down but you're still interested in purchasing the vehicle, you can see the vehicle, hear the vehicle, and have confidence in what you're looking at before you even get here. So that when you do get here, there's absolutely no surprises. And that's why we offer the Summit Auto Fastpass option in the upper right-hand part of your screen. Another link right to our website where you can apply for financing, get pre-approved, see a payment quote, even appraise a trade, all from the comfort of your own home. So that when you get here, all you have to do is drive the vehicle and the whole deal will be pretty much done. All you gotta do is drive it and love it and you're all set to go. Back tires have just as much tread as the front tires. I do want to point out that this vehicle has a spare tire. It, because of its size, it doesn't fit underneath and it does have a spare tire underneath there but we have a matching spare tire that goes with the vehicle. I got it sitting over there so that I can show you the back seats but that does come with the vehicle that's included. The rear gate is in really nice shape as well. And this is a power lift gate. I also have a couple roof rack rails over there that go with the vehicle as well so they come inside the vehicle too. Back storage area is very nice and clean and there is storage underneath there.  Pretty easy. To get this gate down all you gotta do is press this button and it'll come on down like so. It does have a backup camera, you get the blacked out Chevy logo backup parking sensors, full towing package which includes a receiver hitch and seven pin wiring. And as you go down this side of the vehicle just as clean as that passenger side didn't see any dents or dings on the rear quarter. Really nice condition and for full disclosure this back rim is in excellent shape as well and that spare tire is in excellent shape too and that is a brand new Nitto Ridge Grappler 295 65R20 tire on that rim. So that is a brand new tire. Power mirrors, you get the driver side blind spot mirror. Inside the LT1 package gives you the gray leather interior. There are no rips, there are no tears on these seats. Both of these seats are heated. I believe both of them are power. Factory floor mats throughout.  You get the tilt and telescopic steering wheel. Bose sound, memory driver seat, power windows, power locks and power mirrors. As we hop inside the vehicle here you can see that this one has 66,083 miles. You get a digital spinometer and that instrument cluster is very nice and clean. You get the leather wrapped steering wheel. Bluetooth and information center controls on the right. Cruise and forward collision warning controls on the left. Comes with the six speed automatic transmission with the optional tap shift. This one comes with the Chevy MyLink radio which gives you AM FM and Sirius XM radio capabilities. And you can see right there that it does have the factory navigation system on it. This is also where your backup camera shows up. And you do get storage behind this radio, kind of a cool feature. And there is a USB charger in there as well. Has a CD player, dual climate controls, heated seat buttons. You get two USBs and a 12 volt power point there. Passengers side floor mat and seat are in excellent condition as well. No rips or tears on that. Smells very clean inside this vehicle. It does have the Blu-ray player right there. Power sunroof. Here are all your controls for that rear gate. That's why I wasn't going all the way up because it was on three quarters. Home link buttons for your garage door security systems and lane systems. And you do get SOS and on-star assist buttons in the rear of your mirror. We'll take a quick look at the back seats and then we will check out under the hood. Very clean up front. Back seats are just as clean. No rips or tears back here. This is a second row bench. So this is an eight passenger vehicle. Latch child safety system with three latch points. Factory formats and carpeting are in excellent condition back here. You do get rear air controls. You also get a 110 volt, 150 watt plug-in, another 12 volt. A USB and an SD card slot as well as an AUGS and a yellow, white and red hook-up. And then you do get that Blu-ray DVD player up there as well. These seats fold down like so and then they fold up like so. To get to that third row, back carpeting is in excellent shape. Third row seats, no rips or tears back there. Do get a couple indents on them just from being down with that tire on there. But those are in excellent shape. Carpeting is in great condition. And the door is in really nice shape too. We'll take a quick look under the hood. I would personally like to thank you for checking out the video today and hopefully from this HD video. You'll have been able to tell just how clean this Tahoe is all the way around inside and out. Under the hood we have the 5.3 liter V8 motor. Engine bay is very clean, runs very smooth. This vehicle has been fully-safetyed and inspected by our service shop. It has a fresh oil and filter change. All the fluids have been checked and topped off. The vehicle has been gone through mechanically 100% and is 100% ready to go. To see more pictures of this Tahoe or one of our other 450 new and used cars, trucks, SUVs, minivans, Wranglers, half-tons, recoder-tons, one-tons. You name it, we got it. Go to our website www.summitauto.com. Full pictures and descriptions of every single vehicle from two locations all at summitauto.com. If you'd like to check out more HD videos, you can go to youtube.com. Remember to like, subscribe, and share on this video and all the videos that you see there. In fact, in a second you will see a link to subscribe to our YouTube channel on your left. A link to more Tahoe videos like this one on your right. If you have not been to our website on the bottom link to this vehicle on our website, click those. Check us out and we really look forward to helping you with this super clean 2015 Chevy Tahoe. Chevy Tahoe LT1 four-wheel drive in black clear coat. Thanks again.


So who are we? I'm Ramya. I'm a software engineer at Pivotal. My name is Miriam. I'm also a software engineer at Pivotal. So what is release engineering in our context? So our product is made up of many different teams and each of these teams contributes specific parts of the code to us. And then the release engineering team takes this code and runs them through pipelines which are called test suites, also called test suites, and then base them into a product and serves it to the customer. What does this mean in terms of Kubernetes? We're talking about different integrations like Harbor, HCD, and Fluentd, et cetera. So teams integrate with these and share code with us. And then the relinch team runs them through pipelines or test suites, an example of which is on the slide there. To create a product called Pivotal Container Service and then that's the one that we ship to customers. Few things before we proceed. What is concourse? Concourse is a open source tool. It's a way in which you can write tests into different jobs. You can combine them into different jobs and then create a pipeline. This is an example of a pipeline and you can have different jobs sequenced one after another and you can have like resources, resource types and inputs and outputs to this job. So it's just a way of having a test suite. It's anonymous to that. And it's a pretty cool tool. You should check it out at concourseci.org. We'll be referencing this frequently through our talk. A few more terminologies that we will be using interchangeably in our talk. When we say fly, we mean running a pipeline or running a test suite. A pipeline is equal into a test suite and a lock implies an environment or a test bed and a pool is a group of environments which are configured the same. So we're going to tell the story of our release engineering team and our story starts in April 2018. A new team had just started and they wanted a fresh start and we were facing two main problems. We were shipping too slowly and we had a lot of bugs in our code. So naturally we asked ourselves, why is this the state that we're in right now? Well, one of the things is that all of the scripts and things that we were using had worked at a certain point but the teams that were integrating with Kubernetes have been growing and we've been supporting more and more integrations. The size of the relinch team itself had stayed the same. So we no longer had enough people to keep up with all of the work and we had done a lot of things super like duct tape. So now we had a lot of technical debt that we had accrued and this was resulting in us releasing slowly and so our customers were unhappy. We broke it down into these particular issues that we had. We had written a lot of scripts that were not being tested. We had a lot of YAML that was duplicated everywhere. Our environments that were being used for tests were in high demand and we didn't have enough of them. Our pipelines were taking way too long. We didn't have a very well-defined integration process and we still needed to sort out our open source licensing. So the first thing that we wanna talk about is no testing, one of the problems that we had. So why didn't we have any testing in place? When we looked closely, we found that we had a lot of bash scripts written in order to do our testing and bash is really hard to test drive or even just to write tests even after you write the script. It's also intermixed with Linux commands here and there, so it's hard to notice bugs. It's hard in general to just program in bash. So we decided to tackle that first and as a team we decided to change it into programming languages. So what could we go for? The two options were either Ruby and Golang and the reasons for these were because the team had prior experience in these areas. There are a lot of Ruby experience in the team and there are a lot of Golang experience in the team. Also, Ruby is more testable or test driven and Golang is also useful for concurrency and parallelism which we could really make use of. So here is an example of how our scripts looked before. For example, we had an environment metadata file and let's say we had to connect to this environment, right? So we had all of these properties in maybe like YAML and we had to read through it and find out the proxy and the username and password in order to connect to it and this is how we were writing it in bash and as you can see there's like YQ or AUK or SED and there are other commands that we were using interchangeably. This, as you can see, is untestable because we have to test, we have to mark out a lot of things for these. So then we decided to do bash in Ruby and this was an intermediate step. So what we were doing was reading the environment metadata by opening files, reading it into a hash and then getting properties out of it, doing whatever processing we wanted to do and then closing the file and this was happening pretty much everywhere in whatever we were referencing the environment for testing. This is somewhat testable. Like you can still mark out a few things and get away with it but not completely testable. Like for example, there's like splitting on colon and then expecting that the first array of an item is gonna be present, et cetera. So it's quite error prone. So we decided to completely change it to Ruby and what we mean by this is trying to think of the things that you manipulate in your test suites as objects that you pass around. For example, the environment metadata could actually be a lock file class which had properties that you can fill in and this is super testable in Ruby. So we try to think of it as in an abstract way and started creating classes for lock files and pipelines and it was easier this way to test drive it. All right, so the next problem that our team was facing was having a lot of YAML duplication. Doing a quick Google search showed that everyone else was also struggling with the same problem. So why is duplicate YAML a problem? So we were getting a lot of YAML from Kubernetes configuration in general and our CICD tool that we were using concourse is completely configured by YAML. So we were dealing with a lot of YAML that looked very similar and was slightly different and the process of looking at a bunch of files and trying to figure out what was different between these files that created a lot of cognitive overhead. The second part is that we were as part of our process we were copying and pasting a lot of files and then we would just change one or two keys and sometimes our team, which is human, like forgot to go back and change these keys. So we had like mistakes like that. The team was very unhappy. Anytime we needed to add pipelines, modify pipelines, do anything, everyone got very sad about it and it's always important to try to have happy devs. It was taking a lot of time to make these changes and because we were sometimes copying and pasting and forgetting things or were indenting in different ways the pipelines were becoming inconsistent. We decided that we really needed clarity. If we go in and change a pipeline, we need to know exactly what makes this pipeline special from others, which part do we really need to change and did we do it correctly? So the first step is that we had gotten used to having really long YAML files. Like our YAML files were almost a thousand lines and we had a lot of them and it was really, really overwhelming to constantly be searching a thousand lines. So we decided to divide it into as many small pieces as we can and that way we can just see what we're dealing with. After this, the slide cutoff that says embedded Ruby, but after this we decided to use ERB. So ERB is basically a way of using Ruby in YAML. So this is an example where for us we were doing testing and we were testing Kubernetes against all these different infrastructures. And so instead of just repeating AWS, AWS, AWS, GCP, GCP, GCP, we now can just use like these four loops. And so that was really helpful and it really dried up our YAML. But there was still a problem with this. And the problem is that when we had Ruby inside of YAML, YAML cares a lot about indentation, but Ruby doesn't. And so every time we were actually putting YAML inside of ERB, we actually needed to make sure that the indentation matched. And so we were having a really hard time making sure that the indentation where we were actually taking this and moving this had the same indentation as the other place. And this was really hard for us. So we decided we wanted to try something else. And this time we went with anchors and aliases. So if any of you are familiar with YAML, YAML itself has this notion of an anchor. And when you define an anchor, you can just reference it in other places. So we would start having one resource type defined in one place. And then we would just use it in a lot of other pipelines. So this did a really good job of drying up. But we noticed that it wasn't taking care of the cognitive overhead. We were still spending a lot of time opening up a lot of different files to figure out what we were actually including in the pipeline. And specifically for us, we were seeing that certain parts depended on other parts, but we couldn't even tell that they used each other because now we dried up everything so much that we couldn't tell what was even failing or why it was failing. The other part of this also is that we needed a tool that would actually go in and understand YAML anchors and aliases. And so we actually built a CLI, which is always a bad idea. And that was hard to maintain and everyone was grumpy about it. We decided for attempt number three that we were just looking at it the wrong way. We were splitting this up based on how it was configured in concourse. And what we should do instead is just look at how our pipelines are split up. We noticed that there's a lot of parts of our pipeline where at the top we would dry up something and that thing was referencing something at the bottom. In this case, a resource type that's being used by a job. So we were looking closely at our pipelines and sometimes we would have a configuration like this. We had the infrastructure layer, so vSphere 6.7 in this case, and then we had the networking layer, which is NSX. And then we had the version of whatever software we're testing and then the scenarios, like in this case an install. And we decided to have these templates, these templated files, and we're going to just have all of these dependencies in one file. And then when you actually come to make a new pipeline, all you need is a file like this and it's completely empty. And just based on the name of that file, you can figure out which templates you need to use. Okay, so the next problem we had was environment contention. Now to explain what we mean by environment contention, we as a relinch team were testing on many different IaaSes. Some were public like AWS, Azure, and GCP. And then we also had like the private ones that we make sure to test on like vSphere and then NSX pipelines. Nimbus here is an internal tool that VMware uses to spin up NSX environments. So basically we cover all of this in our testing. So why is this a problem if you don't have these environments around? First of all for public IaaSes, like we had many different pipelines, but the number of environments that we had was very little. So the ratio was really skewed. We had 50 plus testing configurations to test, but there was no automation around environments. So we were basically going to the console and then like creating them manually and like making all these configurations and forgetting some parts of it. For private IaaSes, there were other issues like Nimbus like I explained before is based on a quota system. So whoever requests for an environment first gets that environment and let's say the quota for an entire organization is over you wouldn't be able to request for another one. So you just have to wait till someone cleans it up and then you have access to it. So basically we were really sad again. This is causing a lot of human errors and configuration errors and made us sad like the dog in the picture. So how did we fix it? We used Terraform which is an open source tool which can create environments on the fly. So we started having a pipeline for creating an environment and destroying an environment and this was per IaaS. So it was like a one click solution and we just automated it. So we never had to bother about like creating it again unless just going to a pipeline and clicking on a button. Also the pipelines were recycled sorry the environments were recycled after 24 hours if they weren't used. For the private IaaSes that is Nimbus specifically our infrastructure team from VMware came up with this tool called Shepherd which pre-made environments and then preserved it for the relinch team and we basically started having a shared pool for all the teams that were going to test on vSphere NSX. So this way we had on-demand environments and lesser contention because now we were at least we were at high priority. Okay so another problem that we had is that our pipelines were taking way too long. So we were watching a lot of our deadlines pass us by and we were missing them and we're like why are we always missing our deadlines? And also I totally screwed up the formatting on all of these it's fine. You didn't notice, you didn't notice. So we looked at pipelines that are time consuming and we saw that they're time consuming for two reasons. The first is that they're really slow and the second is that a lot of them were really flaky. So we wanted to look at the ones that were slow first. First of all, our pipelines were doing a lot of tasks serially that could be paralyzed and most people on our team were comfortable with Ruby but we decided maybe it's time we actually start rewriting these and going and actually use go routines and by being able to leverage concurrency and parallelism we were able to get a job that ran in every single pipeline from two hours to 15 minutes. Another thing that we wanted to tackle is that in a lot of our test suites we were spinning up a cluster and then we were running tests against that workload and then we were destroying the cluster and then this was happening also in serial. So we decided let's just have one job create a cluster and pass the information for that cluster to all the jobs that need it and have those jobs run in parallel. So this resulted in our pipelines being sped up by four times. The next thing is we wanted to look at our pipelines that were flaky. So we knew that this was the flow that we were experiencing that was causing us pain. We would have a PR get merged and we would build the product and then we would try to run it through almost 39 pipelines against different infrastructures and we would have to trigger this many, many times same exact inputs until it went green and we could ship it. What we really wanted is to be able to merge it run it through all the pipelines once and then see it go green.  So we did 112 runs took about four weeks and we realized that our pipelines were succeeding 55% of the time. So after some very complicated analysis this very small here, we actually realized that there were certain jobs that were failing a lot certain jobs that weren't failing as much and we decided to prioritize these jobs by failure rate. This was actually really successful because we discovered a lot of random errors that really hadn't been looking into apparently running out of disk. Apparently sometimes we were retrying and it was failing and by being able to really look for these we're able to look for those specific teams that were in charge of wherever that integration was failing and they were able to fix these things and this made a big difference to our speed overall. So the next round we had was no integration process. So at this point we had all of these pipelines all the teams had access to these pipelines and we had no idea who was triggering what pipeline at what time. So this was like really initially before we had any processes in place. We would come to work in the morning one day and find pipelines parsed or pipelines run or pipelines deleted. And also teams had no way to test their code. They were pushing changes to master directly and actually debugging after shipping it because they had no way of knowing where their tests were failing or they had no way to run their tests in the first place. So we could fix some of these issues like restrict our concourse access. Like we created, we made it more private and only the relinch team could access it and we started introducing code reviews and PR processes but it still doesn't fix the issue of the teams not having a way to test it. So the teams were making pull request to master and we were running it in production concourse but every time there was a problem with their code the only way they would know it was through relinch because master was broken and we had access to this. So this is where relinch as a service comes in. So basically what the relinch team did was created an exact same concourse that was available to the teams and each of them had their own separate buckets and own separate places to host their code. So basically what they would do was they had access to our code so they would run those pipelines in the separate concourse and they could also run only what was relevant to them. Like let's say they wanted to test only vSphere and AWS. They had access to the YAML files and all they had to do was run it in this concourse and test it out. So this way we just had a script which made this configuration. The teams would change the location of their buckets and their repositories, run the pipeline in the separate RAS concourse and see if it fails. If any job fail they would fix it and then PR it to us and then we would have it green in our production concourse. So this way both sides had more confidence. The teams had more confidence because they tested their code and we had more confidence because we know that they had a way to test it. The last part is open source licensing. So at this point we have pretty much sped up most of our processes but there's still one last hiccup. So whatever we are shipping is this legal and one of the biggest problems of shipping open source software is knowing whether it's legal or not. Whether have you acknowledged for every open source software that you have included. So who's responsible for this at this point? It was the relinch team. So what does this mean when we're talking about open source licensing and why is it hard? When we're talking about Kubernetes especially we're talking about containers and containers are made up of software packages. They're made up of base OSs and it's pretty early right now for us to know what is included or how do we scan these containers is hard. Containers can either be on the internet or they can either be created by the teams themselves and we need to know what's included in each of those containers. So we started scanning them by ourselves. We take each of the component teams code, we scan them, we wrote scripts to scan these containers and find what software was included in that, find the software packages, find their licensing through open source tools and then create OSL. We also became smarter on what to OSL and what not. For example, we started asking the teams to do their own part for the OSL because the teams know what containers they include. We also did OSL only for GA bills and not for only for general availability bills and not for all dev bills that we were like producing. We also started doing this much earlier in the process when we knew that something was a candidate for a build and also started providing teams with early access to the bills so they could scan it after and verify that they had included the containers that they were or scan the containers that they were including. So at this point in time, we had tested code. Our YAML had gotten dried up. Our process was more defined. We've automated a lot of the environment creation and our pipelines were faster and we actually had, we were in good legal standing. So that's always good. At the beginning of this, when the team had started, we were shipping every six months and now we were shipping a new build every 10 days. Our pipelines went from 12 hours to four hours. A large part of that was parallelization and we were able to even have a turnaround for bugs, CVEs, patches of 48 hours. Our release time for one of our builds, 130, was 35 hours from start to finish. That was from testing to OSL. We got it down for release 140 to about 10 hours now. Though OSL is not included in that one, but yeah, OSL, actually OSL is not included. This is just testing part, I'm so sorry. OSL still takes a lot of time. So release engineering as a service, it's doing Kubernetes the hard way, but it's worth it. One of the best parts about all of this is that someone made us pie at the end because we made it a lot faster. Thank you so much. Are there any questions? I'm very interesting. What scanning tool are you using to scan the license? So we developed some scripts internally. We used Bash, but it's actually a really hard process. We did not find any tools that are reliable in the market, so we had to develop it in-house. But the basic idea, it's little manual at the moment, but at least mostly loaded in Bash. Yeah, we used for actually scanning the rest of the product we used for Sology and I forget the other one. Yeah, license finder and for Sology, but they don't do container scanning. For containers, we did it ourselves. Okay, thank you. During that infrastructure as a code part, you mentioned all the test tools will be commissioned after 12, four hours if they are not used. How do you decide? How do you know if that piece of infrastructure was used or not? So we have a GitHub repo that has the actual lock files in them. The credentials are stored elsewhere, but it basically, the act of claiming an environment, you actually move it from one folder in the GitHub repo to another folder. So we look at the time that it was claimed that it was basically moved to that folder and that's the time that it was claimed. So we have a pipeline that runs every hour. It looks at all of the locks that were claimed from that repo within the past 24 hours or more, and then it moves them into the destroy pool. So we basically use GitHub and we look at commits, everything's automated through commits. So that's how we figure out how long an environment's being used. Great, thanks. You just mentioned the third party dependency management. I'm not sure, do we have more information about it? Are you asking around OSL or about another aspect? Yeah, I mean, since you project to rely on some third party dependencies, either Java files or RPM files or some other files, if your project has lots of microservices or projects, all of these services should rely on some third party dependencies, then how do you project, all these projects manage the third party dependencies? Is this for OSL or in general, are you asking? Please go back to your slides. Third party dependencies, I just saw. Just tell me when to stop. No, no open source. Oh, okay. Oh, next, all dependencies in one file. This one, okay. How do you manage this? That's a good question. So, okay, so what we do on our team is every time we make a change to any integration in Kubernetes, what we do is we build our product with that change and we have to get an environment that's configured with this configuration. So what you're looking at here right now, these are all configurations and for vSphere like, for vSphere 6.7, for example, like we actually have our own set of servers locally that we're using to test. For NSX, we're actually deploying that networking layer on top. We do this for public infrastructures as well, like for GCP, for example, but with GCP, we would just do flannel for the networking layer and the OM, that part is sort of a product that's like we're also testing against and the scenarios are just, install is like, we're just trying to see that we can get it deployed, get a cluster, get it to pass against tests. For upgrades, we're just trying to make sure that we can go from one version to the version after it. So for all of these, we basically are using a lot of environment management. Like all of these are just environments. I don't know if that answered your question. Looks like maybe I didn't, I'm sorry. Oh yeah, okay. So we do have, so we have something we brought up and where did we bring it up? Oh, I was gonna say, okay, like just for these environments, I just want it, the ones on the right-hand side, those are all public infrastructures. So we're using like Terraform to be able to deploy everything on those environments. For the left-hand side, we have like internal tooling and we're using that to be able to specify like which specific integrations we want in it. If we like have, I don't know, like Veerli or Harbor or something, then we're specifying it that way. But I guess a lot of these are right now, they're internal. I don't think that we've made them available yet, but some of these things maybe someday when they're like ready, we could share them. Did you ask in terms of YAML files or were you asking in terms of environment? Oh, okay, great, awesome. All right, thank you. You can find us after, if you have any more questions. Thank you.


Okay, this is James P. Madonna of Megalife 21. I'm taking a video of a very interesting public restroom faucet. I happen to like this because it's not low flow. It is not an annoying low-flow water-saving faucet which I despise. It looks like a waterfall. What do you think, Commodore Jeff Sandbello? I like it very much. I wonder if these are available for homes. Interesting. Very interesting. I did not have to take forever to rinse my hands.


Welcome, everyone, to the Tag Runtime Update Lightning Talk. I am Taylor Thomas. I work at a company called Cosmonic. And I'm Alexander Konevsky, or most people know me, Sasha. I work for Intel and Tech Lead for Tag Runtime, co-chair of Container Orchestrated Devices Working Group. So we're both co-chairs of various working groups that are part of this. But the Tag Runtime group is made up of a bunch of different projects that are constantly ongoing all around the runtime space and all the main groups that are listed are all on cloud-native AI, batch system initiatives, container orchestrated devices, IoT Edge, special purpose operating systems, and WebAssembly. You can get this QR code leads to that list and a link to all the relevant information about the different working groups. And just as a general point, these working groups are completely open to the public. For example, I know in Wasm, we run a meeting every two weeks, the meeting's on the calendar. You can come join. That's pretty much the standard practice for a lot of these. So if you never join in one of these, a working group's a great way to get started and kind of show your ability to help contribute to the community. So just keep that in mind. Now the purpose of this is to go over two of the working groups that we're involved in and kind of give you an overview of a lot of the emerging technology that's happening. So first off is WebAssembly. And just out of a quick show of hands, who has heard about WebAssembly? Okay, now lower if you have not used it. Okay, see, there we go, that's everything. So it's emerging, but there's a lot going on. I just came down from Wasm Day upstairs. So Wasm's showing up in a lot of places in the cloud native ecosystem. It's showing up in Docker and the container run times. It's showing up in Kubernetes via RunWazzy and other extension points. So if you look in the ecosystem, you'll see it in places like Envoy, Kubor, Open Function, those kind of things. You'll also see Wasm native application platforms. Wasm cloud is one that I'm a maintainer of. There's also Wasm Edge. Those are the two ones that are part of the CNCF. And then there's plugins and extensions and you just have to go to the CNCF landscape and search for Wasm and you'll find it in some really interesting places. So that's where we're currently at with things. Just a quick update on where we're at and hopefully this kind of is a motivation for people to come and take a look at what we're doing. And so we're working on standardizing an OCI artifact definition. This will become the standard way to store WebAssembly modules in an OCI registry and be able to fetch and read data from them. That might not sound like much, but once you start to learn about what WebAssembly does and particularly the component model, that sounds really cool. Then we have an ongoing thing of WasiCloud. WasiCloud is a group of interfaces. If you don't know what that is, it's okay, you can go learn more. They help us interface with the different cloud technologies, things like key value stores, blob store, those kind of things. And then we've been doing a lot of curating Wasm knowledge and learning. So if for some reason you're a Wasm project and haven't presented there yet in the CNCF, we like to hear from it because we record those meetings, go check them out. If you go check them out, we have people presenting on every project that we could find that's doing stuff in the CNCF. And soon after this conference, we'll be starting work on a white paper on platform engineering and how WebAssembly affects it. So that's what we're up to. All right, so switching gears a bit. I came from a working group which is working on accelerator devices. It's not really emerging, like the work what we are doing, it started like several years ago, all the preparation things, but now with rise of AI and machine learning workloads, suddenly it become like, oh, hot topic. Everybody talks about it. Everybody interested and so on. So we are a small group of people, like mainly Intel and Nvidia, but we have like a lot of fellow travelers from like Red Heart IBM, like from container runtimes and so on. We are patient about how to enable the devices. And we started to look from very bottom looking how devices, how accelerators can be exposed. And we come up like main outcome of a working group is container device interface. It's something which is happening on the runtime level of the CNCF stack, which describes if I want GPU zero, what it actually means for the containers, like what our libraries, what hooks needs to be executed, what permissions needs to be given, and so on and so forth. So main discussion, what how it works and so on, it happening in our tax CNCF repository and everything around it. We can't fit in this talk like with detail description. So if you're interested or if you're curious how devices are exposed in the Kubernetes stack and actually rest of the CNCF, look at the several sessions on the KubeCon, which have like keywords like year A, CPU, CDI. All of those leads to something what our working group is working about. And we have a tag runtime booth. So again, we can talk about more details. I have one and a half minute. So crash course, what CDI is about and how it helps. With CDI is, as I said, like it's low level specification of what accelerator means. We are using it as a basis for year A, dynamic resource allocation, where we can say I want GPU with particular properties, I want X amount of RAM, how to share it, yada yada. At the same time, we are trying not to teach the KubeNet about all those internal details of accelerators and so on. So to KubeNet, it's something seen as a like simple string which identifies the device. And when we have like protocols between the KubeNet and runtime which says this workload requires this particular device. And inside runtime, in container D, in cryo, it got expanded to what container actually needs to happen like on kernel level and drivers and so on and so forth. We have fellow traveler, like node resource interface which can help to fine tune your native resources like CPU and memory to help your accelerators. But it's a bit of side story. Beside Kubernetes, we implemented the same support for like most popular common line tools. So Podman was implemented several years already. Docker, it recently got added. Singularity, in some mode it works. So if you have like common line experience with containers, if you have like HPC kind of experience of containers, I think we have it covered. If you see anything can see I'm stuck, but it's not covered but we still want to use accelerators. Talk to us. We will help you with that. See you all at the tag run, time booth.