Hey, it's time for a tech talk number twenty-eight eight. That's right. What do you eat I? Loose count. I lost counted about five or six We got some why who's counting anyway, this is a man true Well, it's so we can keep track of what's what and where's where so someone says well what episode was that up? Well, it was on number 17 it's cuz a superfan can go you remember back on tech talk number seven when you said that the best microphone for voice That's not what we said really Anyway, we got lots of cool stuff to talk about tonight We're gonna still continue on with this discussion about interfaces and stuff and we've got that We've got an actual man on V4 and an unboxing and an unboxing of the evo Ooh evo for all that coming up on voiceover body shop tech talk From the outer reaches they came Bearing the knowledge of what it takes to properly record your voice over audio and Together from the center of the VO universe. They bring it to you now George Wittem the engineer to the VO stars a Virginia Tech grad with the skills to build set up and maintain The professional video studios of the biggest names in VO today and you Dan Leonard the voiceover home studio master a professional voice talent with the knowledge and experience To help you create a professional sounding home VO studio and each week They allow you into their world making the complex simple Debunking the myths of what it takes to create great sounding audio Answering your questions showing you the latest and greatest in VO tech and having a dandy time doing it Welcome to voiceover body shop Tech talk Voice over body shop tech talk is brought to you by voiceover essentials comm home of Harlan Hogan signature products Source elements remote studio connections for everyone voice actor websites comm where your VO website isn't a pain in the butt VO to go go comm everything you need to be a successful voiceover artist J. Michael Collins demos when quality matters and voiceover extra your daily resource for VO success And now live to drive from their super secret clubhouse and studio in Sherman Oaks, California Here are the guys I'm Dan Leonard, and I'm George Whidham, and this is voiceover body shop or VO BS Tech talk Seem to doesn't seem so big to me, but you know, it's it reverberates it's shrinking Get an interesting email from our friend Jack de Gaulia today He he is a thinker of voice. I he tends to really think about this guess is but he was like You know, he was talking about this this acoustic video from The the sense of fear of the the mosque in Istanbul Though what's it who's the one? Yeah, but it was like this You know what this what this coral group would sound like normally and then what the acoustics were like in this huge Mosque slash Cathedral in Istanbul. That's cool And it was it was really neat sounding what it had to do with voiceover Jack. We're not exactly sure Fascinating. Yeah, because it's interesting because we're acoustical engineers and you're right over there Yeah, and have a little bit of a leg cramp. Oh, man. You need a massage Maybe but we'll talk about that Anyway, we've got you've got an unboxing to do tonight and and it's not a pair of shoes. It's We've got you know, we're gonna talk a little bit more about interfaces And we got lots of great questions tonight from people all over the world We definitely definitely know somebody from Portugal asking it has a question. We have two questions from Portugal in one week It's amazing clearly. They're watching in Lisbon Well, I have an idea. Okay, go for it and the whole idea the way we do our show and being spontaneous Let's try something. Okay, so right now you're hearing Dan's voice on a Sennheiser 416 Yes, you are but I think it'd be interesting to see what his voice sounds like on the vanguard v4 Okay, I think we should switch it out live right now in the air like we did before the last episode Right before we went on the air. Oh, this is gonna be interesting You just keep talking while he's doing that Mike So we got this vanguard v4 in and I thought it'd be fun to hear it in context now Dan's mic right now is Pretty far up and above so when we go to the v4 because it's a large diaphragm condenser We're gonna bring it in a bit closer. It'll probably be in frame and you'll be able to see it but We also I just wanted to make this as entertaining as entertaining for me as possible and it always is watching Dan Replace mics on the air. It's a blast But yeah, we get stuff sent to us from time to time We see vendors at the shows like I was just at Nam show and through those connections. We get to demo equipment We get to demo equipment like this and it's a lot of fun Typically what will happen when we demo gear is we do it in sort of a controlled environment So this is this is more about the experience of using the equipment and things like that But we'll likely do some like booth recordings as well where you get to hear them more in the context of a Voiceover studio. We've done this in the past. We did a a super-duper mic shootout of like five Was it about ten different mics under five hundred dollars, right? This could be a part of a new under five hundred dollar shootout because this is an under five hundred dollar mic Yeah, and this is this is this is a new model from our friends at At Vanguard audio. Mm-hmm. Yeah, they let us try the mics. They send us t-shirts. We go great mics Yeah, this is a great man. I've been playing with this one all week. It's it's it's a little different now the the vanguard The other one that they're multi-directional mic that has the tube like to yeah It takes a power supply and you gotta switch it all out to change the different patterns This one just has a simple pattern, but not So, you know right now it's in cardioid pattern So it's gonna pick me up nicely from you know people forget that a cardioid pattern isn't this flat plane it's a hemisphere Around the mic so I can really move around and I'm still gonna be pretty much in the sweet spot on this mic Yeah, these like a large I cut my mic because I wanted to hear exactly what that might sound like but a large diaphragm condenser the pickup pattern is pretty broad So right that the sweet spot is wider, right the converse problem of that is means it's gonna hear more of your booth It's gonna hear more of your environment You just have to be careful about the getting too far away from the mic right, but it has some interesting patterns then yeah Yeah, it's in right now what right now. It's in cardioid. Okay. I'll do it while you switch it and we can share Yeah, I mean which you know and sometimes we were constantly hearing from people saying why does my mic sound so muffled right? You know you're talking into the wrong side of it always talking to the side with the logo Anyway, okay, so turns out we're gonna switch it now to a figure eight. Yeah, I'll cut the mic Okay, now it's switched. Okay. This is a figure eight pattern now It should sound pretty much the same from this direction But if you're one of those people that tends to talk in the wrong side of the mic, this will cover your butt Yeah, this is this is how you cover your butt because now You know, I get to see my bald head here, but But now it's your you've got sort of a circular not really a cardioid pattern on both sides But but Then it's totally no And why that's cool the fact that it's really dead from the sides means if you've got issues in your booth Maybe you got a big glass window or some reflective surface That's giving you trouble if you put that dead spot on the side where that reflection is you eliminated it Right will not hear it right the key thing is behind the microphone You do have to have that well treated right so the back should be facing towards a base trap or some really well Absorb some good absorbing acoustic treatment right, but it's an interesting way to control the way the mic sounds It also gives it on if you guys noticed There's a little more proximity effect just a little bit. I can hear it It's more like a ribbon mic. Yeah, it sounds which is also in our figure eight usually yes So it's it gives the mic more than one personality right, but it has another personality. What's that? It's you know, it's it's a schizophrenic mic. All right. I'm now flipping the switch again for it Has been flipped it now is an omnidirectional Yeah, so now I'm talking into it as well And I'm talking to it from over here and you're talking to it like yeah, right So I can go anywhere in the room the other people in the studio can talk Talk other people in the studio So that's this is a great thing to do when you've got ensemble and you need multiple people around a single microphone But again now the microphone is picking up everything the air conditioning When I'm further away you can hear my voice bouncing off the walls and there's a lot of acoustic treatment in here And you can still hear a little reflection So you don't want to use on me that often but again, it gives us Mike three distinct personalities That makes it pretty versatile. Yeah, actually what is it? 400 something like that. It's under 400. I think yeah go to vanguard audio and check it out Derek over there who they make their their handmade microphones here and here in California, and they're and they're really nice Yeah, and I like this. What's that back on going back to cardioid here?  Yes, so definitely be careful when you're and when you and never ever turn phantom power on or off With your headphones on right and never unplug or plug your mic on In with phantom power and your headphones on you're not supposed well, I would explain a few things Well anyway, you would hear a very loud pop. Yeah, but I like it. It's it's a very It's a very hot mic. It's good output. Yeah, you know, I mean and now remember all of this stuff is still It's still designed for recording music. We're just using it for our own purposes Yeah, you know, there's your razor blade there. So you can do the unboxing and But I like it.  I'll stand off and just do it that way And did you do that out here in this room? Or did you do it? I did it in the book, you know So you're totally controlled environment. Although it's pretty controlled out here, too So yeah, anyway, so thanks to the guys at Vanguard who sent us that and the check it out Now you get to take it home and check it out. Awesome. Yeah, and I'm gonna turn back on my show EQ Okay, I flatten it out for that little segment But good I add a little bit of top end to it make it sizzle a little bit for the show So now we have another new thing So from Nam show you remember the v4 and you also remember the first segment we we aired was about the EVO for from And as promised I received one so and here it is I've been I've had it for over a week And I've been waiting to open it on the show. So here we go. So I Don't normally do unboxings. I really don't like doing unboxings at home But doing it live on the show something more fun about it, especially using a big open razor blade It just makes it something more thrilling about it. Notice how I'm opening it away from you Yes, and this thing is wrapped in like reinforced not packing tape, but reinforced tape. So It definitely nerve-wracking well George is doing this thing with an open blade Yeah They send us this stuff to test it, you know, then they're not necessarily giving it to us They just want us to test it review it see if it's up to that you know our standards because we have set the standards literally offer voiceover and Of course, I'm always of the opinion. It doesn't really make a whole lot of difference what you use To record voiceover as long as it's good quality a lot of stuff has more to do and the EVO will probably explain You'll probably give us a good example. I open it from the bottom. Okay. It's always a good way to start on unboxing, right? It's more important to George and I it's really more important what the function. Sorry is for These things that's what you're gonna see when you get here. It's a different type of thing you know, so let's take a look at it Hold this in the frame See work smarter not harder. We're all big fan of that. Yeah a few EVO stickers. All right. What'd you say? No extra virgin volume a few stickers will put those on the stop signs in the neighborhood T-shirt. Oh It's a t-shirt. All right. I believe this is the special influencer edition Okay, and the actual actual retail box much much smaller, so let's take that out There we go now we're getting an idea of how compact this thing really is and So You guys saw on the show, but this is a a new product from Audient It is a complete ground-up design product So as far as I understand there's absolutely nothing in common internally or externally obviously to anything else In fact, Dan's got an ID 14 over there and they don't look anything alike So this is a whole new probably different different concept. It's got that nice suction on the lid I like this new theme of really high quality boxes coming from China. It's like automatic. It's airtight. Yeah, it's so nice So this is cool. It's a very small manual This is another trend and and gear mixer face was the same way It just has one card because they realize one most people never RT FM Most people never read the freaking manual So that's one thing and two if it's a good piece of design. You don't need a 50 page manual Right because you should be able to pick it up what you need to do from a very short manual And this one definitely is one of those kind of situations Plug in your computer USB da da da You do need to install install drivers. It says I don't know if that's true on Mac and Windows It doesn't differentiate on here I believe from what they said at the show you don't need drivers for Mac at all But it which is why we like Mac they must have drivers for both. It's funny I just set up a Steinberg for somebody you are 22, right and I did it by proxy. He was in New Jersey It worked great on the Friday when we set it all up, of course Saturday comes you walk away from it It wouldn't it wouldn't detect the device correct And then we I had him install the driver and then it worked So that was a weird mystery. Why would it work and then not the next day? So sometimes if the manufacturer does provide a driver You should probably install it and not do the whole plug-and-play thing. Anyway, USB see so they're going with the modern Audio or modern USB connections to the older micro USB's or others and finally For you fans of unboxing videos Yes, well boy, there are a lot there are a lot of you by the way. Yes. Have you ever seen an unboxed therapy guy?  This thing is Spartan. Anyway, it's very Spartan It's a almost a perfect square. I guess from the side and here's the face it has a So this this is going by that new design where like the Apollo and the what the all The the Apogee and things like that where there's one central knob who does a bunch of different jobs And what that knob does is determine on which buttons you press over here So I'm I'm I have a love hate with that I love being able to tell somebody when they're physically looking at their thing turn that knob dot not You know turn them where to set them it's a little more fiddly But understand why they do it it allows the unit to have a lot more features and be small Right and there's less design costs when you do it that way, but it's got a ring of LEDs for indicators It's got the input selections. This is the headphone level control here The ability to control the one the two inputs independently And then the most unique feature is this green button, which I'm looking forward to testing out the green button This green button is the auto level detect button Interesting as far as I know this is the first interface that is because I've known of another two that were really expensive This is the first affordable one where it has a mode where it will just listen For a little while while you do your script, you know You get your level and the unit will dial the gain in until it finds the best gain setting for the performance you're giving And then when it's done, it's set does it define but you define what those different performances are But voiceover and music are different. Yeah, well, I mean it I think as far as I know It's just listening for a level. Mm-hmm. So it'll You know, I have to use it in the real world to find out exactly what it's gonna do But I imagine it has like a neutral gain and it's gonna bring it up or bring it down based on what you do Now at that moment if you don't do it if you don't do a read that has the dynamic range of that Peace you're not it's not gonna be right. It doesn't do that Dynamically during the read, right? It's a preset. It sets it before you start because I'll tell you one thing You don't want is a gain control. That's constantly moving on you. No, why recording ever AGC on video cameras How bad they were back in the day? Yeah, you don't want that So it just sets the gain and then so that you don't have to go over here set do a real read play with the knob back and forth It's supposed to be doing that for you. That's the whole idea. Still you should be doing that anyway You should be and the other thing is I have no idea what that gain setting is based on is it based on a peak level and RMS How much? Headroom will there be so is it gonna set it so that the peak is minus six I think we need to talk to the three people at Audient May well Yeah, what do you set this thing? We'll probably find out by doing a real-world world test Which right I hope to do shortly and they're counting on me to do one so I better do it. Yeah. Anyway, that's it That's the evo Four from Audient. There's also the evo eight, right? It's called the four because it records four tracks Actually, it records two mic tracks and it will record two more tracks, which are the return from the computer So if you're doing a little casting yeah for podcasting that is cool So it allows you to record the Skype or zoom or whatever coming back from the computer to two additional tracks in your Doll like you know an Adobe audition in multitrack mode, right? You can make tracks three and four the phone call and then when you're done you have separate level control Let's yeah, so I'll be interested to see you know how strong the the preamp is in this and you too Yeah, well, you know is it does it have a lot of good clean gain on it and pass it over to town you too Yeah, here take a look and then feel it up It's pretty late it's pretty late it's probably not that much in there You know a circuit board and another board for the Jackson so that little box replaces This huge box of crap Really wanted to try this and I could not get it to him still not convinced. It's dead the last time I use this task Am It did function when I put it away So we'll try some new drivers. Maybe it's a not it. Maybe it's not a Catalina thing Hey, maybe it won't it was I tried it. I tried it on before I'm a hobby wouldn't wouldn't right? I have high Sierra on my laptop, so maybe we'll go check that out We're gonna do it takes time to prepare for this and it's been a busy time of year folks But we do plan on doing a shootout with things that are from 10 plus years ago to Stuff that's modern. We even have in this box M audio M box 3 which sounds like a new product because it's three, but this is not this is over 10 years Who's an ID 22 is an ID 22 now? This is this is the big brother or the grandfather today grandfather.  Yeah, I met at a conference. He says using a preamp with that. It's well. Yeah, I've got a you know an inline go naked and That was that was a good 10 12 years ago Yeah, they have very good preamps and and and he was right and my audio was so much cleaner Just because they spend hundreds of thousands of dollars on research on these things Yeah, and putting in a putting a preamps into these little tiny boxes The form factor is cool. In fact, this was very revolutionary and it has a my go looks a lot like this, right? Yeah, the problem is is to make something this small all the connections have to be on a little special funky proprietary Breakout cable and that's the part. I don't I'm not a big fan of that that cable connection Freaks out or flakes on you when you're on the road.  Yeah, I mean you can use the internal mic Yes, you could do that in a pinch Which I've literally never used it. It still has the plastic on I've refused to pull that off I just saving that for you. It's been there for 10 years. I guess and it looks brand new Anyway, we'll do a test some Extra week before we do our next tech talk, so maybe we'll have time to All right, thanks for that. All right, so I'm gonna take this by the junk Where we'll be until next time Still be there in two weeks. Anyway, there you go our first isn't that the was that the first real unboxing we've done I think so real unboxing nine years.  You know unreal that we would be doing this and we didn't know we'd be doing it like this like this Yeah, we were in years ago. We yeah, I was sitting in my extra bedroom in Buffalo and you were in your back porch and Santa Monica Yeah, and and now We're in this magnificent humongous studio Everybody laughs when they come on this is it Think of it like the TARDIS. Yeah, really it is kind of like that Moving on moving on Well, we got lots of great questions from you guys, which we have been asking you to send us over the last couple of weeks And we will attend to those questions right after these incredibly important messages from our incredibly important sponsors be right back This is Arianna Ratner and you're listening to VoiceOver Body Shop VOBS.TV VoiceOver Essentials only has five of the mixer face without the SD card version The SD version kind of baffles me for VO anyway being a worth an extra hundred bucks Backup I guess but I always want to record to some device where I can edit Anyway, the no recording version is on sale now at VoiceOver Essentials And they only have five left the regular price 349.94 sale price 329.94 free shipping in the continental US mixer face r4 integrates your iPhone iPad or Android device into the audio production workflow making mobile recording easy for broadcaster musicians and VO artists on the go combined with a long lasting rechargeable battery to Nutric combo jacks headphone monitor and 24-bit 192 kilohertz converters You get an unbeatable mobile production tool right in your pocket get it now Over at voiceover essentials.com and tell them we sent you. Thanks Arlen Everybody it's the time of the show where we get to thank one of our sponsors and tell you guys about what the heck It is that they do over its source elements the creators of source connect and so many I can fix my collar now And so many other great products The one that you're probably most going to be interested in checking out in the context of a voice actor is going to be source connect and It's the tool that is definitely established itself as the heir apparent to ISDN in the vast majority of the pro studios Around the world at this point especially in countries where ISDN has literally been disconnected permanently That is coming to the US very soon. The comprediction is 2020 this year that it will go away in the US That might get stretched out a little bit, but it's going away So source connect is the tool you want to have in your toolbox, especially When you're trying to play a higher the next level up in your voiceover career Land some of the agents agents are often requiring this and there's gigs the bigger paying ones the commercials and Promo and that kind of thing that require it. So go check it out Get a get a promo or really a demo actually At their website source dash elements comm you can get a 15-day free trial and start using it right away And I just wanted to get this in there if you're going to go to VO Atlanta source elements is a sponsor And they'll be there again this year and you get to be there and chat with Robert in person Get a little test of the system and I've been may have a studio there this year They have in years past where you can actually do sessions in a pinch So go check them out and VO Atlanta. Anyway, thanks a lot source elements We appreciate your support of our show and be right back with more tech talk after this Well, hello there. I bet you weren't expecting to hear some big voiced announcer guy on your new Orientation training for snapchat. Were you this is virgin radio? Well, okay. We're not that innocent There's jeans for wearing and there's jeans for working Dickies cuz I ain't here to look pretty. She's a champion of progressive values a leader for California and a voice for America It's smart. It's a phone. It's a smart phone, but it's so much more. It's the files are ready Don't forget to pick up the eggs. What time is hockey practice check out this song It's the end of the road for Rick When hope is lost the I8 from BMW Who said saving the planet couldn't be stylish? Hey, it's J. Michael Collins.  I think they speak for themselves, but I will give you my email. It's J. Michael at jmc voiceover dot com Now if they will stop waxing this mustache for a minute, we'll get back to the show Hi, this is Bill Farmer and you are watching voiceover body shop It's great Well, if we haven't totally bored you to tears especially the podcast people going what on earth is going on I mean, but at least we're discussing how it worked in the mic Demonstration and stuff, but George and I love doing what we do which is Helping you guys do it right making sure your audio sounds the way it's supposed to sound whistle and it's You know again with the the the equipment that we use Is not designed to do what we do Not the evil is getting closer. Yeah a little bit. I mean they're getting there I think the manufacturers have sort of skipped over the pod You know the the voiceover people are jumping into the podcast because there's a lot more people doing podcast, right? It doesn't mean and everybody making money. No, no Manufacturers are Yeah, you know as I like to say that you can everybody can do a podcast It doesn't mean everybody should do a podcast But you know the sure are a lot. I you know I started checking them out like How can you possibly choose between all of this stuff? Anyway, but you guys are voice actors and we're here to help you out with voiceover audio Which is a very very Specific way to record and it's amazing to the two of us how you guys tend to overthink all this stuff Yeah, it's actually quite simple But you have to understand the basics and what George and I do is we get you down to the basics So you can learn a little bit more advanced and how to make sure that your audio is sounding right But it's not the equipment It's the environment and the acoustics how you use and how you use it And that's what we really teach you guys how to do and what we do is Come into your home or your office Your girlfriend's home or wherever it is that we end up your closet and we sniff around we find the best place to record in your particular domicile and we set it up for you and show you how to do it and We charge you for it, but it's totally worth it because we will save you Thousands and thousands of dollars. It's a huge shortcut in mistakes and education and Hours of mind numbing confusion and forum postration. Yes and tech support by committee. No turn the mic around Anyway, yeah Anyway, if they would like to if you if you guys out there would like to work with a professional like us You can work with George and you will find him over at hmm George the tech comm or George the dot tech if you like those short nerdy domains like I do I've got a menu of services there a lot of different ways to work with you And if it all is just too mind-numbing or maybe you've got so many different things that you want to have dealt with You don't know where to start just send me a message We'll let you know where to start, but that's George the dot tech and Dan does it as well on his place on the web and that's home voiceover studio comm And you know I do a lot of the same stuff. I'm a full-time voice actor I understand really what your life is like and how to make sure that you know Your home voiceover studio fits into your lifestyle and and works for you and has the right workflow for you And if you've already set up your studio I'd like to hear what it sounds like and I want to hear you're wrong. This is start. Oh, yeah I've got my specimen collection cup at my home site the home voiceover studio comm for $25. I will listen to your audio and Believe me, it's amazing what I hear from people. It's sometimes like three seconds like you got it. No problem And then it's like oh somebody was listening to their friend Who's a musician who says you got to use this and you got to use that and I think we actually have a question about that tonight Oh, so we'll get to that. Yeah. Anyway We're here for you So let's get into some of the tech questions that had been sent to us by our marvelous worldwide audience We've made them wait long enough. They have You know our first one is from Randy frangelo And Randy says hi guys I built a home studio out in the garage and now I need to build bass trap bass traps But bass traps are for he built a fishing pond. Yes It's bass traps But I read some scary stuff about fiberglass and mineral wool and possible health effects that kind of freaked me out Are there healthier alternatives to effectively treat my space denim hemp sheep's wool Get multiple layers of moving blankets be made into a base a bass trap Or is all this talk about mineral wool and fibers floating in the air a bunch of who we if it matters The dimensions of my room are roughly five, you know, he's gives us some numbers here Which really mean nothing until you really put your head in there and listen to it And there's a fifth wall where the door is at 45 degrees. I think sometimes people confuse the idea of base trap with Acoustical panels now a base trap is an acoustical panel, but not all acoustical panels are base traps Right, right. They're they're too different a base trap is a type of acoustic panel exactly and it's designed to control the lower frequencies in the room which are oftentimes the most problematic in most people's Recording spaces. Yeah, but it's how to make one that's safe You know, I've I've installed some from ATS based some of their base traps. They use the mineral wool Yeah, you don't want the loose particles in the air. No, that is definitely not good and The fiberglass would be particularly bad because of the chemicals the glues things that are in the fiberglass The mineral wool most people don't have any trouble with that issue and when they're built correctly Typically those powered particles are completely surrounded inside the panel You don't want to buy acoustical panels that have exposed material in the back Right, there are some companies out there and before you order acoustic panels ask Are your panels completely encapsulated or covered or are they open because our number of companies that do them open? They leave the back exposed. That's a no-no Right. I do know from personal experience working with ATS that their base traps are fully encapsulated like they have the fabric wraps all around on all sides and I've never seen that one be via problem with the fire fiber fiber not fiber, but the Particulates are the fibers getting out right. Yeah Now I've got these these these panels up here in the ceiling that I built they have Rockwell in them Yeah, I covered the back with weed blocker. Oh, yeah, which worked great Yeah, we can't get through it none of those particles. Yeah, but here's here's the tip of the week If you're talking about building panels or trying to dampen the the acoustics in the room a bit Go to the thrift store and get a pile of old towels Mm-hmm and and layer those and seal those up and put some nice fabric around them Those work great as well heavy dense Whatever towels you can find and I'm if you go to a thrift store, you know You may want to wash them first, but you know But they will do a really good job of diffusing and absorbing the sound in a room You know, and if you're in a closet, you know, how many do you need? You know the two by four closet or something like that And if you're in a closet already has all your clothing hanging on two walls You don't need any because you already have base traps folks best best sounding rooms Yeah, denim. There are some products made with recycled denim. They're pretty expensive usually I don't know of anything that's made with hamper sheep's wool. You could be joking. I don't know But there might be somebody out there But the the towels work the moving blankets also work similar concepts But you need to build up a lot of layers for them to give you the base trapping Capabilities not one thin layer is gonna do that right and I had my studio soup, but I don't sell that anymore But that worked great because that was a combination base trap And it just absorbed frequencies at the right the right right frequency. Nice and it's actually great So those people who still have them, you know what I'm talking about We have a question here from Portugal where I'm going in November with the missus. Oh lovely. Yes taking another river cruise But this is from Jose Jao, I think that's how you pronounce it Portuguese, but Who knows how to pronounce Portuguese except the Portuguese exactly or the Brazilians? Yeah. Oh, no, no, no No, I know someone from Portugal. He says the people in Brazil do not speak the same Portuguese as the people in Portugal.  That's like saying that Canadians speak French the way the Parisians speak Yeah, that doesn't work that I played hockey for a long time. It's it's not the same Anyways, he says I'm a Portuguese VO and over the years. I completely understood that less is more. All right. Who say So right now. I only use my mics an omen TLM 103 and a Sennheiser MKH 416 and my Yama AG03 all the rest if needed is post-production So my question is what do you think about the newest Audient EVO 4 you mean timing you mean the The evo for well, it is cute. Oh, no, it's cute and it's very attractive and light And we're gonna be putting it through its paces one of these days Another new product he's mentioning that just was also launched at NAM was the solid-state SSL 2 Which I'm curious to try out as well. I think it's a different class of product. It's much more heavy-duty It's larger physically So that when I would like to try out in the real world as well as it sell if you're paying attention They could change the way we we think and work concerning audio interfaces or not I don't know if anything is gonna reinvent things. I mean the audience again the evo 4 has that auto level detect That's really the only thing that's truly New and relevant to voiceover that's anyway different than anything else But that's really the only thing that really jumps out at me again that two-track record Return loop back thing is cool But it doesn't work for Skyping and zoom because it's two new tracks so that when you try to play something back It won't be heard on Skype or zoom. So that's something they got to work out still doesn't mean that they won't But we don't know yet. It's too too early to tell but at least you got to see the evo 4 As we did so it feels nice in the hand, right? Yeah, and again, it's a matter of function over form or sound ones and zeros or ones and zeros is there really a difference and we'll have to give this a good test is there a real difference between These interfaces these price points to and you know, that's going to affect the quality of your voice Usually when you get over a certain price point and not a very high one It really doesn't matter You know, I don't want to insult people who make this good. There's really great stuff But they make great stuff if you're a musician. Yeah, and if you're a real audio geek and you really like playing with this stuff It's great But to keep thing keep yourself sane You know, we have another question coming up about that in a few minutes here. Cool. But yeah, I think that You know interfaces or interfaces. Yeah, I mean the auto level thing it could be a game changer or a gadget and we'll find out All right now next question That's a new name for a segment game changer or gadget I like that. I don't know. Take it out of that yet. All right Let's see. This is from Brian in New Jersey. Hey, Brian. Yeah, not that there's anything wrong with New Jersey You've been to New Jersey But you're it's a beautiful state. It's the garden state. It is it is you can ski there and go to the beach Did you know that you can do that here? Not just California? Yeah. Anyway, I recently upgraded from an 06 iMac Is that like the one that's like on the stand? That's yeah, it's probably white. Yeah, it's probably white and audacity Which actually worked swimmingly. Mm-hmm my new beautiful iMac is awesome But alas no audacity on cantilever me. Yeah, not yet. Anyway. Yeah, I figured I would quickly learn logic pro X Oh, but not so fast. Do you recommend any particular resource to learn logic for VO? I also upgraded from mbox mini to an epigee duet. Good. Good choice I regularly use a Neumann TLM 103 and can't wait to get back into it like before any thoughts well Logic is great. If you're producing Your vision commercials or you're a musician posing music for doing, you know total tracks soundtracks For voiceover again, it's a control room for a nuclear reactor to control a hamster running in a wheel Yeah, that's one of my favorite I mean come on now. It's like yeah, you can learn logic It's great if you want to you know learn some advanced recording techniques But don't apply it to your voiceover career apply your time knowledge and and uh in capital Into training, you know, there's certain tools that it just lacks or it has but in the wrong way And the one I'll harp on because I just know this one well is normalize I particularly do use normalize as part of my processing Process and it's part of the it's part of the thing In norm and in uh logic normalize is an afterthought Literally, it only happens at the point of export. That's where normalize happens So you can't even use normalize in the way that we use normalize and all the other Dawes like audition and twisted wave and you can't you just can't and that that's just one example But um, it's also it wants to by default always make a stereo file You sort of have to hack the thing to make a mono file. That's just like garage band Same dang problem. So If you've been watching the show for a while, you've heard us mention twisted wave Just give it a try. I know you already maybe probably already did spend $200 on logic Sorry, um, but 80 bucks for a twisted wave You'll thank us because if you like audacity, you're probably gonna like twisted wave. It's it's similar and it's it's elegant reliable straightforward And uh, it does run on catalina And you only pay for it once you pay for once and then and then they updated when there's an update and there's support And there's support and there is support. It's really good. Really good. Thomas the guy that builds that is a great guy and uh, it's been a When you start to use that there are people say, and I like it It's you know, it doesn't have all the features that I like Like what record stop play rewind edit cut paste I mean, what else is there? Well, no, I've got to do all that processing Yeah, there's ways there's ways that There's ways to do that and he doesn't require that Exactly nuclear reactor to do that. Now maria mochus, of course asked the same question. Have you checked out the evo for? Wow, that's the hot. That's a hot topic. They must have all watched the show last week Yeah, yeah, we're going to for sure maria as we mentioned I can't wait to check it out. Yeah, as I know the people an audience can't wait for us to check it out Yeah, uh, j. Horace black who we haven't heard from. Oh, yeah It's for you. Oh, uh, hey, george. You had mentioned that you liked the a u audio noise canceling headphones I think that was all sounds AU sounds was the name of that company. Um, did you ever get them? Nope Uh, maybe I I'm gonna try to read maybe I'll reach out to them and see if they want to How does demo those headphones? Uh, I haven't made the plunge 300 dollars I think they're still in pre-order until march 1st. I believe so Uh, maybe by next time I'm on the show Man, we'll see and a lot of things to spend money on right now. Maybe not headphones, but I hope to get them. Um, and when I do I'll give you some actual useful information Um, in terms of can you use them for monitoring voiceover? I can't imagine why you couldn't. Um We'll see I believe if I recall they're closed back like these they're sealed Um, you don't want to use headphones for recording voiceover that are open Because the sound will pass out of the headphones and could be then picked up in the mic and that can cause some issues or even feedback Um, but as long as they're sealed like these and as long as they have an analog connection To plug into the computer as these do as well They should work great for monitoring voiceover. I can't see why they wouldn't well Is it monitoring while you're recording or is it monitoring? Listening to somebody else or listening while you're editing. What's the situation? Yeah, I mean because not and sometimes the same pair headphones isn't always the awesome Isn't the greatest option for doing both of those? And of course as we've talked about a lot on the show sometimes you shouldn't even wear headphones While voice acting so it just depends on your situation But um, I'm looking forward to try them for sure and you guys will certainly know when I get to so all right All right, and one from thomas machin. Hey thomas who we know is there all the time Uh, this is it's all magic dan and george the wizards of vio Thanks thomas Uh, lee pinney says great show guys. Hey lee pinney, right? Thanks for being lee pinney. Yeah, come join us here and lee come on You're not that far.  Yeah, anyway All right, I think I guess yeah, that's that's enough geekiness for one week. Uh, anyway, uh, again if you've got questions for us Send them to us at the guys at v obs Dot tv right there and uh, we'd love to hear your questions. You're curious about something Uh, whether it's you know a piece of equipment Whether it's technique because I think maybe we're starting to get the point across to people That it's not the equipment. It's how you use it. It's technique There are certain basic things that you need to know how to do In order to use the equipment. Otherwise, it's just expensive paperweights Right Yep. All right, exactly. All right Well, you got a couple of messages from our incredibly important sponsors And we'll be right back to wrap things up right after this This is bill ratner and you're enjoying voiceover body shop with dan lennard and george widham v obs dot tv Your dynamic voiceover career requires extra resources to keep moving ahead Now there's one place where you can explore everything the voiceover industry has to offer that place is voiceover extra dot com Whether you're just exploring a voiceover career or a seasoned veteran ready to reach that next professional level Stay in touch with market trends coaching products and services while avoiding scams and other pitfalls Voiceover extra has hundreds of articles free resources and training that will save you time and help you succeed Learn from the most respected talents coaches and industry insiders when you join the online sessions bringing you the most current information on topics like audio books Auditioning casting home studio setup and equipment marketing performance techniques and much more It's time to hit your one-stop daily resource for voiceover success Sign up for a free subscription to newsletters and reports and get 14 bonus reports on how to ace the voiceover audition It's all here at voiceover extra dot com. That's voice over x t r a dot com As a voice talent you have to have a website But what a hassle getting someone to do it for you and when they finally do they break or don't look right on mobile devices They're not built for marketing and seo. They're expensive You have limited or no control and it takes forever to get one built and go live So what's the best way to get you online in no time go to voice actor websites dot com Like our name implies voice actor websites dot com just does websites for voice actors We believe in creating fast mobile friendly responsive highly functional designs that are easy to read and easy to use You have full control no need to hire someone every time you want to make a change And our upfront pricing means you know exactly what your costs are ahead of time You can get your voice over website going for as little as 700 dollars So if you want your voice actor website without the hassle of complexity and dealing with too many options Go to voice actor websites dot com where your vo website shouldn't be a pain in the you know what So Level later it did a great job of rms normalization for audiobook content and podcast episodes But it's orphaned software Which means no one's developing it anymore and now it doesn't work with the latest max and catalina mac os So you're stuck right? Well, not anymore Behold the gooey goodness of audio cupcake Visit audio cupcake dot com and download the free audio cupcake app for macintosh Audio cupcake does exactly what level later did so well for so long It applies rms normalization to your audio and it preps your work for acx And it does it so well with mac os including catalina Just like with level later you drag and drop your audio file onto the audio cupcake window and out pops an rms normalized file But audio cupcake goes even further Unlock the premium features of audio cupcake and what pops out audio That is both rms and peak normalized and converted to a 192k mono mp3 file ready for uploading to acx or your podcast platform That's delicious audio goodness audio cupcake is available free at audio cupcake dot com. That's audio cupcake dot com audio cupcake a beautiful simple way to master your audio narration and podcasts You're watching v obs dot tv.  Yes, uh, but hopefully you're you're getting the information you need here and uh, that was fun and it's always fun doing it with you because You're the guy that knows And i'm the guy that does I guess you listen to me. I love it You listen to me We actually learn stuff from each other. We've melded our our brains. It's like, you know, the balkan mind meld Occasionally remember back when jack the gully would like to take screenshots And post them on facebook. I miss them. I still use them. That would have been one Yeah, just now Well, like we can still go back and take any one of those Anyway, uh next week on this show we're gonna we're gonna play one of our great all-time Interviews i'm not sure which one it is, but we'll put it on there if you got any ideas. Yeah, uh, and then the following week Uh march 16th, which is next week if you're watching this next week because you are watching this next week I'm trying to track. Yeah, simon vance will be our guest simon vance one of the top audio book narrators on the face of the earth prolific And he just built himself a new studio in his backyard Just a standalone studio like this marvelous studio here. And uh, I imagine it'll be a little further along in its progress He's been posting pictures on facebook, but we'll have a video and we'll hear simon's amazing voice Which is always a pleasure to hear going to audible and type in for author search sign advance And gets bold popcorn And eat popcorn and start scrolling Exactly, you'll you'll finish the popcorn. Yes So who are our donors of the week? Yes, quite a list this week. Yeah, natasha moshuka. Am I saying that right? Please have See you're too nice and i'm too slow to learn how to speak russian. I'm sorry next polish and I misidentified the race so swing I get one more chance at bad here Natasha or chev cut. There you go.  It's thank you rob. Yeah Well, we're really appreciate it. We were in his studio last very good You know like now for tech talk, we like to have these technical backwards. We like waveforms. Yeah, but uh, you know If you have something interesting you want to send us Plan it to us at the guys at v obs.tv And maybe we'll use it and maybe we'll go. Oh, that's a really interesting picture Anyway, if if you'd like to be here in the studio to actually watch us make fun of each other every week Uh Come on down write to us at the guys at v obs.tv if you're in the greater los angeles area because there is no lesser Los angeles area Uh, and not yet. Anyway. Yes waiting for the big earthquake Yeah, on a monday on a monday afternoon and come join us live and watch this mayhem as it happens Uh, anyway, let's see what else we need to think We need to think about but while we're thinking about it We need to thank our sponsors. This is what happens. That was almost a good recovery. I don't know I may not have to give you points for that. Okay. Anyway, let's thank our sponsors. You go first. You go Harlan Hogan's voice over essentials voice over extra sellers voiceover heroes dot com of voice actor websites dot com and j m c demos Alrighty and of course the dandan marsy lennard foundation for the betterment of live and recorded web end pod casting And we're building a new a whole new wing for the museum. Good. You're gonna need it. Yeah We need to thank jeff holman doing our great work in uh in the in the chat room tonight And our amazing technical director who just makes it look effortless As she's there in flop sweat Uh, sue merlino Thank you sue. Great job tonight. Say hi sue Hi, sue. That's better. Okay. Good. Thank you. All right. I hooked up that damn mickey better I could talk a lot more if you want. That's okay. That's right. The show's over. Yeah, we're out of time I'm dan lennard. I'm george woodham and this is voice over body shop or vio bs tech talk tech talk tech talk tech talk We'll see you in a couple weeks guys. Have a good one. Good night


I have noticed the communication you have with the parents is so huge, so I'll set aside time, I'll show investment, and it goes the extra mile. The text messages back to the parents, that kind of stuff, they'll text back at just how involved, how much they love their kids, and then honesty sometimes when they're over pushing them, and sometimes a lot of times they want you to be the voice that they can't be, because you know what I'm saying, so they're actually communicating to you what they're telling their kids, but they know that their kids aren't listening to them, you know what I mean? You're the middleman, and that's okay, and I get it just being a father that your voice can kind of get drawn out, and actually that's the really good coaches, the really good coaches in team sports, that's why they have assistance, and they know how to utilize them.


Welcome back to Clearwater and Sunshine. I'm in the middle of Texas, baby, on Lake Travis, one of the clearest bodies of water in the state. And we're gonna do a little bit of Clearwater dangling for you today. So, there is no fishing boat trailers in the parking lot that is indicating that the fishing is probably not the best out here. There is some huge bass in here. There's just not a large number of them. There are a high number of small bass in here, including our state fish, which is the Guadalupe bass. I haven't caught one of those in a long time. So, I'm pretty excited about that. I think we're gonna be doing mostly finesse tactics, always good to brush up on that finesse stuff. And there's something about going to a Clearwater lake and the hot, you know, heat of summertime, jumping in the lake. I don't know. It just feels like you're at a swimming pool. It's good times.  That's just me. I don't know. I like that more than spring. Fair off the murk. Let's get to cranking on them. A rattling net. Just because I think it's gonna be one of those things that'll catch fish in this Clearwater. We're talking, goodness. Keeping that five foot there. Five foot down. I can see that net there. It's probably more like six, seven foot. I didn't have the glare right there. So I've got the boat sitting in 53 feet of water. And I'm just fishing a little bluff type area that has a creek channel swing next to it. I think a good question for Clearwater is where do you even start when water is this clear? And you've got a ton of water. And I'm gonna say for this time of year is you gotta have deep water access. That's really like three things in the summertime. They're like oxygen.  Then you got shade. Shade's gonna provide cooler conditions. And then you have just deeper water. That's gonna hold higher oxygenated water and it's gonna be cooler, which kind of go hand in hand. So a little bit of bait on the electronics right here in like 30, 35 stagnant, not moving. It looks terrible. It looks terrible at this point, but I don't know if you guys can see this little color change right here from the boatwakes. I can sometimes be your friend in the summertime. You don't have a lot of wind. You get some boatwake and can help stir some things up. So I'm just gonna start with this Ned rig, work some of these little points plus and then the dock type areas. So while we're working this little bank here, which I'm not too excited about, but just while we're kind of messing around here, it's the start of the day, just got the okay to talk about these rods. I know a lot of you have asked about those, one of those black rods that you're using in the videos. And when I was in the office the other day, they said, so I can go ahead and start talking about them. So this is going to be our next series of rods. So you guys are familiar with the green series, which are, that's our baseline of rods, the green series and the gold series, which is a step up from their own blanks, but the really the same types of actions. There's a little bit of differences between the actions just due to the materials, but they're really built on the same basic, you know, five, six, seven rods that you can pretty much do 90% of bass fishing with. And I love those, I still use them. I use the golds a lot, but I've also been using these black series, obviously testing them out and making sure the actions are good and everything, but these are more advanced technique specific rods. So for example, this right here is a seven, six, medium light, extra fast. So it's a little long for drop shotting. It's really more technique specific for doing stuff like this. Now throwing a finesse jig, throwing a hair jig, throwing a net head, throwing shaky heads on light line where you've got a light lure, you know, you've got to really get that lure out there. You got an extra six inches or so on top to whip it out there. The action is correct and it's also very light. So not only are the actions different, but these blanks are really light, they're very sensitive and stepped up the real guides as well. I can't remember which foodies they are, these are, but they're top notch. You'll notice it when you're drilling your line through the guides, how smooth it is. So those are just some new series of rods that's going to be called the black series that we're going to have coming, I think in the fall. And they're legit guys. So anyway, I was hoping to catch fish while explaining the new rods that are coming, but it's not happened just yet. Let's keep going. Got to find the three things of summer bass. Shade, oxygenated water, moving water. Deep, current or shade. Just got this fish on live scope. It came up and grabbed the net. Suspended 100%. Hard, hard fish to catch. It's fighting kind of weird. What is this? You got to be kidding me, Sally. I am on a drum kick right now. Drum kick for the ages.  Oh my gosh, I thought you were calm. Look at that. Rattle and net, 100% suspended fish out there, a 30 foot of water. Just nailed it. I mean, just like a bass would. I dropped it vertically over its head, rattled it, fish came, wah-bam. All right, joke's over, fish. Come out now. It's okay. Not here to harm you. My goodness, man. This place is the opus of nothing. It's been very, very difficult to see. I'm wondering if I, what I was seeing, if I was seeing bass that would follow my swim bait or if those were drum. Going here, we got a little school of shad getting pushed. Oh yeah, eaters. Oh my gosh. Wow, that was explosive. Literally, as I was saying that, I'm like, where are the bass? There's one of them little devils I've been looking for. A midday schooler. You can see the ball of bait. It's real dark up towards the surface. That guy just torpedoed out of the depths for it. It's a largemouth. That's what I was like, oh my, can you imagine? If that guy got me. Good bite. Four of you. Doppler in there.  Filled for the old wacky worm. I saw that fish on the very end of here on the last go. There was another one with it. See if we can get one more. Not very big. Not very big at all, but I will take anything right now. Looks like a lot of bait or maybe even crappie. Maybe bluegill down there and there's something. Holding a little bit deeper. I only fish this in like really clear water. This is the baby bass color. This is the four inch. Normally I'll just go ahead and throw the five inch, but these fish are just finicky and I've got a little band I'm putting on it. I like to do this when I'm fishing it around marinas. You have to skip it, docks, marinas. All right, there's a couple of fish sitting up under this little walkway. Just don't know if I'm gonna be able to get them. Really far up under there. Ooh, just had a bass. Chased it all the way out. Could have been one of those ones I was talking about.  I've seen where you reel in one. You see like four or five more or you see a lot on the electronics. You'll see them school every once in a while. They'll come out, get some shad, but this is pretty tough. Fished a lot of lakes in Texas, but this is probably the most impressive on houses. Wow. Okay, trying another bluff wall. I kind of just wanted to stare at this house, so crazy. But this is about as steep as I have seen, about as steep as a bluff wall as I've seen. I'm sitting at 40 foot. It's not crazy. I've seen steeper bluffs on other lakes, but just looks of it. There's the creek channel runs right by this bank. And I'm trying to get lots. The state fish. Stuck like chuck. Son of a buck. I got it out. Yes, that's the fish I seek. Been searching all over for you, Magoo. That is the state fish of Texas. The Guadalupe Bass. Did you not get very big? I've caught one that's probably two to three quarters, maybe three pounds is the biggest one I've ever caught. Or maybe that was some sort of hybrid. But that is a Guadalupe Bass, man. Pretty, pretty fish. State fish of Texas, baby. We're done. Let's go. I'll catch those all day. Those rapid fire. Do, do, do, do. On finesse gear, makes it fun. Which is cool. Cool to catch a state fish. Can't do it everywhere, you know? Don't have them around where I am. They're really in just central Texas, but I've heard there's a lot in this lake. Just found one. There we go.  Oh, that one felt juicy. What happened there? He had it, 100%. I am packing it in, y'all. Steamy hot day, got to fish really the whole afternoon. Almost still dark. And just, you know, just never really got home. I did get to catch the state bass. I had another one on, I don't know what it was. It felt a little bigger than a Guadalupe, but bite is tough. Fishing deep bluffs. I fished deep marinas. Letting my weightless bait sink down there sometimes to like 20 feet or so. Pretty much just fished finets.  But that was about it, y'all. So I never really got, I never got into the full Guad swarm. That's what I really wanted. You know, I literally set up some finesse rods to get into some Guadalupe bass. You know, because they're not as big and they still fight hard, but get a good fight on the spinning tackle, light line and all that, but I just never got into it. So I'm actually gonna go out again tomorrow. That's gonna be another video.  We had like a $10,000 bait and tackle bundle. It was like just an unlordly amount of things. And anyway, this person bought one of those and used my promo code, LFG. So they're from Austin. So I was like, hey, let's go out fishing. So that'll be on the next video. Hopefully they know where the fish are. I will end with this. I love, now that I don't tournament fish really, I just fish team tournaments, not individual tournaments of high level stuff. I love the mystery of the lake. Like you literally just don't know. You don't know what you're gonna catch. You have a pretty decent idea. You have a target species that you're going after. You can catch 40 pound flathead. You can get into a giant school of stripers, get into a giant school of crappie, just oddity things. You can catch a six century, you can catch a giant bass. I love the mystery of it. I really do. Now that I don't worry about tournament fishing and just catching like three or five pound bass all the time, it's really a lot of fun. So enjoy the dangle. Thank you guys for tuning in. Every single outdoor video of mine. God bless you guys. Speed. We'll see y'all next time.


Alright, hi everyone. I'm Flaviu and together with my friend JP we're very excited to talk about some of the infrastructure elements They're going to fine-tuning large language models such as MetaSlamma2 on communities with Argo workflows and Hera We'll do some quick intros again. My name is Flaviu. I'm a staff engineer dyno therapeutics a biotech company in Watertown, Massachusetts where we focus on Designing delivery mechanisms for gene therapies and my name is JP Zivillich I am the CTO and co-founder of Pipekit where we do expertise services with Argo and Additionally provide a control plane on top of Argo workflows Alright, so doing a quick outline first We're going to talk about the motivation for this talk Who it's for and what you should expect to get out of it Then Flav will do a bit of a walkthrough of what foundation models are the process of fine-tuning and why you would be interested in fine-tuning I'll be handling the infrastructure piece and then lastly I'll hand it back to Flav Who's going to do a full-on walkthrough of the heracode needed to actually do the fine-tuning? Cool, so doing the motivation So we wanted to show in this talk how to do scalable distributed fine-tuning for LLMs so The target audience is really going to be these like individuals teams companies who want to use LLMs But need to do some additional customization like let's say for example, you wanted to customize an LLM to represent like a certain tone of voice maybe Alan Arkin from the 2012 movie named Argo perhaps or Christopher Walken right that would be a like a good use case Also, if you're just interested in distributed model training generally you're going to get a lot out of this talk All right, we'll get started with foundation models Which is a relatively recent term that was introduced to describe these generally available open-source models that have been trained on vast amounts of data For tasks such as image generation or or text generation. These are generally very prohibitively expensive to train they require a lot of in-house scientific and engineering expertise and they take a lot of data and a lot of time which Essentially provides the motivation to take these models essentially off the shelf and fine-tune them on your own data Which is a process that I'll explain in a minute But it essentially boils down to taking these models feeding your feeding them that your own data so that they you improve them at a Particular task that your business might be interested in for the applications that you build for your customers And this can be applied to a variety of domains such as you know If you have medical notes or support tickets or access logs or something like that that you want these models to Essentially perform Q&A on They're very good for for that Fine-tuning generally speaking It is a transfer learning technique You're taking essentially the knowledge that's embedded in a model that has already been trained on a particular task and you're transferring that to your specific domain and By fine-tuning by fight fine-tuning the model by giving it more of your own data You're essentially making it better at a task that you're interested in of course for example If you take matters Lama to Lama to hasn't been trained on your proprietary data but you can actually take this open-source model and If you did some of your own examples in order to do better at your particular application There's multiple stages that go that go into fine-tuning such as setting up your infrastructure You know getting access to these specific models and then feeding them your your own data And in this talk we're gonna focus on as JP mentioned on the infrastructure of that which JP will start describing now Thank you Flav. All right, so given this is Argo con and kubcom we will need a kubernetes cluster. I think that one is a given So we'll need some GPUs in which we're going to be doing the actual model training on Within that kubernetes cluster. We're gonna need principally three things We're gonna need a custom storage class. We'll delve into the reasons for that in just a minute We're gonna need the GPUs as aforementioned and then lastly we're going to need Argo workflows installed Which is going to be the workflow orchestration system of choice that will allow us to string all of these pieces together Outside of the kubernetes cluster. We're going to need a hugging face account Saw the hugging face individuals earlier this morning. They have very distinctive shirts love the logo and Lastly we need approval from meta that you can use llama I think that's a pretty simple process, but it is a checkbox that you need if you're using another LLM you might not necessarily need that Cool, so we will jump into a quick architecture diagram of how this is going to look so the data scientists or the data engineer in this case the you with the Exclamation point on this one is going to be submitting the workflow that Flav you is going to demonstrate to the Argo workflows server The Argo workflows server is then a wrapper for the Argo workflows controller Which manages the workflow state across the kubernetes cluster that is going to be creating the workflow And this workflow is going to consist of three parts. The first is creating a distributed key value store That is going to be handling the metadata for the model training In this instance, we have chosen to use at CD to clarify We are not using the at CD that is coming built in with kubernetes, but rather standing up a separate at CD instance Right at CD is really just a distributed key value store So why not use it to do other things than just storing kubernetes state second We are going to be creating four nodes each with four GPUs to specify We were talking about server nodes not nodes within an Argo workflow. So these are four GPU instances This could be more about we're doing for for this example And then lastly at the end we're going to be deleting all of the resources and treating each of these workflows as ephemeral So next touching a little bit more around the distributed key value store So the problem is that we want to track which shards of the model have been trained on which sections of the data set and make sure that each section of the GPUs can share metadata With one another So again, we want to be using a distributed key value Store and it could be any key value store We chose to use at CD in this instance because we are familiar with it There is existing by torch documentation on how to use at CD and we found that spinning it up on kubernetes was relatively easy So again highlighting that this is separate from the existing at CD instance that kubernetes Uses so we will refer to this as the training at CD instance just for clarity's sake All right, so next i'll be handing it to flab to talk about the workflow steps All right, so now that we have that etc the deployment actually available in the cluster Which we're going to cover how to do in in a few minutes I wanted to illustrate what happens with these Containers once they spin up so each container here mounts four GPUs So we have 16 GPUs in total each container talks to etcd the moment it actually spins up to communicate via etcd it's about its existence and Readiness and it just waits until all of the other neighbor peer containers are actually spun up as well Once that is done all of them start downloading the data and the model And the data will be sharded across the independent GPUs And in addition the model will also be sharded across the different GPUs Because the model such as lama lama 2 that has 7 billion Parameters does not fit on its own on a single GPU so There are different approaches that you can use to take chunks of the model and then put that at a time on the GPU And they're just trained and essentially pass data just through that specific That specific shard I wanted to show an example About how this actually works in in pytorch, but they have an amazing diagram that illustrates just this So we're going to walk through that Right now this is taken directly from the pytorch documentation. So this process is called fully sharded data parallel And here we have an example where we have two parallel processes processing data And different chunks of the of the model that we're training. So at the top We really just have a single GPU and at the bottom we have another GPU That those pull from the same data set they pull one simple One example at a time One of the the the chunk is loaded Or the model shard is loaded onto the GPU There's some synchronization that happens across the GPUs, which is where Etcd is useful because it stores things Metadata for communication and synchronization across the different ranks Data is actually passed through that specific chunk And the details of what happens, you know, with the forward and backward and gradient computation Don't generally matter for this in this specific diagram the take-home message is that at different stages in training There are different Synchronization steps such as this gathering of weights, which is ensures that the parameters are consistent across the GPUs Because the they they can process different data sets they'll get different weights which have to be Again gathered and synchronized across the ranks And then once we actually modify the network in the learning process We have to sync the so-called gradients As well and once that specific chunk is done it gets offloaded Optionally to to to cpu memory and then the next chunk can be can be processed Now, of course, we have to take all we can easily we can easily spin up all of the resources and infrastructure Necessary for actually running this on Kubernetes But we also have to take it down because it's very expensive and I'll pass it to JP to show us how we do that Aren't you glad we scheduled a very light and fluffy talk to be the first talk right after lunch All right, so I'm going to go over the teardown of the workflow that you just saw earlier. So teardowns are Where we're going to treat each of these workflow instances as being ephemeral So walking through the steps in the teardown First the training at cd instance is going to be torn down at the end of the workflow using an exit handler In harrow or argo workflows For those not the most familiar with harrow or argo workflows This is a concept where at the end of a single workflow run. It's just a cleanup step It can be a teardown You can tell it to do other things like post a slack etc etc etc But whether a workflow succeeds or fails it is going to execute Second the cluster autoscaler is going to be tearing down the gpu's Since they are no longer needed right like this is a Pretty normal kubernetes concept. We have several nodes. They are now no longer going to have pods scheduled on them We don't need them anymore. Don't want to get billed for them next We are Guaranteed that the teardown is going to happen regardless of the success or failure of the workflow run itself again Since we're using that exit handler concepts right whether it succeeds fails Doesn't matter and then as a general rule we want to treat all of these workflow runs as being As ephemeral as possible. There are some situations in which you might not want to do that which we'll cover a little bit later But for now we'll keep them ephemeral All right, so next we'll hand it back to flab who's going to actually walk you through the harrow code on how to accomplish this All right, so now we're going to step to the deepest level and actually go through the code that is used to schedule All of these resources spin them up on kubernetes via harrow and argo workflows and talk about some of the requirements so Of course, you can access this. There's a public repository of all the code And for those of you who are stalker is stuck in yaml land. Welcome to python. It's much nicer here So very briefly harrow is a is a is a python sdk for argo workflows that allows you to Set up basically your own mini platform through harrow by Importing, you know things like global configs hooks and things like that to to set up everything that your internal users might need So such as authentication the host the tokens and things like that so they don't have to worry about it So I wrote Such a wrapper just for the purposes of the talk and we're going to go through now So there are requirements such as setting your host Host of your argo server your token in case that is actually necessary. Of course, you can connect through local host as well You need kubernetes namespace where all of these resources will actually be provisioned And in this case I created a single docker file that I am using for all of my all of my resources that is set globally here Then I have some hooks that will intercept any Containers that are created and container here is essentially the same It is one to one mapped with concepts in argo workflow. So container is a docker container that is created by argo workflows I'm always setting an image policy of always And I'm also adding the necessary tolerations because if you're using specialized infrastructures such as gpus kubernetes will often have Tains and it will require tolerations to be set on your pods in order for those pods to be schedulable on those specific nodes The dem the talk was actually run on gke So I'm adding a specific node selector here if you have a different crowd provider This might change for you, but it is it is easily adjustable for for your own infrastructure And I'm also adding this A very important bit. It's an empty directory that is mounted to slash dev slash shm And this is the shared memory space of the node which is required for Inter gpu communication on the same node on the same kubernetes node And if you don't set this you'll get things like python bus errors, which are incredibly hard to debug The next thing I'll talk about is how we actually spin up the etcd resources So one of the great things about argo water flows is that if it doesn't have a primitive such as a service That you can just like spin up dynamically You still have the the liberty to use a resource to give it a specific yaml Definition and it will just create it for you. So I'm creating that here I'm creating a load balancer for the etcd instance that jp described Then I'm just defining an etcd stateful set that actually creates the replicas the the replicas that are the independent etcd Workers and these ones mount An ssd and this is required Well, it's not required. It is recommended by etcd because etcd writes data to disk So that depending on the the performance of etcd is tightly coupled with the performance of the disk itself And then finally deleting those resources And I also have this magic container that runs a very ugly bash command And the reason I have this is because waiting for I we need During training the ip of the load balancer and getting that ip actually provision takes a bit of time So I added a bit of a a small container that will wait For that ip to become available All of these are independent components we're covering right now And we're going to get to how we're actually putting all of these things together into a dag on argo workflows in a second I'll show you really quick how the ssd is defined We're using An ssd on gke that has a volume binding mode of wait for first consumer So that the disk is actually attached to your it is provisioned when a specific pod actually wants to mount it because otherwise you can get Pods and disks in different zones and they become they're not mountable in that case All right, so finally the the actual training workflow So before we actually define the the actual workflow and all of the dependencies I'm going to show you what we actually create for the independent containers that we previously talked about that have the four gpus attached to them So we have a fine-tuning container That has some environment variables for the hugging face token that jp mentioned It has an image we're using torch run as the core command, which is part of the container This is the one that will actually run the script and do the message passing and the synchronization between the gpus and it's part of of pytorch It has specific flags to actually inform pytorch how many how many gpus there are how many ranks there are So how many how many containers how many nodes? And that is specified through n nodes here. So in our case, we have four of course we have four processes per node or four gpus But we specified that the rendezvous backend is etcd the rendezvous backend is used for synchronization By pytorch between the different ranks We have the etcd ip that is actually passed as a parameter from through our go workflows And we have some extra parameters here that represent the id of the training session Which is like a unique identifier in etcd for all of the metadata of the training run And finally we have the actual fine-tuning script Which we which we borrowed from Lama recipes that has the actual training loop that For the model and the download of the data set We're focused on the infrastructure aspect if you're curious about the details there the the details of the training loop It is a great Lama recipes is a great repository for getting some of those details Okay, so back to back to training We have some impulse defined which are the input parameters to the specific job And of course we have the actual resources that we have to use so we have four gpus We need quite a bit of memory for this so 120 gigabytes and several CPUs We also mount some Some volumes that are dynamically provisioned this container actually turns into a argo workflows template inside the workflow itself So if you just use dynamically provisioned volumes all of the container copies will actually mount the same volume So you need to create volume mounts that mount dynamic volumes dynamically which are created through the workflow All right, so here's where we define the workflow through hera We have several volume claim templates that one volume claim template that is defined for every container that we create So here's where we say Give me a persisting volume claim called rank i for zero through three through three Each will have 20 gigabytes And this is useful if you want to save the the model to disk after you actually train it to save it in your own buckets or registries or something like that So these will this will Tell Kubernetes that four of these Containers four of the volumes should be provisioned and this volume mount will actually mount the The independent volumes to the right container based on the the name So finally we get to the part where we actually define our dependencies. So we have a workflow We define a rendezvous ID, which is the training session ID that goes to etcd We define an argo workflows DAG called fine tune We first the first step we do is create the ssd storage class That that that is required for for etcd In parallel, we have this list here. We create the etcd stateful set And the load balancer in in parallel Then we have the the independent container that I showed you to wait for that specific load balancer ip to become available And then it has an argument called etcd service name So that specific job requires an input for the service name that it should wait And monitor for for that specific ip to become available And finally we have in parallel In here in a loop we're defining we're calling fine tune Four times of course or up to the number of nodes and we pass in the rendezvous ID The node rank the node volume that should be mounted and finally the etcd ip that it that each container should connect to And as jp mentioned, we have to somehow actually delete all of the resources That that are spun up dynamically for for the training session So we Called the delete etcd resources in the end as an exit As an exit DAG in our go workflows, which will execute irrespective of the result of the of the workflow And then finally we create the training workflow now. I'm not going to run it at the moment because these are First of all expensive resources second take a lot of time to actually spin up And it would be dreadful to wait for them to actually come up So we'll go back to the talk None of this is actually it would have been possible without a lot of community involvement So we're very thankful for contributions to hera argo workflows pie torch the open source single lama 2 Which are all amazing resources that you should definitely check out We'd love to hear your feedback. Um, and here's a qr code for the repository As well. Thank you for the crew for welcoming us in our talk and thank you everyone for your attention I think we have about three and a half minutes if anyone wants to ask some questions Thanks everybody. I I see you you want to answer ask a question. Yep I'm going to hand you my mic like in the center How many different uh instances have you tried running with this because I imagine now it's easy to spin up many different models When you say instance do you mean like uh different llms like lama versus other models? Exactly Yeah Yeah, we've only focused on training lama 2 The train the the training loop that you find in the lab recipes We'll provide a lot of examples of how you can actually wrap other llms through through a similar infrastructure And and pie torch and all of this so you should be able to train From scratch if your business is interested in that or fine tune on your own data using this type of model as well And that's regardless of the model or their specific limitations on which models can be used This type of infrastructure is useful for large models If you don't have models that benefit from sharding independent chunks of a model to a gpu then don't use this If the model is small don't use this. Uh, it's going to be very costly for the actual requirements of the model Looks like we got another one. Yep Imagine that your gpus are mostly reliable and you have now a very large expensive job and your gpu number three Just failed that you don't want to lose the inputs that you had on the other three gpus So what's the story here? How do I rerun exactly that chart in that chart only? There's actually a retry flag on the On the on torch run itself and I believe it keeps track Of the failures within a specific rank or one of these specific containers and it will try retry that specific job Or continuing fine-tuning from that specific point And there's also check pointing that you can use and I believe it uses that as well which Saves the models to the disk and then it really is it really loves them upon retrying All right 50 seconds any other questions Going once twice thrice again. Thank you guys so much for coming out and checking out the talk. We really appreciate y'all


Mae cyntafol yn galluikesgwch ieich FOneid Od처럼, ond'u cael ei wneud cyfal taith sydd fawr a ffair a Fсть trackl y byddai'r hawm. Taith y fawr yn cymuned ieich FOneid Yr Ymysgfaith F? Armaen nhw fawr efallach am ei wneud cyfal taith yn zefnwyachxesgwch ieich Foneid Yr E니까u..... Efallai hwylter Nord brakel, ei ddarparu a ddawdd a fawr o disclose a sew Down sounds, ond a마 efallai fawr yn cyfbras Paul Rees. The next item of business is consideration of business motion 397, in the name of George Adam, on behalf of the parliamentary bureau, setting out the timetable for a bill. Any member who wishes to speak against the motion should press their request to speak button now. I call on George Adam to move the motion. There is absolutely no case for bypassing due process, and the laws as they stand do not need to be renewed, they do not need to be pushed, steamrolled through this Parliament in just a few hours, because the provisions of the acts do not expire until the very last day of September. There should be proper consultation, and Parliament should have adequate time to scrutinise this bill. I call on the minister to respond. I feel as if I'm in Groundhog Day here at the moment, but just to go through the process again, this has been discussed at length at bureau. We discussed the whole process, we discussed how we would take this bill forward, and for Mr Kerr to come here and say that it's been bounced through and rushed through is just complete and utter nonsense, and really we're wasting time that we could actually be using to discuss the matter itself, which is more important to the people of Scotland. The question is that motion 397 be agreed. Are we all agreed? Parliament is not agreed. Members should cast their votes now. The result of the vote on motion 397, in the name of George Adam, on behalf of the Parliamentary Bureau, is yes, 85, no, 31, there were no abstentions, the motion is therefore agreed.