And welcome to the homelab show who forgot to look at the episode number, but I think what are we up to Jay? This is isn't it 62 or my 62. Wow See yep, it is 62. I got distracted because we started a couple minutes late. Welcome to the homelab show episode 62 Q&A episode We love when all of you contact us and message us and especially when you put it in the form on our website Which is the Google sheets for fold disclosure because it's easy and it collects all that data for us And we like marking these things in a spreadsheet. So the Questions we have enough of them to do a Q&A episode and before we do that Let's really quickly think a sponsor show and that is Linode Linode has been with us since pretty much the beginning here I think only one or two episodes wasn't sponsored by Linode thus far and we've been happy with them and I know a lot of you have Had a good time setting up and using some of the tools with the node to put some of the projects We talked about in play or you know some we get it's homelab But occasionally some things do need to go in the cloud and you need to put it in the cloud The load's a great place to do that We have an offer code of Linode slash the homelab show if you'd like to sign up with them We thank them for me a sponsor to show and let's get started with the Q&A Yep, let's do it so There's a several different ones we could start with but one thing I do want to mention is that There's like and this is interesting It's almost like a lot of the fans just got together like they met up somewhere And we need to all ask about the same thing because I don't think I've ever seen like so many things about the same thing Yeah, literally actually two things that that came up was RDP or similar related things like Connecting remotely and also like user directories came up a lot So those are definitely things that we're going to be considering for future topics. Yes The fact that we had like I don't know how many like a lot of the same question I'm just joking. It wasn't quite 15, but yeah Yeah, well, that's definitely something we will consider and we come and I just need to talk more about it and plan it Yeah, and those topics are gonna be around how do you essentially manage users whether or not you use active directory These are some weird questions for homelab people So I'll go on the assumption that they're homelab people wanting to get more into the enterprise side of when you do need to use and Federate with active directory and things like that, but we'll we'll make those some topics So there's there's a lot of discussion to be had around doing centralized user management and IP management and radius servers and LDAP servers and free IPA and PAM authentication and Just goes on so many things and I love these topics because they You know, I'll be the first to admit I am not an expert on LDAP.  I have managed it. I have managed active directory. So I do have experience I don't have the experience of like a windows domain admin that you know, someone that does this every day but These are the type of topics that that caused me to learn more which is always a great situation when I Learn more myself and then I pass that on to other people, which is exactly why I do what I do So this is another opportunity for me to you know, be more Go go deeper into the subject. Yeah, the Directory management is a little bit of a challenge though because this I was at least get this answer right out of the way for people There is not a direct One-to-one type of option inside of Linux that works as well or as thorough is active directory does for windows That's a thing that a lot of people ask this is a general broad question But I'll throw it out there and everyone's like well, you can kind of do it with radius You can kind of do it with that, but there's not really There are of course servers and someone's gonna bring up. I think it's called Zental server It can emulate active directory, but I'm really not sure what the future looks like for any type of System or platform on Linux that emulates active directory a lot of times you just go with Active directory if you want the full features of it and it's usually because you're dealing with mixed environments because you can tie Other things in the open source and homelab world to active directory true. NASA's an easy example PF sense can be tied to it lots of other projects have the ability to interface with it for central authentication But the opposite is very much not true It's not Windows doesn't necessarily work nearly as well with any of the smaller solutions that run on Linux to Manage your Windows servers. So and if you get into we can we can complain all day about Microsoft But if you want to work in the commercial space and you work with end users You will run into the Microsoft beast that is active directory So or even the Azure 80 the cloud version of it So there's it's definitely something you can't ignore if you plan to work in a space Unless your space is only the Linux space not anything end users do so It's not going away anytime soon, but it's so really cool And I do like it. We have one it career questions in the chat and he said novella is making a comeback That'd be a comeback story of the year So yeah to build further on what you were saying With the disclaimer that I feel like I'm at best Intermediate with LDAP because I'm going to be full, you know, people say, oh, you're the expert Well, I might be on some things, but I am also human and anyway what it seems like to me is that When it comes to Microsoft and I don't I don't really know if they're still like this now because I got to give them credit You know, they've done some pretty good things. I mean, they're they've not been, you know, nearly as Toxic as they have before they love Linux apparently. So there's that But it seems like they get technologies that are cross-platform and they make them proprietary which You know, it is so true that active directory works better You know windows works better with active directory. I mean, that's just the case but then also keep in mind it's like It's almost like Microsoft gets LDAP which is not system specific platform specific. You could just use it It's, you know, pretty much a standard But let's just add some proprietary bits that are windows specific on top of that And of course the entire industry is going to hook into Microsoft because, you know, they're Microsoft So then active directory itself, even though it you could argue that a lot of the underlying technologies are exactly the same they made it custom just like with You know, Microsoft sequel the syntax is pretty much the same, but it's Microsoft sequel. There's some specifics there When it when compared to my sequel a number of other things. So To me, it almost seems like there is a standard But Microsoft just takes the standard and makes it like just focused on their platform And unfortunately like you're saying, I mean when you buy Like pretty much an enterprise device It's it's wanting to hook into active directory because that's what they assume that everybody has So it's going to be one of those situations when we do make an episode about this topic where There's going to be some situations where some homeland people may not want the burden But then also they might want to run active directory if they have that at work and they want to learn it So there's there's different angles we can approach this and I think it's going to be a Actually a fun topic when we when we get time to do that Yeah, it's also I'll answer this question working in the enterprise space even with some very large companies brands you may have heard of Um, people always ask but Tom don't they have all their ip addresses and mac addresses in a radius server And everything's properly assigned to every network and everything goes really smooth I'm like, you know, we started out that way There were great intentions and then reality set in and my favorite phrase no battle plan survives contact with the enemy And that enemy is trying to manage. Oh, I don't know thousands of mac addresses in a radius server Everyone thinks they're going to control their environment perfectly. It is a Worthy goal and if your budget is unlimited you may achieve said goal Then you may realize the it is a lower priority because it's not the biggest security risk you face And uh, it is not the biggest challenge you'll face in enterprise, uh tech So while some places may have that under managed and I guarantee many homelab people do Myself included every mac addresses Nicely reserved and organized in my network. Um, that is manageable and reasonable as the network gets larger Or to the scale of these large companies. Um, the goal is still there often to do that But then they start breaking them apart from what can we manage? What's reasonable to manage and what are they going to give me the budget to manage? You know, I mean budget if we had unlimited budget, I mean imagine what all of our homelabs would look like But in addition to that I mean we have so we have different homelab types of people, right? You have the ones that have a very functional basic homelab does what they needed to do three four different things in there Works fine And we have other people that you know have something that's gloriously overly Complicated in all the right ways and it's like your network is better than the majority of corporate networks like the consistency that some people put into this and um, I just love these types of things because it's just um The line between company and home You know is thinning and i'm not saying your homelab is going to be part of your company But there'll be there's so much in common with the technologies on both sides, but yeah Actually, I actually am a laugh at jimmy the it guys comment here You need a radius department just for managing a large-scale deployment So you'd have a chief radius officer. I imagine as well So you'd have an entire department just in charge of asset inventory and mac addresses That's someone who's worked in corporate it that knows oh, yeah, it's a Everyone thinks it's just going to be magic If you have a if you have radius in your job title And you talk to somebody who doesn't know any of this at all It sounds cool. Like I am in charge of the radius. So that sounds super important to the the security of that pretty much everything Um, but it's not quite what people think. Yeah. Yeah, we'll make a better show topic though Because I think it's a great I think it's a great idea because there's a lot to unpack So me and jay will definitely um, we have enough people asking the questions There's obviously a demand for the knowledge. So we'll put together some show topics on that and kind of explain them Some of the options in there. So absolutely That's um, I think I gotta do a video on on this because apparently when this many people are asking about it It's a big deal. I need to totally we need to get on this. So yep Yep, um, let's see. I think also you would end up covering, uh, how to do it with ansible I think that's we started talking about that before the show this we got a little Jay jay would show how to do a management via ansible to how to manage users and trigger, uh Trigger changes Yeah, but that's gonna see why I feel like when I get to that topic It's going to trigger people in other ways too because part of my argument Is going to be that I don't feel active directory is necessary for as many things as people use it for I'm not saying that there's no value. I'm not saying there's no benefit There's no reason to use it. I'm not saying any of that But I do feel there's situations where a configuration management server can do what a company is currently doing And they don't need active directory, but I know that's going to take people out of their comfort zone So, um, that'll be interesting to see the reaction. Yep. All right. What's the next question or topics here? So one thing I could just bring up. I don't think it was so much of a question But uh curt wrote in regarding his home lab and and other people have too I I feel like I need to pay more attention when people Talk about what they're running because this is something that could inspire other people That might want to check out some of these things some of the things that we may not always cover but Well, he wrote in and I'm just going to give some of the highlights here TrueNAS core Plex file server Um open this is interesting open media vault number one and number two Now I've heard of many people running open media vault and it's pretty cool. I like it. I've used it before I'm having two of them. It's like I don't know how like are they clustered? Are they just two separate things that you have things? I guess one could be a backup Yeah, but but according here. I mean, it's like they both they're both running docker portainer Shouldo be for the cctv, which I haven't used yet myself Portainer, I haven't used yet either docker. Obviously. I've done a whole series on that um, there's there's a lot of cool aspects here about About his homelab and I think that to me stood out like two open media vault servers. That's pretty cool I'm just trying to think of all the things you could use a you know multiple Open media vault servers for that's pretty cool But for the most part, I mean it's pretty neat to hear what people are running and how they have things set up So that was pretty cool. That was in response to Another inquiry in the past So I figured I would mention that if if people want to check out open media vault That's the main reason why I picked that question or that statement You know true nasty is great. It's what I use. I love it but Open media vaults cool, too So if you want to go that direction, I don't feel like you'll be disappointed in that decision I think it's it's working pretty well from everything I've known about it So that could be an alternative for somebody that Might want to go that direction and one of the cool things about open media vault That does set it apart from true nasty is that you can run it on a raspberry pi doesn't have to be x86 That that alone could get that could make that an option for some people that didn't otherwise have an option for that so Yeah, it's the arm compilation is pretty awesome because you you can also look up like some raspberry pi storage projects You'll land eventually on jeff geerling and the pet of the pet of pi project. I think it is But not everyone's gonna build Yeah, he if you were interested in building one jeff's definitely got that covered But it's cool that you can look up a few of these really neat little projects based on single board computing And low power that you can build with open media vault like I understand that use case for because I I don't use Open media vault, but people say why don't you use some videos on it? I just don't use enough I think jason a couple and there's a couple other people have done videos on it And they're almost always centered around some of the lower power stuff. Which is great. I mean, it's got it If that's your use case, absolutely. I'll see a reason not to use it. I just don't really use it I think the only reason I might not use it, which I don't even know if this is a valid reason to you know I'll give that disclaimer. Last I looked Admittedly over a year ago The development team was very small. I want to save it as one person or maybe just a few And as a business is kind of hard for me like When you have a smaller development team, it's like if there's any vulnerabilities Are you keeping up on that now a common rebuttal to any Statement like that is going to be well. It's open source is based on debbie and debbie and gets security updates So yeah, of course, it's secure But then I counter with that usually with But there's custom code That isn't part of debbie and that custom code is where Some vulnerabilities or vulnerability chaining can happen. I'm not saying there's any security concerns with open media vault at all So take that with a grain of salt But I do think sometimes you have to at least look at the environment to see Um, how active they are in security and maybe everything is fine and they're you know on top of their game for all I know I'll just assume they are unless I find otherwise But that's the only Thing that would probably make me caution but uh again if it's uh if they're on top of things take that with a grain of salt it's uh, definitely a great solution every time I use it like Um, I may not feel like the interface is as good as true and as but it's pretty darn good Especially when a lot of commercial ones have a horrible interface and you pay money for that. So Um, that's a completely different thing now The next one I want to take I'm going to try to summarize this as best I can because I feel like we're going to do an Entire episode about this I guess you could kind of consider my answer to be a preview of you know, the direction I might go with it but the question was um in regards to It looks like from chris in regards to team viewer and this is one of the many remote management questions on here and What I want to do is kind of set some Initial picks that people could take away right now and then in the future. We can deep dive into any of these Um, specifically ramina and x2go are two things that I'm going to bring up now What I like about ramina is that It's basically a client to connect to something that's serving a remote desktop whether it's vnc rdp x2go ssh Whatever it doesn't matter send your repositories if you're running linux And what I like about this is that with ramina you can have All of your connections saved so it doesn't matter if you have like some microsoft servers some linux servers in one Dropdown one menu you can have all your servers listed regardless of their os click on the one you want Even if it's just you know straight ssh over a terminal you can add that in there too And then in one app you have all your remote connections regardless of the back end technology that they use so I think it's like Instead of having a vnc client an rdp client and all these others You can have everything in one. So I think that as far as connecting two remote desktops Ramina definitely the way to go as far as like You know presenting a desktop to share um I isn't it true that rdp is open source now or am I mistaken? I don't think it's I think it's been reverse engineered So I wouldn't say it's open source But it I do know and I have not spent much time testing it that a bunch who has the ability in there now to Emulate rdp so you can connect using rdp But then you're also using a reverse engineered open source client like ramina that also reverse engineered it So you it's kind of how um smb works It's microsoft's protocol, but you can actually start a smb share on true nas and then use it in Samba to samba, you know using it in linux, but technically you're just two pieces of reverse engineer running on a protocol That was reverse engineered and it's kind of how rdp is it's not to my knowledge open source at all or open Well documented standard, but it's not something microsoft is pursuing people for for making for using it as I understand That last thing is key because I i'm trying to get rid of some of my bias here and i'm going to admit some bias You know against microsoft, but also understand the reasoning for this because When I was first starting out with linux rdp vnc pretty much the only thing that existed at the time And even vnc was kind of hard because you had different like providers and different packages and things which made it a little confusing But when I started with linux microsoft was actively suing companies for using it Okay, so yeah because of that um in that time period, which you know, we're talking what i'm just going to guess and say probably Um early mid 2000s. I'm kind of guessing here My mentality was if it's microsoft it's gone Like I I don't want anything microsoft installed on my equipment because they're suing my favorite operating system or people for using That operating system. They're not really as far as I know they're not doing that now So again, it's just sometimes it's hard to get rid of that But because of that i've always been I don't know not rdp though. That was that was my whole slogan with You know remote management and then I discovered x to go And then there's been no reason for me to consider anything else nowadays I feel like microsoft is not like they used to be So maybe rdp is totally fine to use now, but with x to go I love it. It's great. It works. Well, you could present the entire desktop or an app You could you could have a headless server And it's and I used to install like kaden live for video editing. Yeah, and tom you remember this You already know what i'm where i'm going with this Um where I would have like a server that was completely headless But I have kaden live installed on it and I would be able to share kaden live with x to go So the app appears on my desktop as if it's running locally, but it's actually running off that server And then what I would do is render the kaden live job On that server which had like 40 something cores and I'd hear the fans go crazy. It was great But x to go my point is is that it gives you the option of have of sharing just the app Where you can present the app there's a word for that.  So Considering that x to go has all this functionality It's still hard for me to recommend rdp Not now now not because of you know bias against microsoft just because x to go works so darn well that it's hard for me to consider other other things That being said rdp exists and it's something that will will need to be covered and Leaving the bias out people want to know about it want to know how to use it I think it's going to be a great episode but I just wanted to get that x to go recommendation out there because if Right now you just want to know what to try ramina next to go ramina to connect to things and x to go to serve the things Now and I'll mention when marx is kind of on the same topic and someone mentioned in the comments But it's something I was going to look at because it looks novel and a lot of people It's becoming popular. I really don't know how good our security is so please I'm not giving this a security endorsement because until things get code reviewed. There may be some major flaws in there to allow for Well bad things to happen, but rust desk dot com the open source virtual or most desktop infrastructure It's kind of like a team viewer so people can get remote support I have not used this but a few people have mentioned it to me I think it's come up in my forums a couple times I did see I think awesome open source did a video on how to get it set up There's another youtube channel called that just type in rust desk is what it's called But it's it looks pretty cool. Um, the challenge with any of these is like x to go is going to use ssh as a transport layer ssh is doing security handling at that point So you're less concerned when you use some of these other third-party products everyone's excited about features But I always first question asked is security Um, if I expose this or if it's not if it all behind a bpn Less of a chance you've reduced your threat surface But if you're going to host this publicly or someone had commented they use it to help relatives remotely Yeah, well, hopefully that protocol has been vetted and secured and someone's taking a look at it So just a heads up on that but rust desk looks pretty cool But i'm not endorsing its security for people looking for Because I know home labbers one of the things you get to do is be tech support for family I completely understand why that comment came from If if you chose that path and that then or that path has been thrust upon you Then yeah, you do need some tools like that and team viewers become kind of expensive. So it's worth taking a look at Q warning one more time about security. So I always talk about security first No, I mean and that's a that's a great thing one last thing I wanted to mention about x to go Is is just use your imagination and one example. I'll give I did a video a long time ago You know, I don't know if you ever forget video like what you covered in a video because you've done so many I just know that I've covered this Where I had a raspberry pi Which of course might be memory constrained And then using x to go I had firefox running on a server with a bunch of ram And I used x to go to present firefox from that server onto the raspberry pi desktop As if it was a native install of firefox Now what that means is that the memory it's going to use some of the memory of the raspberry pi it's not like completely free but The majority of the processing is being done on the server, which of course is crazy to have this like really expensive or maybe not expensive but a Huge server and then a raspberry pi is a front end, but it's cool and it gives some it gives the raspberry pi like almost like a thin client kind of Thing and that could be something that somebody might consider Maybe you might have a server and then have raspberry pi desktop. So the app's running off the server. The only problem though is Um watching youtube videos When you're sharing it out from a server you could probably expect choppiness and or the words not matching the lips If you're watching someone talk on youtube if it works at all So it's really horrible for video. So so keep that in mind But there's so many different things that you could do with these technologies I think that with homelab we just love to Find new ways to use things whether it was intended for that or not and that's another example of that. So Yep, um What's the next question you want to do 119 Yeah, I was actually thinking about that. Um Now this answer isn't going to be what anyone wants to hear, but it's just the honest answer I really don't know anything about salt stack. So I'm not in a position to teach it at this time It doesn't mean that I won't get to it. I want to cover it Yes, I'm Probably in favor of ansible because I'm obsessed with it But I do understand that everybody wants to use ansible some people want to use other things And if nothing else and I think that's what this person is alluding to they want to know like What's the difference or how it compares? Which I also want to know as well The issue is my backlog is at least 30 somewhere between 30 to 50 topics deep right now. So um, I wish we had more hours in the day like, um You know, it's like one person trying to cover all of dev ops all of homelab all of linux. That's a Very big uphill battle. So I'm hoping to cover this It's definitely on my radar. It's on my list, but In between the honest probably find an expert we will bring them on the show though. So we we happen to yeah I'll have to see because I don't know I can't think of any of even in my fairly large circle of developers I don't know any of them with the exception of david berke who runs security onion who uses salt stack He's the only person I can even think of and he's focused obviously had security onion But I know I think they still use salt stack as part of their build process in there But it's I don't know that it's a lesser use tool. Um, it's used in big projects for sure It's definitely out there. Um, but I don't have an expertise it needed a j in it So we're not sure we're gonna be able to do a salt stack episode anytime soon Yeah, it did you hear that like I'm hearing this strange sound It sounds like a bunch of keyboards going probably if I had to guess twitter people That are on twitter that are listening to this At tweeting somebody who's probably some kind of expert and chances are by the time we're done Recording we probably have like four tweets from somebody and I'm hoping that's the case because that would be really cool to To know who it is people want us to bring on and if you can connect us with a salt stack expert We'll bring them on the show because Is of all the people I do know for all kinds of different things that's not one of them It's the same with open stack. I I love open stack. I don't feel I'm At the level I need to be today. So we had somebody on to help talk about that So that's absolutely a great way to handle that so people can get the info they need without waiting so Yep All right. All right for the next one I am scanning the questions right now 120 is interesting and it's kind of a new question, but hey guys love the show fellow michigan der so awesome Uh question about docker compose containers. I'm using snap rate running sync on my nas nightly But sometimes my containers writing data interfering with the process My thought is to run cron jobs to start and stop those containers to set time But having issues getting it set up any advice about the right way to go about this So it sounds like they want to snapshot their containers, but have them and I'll stop state to do it. What's the best methodology? Well, I think it's I'm not really sure if I'm going to be the most helpful on this one because it seems like There's it almost seems like it's worded as a race condition It's writing, you know, the individual says sometimes my containers are writing data and interfering with the process So I haven't used snap rate personally So I'm not sure if this is a snap rate specific thing and let me know if I'm interpreting the question wrong if you're interpreting it different tom than I am but I'll just I'll just say what I do There's probably no value to this individual because I'm running true nas and they may not be running that but what I like to do is have a nfs share where you have a directory like in true nas you can have different data stores And then have the directory named after the app and then all the apps kind of just have their config files there So the beauty of that is you can even if you delete the container since the storage is on nfs The container comes back up it reads its data then it it's like nothing ever happened But you can also have a snapshot schedule on true nas to snapshot that data directory So if you want to You know revert back to a previous config you just do that in true nas and the next time the container comes up It'll do that I don't know about snap rate in this regard though the problem individually this person's having Yeah, and I imagine so it should be and we'd probably have to understand the error message better You can just start and stop the Docker containers That should be straightforward enough to have something that stops them and then the backup process run And it's just a timing issue at that point. So no data is in flight As I've seen if the docker container and this is an example for you I use bit warden I don't want to snapshot anything in bit warden Because it's a database driven at the back end So you option a start and stop it option b is what I chose Option b is I export and do a database dump to a file. So I don't worry about data in flight There's often I mean if you're starting and stopping vms If that's the only way you can do it then that's the way you chose to do it ideally because the containers and all the docker stuff should be ephemeral and just be whatever I can destroy and rebuild them any time focus on the data itself and Outside of databases most of the reading and writing you can just grab a snapshot of the data at any given time But transactional things like database the best way to back those things up is to Run an export tool against the database so I can grab a snapshot even for example with our database that runs our invoicing system invoice data We're running regular backups of the database you can set up a either a whole separate database server to send it to So you always have the data and it's doing a commit of that time as opposed to the less ideal Especially when you get into the enterprise space situation is shutting down a vm I mean you don't want to do that But of course you want to have those backups so think about the things that are in flight Or how you can take wallage running and do an export to a location as your backup That's a just a better overall strategy for backing things up too Yeah, and and cron is probably I mean if you don't have kubernetes or some kind of containerization thing going on Which let's face it not everybody needs that right unless you're wanting to learn kubernetes. It's like Do you want to maintain that I feel like the problem This individual is having would be solved with kubernetes But then I'm not going to recommend kubernetes because I could be recommending a burden depending on If they like that technology or want to run that So in lieu of that I would say a cron job. I can't think of a I'm sure there's an easier better way But personally I think that's going to be probably the best way to go for now at least But maybe if somebody has a better way they can write in and we can share that Yeah, um, did you have time to read the person's the there's had a whole blog post right up on 121 I just noticed that had that on there Yeah, I'm going to scan it very quickly here and see if it comes up You do that and let me explain the challenge to our audience here I had a problem struggling with and recently came up with a solution Wanted to get your guys opinion on it. How do you handle scheduling ansible playbooks when my ssh key has a pass phrase on it Yeah, and I think Normally use an ssh agent on my dot bash rc So I can type it once and done for the session But there isn't a session to speak of when you're running it via cron And I believe without even while Jay's thinking about this I believe the way that a lot of people handle it is you just have multiple ssh keys and you have one that's designed with limited Scope to the ansible system So you'll have a different ssh key with a more limited scope of what commands it can run Tied to the ansible is that is that me guessing a right way to do it? that that would work but immediately i'm thinking that's going to be like a huge, um Administrative burden because if you think about with with sudo right when you allow things Most people just allow a user access to everything a better way to do it, of course is Allow the user via sudo to Run binaries that it wants to run So for example If all you want to do is shut a server down Then you're going to give the user access to do that one thing If you want if you're running on debian you want the the user to be able to install packages in sudo You have to give them access to apt so they can do that but the problem though is You're going to keep adding individual things to sudo and it's just going to be ridiculous to maintain but So but if the ssh key is definitely a problem if there's a passphrase the passphrase What you hope for is that if the key leaks that the individual is not going to be able to use that key because they don't have the pass Ray so right the mindset is great. This individual's got their mind in the right place for sure But the issue though is you're not present to type that key in now My opinion Is that just don't use ansible ssh? I know most people are like spitting out their coffee right now There there's probably some people raging like that's exactly the way ansible was designed You're telling us not to use it the way that it was designed Um, I've had nothing but problems with ansible using it with ssh for example Like I've talked about before my workstations laptops and servers are all Managed by ansible but servers it's fine because they're usually on all the time But what if you you turn your servers off at night? Then your ansible server if it's separate Can't access it it's going to keep airing and send you air messages or you know if you have like alerting via emails It's going to send you airs So you have you wake up in the morning and try to reach the server 10 times or the servers off because you're saving power It can't access it it gets complicated to tell ansible to run only certain during certain times But then it's worse if you also maintain your laptops and desktops What if you take your laptop with you you put it in your bag it's suspended It can't be reached there's going to be errors So and then you have this problem here with ssh passphrases Where you know, you don't want that ssh key to leak out Obviously, and if it does you again, you hope that there's there's a passphrase on that The person who stole the key doesn't have that passphrase so they can't use it But then you have this problem right here where you can't really use it unless you take the passphrase away Now my solution is ansible pull. Yep It's not a common way to run it but the more I mean I've been using that solely like for the longest time I started with ssh as my method as well And I've had nothing but problems like I mentioned so what ansible pull will do is it's kind of like the inverse of ansible Where you give it a git repository and you could tell it to only run if changed It could pull that down and there is a way to make the configuration different per machine Which is you know, the first challenge people run into with it that can be solved And then what you have is that when the machine is on it pulls down ansible runs it via local host or against local host There's no ssh keys at all in this situation and as long as you make the The git repository private no one else can grab that and use it But even then you could use ansible vault to encrypt things that are secret in the repository Even though it's private does someone still manage to get it and you encrypt all the things that are secret Well, I mean they can't read that information. So you're protected in that way And then you have all the machines just using ansible that way and it just works so much better But then the problem becomes You know, you have some servers where you do want the error message because they're 24 7 And you don't want like ansible to silently fail because if it's not able to pull it down and run Europe you might not also get You know, you might not get an alert. So the solution there is to use health checks.io which is Something that you can hook into any cron job or any task or something you're running And you could have one task per server and and tell it if you haven't received a ping from this ansible script And you know, I don't know three four days send me an email. So that way if something is, you know bailing silently health checks.io will give you a message that it hasn't heard from that server in a while So that way if there's a specific issue, you can go tackle it So and I feel like ansible pull solves all of these problems But since ansible is heavily designed to be used via ssh and that's how like every tutorial On the planet uses it I feel like a lot of people don't even know that ansible pull is a thing. They're just not aware of it Right, that's what I would do Yeah, definitely that's an easy solution For that one ansible jay's got a whole bunch of videos on ansible as well Including, you know, some of the whole auto deploy scripts and everything else the Kickstarter scripts. So Check jay's learn Linux tv channel for that and you'll find plenty of information on it There is an ansible pull video there too. So I have like this entire series But then there's some side episodes that are not numbered as part of the series But they're considered part of the series one of which is ansible pull There's one an ansible bolt teaches you how to encrypt things within your Configuration, so if you go through all that you'll definitely know how to use it Yep And I'll sum up a couple of the questions together someone said they had a lot of trouble getting Things set up in butterfs jay has a whole video on that as well So he's got the tutorial put together and and dives deep into getting butterfs going on a linux install so that'll hopefully answer both of those and The butterfs nas projects the only one that i'm aware of because more than one person messages But it's rock store our ock stor We haven't tested it, but it is something uh from our butterfs episode that people ask like what you know What nas has built on this and that's uh, so far the only one that anyone's messaged us so And I mean this is the synology, of course, but talking specifically about things you can run yourself on your own hardware Yeah, yeah, which I assume is what that is. I haven't used it. Yep um Next one is we covered this back on What was it to do? would have been episode 28 about owning your own domain someone had asked and So we won't dive too much into it, but someone asked about you know, should I get an own domain for my email? Yes, you should because that way you own your email address, you know at your username.com or how we are You know, you know at each of our own domains But then the next part is what about hosting the mail server yourself? We don't really recommend people to do that. I I still get a lot of people who They try to set my forums my forum will see that you're on a blacklist for spam And if you're on the blacklist for spam, I just tell people sorry You're probably a spammer, but then I'll realize it's their name and they'll message me via twitter or some other method How do I get off the spam list? I've been you know, I can't people can't email me I can't email back because I'm on spam list It's just the tediousness of running your own mail server. So we don't really encourage that part as much I used to be a mail server administrator. I'm not talking because I just don't like doing it I talk about the impracticality of managing a domain and Wanting emails to go through if you look you'll see that launch systems.com is held by google So yes, even myself I have surrendered in for years I was hardcore run your own mail server guy because I was a mail server admin starting in the 90s That's what pulled me into a heavy linux stuff. So yeah, it's still not something I'd really push or recommend here in 2022 What I would recommend though is personally I've used both hover and I've used a fast mail Fast mail is my favorite. It requires a little bit more setup But the interface is called fast mail for a reason the web interface is really fast And you could just get your own domain Hover is a great place to buy a domain if you don't already have one and hover themselves They offer an email service. So they maintain it with and then I think it was Last I looked it was 25 dollars a year for a mailbox and then whatever your domain renewal is on top of that So that could be a great way to go You can set it up with thunderbird or whatever or fast mail pretty much the same thing But you can pretty you could pretty much create the experience of running your own email server by For subscribing to one of those technologies And then you could set up a next cloud server and you could actually Have it pull in your mail via like an smtp client and give you that web mail experience If that that is if you don't like the web mail that comes with hover and fast mail because they have their own But you could have that as part of a Suite like g-suite but in next cloud you have your online document editing You have your email in there as well at your own domain And you get all the benefits of running your own email server without actually running your own email server Someone else's problem and you get all the benefits with that method, so I would recommend looking into one of those and possibly even next cloud on top of that Definitely some options there. Um Wendell did and I I like the video series a lot the forbidden router and I may Readdress some of that because I did a video a long time ago about how to build everything into one box Um and Wendell covered it to Wendell didn't cover a couple of the details of it So I thought maybe that's an opportunity. I'll talk to Wendell and maybe meet him and clav on a little bit But I don't have a problem with it But people ask like why am I so hard on people about don't virtualize it I'm like, well, it's because of the extra complexities that type of thing adds So it's a it becomes a real challenge to troubleshoot things Wendell's a little mitigation of Doing a drive pass through so you can reboot your virtualization directly back to A pf sense so you can get things up and running fast. It's kind of a clever workaround for it, but it's so Just that it's a workaround for when there's problems with your virtualization system Or the fact that you lose internet every time you have to update your virtualization server um Yeah, it's something I I don't know if it's a really a homelab show topic, but maybe I'll do a video on it for my channel I think you I think it's important to understand I mean when when we're asked like why are you you know, do you have something like we have stigma against that? I mean we kind of do but part of it is because these Services and these soft pieces of software. We're not made to run like that So you are running them in a way that the developer did not intend But then again, um, we always run things in ways that weren't intended. We're homelab people I mean technically having a homelab is is not the norm So I totally get the desire and and everything to do this because You know, I don't think you're not supposed to do it like that has ever stopped a single homelab person That probably makes them want to do it more But you you just understand you will run into additional challenges And this isn't the way the developer intended if you are okay with that and you're clever like, you know Wendell obviously is and you want to find a way to work around those types of things It will be a challenge. Maybe that's what you want and that's fine. If you want that challenge, absolutely Sometimes the challenge is fun, but keep in mind Um, I personally don't think and maybe I'm wrong and you could correct me. You probably know I don't think pfSense has developed For virtual machines. They're gonna be running a vm. I just don't feel like it is it's usually not for them They have a specific, um support for vmware, but everything else is kind of You know, this is this happened with one of the updates when it broke everything in hyper v So people that were running hyper v ran to a bunch of problems with the update And I think they just don't spend as much time testing Uh, like the hyper v solution and I feel like they'd spend even less time testing the xcp ng solution Although I've got it running in xcp ng and there's a write up on how to do it Which is what wendell shows for his forbidden router. So it's pretty cool. The last thing They asked in here and I have not had experience with it. There's actually a few different versions of this I think these are probably forks from each other But I don't I don't have the whole genealogy of how they got here because there's certainly some history But neth server and eth server dot org This is one of those build it all into one box. It's your file server. It's your mail server It's your vpn. It's your firewall. It's your groupware server. It's your active directory emulation It's your chat server. These are kind of like appliances that throw everything in one box, which are kind of novel They're only as good as their support. There's been a lot of projects like this over the years that come and go Unless they have a good business model around them They just they're hard to maintain because you're throwing so many different functions into a single box It also has a interesting threat surface because now your file server is also your firewall server Now provides your firewall servers maintain well great But you have one box to do all the things but you only have one box that could be You get owned and all your things at once simultaneously. I don't know really how I feel about those anymore I really comes on to how well they're done. So I haven't particularly used neth server There was another one. There's another company and I can't remember their name But hp bought them they rebranded it so there was another open source project to became an hp project Which is they have a nursing license model where they sell it now But if you buy an hp server it comes free with certain hp servers So like that was like their business model like hp owns it to try to push hardware I I don't know I haven't used any of those projects when they throw everything into one appliance The one challenge you have is when you want to do something custom and you start tweaking it sometimes the updates break Because I've had people contact me about the previous one. So you pop in a few school districts Um beat build changed a couple things and the whole thing fell apart And then they had to buy a support contract to figure out how to put it back together Um, it was a lot of problems with the upgrades. So use at your own risk I don't have any direct experience. Maybe they work even better than I'm suggesting I just know my past experience with them, but I haven't specifically used net server any th server. So One one thing I'll mention quickly about I thought was pretty clever and borderline hilarious When it comes to running psense and a virtual machine an example of it working very well actually is um a company I worked for quite a while back And I don't know if open vpn is still hard to build custom But back then the team I was working on was tasked with setting up a vpn server They didn't want to renew whatever hardware they had. They wanted just to do it um So someone on the team looked at it at open vpn and what's required said it up. They're like, uh, no I'm not doing that. So they just installed psense into a vm and then only used it as the vpn server for the company And no other function on that Entire installation of psense was used only the open vpn because back then at least I don't know if it's still the case now It was easier to set up open vpn on psense and it was to build a open vpn server so in that case it worked fine because it was just the company's vpn server, but um You know, that's there are some edge cases where it might work well But in general tom and I we I mean we we don't we can't cover all the edge cases. So proceed at your own risk Yep um One of the other questions in here, and I think this last one we have is We covered all the um Pam and access management systems, but it's is there any open source networking gears such as Uh switches access points besides ddr at or tomato or roll your own um vios is the one that comes up so that uh Is it still called vios? Yeah, the open source firewall router platform but There's some challenges with using some of these and there's a couple other ones out there one of them is even stranger Serve the home actually covered it. Um The name eludes me at the while But there's a couple of these unique softwares that run on some of the del switches But they run specifically on these enterprise switches vios is Definitely among those, but they're different open source switch software and the enterprise space actually does use some of this Because the way they're custom signing hardware and they have various niche use cases. These are not My understanding even from watching serve the home. They're they're not like extremely well documented, but a lot of people use them in that The hyperscaler space the facebook's of the world the big big companies are using some of these So it's not it's not like the same as just grabbing some uh dwrt There's like this small like we can reflash these small routers or we go to the Enterprise market, but the everything in between I haven't really seen a lot out there now. Maybe it exists um, but i'm Not really certain about it, but v y o s Uh dot i o is their site. They have a rolling release for it. They got a bunch of it's been around for a while Um, it used to be there's a whole history of it used to be something else because it's still the basis of it Is actually what runs like your edge routers from ubiquity and some of the other ubiquity was based on some of that so It's it's interesting because there's anything out there, but it's not it's not like uh It really mainstream. Yeah, that's what I can say where you can just go Hey, i'm gonna grab a switch and make it open source and the other challenge with that is many of these switches have A lot of very specific design hardware at them And so you can't get each of these companies to agree how to design them universally So it's hard to write an operating system that universally works with them That's true. I would like to see If again if it doesn't already exist Even if it's just x86 based because you know we can We can get we could find on ebay a small desktop that has multiple gigabit ports on it the protectly ones are an example of that but there's many others and Ability to have something like pf sense. That's just a switch That just grabs a network ports and gives you a menu and can integrate into They can integrate together and it's all open source. I mean something like that would be I would I would say an overnight success in home lab if it if it something like that came out And they did it right. I feel like every home labber in the world would definitely want to try it out So i'm hoping We get something on the level of pf sense eventually but you know for switches as well router switches and have different versions or something but There's no shortage of hardware and things like that keeps hardware out of the landfill And we can use things that we wouldn't use otherwise So I hope if it doesn't already exist that it does I don't know bios is that thing but if anyone knows of anything we don't know then let us know about it Yep, uh, the other one is I see someone posting her now. Remember this one, uh, the Disaggregated network operating system danos project enables community collaboration across network hardware forwarding operating system Layers danos is initially based on the AT&T D NOS software framework of a more open cost What the trick you have to learn with some of these is what they work on and Find that now the good news is you can usually find some of this hardware on ebay, um, but it's It's some learning curves to uh mess it around with it. So that's definitely a uh, That's gonna be a challenge. It's not like there's That's it's not as well Ubiquity is an easy example of company. It makes things really easy with their unified system And even then if people get confused in how to set up VLANs and networking when you start doing this all from the command line Cisco's well documented for how Cisco iOS does things in the way it does it, you know, it can be Argumentally complicated or convoluted, but there's still a lot of documentation you narrow down further when you start using these There may be less documentation you can find on it. So uh, it's starting at the deep end But if your goal is to work in the enterprise data center, yeah, you'll probably it's probably worth learning. So yeah Absolutely Lots of stuff out there Absolutely so much stuff on there sonic os is another one too. So wow, we're getting a lot of good recommendations Yeah, sonic os is I think that's the one that served the home covered is they did a video on using sonic os So they will serve the home has got a great enterprise channel for covering a lot of they just Patrick dives deep into a lot of the switches I imagine there's a massive crossover from our audience because he's covered so many of these devices and things like that And I know he did one on specifically. I think sonic os running on some del switches So yeah, then someone did say yes serve the home to cover it. So cool So there's there's some more reading on there. I don't know anything about it because I just don't use it Um, it's not it goes outside of the things that we consult on so it's not something we run Into very much. So I'm really hoping that sonic os has the classic, uh, sega jingle from the 90s when it boots up Because I know that would be a missed opportunity He's got to do the little uh Sonic Come on guys, you got it. We got to have that patched in it's open source. Someone's got a Yeah, yes All right, um last couple questions. I've seen them roll by and I don't want to leave them unanswered one of the questions people had is um Talking about in the home lab does pf sense cares much about some of the different network cards I don't think most people will ever take advantage of some of the more advanced intel network card Where it has certain offloading on there for their home lab. I don't picture them doing it. I'm saying you can't so I Don't really don't fret too much about that Basically go get yourself in Most of the time and please stop routing storage home lab people I need to do a video on it So I will but um I'm going to do a video on storage design because a lot of people seem to want to go 10 gig in their pf sense When they don't have a 10 gig internet coming in because they want to route everything that sometimes includes Things that shouldn't be part of the route. Uh, you can do that if as an exercise But it's also not the more common way to do it So you're fine with your unless your internet's faster than one gig one gig is fine for your internet on your pf sense or even your internal you can have all the fun 10 gig stuff You put all the same things that need to talk at 10 gig or faster 40 gig for those of you that are a little more advanced You're 25 gig and 100 gig connections for those of you with homeland with the budget But yeah, it's a stick with your standard one gig in tells and you're fine It's even what we're using mostly at my office because we're not trying to route We're not trying to route 10 gig through the pf sense. Although we have 10 and 25 gig at my office It's interesting you mentioned that because I I I now have that problem. I now have a connection. That's faster than a gig um Long story how I ended up with that subscription, but the same problem my uh pf sense is one gig So i'm not going to benefit from that extra 200 megabits in my case. Yeah I need to solve that problem myself actually. Yep Yeah, all right But we have a routing storage someone said ouch. Yeah, that's Look, we we see a lot of Consulting that's where for those of you don't know me. I have a business We do a lot of consulting work and sometimes when we've seen so many things go wrong That's what you see as a consultant. They don't call you because they're happy They call you because something isn't working right and we see some interesting things So routing storage, it's a it's not the right way to do it, but boy is it popular Yeah, if it's not something like that is definitely dns. Yeah. Yeah, if it's not that is dns So or mtu everyone tries to mess with the mtu One thing about mtu has to be consistent across all your switches or you'll have really interesting results so It doesn't work under heavy load, but it seems to work under light load. Do you mess with the mtu? Mm-hmm. Don't do that. Yeah Uh, I said a lot of things back to default. That's um, that's the secret of Consulting for anyone wants to know the secrets of my business. We set lots of things to default Could just know those knobs until you know what they do once you know what they do then twist all the knobs So that's a homelapse for twist those knobs have some fun play with some projects And I think that's all we got Jay, right? Yep. Thanks. Oh awesome See you guys next week and uh, me and Jay will hammer out what we're gonna do next week Because we have to dive into one of these topics that we talked about. Oh, yeah, we gotta definitely cover this stuff All right. Take care everyone. Thanks


Okay, everyone, I'm going to get everything moving a little bit quicker, but I've got a big slice of pizza I want to get eating. And I will hand over to everyone because I've got a big group of people next to me. So rather than me introducing, I will just hand over to the bug crowd lot and let them introduce themselves. So without further ado, thank you, everyone. Can you hear me okay? Alright, I'm really excited because it's many of our first times actually talking at DEF CON. So this is pretty exciting. I'm going to just quickly do a quick intro and then have these guys talk a little bit more about themselves. My name is Chloe Miss Daughey. I work at Bug Crowd. I'm a secure researcher advocate. I'm also one of the founders of Women in Security, WOSAC, and also the founder of Women Hackers. And I'm really happy to be here because we're going to dive into our, these are three of our bug crowd ambassadors that I get to work really closely with and I'm so excited for this panel. I have Jesse all the way down to the right. And then in the middle I have Sam. And then here I have Darryl. Anyway, I'm going to have Jesse, can you please go first and tell a little bit about who you are and your background? So that's my gate job. So I spend time making sure our product line is secure. I also run our bug bounty program. And then I spend nights and weekends hacking against other companies on bug bounty programs. I am Sam Curie. I've been doing bug bounty for about three years. I started off doing full time research for about a year and now as a security analyst, I work three hours for another year and I have been doing bug bounty for a year. I have been a bug crowd ambassador for about a year, I'd say. Yeah, and it'll do some really cool stuff as well. My name is Darryl Lopstad. I've been on bug bounty for like about two years. I currently work as a senior penetration investor and I've been doing under like seven. Nice. Okay, so tell me how long have you guys been an infosack? Seven years. What about five or six years? Yeah, I think I've definitely dropped continuing the market at this point. Nice. Okay, so let's start with you Darryl. Oh, actually Sam, let's keep it with Sam. Okay, so Sam, tell me when did you start hacking and what were the moments that led up to that? Yeah, so like, probably my sophomore year at school actually started messing like what up stuff. I moved into Brooklyn, I think like when I was senior in high school. I like spoke with some friends of mine who have been doing it and I kind of taught them how to do it. I remember I was working at like Darryl King. It's a good time. But I got paid for like about five hundred dollars. And at a time I was like, two weeks back at Darryl King. So I was like, the static. So that's how I decided like spend a lot more time doing it. And then after high school, I just did it full time. So I went to university. Nice. Jesse, what about you? Alright, so yeah, I would say like the first time and I would like consider myself like happy. It's also in my school. So I thought it was a watch that I could program my program to control all the teams in my school. And so I definitely got attention about that. But I was kind of like my first experience actually trying to break something. And that's why I didn't just be interested in security in general. So I started doing a lot of research on different things, mobile app and that kind of stuff. And then that led into my career starting when I spent five years working for the Department of Defense doing some of the security stuff. And then move over to private industry after that. All right, Darryl, your turn. So I got started a little later than everyone else here. I can start out here doing anything in for a second till I was about 25. So again, into the main hall later. And after I figured out that I had set up enough deadlines, I went over and started doing it in for a second, a lot more and then go to the private sector as well. Nice. Okay, so what tools did you start with? Not. Anna and Bert is where I started and that's it. There's just a hard one called live HTTP headers. And that's kind of accepted by a KQ class. So I think first of all, I used to call it that. And it's like to, you should be able to, there's just like really, really the easiest way to learn for websites these capabilities like the problem of transactions you were trying to live on. But yeah, I'd say go to you for sure. It is definitely my first tool for me.  And just using my Chrome Dev tool for something is what you're doing with that. Nice. Okay, so when you started learning and even today, how do you keep up like with everything? Do you use Twitter? Do you follow certain blogs? Are you one of those people that likes the vlogs more on YouTube? What do you tend to use? There's a lot of, there's a lot of cool stuff. But most of it Twitter, right? I follow like, probably like 400 or 500 people overall into a center lid.  Yeah, there's simply two videos in particular, typically on YouTube. And then more in that you can typically go like, you know, like, Yeah, I agree. Twitter is like my go to. It's kind of how I stay in touch with everybody. But another thing that's really important is like the sense of community, right? So going out and like reading other researchers reports is critical to learning, right? And you use those baseline, because then you can take that attack and try to be more with it and build upon that and then share back. So I think that's really important how to stay engaged. Yeah, Twitter for sure is, and then I have a awful amount of blogs that I had in my RSS feeds that blow up. But like Jesse was saying, I kind of I go in with lots of other people do and then I kind of put myself in that like the mindset of what I found it like that. Usually no, it's not I would know. And I try to figure out how they came to the conclusion of trying what they did and try to like supplement what I actually do to at least add that Nice. Okay, so tell me since you guys shared a little bit about your like personal experience with being hacker and like the resources that you use. I'm just curious now because I just did a talk on safe harbor just like a couple hours ago. And I'm just curious. What was the first bug you ever submitted of all time? The first page submission I had was a sequel injection. That it meant 10 days. Is that up? Is that what you wanted? Yeah. How was it? How was it? Was it exciting? Was it like holy crap? I think like the first page bug I had was like the SMTP induction on hackers. And it was like, is that the kind of thing that basically you can serve character online feeds and like an email header and modify it. And like, so you could like, if 200% before you could like modify it was from and it's really cool because it was all silent stuff. But it was like $1,000. I'm getting my freak out. Yeah, so the first program that I had was actually on Starbucks. Like why not? So Starbucks, I don't know what I was doing. So I just threw in a report like, you can actually get paid like 100 bucks or something. That's a lot of coffee. So keep going and try hard. I learned from that right because I all the questions are taking back and asking about my report. I could take that and like, okay, this is how I need to address this and further reports that I submit to the learning experience. So Jesse, have you ever been scared when submitting any sort of bug of being prosecuted? Yeah, so a couple of years ago, I actually presented over Skybox about a big vulnerability that I found across like 10,000 different websites from the government websites, right? So that was super scary, because I had to find a way to disclose all that. And it was very nerve rapidly, because like most of these companies had no idea how to fix this stuff. They didn't have a vulnerability management program, right? So it was kind of shooting the dark and like, I hope that a fear of nothing happened. So that was a little scary, but I pulled it off. I'm here today. So that's good. I think when I first got online, there was a lot of programs, very particular guidelines. And they didn't mention anything for like Google stuff. And I was just kind of like, you can only hack on this. And this is the same, like, I found this like secret, and I could get up and I was like, I submit this, like I said, I'm like, I don't know, I was, I was, I don't think a fear of prosecution, but like, I felt very like, or did I guess for me, I think it was another secret injection that I found that the bounty brief was, it was, was a very specific about what they couldn't do. But it did say something along the lines of, you have to get the whole whole database to get paid for it. So I was a little more overzealous than I probably should have been. And looking at it now, I think probably should have been really pissed. Okay, now I have to ask this. How do you guys feel about safe harbor? What are, what's your opinion on safe harbor? I would have liked to have been on the program that got, that could have gotten really risky. I think it's great. I think like, I think when we're in Donna's first game out, it was very like, like, oh, yeah, we're on a frosty, you don't worry about it. It's like, okay, yeah, but then I know it. There's not a lot of legal research and like, it's really not legal. And it's really cool to have like an actual guide on for it. One of my bugs recently, it's really cool. It was like, the program had like, very extensive, like, research or protection. It's like a Tesla program. But like, like, yeah, like, you know, when I cross to you, here's the actual legal guideline for it. And then like, if you break your Tesla, like, we'll help you fix it. That's really cool. But yeah, the direction it's going is like, fantastic, I think. And I think it's really powerful. I'm like, going away from like, the, hey, if you connect to your like, I don't know, neighbor's Wi-Fi, you're going to do it all the way, like that whole like, recording. So I don't know, it's crazy. Yeah, I mean, I think it's great. And from like, a program management perspective to, you know, you have safe harbor on your right, you're gonna need more hackers on your phone. So it's great. But when like, a lot of I try to look by right when I'm packing, if I ever hit a gray area, I stop and ask myself, okay, are you being an asshole? Maybe like, actually, you reach back out to the program and say, hey, you guys want me to keep going with this or whatever, that way, they realize like, you really do care. You want this trying to like, interact with them and engage with them. It's pretty cool.  That's fantastic. For those that don't know, has anyone ever heard of disclose.io? Raise your hand. Okay, we got a couple people. This is great. So those that don't know Disclose.io, you can go to the GitHub page and actually look at all the different companies that practice safe harbor. So if you ever want to like, try to find vulnerabilities or hack on something, I highly recommend going to that list first and checking it out to see which ones are the much safe ones. Okay, so since we dive into the background now, let's go into bug bounty because bug crowd, first of all, and we're on a bug bounty panel right now. So tell me, you guys, how did you hear about bug bounty? And how long have you been a hunter? I think you guys kind of briefly talked about it, but give us a little bit more details. And Sam, why are you, since you're holding the microphone? Yes, when I first like, got started in like, sending security stuff, like I play video games a lot. And like, there are people in the community who are like, sort of into web stuff.   And it's like, might be the exploits on stuff. But like, one of the guys in the chat was like, hey, you know, first one of mine, like a vulnerability on the form hub, you get like a hit rep, you know, like, in my community. So like, that's like, all right, let's let's do it. So I went and I felt like a cross-expecting vulnerability. And like, I don't know, I'm kind of excited because that's the first time that I get to that interview, just like, bugging itself. But yeah. So I've been, hopefully, active for three to four years now, I'd say, in like, bounties in general. But I really got my started running and I took it out on long-term projects, creating more of a sale course, myself and run my bounty program. So I got to see how that worked out in a large enterprise perspective. And then that's when I started hacking in and of myself, because everyone was like, how powerful this is for this, and what needs you to do it? I've been on the boat crowded and balance for about two years, or, I think two years exactly yesterday or something. I had a pop-up from Sandy and saying, congratulations. Okay, well, I'm like half there, I'll like that. Not all at once. But yeah, I don't think I had any really big thing other than my wife went to night school and I went forward. So I started hacking on it. And that's where I got my nice. Okay, so let's start with you. Since you got it, you're ready. So tell me how many bug bounties have you been awarded? And that's single bugs. I think I'm at 70 ish at the moment. I think somewhere around here. Nice. Okay, so ranked in terms of priority and importance. What are the top five tools that you use? And I know number one is always going to be burp suite.  Like your search and sell a search. And then there's some other like interesting tools that like, I think obviously going to be a top five for some people like that. And then like burp extensions, right? Like, I'm definitely going to like my top five is going to be awesome now. But brand mine, I'm still like that. Yeah, I really, I'm kind of as a research analyst, I tend to focus on tools too much. I really like doing like manual boot dives. I think I talk about most of my bugs, but like they're definitely situations like where you're getting interested in trivia or whatever. Yeah, those are like my top five are cool. Yeah, so of course, so I would say like, aside from that, I love working on programs that have a really huge scope.  So you don't have to go through it manually to look at the results. We'll just put it next to a report. You can scroll through and see if there's anything valuable like adding manuals or whatever that's exposed. So I use that like to research. That's it. But one like general thing to do is like, a lot of people don't need to automate things essentially, they're having this program. So in a sense, you can do it if the program allows, but I think the tools as it's like if you're lost in the word trade, your tools are accomplished. It's going to play the general direction of law, but it's not going to help you navigate all past, right? So you're still going to have to go do the manual analysis of like different vulnerabilities and stuff. They try to actually figure out what you're doing. The tool is not meant to end all the off everything, unfortunately. So basically everything that was just said, but probably the go to is that I would add there for a walk and said that I found some a ton of things because I didn't rely on the things that the tools that everyone has on GitHub and all that and just did my own kind of projects. And that's where I found the bulk of my bigger bugs. Okay, Darryl, tell me what is your favorite bug you've ever gone? So probably it's being recorded. Sorry. Just actually let you know. Okay, well, I can't do the same thing. Okay. So probably the favorite one that I've worked on was we we had a bug where we got some deep we found some default credentials and logged it into the site. And then the next day when he came back to it, the license expired. So we had to go look on showdown for another like server. We stole the license from now and applied it to the one that had expired. And then I found an authenticated SQL injection in it. Well, I think I think my favorite book probably was when I found that recently. Basically what's happening is there's like a login functionality for website and it's also registration. And I think they're using with a very old system for the actual core handling of authentication. And what if you sent like a front sense ABC for the agent and there's taken it, you send it, you check it and pull back, right? And something interesting about this was if you sent ABC percent zero zero, it thinks the string length is forward. And then it goes down thinking this thing is forward somewhere in translation that gets turned into three. And then the server says, okay, well, we can send back four bytes for you know, a memory. So it sends it back. And then you get a random piece of memory, right? So instead of sending it to you see you send thousands and megabytes of null bytes. And the server's like, all right, you know, this is a too long to use your name. We've checked if it exists or not and hit that system. We're going to send back the data. And then it just sends back raw server memory.  And then you're sitting looking like 10 gigabytes of server memory.  And all right, so let's search my password. Let's see if it's logging passwords. We need to find a password. So it's right script to automate like pulling keys using passwords. And it's just a really unique bug. Yeah, so mine's more like a general classic stuff that I've been looking at recently. So a lot of companies are using Slack that communicate it with their structure, right? So we can use that team to work on Slack and it's pretty much a bold one. So I love recon. So you go out and you work for different Slack opens and like get hub or did like their repositories, right? And you take those and a lot of times, but they're still active. And it will just give you full access to your core Slack. So that's a lot of fun because you know, there's all kinds of stuff in there and customer data and kind of stuff. So if you can you can get into Slack, you can pretty much own the phone. Nice, that's very much true. So okay, how much time do you usually spend a week hunting? It depends so much. Sometimes it's like none. Other times it's like 50 hours. Like I feel like acting for a lot of people is like, you spend two weeks dead. It's like I don't do anything.  But probably you're not smarter than like 20 or 30 hours. Average probably up two to three or as many as many hours that I just go without sleep to supplement that with. So I would say on average, it's probably 15 to 20 hours a week. But it's kind of like watching like then watching TV, right? There are some weeks like just spend like way too much time hacking. And then other weeks I don't touch it. And it's a tentative thing to step back and take a break from that because you can really separately burn out in this industry. I'm glad that you guys kind of talk a little bit about like how you need to take time off sometimes. I always like to ask this question, even though it's never part of this topic in particular. But because we are in the hacker community itself, how do you guys deal with mental health? Well, all the time it's just like distancing yourself for a little bit like what you're saying like, you know, it's going to be like doing it crazily. Like being able to like correctly like decide when enough is enough. I'm like, let's say you're like, I don't know, not like anything. Now that having a good time, you're kind of forced yourself into a schedule. It's good to always step back and like maybe, you know, try to spend time, you know, family friends and like that. You know, it's like if I don't have that like kind of like, you know, I'm really excited to do this. It's simply hard to find those. So like the more I spend time doing that, it's just like the more I'm stressful, the more like feeling kind of exhausting it is. But just knowing when to step back, I guess. Yeah, so I think taking like disconnecting is very super good. So when I get back in the day and I'm completely disconnecting, at least, I mean, in my detox and then probably where that happened again. But definitely step in the way is good because if not, you will get burned out really, really quick. And then like through the day to unpacking on something and I'm just like, you know, I'll step back from that and I'll go like, move along. And I will solve so many problems, one out on the lawn and come back and just be able to hear it. And so it just stepping back sometimes is really, really good for me. I see a lot of people on Twitter said like goals for themselves, like I'd better find 100 bucks in 10 minutes or whatever they choose to say. And then they update everybody every whenever they do. I don't generally set any any kind of goals for that. I find something that's fantastic. It's going to pay for something like I'm going to go over, but I don't really set any kind of goals like for that, especially on realistic ones that a lot of people could probably get in there pretty fast. Nice. Thanks, you guys. OK, so since we are in Recon Village, I have to ask what tools do you tend to use and why are they better or more effective than others when it comes to recon? I think we did touch a little bit on that, right? But I'm going to go right back to set off and grab in off just the things that anything bash related is is my favorite. It depends. But yeah, like I think it's really about like all those tools now. So just like a lot of times like the building tools for like specific things that come up for specific mentalities, for instance, like I feel like until a couple of years ago, like doing good luck, like process and kind of cool secrets from people wasn't as big as it was. But as that time went on, people felt these really great rules for like you can auto generate the guy who is going to pull them and run it for people for possible ways for like specific terms. So typically, like it's just like kind of going to lose for a particular program. But honestly, like one thing I think that kind of difference for me, maybe other people is just like taking a particular program and sticking to it. And that's kind of their constant is just like learning over time all the quirks and like the functionality. Like for instance, there's a group of researchers who like only hack on Facebook.  And like it's very particular and you can tell they've spent a lot of time trying to understand it. And I really understand like the ecosystem. So I think that has like a cool kind of thing. But yeah. Yeah, so the get up integration pools, S3 pocket integration pools are called like my code too. And like if I'm approaching a target, those are going to give me a good idea. Like if this target pretty hard to get over, is there a possibility is here if I'm the first up? If I start seeing them like S3, I guess they've been something bad for some of them. I think they're great for other problems because the problem number is here. But yeah, so just a new integration stuff. I love Recon. And I think that you can get a lot of stuff out of it if you really focus on it. But I am looking at myself. I like to jump across multiple programs. It probably isn't a good thing. I'm showing people on my money set on the focus. But when you have these tools, it's easy to just kind of like spray and spray on different sites. So you can see what you find. Nice. OK. For those that don't know, by car, we actually just released a basically Recon video, Recon Discovery video module. So by car, we have by car to university. It was kind of started by Jason Hattix. Who knows Jason Hattix? Cool. He's my mentor. So he basically grew up by car to university, get back to community, give him more curriculum in a sense so you can all learn. And for other people around the world can learn as well that don't have that access. But yeah, we just posted it yesterday. So there's a Recon and Discovery video, which is really helpful and very useful. I just want to kind of end with one question for you guys. What do you see the future with by body? Oh, that's interesting. I feel like by body is a great tool for like discerning people who are passionate about security, right? Like I think what's interesting about money and like the future for by body is just like this hotbed of like research activity. Like in the past couple of years, there's been so much like people who have been introduced and like this really good about money. And so I feel like by money and like as time goes on, it's kind of turning into like this like really, really cool community of people who like find a little stuff and like work with each other to find bugs. Like the future of like actual programs and stuff like it's been scaling up so crazy like like work with 100 companies and stuff. I just think it's going to get bigger and bigger and it's going to be like a great it's kind of like a really great tool for like community stuff and like people like put something to it. Yes, so I think more more companies are going to have a bunch of programs, right? It's kind of the trend, right? It's thinking about them and actually trying to be proactive. But I think on top of that, they're going to be companies out there that are going to start thinking about one kind of programs in different ways and try to get creative with their approach. So I see more and more companies maybe leaning towards using it more instead of the like reactive approach like people finding bugs and stuff that's already out there.  Have your hackers come in and test that and give you feedback and interact with them. So I think the engagement isn't out of ownership for the next few years. Yeah, it's going to explode because it has been and it's going to continue. I don't see it stopping. I mean, I'm a pen tester and I think it's a great supplement for that for sure. It's because like you have so many different people with a set of eyes looking at it instead of you got someone, one guy looking at it for maybe 30 hours with the curve for a given job, pen testing engagement. So I think it's a great supplement for any basically anyone. OK, I know I said the last question, but now I was just thinking for advice for those that are new at Bug Bunny. Besides, you know, keep pushing. Don't give up when they first start. Is there anything else that you guys want to add on to that? I gave it once and then I got back into it after a couple of months because I got bored and I sort of find stuff right. And then I would prepare to think about other things not to put yourself to the James Petty over friends, friends of the world. I'm never going to be that good. I already know it, but there's so many things out there that you can still find just based on watching different techniques that have been released to everybody. I think like sometimes really recommends yourself, but all the other researchers have taken on the books. There's still, there's still a lot of stuff out there, right? When I first started, like, I remember like looking at programs and like, oh, you know, there's so many yourself, I was like, I'm never going to find a thing. And like day after day, there's like 20 more about issues, 20 more about issues. It's like, wow, what's, what's separating me from that? Like, how do I do that? And it's just like the dedication, right? Like not giving up. Like, I think there's a certain threshold and like right after you find your first bug, it gets so much easier. It's like, it's actually a possibility. Like, you know, like you can kind of convince yourself to put a little more time into it. But like, find your first bug. Like, it takes a little bit, but once you do that, if you're really into it, like, that's, I think that's going to happen. And also just to give you guys something that we're talking about a little bit earlier, finding your friends to do it, you know, like if you're in the university, like in your hacking club, like I guarantee you there's a couple of people who are like, yeah, like I want to spend time in the hack websites. Yeah, it's like cool. Yeah, so I would say like, if you start out hacking, right, you can be a little donkey and you can be like, what in the world am I doing after the first ball here? Just step back from that and think about what you're trying to attack, right? So let's reach your website, right? Go out and learn how to build that. Go out to AWS and deploy on a server because you're going to see these like things that the company's going to miss, right? The security misconfiguration is like the security, so you can go back and check that in your target. So go learn to build this stuff as well to be able to break it successfully. All right. I just want to say thank you guys for taking the time and being on this panel. And I want to say thank you to Recon Village for existing and also all of you guys, thank you for existing.  So I've been hearing too many of terrible stories this week on mental health. Just know that you're not alone. We're in this all together as a hacker community. So thank you once again. And if you have any questions for them, we'll probably be right outside that door. And BugCrat has a suite that's open in this hotel. The suite number is 3151. Once again, it's 3151 and there's free drinks and a lot of cool news swag and you can get your stickers there. Thank you so much.