Hey everybody, welcome to the Linuxcast. I'm your host Matthew Everett. I'm joined by Martin Burke. How are you doing, Martin? Well, I'm doing well, Matt. How's yourself? I'm out of practice. I don't remember how to do a proper cast. It took me a while to figure it out. Uh, it's been a while. Uh... Oh no, yeah. That's rummaging around for the microphone to be fair. I couldn't find it. Well, it's just been a while since we've done a show. It's alright. Stuff happens. So, uh, you've been doing anything cool on Linux lately? In short, no. Um, still under lockdown here. They're lighting restrictions. So, I've been trying to escape my hole and get out in the garden and do some things. I have dusted off my Pinebrook Pro, though, and in the hopes I can do a bit of work outside. Um, so, I mean, I'll probably will stick under the distro, won't it? But, currently, it's really in Manjaro. It's all updated fine. But, if it isn't broke, don't fix it. I think Manjaro, at the moment, is about the best for the Pinebrook Pro. Uh, just a quick note. I think they're releasing some for you guys, aren't they, with your keyboard? I think they've put some pre-orders.  I mean, it's the best bang for the book you're going to get, to be fair. I'll take a look at it. I wish you had more RAM. Yeah, that's what I think. I don't know whether to sell it, get a good price for it, because, I mean, it literally does sit here doing nothing off, getting out now and again, and just wait for the next iteration of it. But, no, it's not too bad, to be fair. For what we use it for, it's fine. But, yeah, I know what you mean about the RAM, but it's just going to have less tabs open, really. Well, I won't be able to do my video editing. What do I look, you know? No, no chance. Um, I think it was Jason. How about you? Hey, wasn't Jason Manjaro that did the, he tried to do, um, Kaden live on, like, the Raspberry Pi or something. That was fun. Uh, so, I did my first live stream this week on the YouTube channel, which was a disaster. Um, I mean, it went fine, but the audio was horrible. Uh, and during the time, I looked at the, at Awesome Winner Manager for the first time, and I cannot get my head around Lua, like, at all. Like, it's just completely beyond me. It was, that's where really the disaster came in, because I was just looking at the docs for Awesome, which is written in Lua, right? And I just couldn't figure it out. I mean, granted, it was my first time, so I didn't, I haven't spent any time with it beforehand. So maybe if I spend a little bit more time with it. But I think Lua's going to be one of those, um, languages that I just can't get a hold of. Kind of like Haskell, which is what Xmon has written in. I just couldn't get my head around them. So that's what I've been messing around with. I've also been scripting a little bit. Uh, but I'll talk about that a little bit at the end, because I actually wrote a script that I'm using as my app of the week this week. That's something. Oh, cool. I see a certain YouTuber was bigging you up. Yeah, yeah, this sort of too, mentioned this time too, which is crazy. Went from 400 subscribers before he did that, and now we're at 2220. So. And the main thing is that you kept your subscribers tuned in, because I'll do that. I'll check out this channel and you'll go, oh, yeah, this is okay. No, actually, I'll unsubscribe, but you kept them and grown on it. So props to you for that, Matt. Yes, it's been pretty good. I was actually, I mean, you should have seen, because I watch all of his videos, right? So I, uh, you know, I watched it on my phone.  Like, like, yeah, this is cool. Like, there's no, no way. I didn't even think I was like, well, I didn't think about, you know, well, maybe he'll mention my channel. And then what's great though is he, I was first on the list and that kind of helped because the ones that were further on the list didn't get nearly as many subs. So that was really good. To be fair, I've got, yeah, I do watch a DT, Mindy for the Fact. He talks sense and there's no drama. That's what I like. It gets to the point, even though he's heavily arch related, I do like the guy. He talks a lot of sense. Yeah, great stuff. With that in mind, let's go ahead and jump into the contact info. You can contact, you can contact us on Twitter at the Linuxcast. I'm at MTWB, Martin's Martin to you. You can subscribe to us on all of our feeds and stuff at thelinuxcast.org. That is going to get a website soon. I promise. Eventually. I'll do it eventually. I want to do a video on Hugo or Jekyll or whatever the static website generators, but I just have not gone around to it. You can also contact us via email at thelinuxcast.gmail.com. And you can support us on Patreon at patreon.com.linuxcast. I'd like to thank Devon, Zach, Marcus, and Kamp for being our patrons. You can follow us on Facebook at facebook.com.linuxcast. This stuff is all out of order. I need to redo that. And you can subscribe to us on YouTube at youtube.com.linuxcast, where you'll find Linux related content every day of the week. I've taken a couple days off now. I had a streak going. I started December 24th or 6th or something like that and did a video every single day until like March something. And then I took a day off because I was sick. So now that the streak is over, I feel a little bit less guilty about taking a day off every now and then. So every once in a while, you'll see a video. It's not a day that doesn't have a video on it.  And we do obviously a weekly podcast when we can. So every week, we pick a link of news-ish, news-ish, that we can talk about. So Martin, what was your link for this week? I'm 40.  If you sign up to beta, get that running. You've got now I'm 40, now I'm 40, whichever way that you want to say it. I don't know whether you've had the chance to have a look at this, but I'm quite impressed actually. Really? Yeah. I'm really, I'm quite impressed with it. To say that the baggage in the history of it all, no, blah, blah, blah. But it looks modern, looks up to date. Reminds me, well, obviously, they've nicked a couple of bits and pieces from KDE, but it just looks really smooth, fluid. I mean, it's going to be on the website, things like that. But I think a couple of used tubers of the work of a way through, but I think it's a generic thumbs up for it. What do you reckon on that? So I did use it. I installed it. I installed GNOME OS on boxes and made a video on GNOME 40. And I came to the conclusion that it's GNOME. So here's what I think about it. But if you like GNOME and if you've been using GNOME for ages, you'll probably come across with your opinion about it, because it will feel different to you because you've been using GNOME for so long. To me, it was just very similar to what GNOME has always been. They just moved the dock to the bottom. That's the way it felt to me. But like I said, I've always been a KDE guy. So KDE has gone through many changes in the last, I don't know, 10 years. I mean, KDE 3 was released in 2009 or something like that. And that was 2004. Or excuse me, it was KDE 4 was released then. And then 5 came out in 2015. And while they share some similarities, they've made visual refreshes. GNOME's looked the same forever. I mean, since GNOME 3 was released, this is what GNOME's looked like. And I think if you've been using GNOME for a long time, like I said, you'll find that it does feel fresh because it does feel different. Now, I will say this, if you're one of those people who uses extensions a lot and you always use dash to get your dock to the bottom anyways, you're going to probably feel like this is just the same old, same old. Now, my impression was that it was slower than it has been. Because I've been going through GNOME for the last two or three years and trying to make it faster again. It felt slower to me. You use a decent machine, haven't you? Well, yeah, but it was in a VM. And it's beta software. So I don't really want to judge it. So I'll wait until it comes out on Fedora 34. I couldn't get Fedora 34 to install in a VM. I didn't try in boxes. I tried in Ritualbox. So I'll wait until it's final software before I make a final verdict. But it just felt slower, whether or not that was because of the VM or because of whatever. I don't know. We'll see. It's food for thought, Roy. I mean, obviously, I'll have a double. Troy it out. Well, it'll be new to me, so to speak. So I mean, if it fits in, but if it's going to be a memory monster, then no. Yeah, we'll see. But I mean, from what I've saw of it, it looks, well, a lot slicker. The rounded corners and everything. Yeah, see, that's what I thought. When I just first saw the announcement before I used it, I was like, oh, this is great. This is a whole new look for it. But it just, I guess it's the way it felt. It felt the same. The real question I have is what Ubuntu is going to do. So Ubuntu is not going to do Genome 40 in the next release that comes out next month. But we should see it in the fall. So now if we want to talk about a distro that's looked the same forever, that Ubuntu has looked the same forever. Now they've changed themes recently. They went from the whatever it was before to Yaru or Yaru, whatever. I don't know what that was called. But in terms of layout of stuff, it's looked the same since, I don't know, 2004. It's basically the same. It has the icons along the side and the bar along the top. That's the way Ubuntu looks. So it'd be interesting to see how they go through and implement some of the Genome 40 things into Ubuntu. If they make a similar UI change, I mean that'd be really cool if Ubuntu decided to do something a little bit fresh.  So that'd be interesting. All right. So my link is... I debated putting this in there because really I don't want to get into the whole Richard Stallman thing.  Excuse me. I don't care about Richard Stallman at all. I have no horse in this race. I don't care if he stays or goes, but he's back at the FSF. And you got two factions within the Linux community. Some of them want to get rid of them again. Some of them are trying to keep them from getting canceled because of the cancel culture, the whole thing. So I just thought I'd put it out there and ask Martin, do you have any opinion on the Richard Stallman fiasco? None at all because I've literally had my cake day the other day. I was using Linux. I looked at the date. I think it's about the 24th or something of March. So now I don't know any of the history. Just having a quick check. Is he a more type of person? You either like him or you don't. So Richard Stallman founded the GNU project. He founded the GPL license and he founded the FSF. So he's kind of like the Linus Torvalds type guy. And he's very, very smart, but he has no social filter. So he's come out and said things about pedophilia and stuff like that that he really shouldn't have said. But those things were said like, I don't know, 15 years ago. So people know this about it. It's not a secret that he's a weirdo. I mean, and he is. He's a weirdo. What he got canned for a year and a half ago was saying stuff about that Epstein guy who had a harem and was pimping out little girls or something. I don't know. Stallman came out and said something about him, something about, he didn't really defend him. He just said something about semantics or something. And that got everybody up in arms. And he was forced to leave MIT, which is where he was a professor and he was forced to leave the FSF, but now he's back.  I've tried, like every, like, Linux YouTuber, like the Linux gamer guy made a video on this. DT did a video. Brodie Robertson did a video. I decided not to do a video, because honestly, Richard Stallman's like 70 years old or something. I don't really know how old he is, but the man's ancient. If he gets canceled or whatever, that's not going to affect Linux at all.  I don't care. For me personally, I think that he probably should retire just to bring in some fresh blood, you know? At this point, he's a legacy. What is he doing? At least with Linus Tarval, he's still in charge of the actual product. He's still pushing out the Linux kernel, like he's been popping it out for the last 30 years. He's still doing stuff. If anybody asked, I mean, you're fairly new to Linux. I've been using Linux for three years. I think if you ask the majority of people who use Linux what Richard Stallman does every day in order to support the Linux ecosystem, they have no clue. Maybe they know that he's part of the FSF or maybe they know that he was the founder of the GPL license or the GNU project, something I don't know, but they're not going to know what he does now. I don't know that he knows what he does now, other than causes a lot of brouhaha. I don't know. That's where I fall on it. I don't care. That's why I was... I thought about not even putting this link in there because it doesn't really matter, but I figured I should cover it somewhere. I don't know. Yeah. I was just flicking through having a read. Yeah, definitely. If you really care...  He's not necessarily... He's a weirdo.  He shouldn't... Yeah, just be behind the keyboard. He should be coding or something. Social media. Yeah, definitely. That's the way it is. All right, let's go ahead and jump into the main topic, which this week we're going to be talking about the things we do when we install a new distro. So, Martin, when you install a new distro after hopping, what do you do? Well, prior to that, I mean, I could say go to Distro Watch and check out, but I do go to Distro Watch, obviously, and I just check, mainly, the new releases, see what's coming through. Usually, your favorite YouTubers will be doing something. First look and all that. I check through about two or three YouTubers just to get some sort of opinion. Let them do the hard work to save me doing the install, so to speak. Then I'll try out the distro, bang it on a USB but I would stick it on a USB just to just get a feeling of it. I know there's going to be a bit of a lag on using the USB and what not. And then what I'll do is I'll obviously install it. Ideally, it's best to have a spare drive, so what I'll usually do, I'll back up all my pictures to the cloud or a USB and I'll just literally drag my home directory onto the spare drive. Make sure everything's copied over. Unplug the spare drive just in case. Obviously, go through the install. I mean, the hardest thing to do is to get your new distro set up to exactly your workflow.  What are your thoughts on where I've got to so far? Are you much the same? I guess when I've changed over the years, so when I first started using Linux, I'd do the same thing. I'd go do the research and stuff, but now I pretty much know what distros are out there and they're all things that I've tried before. Yeah, I know what you mean. The main ones. When I hopped last month and I was just leaving Arco behind because I was having a problem with Python or whatever, I pretty much knew that I wanted to go to an ArchBase distro, so I just randomly pulled up one of the ArchBase distros and just tried it, so I think for an old timer, it's a little bit different than I should say that it did try now that I've started the YouTube channel, I've tried more like not necessarily ones that I haven't tried before, ones that are new to me, and then yes, I'd go to DistroTube all the time and see, that's how I got to use MX Linux for the first time because it's at the top of DistroWatch all the time, so yeah, I've do the DistroWatch thing, yeah. Right, so once I've installed, I mean I am on Rint at the moment using this Cinnamon, I mean previously I was on KGE, but I was doing a lot more Linux stuff, so I mean I will install Redshift just to reduce eye strain, so that just goes with the phases of your light levels and just reduces the glare and the blue light, I think it is late at night, just because there's nothing worse of frying your eyeballs late at night than watching YouTube or chatting about. Otherwise, download a clean copy of Firefox. I do use a sync, so I'll log in to get all my bookmarks back up and extensions running. I do install Timeshift as well to take regular snapshots, I mean I'll do I think it's something like two quickly and I'll do two daily and just keep them running so I've got anything to fall back on. I'll install Web Apps as well, so I can stick my YouTube on there without Google spying on everything and shopping. I don't ever really log into Google from my browser of choice. I mean I've usually got two running. I've got Browse, Brave and Firefox. So they installed Web Apps and Rambox I spoke about in the previous podcasts. It's just basically containerized systems. I've got WhatsApp Outlook, Gmail Skype, Skype played up the ones, so I've got rid of that. Banking if I wanted to stick that on separately and then just as I go along I will install programs as and when I need them. I mean when I do shift to distros, I must admit like everyone, I do like a light distro. I mean I know they've got to include this or you're used to this program and you've got that and you've got to go through all the pain of that free software that you've got reinstalled in a different version, but yeah that's mainly how I do it and how I like to start my new distros. I mean what are your thoughts? Do you just install as and when you need or do you just do one massive install of all your main programs? Well, so I'm always quite surprised about how few programs I actually need to download when I hop. There's only like five or six. I need Firefox, I need LibreOffice and I need Cayden Live and Audacity. Those are really the main graphical ones that I use and then there's a whole bunch of terminal ones obviously that I have to install. But the big thing that I always download is my .file. So I always go to my github page and download the backup of my .file so I have all my window manager endings and stuff all right there. And then I'll always go through and just install the few programs I need. Like I said, I need Firefox and LibreOffice and Cayden Live and Audacity. And then I always download like Todoist and Zim and I do MuttWizard for mail and I have a Notion app and I have Discord and then the messaging apps that I try to avoid like to play. So I have WhatsApp and Telegram on my system now but I do not install those until I absolutely need them. I don't like them especially WhatsApp. WhatsApp has a horrible app on Linux. It's just not good. I know there are third party ones but they're all just kind of terrible. So other than that, that's like maybe like 7 or 8 apps that I install. Other than that, I can get to remember most of the time I install Arco. So Arco comes with a ton of apps installed. So most of the time I don't have to do Firefox. I don't have to do LibreOffice. Those are things we usually install right from the beginning out of the box. And Firefox is usually the default browser on every Linux system. So that's something that I don't have to do. Now if I ever get around to actually switching away from Firefox to something more privacy focused than it's possible that I'd have to go through and install that but I haven't actually done that despite threatening to do so over and over and over again. Yeah and I'll look into my P close account with what I was doing when I was hopping a lot and I've got a text file with all my terminal commands just to install my various software but I just do it the lazy way and use the GUI and click install and do that but previously I used to have a list of my commands and I used to run the file to install it but yeah I'm a bit lazy like that. I'm not like you with your terminal guru and that's about it really and nothing amazing I mean I would say I mean I know we did a podcast about security and firewalls if I remember to click on the firewall I do I mean I usually do bottom click just see what firewall is installed but as I said on Mint it's already installed so just another click and it's up and running and that's about it really yeah a small bit done on that. Yeah so what I've been thinking about doing is creating a bash script to just install the programs that I need for me but the thing is I'm not sure if I hop distro is often enough to care right so maybe when I first started out and I was hopping every two or three weeks it would have made sense but now I have no intention of leaving what I'm on right now I have all of my stuff here it's working fine I'm going to stick to it and that's a completely different you know outlook on Linux than what I had even a year ago like well maybe not a year maybe like a year and a half ago because about a year ago I switched to Arco for the first time and then I used it for a whole year like without hopping and then I hopped crazy and end up back on Arco so like maybe like a year and a half ago this a bash script to install my programs would have made sense now I'm just not sure that I actually need it maybe it's just something I'd you know create just for fun and then put up on GitHub that way if I ever do hop I have it I can just you know run it after I you know install I don't know what to think about yeah it's what I did as well I did my little bash script that used to throw up errors because some of the stuff was out of date after a while and I had to go in and edit it and I just thought back in the times when I was confused with all your various software stalled names and this and that and trying to find it I just did that but yeah just very lazy just use the software stalled of choice yeah that's it and I haven't hopped recently I'm still on Mint which is probably longer than I was on Mint last time to be fair but then again I haven't done a vast lot on it currently so if it ain't broke don't fix it is the way I'm looking at it currently yeah well I mean I think I said I think I'm going to stand this for as long as it possible I think that's what I think that's as you use Linux you know the longer you use Linux the more apps you are to just find what you like you know you get everything dialed in and you don't want to leave unless it breaks and even if it breaks you spend you know at least a day trying to fix it before you say because hopping becomes a chore I mean even if you own an only install if you're like us you only install five or six apps and you have a script or whatever yeah you still don't want to do it whereas hopping used to when you first use Linux hopping seems like it's so much fun because it's all new and it's going to be exciting and you're going to learn a whole bunch of stuff and yeah it's going to get some work done yeah but eventually it's going to I mean even if you just nuke and pave it and reinstall it you know the same old same old as well yeah it's kind of a pain alright let's go ahead and jump into apps of the week Martin what was your app I've installed this have you installed this sorry yeah just now I'm looking at now oh excellent right so it's a bit of a cheat it's not an app extension of the week it's called dark reader now this is a dark theme for every website you visit now you might go into random sites I mean I think distro watch it's like an acid trip whatever one of them is like it goes a bit funky so I mean you can turn it on or off and things like that but there's it's just spot on Amazon things like that just to give you the the dark dark web page instead of your bright one so I know are you more of a dark mode do you switch everything to dark mode or do you just leave you as is as much as possible yeah I use dark mode mostly because it helps with my phone yeah oh yeah everything on the phone as much as possible yeah white just sears your eyes and makes you want to die yeah but I mean there's some more and more websites have got it you've got to click on it but then it's still storing it in your cookies once you clear them you log back in and you've got this bright white screen but yeah it's really good I've been using it for the last three weeks and you just forget about the funky the odd couple of sites you just carry on it's just not to actually just see and read without blinding things on the other side especially if you've got your ad blocker so all the the ads are gone I'm not sure if it's got a slight ad blocker in it I don't think so but yeah that's my app well not app I'm quite pleased with that how about yourself what terminal related app have you got for us it is terminal related I know you're taking it out of you but yeah make fun all you want this is actually a little bit of self promotion because I wrote this script I'm not a good bash I'm sure if someone who is actually a really good bash came along and looked at this script they'd be horrified by how amateurish it is and I'll freely admit that but it does work and I've tested it in Ubuntu and on Arch so this works there are instructions on the github that I've linked in the show notes basically what this will do is it will go through and it will download my setup on DWM with all the dependencies that you'll need run the script and you'll be able to use my version of DWM out of the box and it's really good the reason why I made this was because I've had a couple people ask on the youtube channel how they'd go about getting my DWM setup so I was like here's a script it's so easy I could do it so I don't really know why you need a script but it's been created I've made a video about it I'm going to upload that today by the time this goes up you can actually go back to the channel and actually watch me run the script on Ubuntu and on Arch so like I said it works on both and any related distros obviously and it's really good now obviously just because you get DWM installed from the script doesn't mean you're going to know how to use it so that's something that I'm not going to help with you're going to actually have to find a tutorial or something after you get installed but for the most part it seems to be something fairly easy that most people can pick up so yeah just one question what's nerd fonts is this something I'm missing sounds really good do you know what awesome fonts are um so basically nerd fonts are just awesome fonts are the fonts that have little icons they're not really old yeah kind of like that but these are a special brand they're usually meant for Linux in the terminal in order to show icons like file icons and stuff like that nerd fonts are basically like that it's a huge collection of fonts that um a lot of them are mono space fonts but some of them are not it's like 8 gigabytes worth of fonts it's huge that's the reason why in the script I only downloaded the ones I needed because I didn't want to go through and say I didn't want to go through and get clone the entire repository so it's just the four that you need and you have to install those on your own yeah that's what nerd fonts are it's just the little icons in the bar yeah it's with the amount of gig yeah that's cool best recommendation for nerd fonts is just browse through the ones that they have there and download them specifically instead of downloading the whole thing now I download the whole thing speaking of every time I hop I always just download nerd fonts every single time it usually takes longer than installing the distro I'm just going to put that out there it definitely does so that's the thing it happens alright anything else to say today Martin or are we done no not from our side I did not want to put a call out oh right so Martin's leaving us he has to go back to work I don't know why he needs to do this work thing I mean obviously the YouTube is where it's at anyways he's leaving us so we are looking for or I suppose I am looking for a new co-host so if you are interested in doing a weekly podcast I'm fairly flexible on times I mean Martin we've taken many many weeks off here and there right I mean you started in like I think you started in October October was it yeah so we made it a good portion of the year but anyways I'm very flexible in time it takes about an hour a week if you have a microphone and you're interested in Linux give me a contact at thelinuxcast.com or contact me on Twitter that's honestly when Martin started I hadn't really started the YouTube channel yet so he had to contact me on Twitter to check my damn email in order to see the email he sent me which is hilarious I still have that email and I still have the tweet so anyways yeah if you're interested give me a contact thelinuxcast.com and Martin you'll be sorely missed I know and anyone that's interested you can do better than me believe me but yeah drop my toline and talk Linux for the better they can do better than me too I mean no you don't want anyone better than you you want someone just below you but yeah it's just there's only so many Linux videos I can watch not that you can tell but just recently it's just slacked off and just crawling out my hole into the brave new world now so yeah gonna hit you on the head for a time being I will obviously keep in touch see how it all goes but yeah by all means jump on board and either learn Linux or teach Matto to use the GUI we definitely need to make sure we have a mint fan it's got to be a mint fan because otherwise we can't have two people here that don't like you can't have two archfront oh we could it would be so much better hey come on man definitely your biggest flaw is the mint thing I mean come on alright I think that's it so thanks for listening thanks for watching if you're watching on the youtube channel make sure you subscribe like all that kind of stuff thanks for listening and we'll see you next time guys cheers take care guys


Hi everyone, I'm Nikhil Mithil and I'm presenting on hacking with human interface devices. Welcome to the hardware hacking village. A couple of things about me. I'm a hacker. Still not audible? I'll be louder. I'm a hacker. I'm a trainer. I speak at conferences. I'm at the two open source toolkits, Kautilya and Nishan. So Kautilya is the toolkit which we are going to have a look at today. It could be used to generate ready to use sketches for Arduino like devices. And those sketches could be used for penetration testing or breaking into machines if you are optimized to do so. This is my Twitter handle and that is my blog. So follow me. I'm a sucker for followers. I'm generally interested in offensive information security and new ways to get into systems. And I've spoken at a couple of conferences previously as well. So what we are going to have a look at today is why should we use human interface devices? What is a human interface device? And obviously a lot about my tool. We will also see some live demonstrations and hope they work. I've just tested everything in the morning. So if something doesn't work, assume and trust me. And at last we will talk about a couple of defenses against these attacks. So I missed a slide. What is a human interface device? So those of you who don't know it, it is something. So we as humans interface with computers using different things like mouse, keyboard, touch, joystick. A human interface device is something which allows us to do that. For example, a keyboard is a HID, a mouse is a HID and so on. So why should we use these things in penetration tests? First and foremost, they are really powerful as you will see in a few minutes. Counter measures like antivirus, Hips, security teams, they trust these devices. For example, when you plug in a programmable keyboard, it's obviously detected as a keyboard. And I rarely seen any corporate environment of any of my clients who block keyboards to be attached to the machines. A lot of people block mass storage devices but no one really cares about keyboards. And the last thing is, it's really cheap. So the device I'm going to use is a TeenC3.0 from pdrc.com. It's just like $20, $22 device. So you can literally use dozens of them in your penetration test gates. There are so many HIDs or human device which hackers can use. I've listed a couple of them which are really cheap and small in size. For example, TeenC which I'm going to use today. Open to Boss, Arduino Micro and a lot more devices. In fact, I've not listed here of them. There are so many devices. Just keep in mind that if you really want to use them in penetration tests, the physical size should be small. For example, I've seen using Arduino Duo as a human device but it's like this big. At least this big, like the size of a palm. So it will not be easily hidden inside other devices. So about my toolkit, Kautilya, it is named after some Indian guy who was a political thinker back there. So I do not think about it. It is written in Ruby and it's available on my GitHub. It contains like 54 payloads right now, 45 for Windows, 5 for Linux and 4 for OS X. So your mileage may vary on OS X because I just ported it on VMware. I own a PC, I work on Windows technologies but Windows payloads should be quite stable. So Kautilya currently supports TeenC++ 2.0 and TeenC 3.0. So as you will see, we can just input some things to this tool and it has an old school GUI menu driven thing. I am scared of this thing. I am still audible. Because Arduino supports only TeenC devices, there has been a port of Kautilya for Arduino Micro by some awesome guy. So you can check that one as well. And that brings us to the pentas scenarios. So we will be having a look at these things. So these are the things which I have tested today. There are a lot more things which you can do with Kautilya. If there is any user of Kautilya, you can ask for a demonstration and I will do that right now. And the target I am going to use is a Windows 7. It is not pirated. It is a demo version but it shows that it is not activated. So do not think that it is a pirated one. I just created that 30-40 days back and it has expired couple of days back. So it is not a pirated thing. Do not complain to Microsoft please. This is going to be our target. What we are going to do is we program our HID and connect it to this target. And let's see what happens. So this is our device. As you can see, the target must be unlocked. Only in that case you can use this. I mean it works even if the target is locked but you won't be able to type anything. So the target must be unlocked. And this is not a pirated version. So this would be our target. This is how Kautilya looks. It looks a bit ugly here. It is designed and developed on Ubuntu. I am using it on Windows right now. If you have Ruby installed on Windows, you can use this on Windows as well. So our first fantastic scenario is getting access to a target environment. And we would use a couple of reverse shells. That is the shells which connect back to us. And the target which is this machine. It is a fairly restricted environment. On both TCP only 80 and 443 are allowed. On UDP only 53 and only ICMPV4. So that's quite a restricted environment. So Kautilya has a really old school GUI. You have to enter your options like this. Is it visible to the back? You save one here. Then the shells are in the execute category. Press 2 and we have 11 options for shells. Let's use a reverse TCP which is the seventh one. It shows a small help and the options you can enter. So right now it is asking for the IP address to which the reverse shell will connect back. So let's use 443 because it is generally allowed through the firewalls. Now it asks for the type of board you want to use. So let's go to IC3. It has created a sketch which could be used in the Arduino IDE. And the sketch is in the output directory of Kautilya. So let's just open that. So this is a sketch. We'll just either open it in IDE, Arduino IDE or just copy it and paste it here. Now connect the device. First I will upload the sketch to it. And because there is a sketch already loaded on it, first I have to stop the device after inserting it. So I will just disable the auto reboot and stop the device. It started working. So now the device is now ready. Now we can upload our sketch to this device. Just press upload. Now stop, stop, stop. I believe the sketch has been uploaded. Let's just try it out. So I entered the IP address of my host machine and put 443. So let's start and listen to it. Okay. So I'm going to use PowerKit which is a NATKit port to PowerShell. Or you can use your NATKit on Linux as well. So it's our choice. In fact let's use NATKit. So for that I need to just change the IP address. It's gone. So this is IP of this Kali Linux running over here. Let's start. Let's start a listener output for 443. And now if I connect this device to the VM, that is our target. Let's see if it works. So what it will do first, if it is for the first time it is being used on a machine. It waits for the drivers. It minimizes everything and opens up a really small command prompt. Types in dark blue on black so that it is less visible and does the stuff. Let me just disconnect and reconnect it. Okay. Let's see if it gets detected. So the idea behind this thing is you can program it as a keyboard and it can type really fast and without errors. So in internal gigs you can assume it as you have access to a machine and you want to really fastly add a user and do something like that. Then this device is for you. Okay. Let me just re-burn it to the device because it was the different IP address previously. I use it in external pentas gigs is I take a couple of dozen of these things hide them inside either thumb drives or USB toys and things like that. And then I drop it in my client's parking lots, cafeteria, etc. or near the office and the success rate has been quite high like 30 to 40%. So that's really really good because once you have access to even a single machine you are in. So let's see if right now we have something.  Okay. Let me start the power cat listener. Okay. So it drops partial scripts in the temp directory of the user because this is not working right now. Let me check that script if it is something wrong with the script. I'm trying to make it file less but right now this is what it does. So it dropped these partial script and VB script in the temporary directory of the user. And I know what's wrong with this. Nice. It connects to my, okay. It's there. So it took some time. I'm sorry. So we have this interactive PowerShell connect back from the target machine. It took some time. So now we can run the native commands or PowerShell command lets as well whatever you want. And once you have a PowerShell session, you can really like in couple of seconds upgrade it to a metapeter or in fact only using PowerShell you can do really cool stuff. And I do a training for that so please join that. So this is a reverse TCP proxy of your shell. You can use it over internet in your external gigs or whatever. Another thing I would like to show you is a reverse ICMP shell. Please note that that reverse ICMP shell is not tunneling anything. So if someone is looking at the network, I'm sorry to say he will see everything, your commands, your responses. I'm working on encoding things. Right now the problem is if you encode at least base 64 encode things, they would be so huge that you will generate malformed ICMP packets. So that is not what we want. So is this good? I'm going to close this shell. Okay. So if you want to have a look at it, just in case you don't feel like that. It's off. I'm sorry. I'll just turn this on. And if you see we have only couple of outbound rules. Let's enable outbound ICMP as well. So for an ICMP reverse shell, we'll again go to payload for windows. Execute. And now let's select the reverse ICMP shell option. Or if you want, I can show reverse HTTPS shell as well. So the reverse HTTPS shell is a certificate paint completely encrypted shell. So really hard to detect. See both of them. Okay. So nine. Notice that in this case, you must have the listener on a Linux machine because you cannot turn off ping replies on windows. So this is my Kali Linux, couple of other options. And we have a sketch. Let's just copy this sketch to the Arduino IDE and let's just upload it to the device. Now we must start a listener. And if you notice the help here, it shows the command you need to execute on your Linux machine to turn off ping replies. Let me just copy this. We have stopped ping replies by the OS. It's our server which will reply to the pings. And we need, as a server, we need the server from ICMP as such tool suite. So let's start that. So now our server is listening for ICMP probes. Now if I reboot the device that is connected to the victim, it again will open up a command prompt and type some stuff. Because in case of windows, most of the payloads are in PowerShell in conjunction with VBScript. And as I will show you in couple of minutes to get away from the USC prompt, it uses some need break. When it needs to open an elevated shell, for example, to dump user passwords in plain, it uses the... Okay, so we have our reverse PowerShell shell here. For example, let's run GUMI again. So this is completely ICMP. Let's just exit this one. So that was our first pentest use case. We must shell or shell or shell from windows machines. Let's use this second one. For example, in a social engineering campaign, you may be asked to gather user credentials in plain. So let's try that. So for that, you need to go to the gathered category, which is one. And then the ninth one, which is dump passwords in plain. So what this script does is it needs to pull Mimicats ported into PowerShell, which is a script called invokeMimicats. It needs to pull it from the internet. That's a bad part. It executes it in memory on the target, dumps passwords and paste it back or exfiltrate it to whatever you want. So let's use our local web server. So this is the URL of our script. We need not pass any argument. Let's exfiltrate to our local web server. So that would be option number three. Post request to your own website or web server. And that is my Kali Linux. So we have a sketch. Let's open that sketch. Place it here and upload it to the device. And now if I connect this device or reboot it here. Now the mode of operation would be a bit different. It will open up an elevated shell because that script, obviously like any other method, needs administrative access to do that. But I don't think this user has administrative access. Let's see. I don't think this user, okay, I don't know. Okay, it does. Now it presses left arrow and y to say yes to this UAC prompt. And that thing right now is an elevated shell. So it types the PowerShell code for downloading a script which we specified and executing it. The time it takes prior to closing this is because of its weights if the driver needs to be installed. Also it has a reliability check that if a user clicks in between where it is typing, it restarts the process. Okay, so as you can see it was downloaded from my local web server and should be executed. Let's check. Okay, it is, it should be this. So this is obviously encoded output. Let's just copy this. So Cortelia in the extras directory has scripts to decode things. It has written the decoded data to decoded.txt. And now if I open it, you can see that we have password in plain for that user. So it's quite powerful, but you must have internet access to pull the script right now. What I am working on is to mount an SD card over this device which pulls the script from it so that we need not depend on access to an external web server in case of larger scripts. We used, we downloaded the script here because it is so huge that if, I don't think the device would be able to hold that. And even if it holds that, it would take so much time to type that out which is non-practical. So that was just another case. Okay, the third one is obviously I cannot show it at Defcon because both of these are my favorite ones. Not really practical, but really cool. For example, this one, the first one, which means script, which means secret in my language. That reads SSID names. So the idea behind that thing is it looks for a specifically crafted SSID name and the command which it needs to execute the target needs to execute is within that SSID name that it never connects to that network. So there are slight chances that of WIPS or any Wi-Fi countermeasure may detect or may not detect it. So I'll show you how to configure it, but obviously I cannot run it here. So these are the banglers available. So this is how it works. You need to specify a magic string, a four character magic string. If I go by the example in the payload itself, nothing. Okay, so what happens now if I connect it to a target is it will look for an SSID whose name, whose first four characters are open, and if the fifth character is C for command, then the rest of the SSID name up to 32 characters is considered a command. For example, if I run this on my machine and you start a hotspot called open C, who am I? It will execute who am I on the target. And to execute scripts, unfortunately, because SSID names are limited to 32 characters, you have to specify open view and the shortened part of Google URL shortener. So it takes that URL shortener, expands it, and pulls down the script from there. So I like it because it looks cool but not really practical unless you are sitting with a targeted antenna in a cafe across the road of your client, which is always possible. So I cannot show the next one as well, the rogue AP. So I can explain what it does. A Windows 7 and onwards, there's a feature of hosted wireless networks. So what this rogue AP does is this. You can provide SSID which would be created, which we used on the target machine. And that, and then there is a meta-printer bind started on the machine. So it is like you have SSID with proper key on the target and you can connect to it. And as soon as you connect to it, there would be a bind meta-printer. This payload adds firewall exceptions as well. So there would be a bind meta-printer listening for it. You can just connect to it so that you have a private network on a single machine with a meta-printer session. So again, because my VM does not have a Wi-Fi card, unfortunately, I'm not carrying an external one. So I have to create it on my machine, so I'm not doing that. So just close this. What else? We can drop infected files as well. Again, in case of Windows, PowerShell allows a lot of things to do. So we can drop a Word, Excel, Compile, Help file, LNK, the shortcut files on the target which are infected. Word and Excel files use auto-executable macros. So if there is something which detects auto-executable macros, it will catch that. But I've not seen any AV flagging it yet. So we can use it. Another thing is, as far as I know and I tried, it is not programmatically possible to password protect your macros. So someone can open it and have a look at your macro. So let's just try this one out. Okay, I forgot the HTTPS. So I'll demonstrate the HTTPS shell first. I just missed that. So this is our listener. So this needs to be executed on the attackers machine. And this is Windows specific right now. So we have our listener running on our attacker machine. Let's choose time with this reverse HTTPS shell and the IP address of our machine and the port. So let's paste it here, upload it to the device. Now if I connect it to the target, let's see if it works. So this is one of the smallest payloads in particular. Just two lines needs to be typed on the target so it's really fast. Doesn't need administrative privileges or anything like that. Okay, now it didn't work. It's perfect, it should work. There could be some issues with the payload. Let's not waste time on it. Keep one thing in mind that if you're using, if you're trying to use this with Arduino, you must install TNC-DUNO and things. So that is written in detail on the website of these vendors. Make sure that you use the current board which is in this case TNC-3.0 and the USB type. So these two things should be kept in mind. Okay, that could be firewall here. Just let me see if I have an output 4.3 or not. Okay, it's allowed. So it should work. I don't know why it didn't work. Okay, again sorry, it took time. So this is a completely encrypted certificate pinned session. It should work. Okay, so this was our reverse HTTPS shell. Now on to dropping infected files. So the idea behind dropping infected file is just to have another way of breaking into a system. For example, if you can have a reverse shell, there is no need to basically drop files on a target. But let's see it anyways. So these are the options which we have with us. Let's go for the MS word. So it asks for a PowerShell one-liner or native command whatever you want. So let's use a reverse shell. One-liner reverse shell, let's see if that works. So you can always copy it from here. Let me copy one-liner reverse shell that connects back to again 443. So if you want, you can change the name of the file as well. And if you see the sketch, you will see that it is nothing but just a simple PowerShell script which drops the file on the target. So let's just try this thing out. I need to execute this on the local machine because there is no MS Office on the target. So let me just reboot it here. So it will drop that file on the user's desktop. Hopefully. So it is a good thing to have if you just want a different way to stay in a client's machine or in a target machine. So the word file, this one, is the infected one. And if I start, let me start a new one. I have to start a listener before I open that file. Okay, now let me just close this one. So it connected back actually. So this is how our infected file worked and hopefully we will have a reverse shell here. The flip side with this thing is this, anyone can go to the macros and see what we are up to. So if you are really using it in a social engineering campaign, then you have to password protected. Otherwise, anyone would be able to see, for example, if you are downloading a script, anyone looking at the macro would be able to see your IP address. So that's a flip side. Maybe it is taking time. Let's come back to it in a couple of minutes. Okay, a reverse meterpreter. So there are two ways of getting a reverse meterpreter using this thing. The first one is, which we saw a couple of minutes back, get a reverse power shell and use it to upgrade it to a meterpreter. Or you can use this one. Sixth, it is execute shellcode. So you need just create a shellcode using MSF Venom, paste it to... Okay, not this one, sorry. Oh, not this one, not this one. It is... Okay, so it is the first one. You can create some MSF Venom supports creating meterpreter in PowerShell. You can just create that and execute it on the target and you will be fine. I am not going to demonstrate that. We have already seen so many reverse shells. So defenses against these things. You can always control installation of devices using group policies. But I have seen in real corporate environments and in my clients, what they do, they match the block mass storage devices and nothing else. So if someone is really smart enough and blocks installation of any hardware device or hardware device by class, then this thing could be blocked. If someone is using, for example, so many antivirus vendors, block devices by VID, vendor ID and product ID. You can always change that by going here. And I do know hardware teensy course and this one, not this one. I believe it is this one. So it is there on my blog. I am not going to recall it right now. Here it is. So you can always change these vendor ID and product ID. So if there is, you know that there is an antivirus or any endpoint protection which blocks a particular VID and PID. You can always mimic HP keyboard or Microsoft keyboard or whatever you want. So you can just change it here. So there is a list of vendor IDs and PIDs on usb.org and so many websites. So one difference is on Windows machines is to use ProPolicy or Linux is to use UDAV. But the best thing is just put glue on all the USB ports but one and lock that USB device to the machine. So that is the best difference. If you have USB ports open, there are so many devices. As we saw a couple of minutes back, you can do really nasty things with them. And the best thing is to make your employees aware of the risks of attaching unclustered or new USB devices to their work machines. I mean I've seen like in one of my pentast gates, this device didn't work. So it was, I used it that way that I used a two port USB hub and connected the toy and the device together inside the toy. And there was a single cable coming out of it. So it didn't work on one of the guy's machine. So they connected in the colleagues machine and I got like three shells from a single device. So employee awareness is the best defense. And couple of closing remarks. Bad guys are already using it. I have myself seen this with couple of clients getting hacked by this thing. Even good guys are using it. It's not really something groundbreaking. As long as operating systems and system administrators and counter measures keep trusting things, these things will work because security ends with trust. Okay, so some shameless self promotion. I do a training for this thing. So I have a look at it on my blog and these are my contact details. You can find quarterly on my GitHub. There is another tool open source tool I mentioned. You can find it there as well. So that was all. Any questions? No questions? Thank you.


And it's my pleasure to introduce Johannes Koffler, who is going to talk about deep learning for quantum physics and astronomy. Please go ahead. Thank you very much for the introduction. Good morning, everyone. Welcome. It's a great pleasure for me to tell you a little bit about the research I've done in the last couple of years at the Institute of Machine Learning in Linz. Now I'm, after a parental leave, here back at Quantum Information in the Institute for Integrated Circuits. So the task of today in the first 10 minutes or so will be to give you an overview of some fundamental neural network architectures.  Residual neural networks are maybe not so well known outside the field of machine learning. And then I will talk a little bit of recurrent neural nets, in particular the long-short-term memory, and then two slides on graph neural networks. And it's actually the residual networks, the long-short-term memory, and the graph neural networks, which I will then later use in showing you three applications. First is modeling multi-particle high-dimensional quantum experiments. Second is predicting outcomes of pairwise planetary collisions. And the third is to learn the ground state properties of quantum Hamiltonians. Okay, so without much further ado, you all probably are aware of this picture in one way or another. So about 10 years ago, the deep learning revolution has started because we had fast enough computers and large enough data sets. Actually, deep learning is a subfield of machine learning, which dates back to the 80s. And machine learning is one pillar and one subfield of the big field of artificial intelligence, which dates back to the perceptron and the ELISA back in the 50s. Feed-forward neural networks are, so to say, the vanilla standard setup. You have an input layer, then some hidden layers, and then output. It's known since Kurt Warnick's work in the beginning of the 1990s that feed-forward neural networks are universal functional approximators. So even one hidden layer that is arbitrarily wide is enough that such a feed-forward neural network or a multi-layer perceptron, one hidden layer already makes it a multi-layer, is enough to approximate any function you wish, in principle. This is just an existence proof. It, of course, is an art and not easy at all to do the learning algorithm and to have this universal approximator in the end for your data. The learning of a neural networks usually works via backpropagation. This is an algorithm dating back to the 80s. Joffrey Hinton, I think, and others invented it. So the weights are updated via gradient descent with an error signal. Very early, and this led to somewhat dying out of deep learning in the 1990s, is the so-called vanishing gradient problem. If you have a deep neural net and you have an error in the output layer, how does the error signal propagate back to the input layer? It usually gets weaker and weaker, and you have to fight against this vanishing gradient problem. And there are many solutions. Many of them developed only in the last 10 years or so. There's batch normalization, good choices of activation functions, but also LSTMs, where they are back in the 90s and residual nets. Convolutional neural nets are your networks of choice if you deal with picture data. So even a low resolution picture with 250 by 250 pixels with three-color general is an extremely high input dimension, almost 200,000 input nodes. But the good thing about pictures is that pixels that are close to each other are correlated. And some basic patches like edges and corners appear again and again. It doesn't matter whether you train on cats and dogs and horses or on houses and airplanes. Even in our brain, the very first things that happen in our brain after the visual signal is processed is to get corners and contrast changes. And these would be the low-level kernels that you train. And the miracle or the magic of convolutional neural nets is that you apply small weight matrices, also called kernels, also called filters, and they are usually just three by three at most, five by five pixels. And you slide them over the whole picture to get the next feature map of the next layer. So you reuse only a very small number of weights. In this picture, it's only nine weights. You reuse them again and again. This is in very stark contrast to a fully connected network where every pixel from the source image would be connected to, so to say, every pixel of the first feature map. And that would be a quadratic number of pixels. So this weight sharing is the magic of CNNs. And since about 2012 or so, all vision challenges are won by convolutional nets. So this has completely changed the decade-old game of vision tasks and everything today, facial recognition segmentation is done by convolutional nets. Another architecture that is very powerful are so-called residual networks, or rest nets. And the trick here is that you have so-called skip connections, or highway connections, or shortcuts, and they jump over one or multiple layers. So you not only now have layer connections from layer to layer, but you have these highway connections that jump over multiple other layers and make, so to say, a fast, possible error signal to the lower layer. And in the initial training phase, there's a fast adjustment just of these few layers that have these skip connections, and only later there's an adjustment of these skipped layers. And this leads to much stabler gradients in learning and also then to faster and better learning. And these residual networks have allowed and opened up the path to have deep learning architectures of thousands or more layers, which is really impressive. And especially this combination of residual nets and convolutional nets is extremely powerful. Then another very important architecture are so-called recurrent neural nets, RNNs. So on the top right here, we have the picture of a feed-forward net. We have X1, X2 as input. Then we have a hidden layer here with activations A1, A2, A3 and then output. And in a recurrent neural net, in every time step, you would allow a feedback from every node of the hidden layer to itself and every other node in the hidden layer. So the nodes are connected now to each other in the hidden layers, and this allows for some kind of internal state or kind of memory. And also you can now process arbitrary or not predefined length of input. So I can input one input vector, in that case only X1, X2, so two words for instance. And in the next time step, I input another X1, X2 entry and another one. And with every tick, the recurrent net sees the new input but also remembers the past inputs with this internal connection in its hidden layer. And this is the most promising architecture in general for this classification, sequence generation and meta learning and so on. This is still a standard architecture for translation. You would enter word by word here in red for instance, then the recurrent neural net would say tick, tick, tick and then it would output the words in the other language. Or it would many to one, you input some words and it would output you only which type of language, English, Italian, French. So there are many different ways these networks and they are usually trained via back propagation in time. So very similar to the standard back propagation algorithm you just have to be a bit more careful to really tick it back in time until the first input. And one particular variation of the recurrent neural network is the so-called long short-term memory LSTM which was invented beginning of the 90s and it is a very powerful architecture because it can store information indefinitely by an integrator but it can be selective about what to store via gating. And this is actually one of the holy grails of RNNs that you have the selectivity and the integrator is just a cell state being added up to its own past so it stores information over time by just adding up and there are no vanishing gradients. You have this stability here in the LSTM learning because of this integrator property and then you have gates, an input gate, an output gate, also a forget gate and they serve as an attention mechanism. There is a feedback because of the recurrent hidden state and the output gate can influence the values for the next hidden states and all these gates, the input gate, the output gate and also the forget gate, they are full neural networks themselves that you have to learn. So the LSTM in some sense is a recurrent neural net where other networks learn what to look at, what to forget, what to memorize and how to weigh the importance in the future. And until the transformers came up a couple of years ago, LSTMs were the state of the art in speech text generation and recognition time series prediction. So all your voice control in the mobile phones, the translation that happened between in all the 2010 to 2020, so to say, this was all based your Alexa series, whatever you had, this was all based on the LSTM architecture. Now I want to say a couple of words on graph neural networks. They are particularly interesting, I think, for physicists maybe because they have a very good inductive bias and are very well suited if your process data can be represented on or as graphs. So, for instance, you could want to learn the toxicity of molecules and then your molecule would have an adjacency matrix. You could put it mathematically in a graph with the adjacency matrix telling who is correlated to whom or who is connected to whom. And then there are three possible things you can learn with such graph neural networks. One would be a global graph property that could be, for instance, the toxicity of the molecule. It would just be toxic or not toxic. Or it could be the phase of a many-body system. You enter a many-body quantum Hamiltonian here and you just want to learn whether this is solid, fluid or whatever. So that would be a global property of the whole thing, like a phase of toxicity. Another way to use graph neural networks is in node-level tasks. So every graph consists of nodes and edges. And in node-level tasks, you want to predict the nodes. So, for instance, the spin values in the spin lattice. And then in the node, the spin value would be represented or embedded. And the last way to use them are edge-level tasks. And then you would want to predict the edges. And that could be the links in the knowledge graph or the coupling in a Hamiltonian lattice, for instance. So, in general, graph neural networks are optimisable transformations on all these attributes, the globals, the nodes and the edges, preserving the graph symmetry. So, in every layer, you have this graph symmetry and typically you do not change the connectivity in the layers. You could do that, but that's typically not done. And now I want to tell you one more slide about how these graph neural networks work. This is actually a new architecture. It's not older than 10 years. The main idea here is pair-wise message passing between the neighboring nodes or edges. So, let's look at this simple graph here in this simple graph, we just have five nodes, X0, 1, 2, 3, 4. And everybody is connected to X0, but no other connections. So, that would be our simple graph here. So, we have a node set and an edge set. And how now do we update? How does the learning algorithm work? We would look for every node at its neighborhood. So, let's look at node X0. Who is connected to 0? Well, 1, 2, 3, 4. So, we have to look at this whole neighborhood of all four others. And then, we would take for... To compute the next internal hidden state or node representation of X0, we do what? We take the representation at the given time step. Then we take the neighbor and the edge to its neighbor and put it in a message function. So, there's a message, so to say, from X2 to X0 that takes its values and how strong the edge is. So, we have a message from X3, from X4 and X1. These messages are aggregated up with this operator here. And then, together with the node value itself, then there is another function, the update function, that then computes the next node representation at you. And now, again, this update function phi and message function psi are neural nets that need to be trained. And the advantages of such graph neural networks are that many problems in physics, chemistry, territorial optimization, like traveling salesmen and so on, have a natural graph structure. So, gene ends are very often a natural approach for such problems. And they allow to adaptively learn the importance of each neighbor. So, there's a very close connection, maybe to the real-world problem at hand. The disadvantages compared to other approaches are that they are computationally quite expensive. Typically, they can only be implemented in a shallow way, like three, four, five layers, so that limits the performance on large datasets somehow. And gene ends are fundamentally limited due to some graph theoretical theorems. In particular, they can't be more expressive than the so-called Weisfeller-Liemann graph. Isomorphism test, yeah? But there is a way out. People have and are developing so-called topological nets that allow higher expressivity. But, again, no free lunch, some advantages, some disadvantages. Yeah, with this, I am done with the introduction to this talk. And now we will come, ah, sorry. I forgot, of course, the last one I just want to mention. Now, everybody knows this now. Transformers, I think they have been put forward in 2017 or so, but they have completely revolutionized the field in the last year or two. And now it's almost every day in the news about something about large language models, JetGPT, and so on. So this is yet another architecture that is closely related to recurrent neural nets with the difference that you don't have a sequential input one by one, but you have a massive input all at once. And the self-attention mechanism is somehow dealing with this massive dataset at one. It can learn how to weigh the input according to already learned measures of relevance. Yeah, and to close this introduction, there are many more architectures that I have not shown today, of course, and you can pick your favorite network from a big list of options. There are many other architectures, but the one I have just shown in the last 15 minutes certainly make up the bulk of the workforce, so to say. Okay, that brings me to the first project I want to talk about. It's called Quantum Optical Experiments, modeled by a long-shocked memory by LSTM. And here we were motivated by the question, can neural networks design quantum optical experiments? This was our main motivation. So the physical scenario we were dealing with was we are interested in three photon states with orbital angular momentum. So actually experimentally, it's a four photon state, but the orbital angular momentum is used as a trigger to know that the other three photons are there. And orbital angular momentum, as probably most of you know, is a degree of freedom of a photon that is measured in multiples of h-bar, and in principle it can take any integer value. So it's not a Q-bit like polarization, but the orbital angular momentum is a Q-dit, where d is in principle arbitrary large and experimentally, d equal 300 or more has been reached. So this is to find interesting high-dimensional entanglement. So every experimental setup consists of a sequence of beam splitters, wave plates, holograms, and so on. And given a setup, if I give you a sequence of these experimental building blocks, then it's very easy to compute the final quantum state. You can put that in a computer and clearly do the unitary, make your matrix vector multiplications or whatever, the final quantum state. The reverse task is extremely hard. If I give you a high-dimensional entangled quantum state and I ask you, how do you build that? This is close to impossible. This is extremely hard. And what Mario Glenn has done for a couple of years already, he implemented a random search. The algorithm was called Melvin, and it just tried out random setups, looked at the final state and classified its interest. So how is it, is it a maximum entangled state or not? What is it, Schmitt rank and so on. And this is very costly. It costs about one second per simulation. So it is straightforward, it is easy, but it takes time. And our goal was to improve this through machine learning because inference times in a trained neural net are very fast, milliseconds was less. So our quantity of interest was the Schmitt rank vector. This represents the dimensionality of the states of photons. So let's look at this example state here. It's 0, 0, 0. So three photons, 0, 0, 0, 1, 0, 1, 2, 1, 0, 3, 1, 1. This is a maximum entangled state. And its Schmitt rank vector is 422 because Alice's photon really uses up the four-dimensional space, 0, 1, 2, 3, that's four-dimension. Bob's photon only lives in 0 and 1 and Charlie's photon also uses up only 0 and 1. So that we call a 422 state. And our goal was to find maximum entangled states with high Schmitt rank vectors, 7, 8, 9, 10, and so on.  And each sample consisted of the experimental setup as the input. So this is a sequence of these components, wave plates, holograms, and so on. And in the end, it had a target value, a supervisory signal, a flag, which said positive and negative, indicating whether or not it's maximally entangled. And a three-tupper, a vector, that says the Schmitt rank and mk. And we had, yeah, like millions of examples. And the leading Schmitt rank n was from 0 to 12. But you see in the higher samples, you don't find maximally entangled states so easily anymore. Yeah. And we used, we clustered the data by the leading Schmitt rank and did some cluster cross-validation on the fold 0 to 8. Cluster cross-validation means you train a network on the folds with Schmitt rank 0 to 7, and then you test it on 8. Then you train it on 0 to 6 and 8 and test it on 7. Then you train it on 0 to 5, but not 6 and 7 and 8 and test it on 6. So you always leave one out and train on the others. This gives you very good generalization characteristics. And moreover, to not only have generalization, but really out-of-distribution capabilities, we did not at all train on larger equal than leading Schmitt rank 9. So we used that as an extrapolation set or so-called out-of-distribution set. Yeah. And as an architecture, we took an LSTM. So x now would be the sequence of wave plate, beam split, hologram, and so on. And these are variable sequence lengths. Some sequence have 10 elements, some have 20. And the y hat would be the output signal. It would say whether or not it's maximally entangled and the three tupper of the Schmitt rank vector. And setups are then proposed after training. Setups are proposed randomly, and we characterize a setup as interesting when it's classified as being maximally entangled and when a Schmitt rank vector is in some closeness to uncharted territory. So we were interested in finding new states. And these are then checked by the simulation on what they really are. And here I show some results. There's some trade-off between a rediscovery ratio of what we knew already and the precision. But in general, the model performed very well on a wide set of parameters, so true negative and true positive rates and precision were all very well, especially when allowing some distance to uncharted territory. And this is again, this is the cluster cross-validation. So it shows that the true negative rates are high for all validation faults, and in general, all metrics are very good also for the extrapolation set. So this is quite remarkable that this LSTM performs very well on the data in a part of, so to say, the Hilbert space, which it has never seen in training. So we demonstrated that LSTM can learn to predict the characteristics of such high-dimensional quantum states without any explicit knowledge about quantum physics. And this, in principle, can be used as a generative model where you can, yeah, the training set was calculated by Melvin over many months, and it was just randomly selected sequences and then computing the target values. This was the training set. Yeah, yeah, yeah, yeah, yeah. Training set is very important, and this is a huge amount of computational power in the training set. But once you have it, then you can have very fast inference. Yeah, and you can build a generative model out of the discriminative model, where I would give you a certain number of elements and then the LSTM would predict the next best element, like then you should put a hologram or a wave plate or whatever. Okay, this closes the first application. Now to the second one. This is ResNet for the prediction of planetary collision outcomes. This was a work mainly pushed forward by Philipp Winter, who had an astrophysics background and then switched to machine learning. So the motivation here is we want to have a fast and accurate treatment of collisions in n-body planet formation. And this is a very challenging task because there is a very fast way to do that. It's called perfect inelastic merging, but it is also very inaccurate, yeah? Yeah? Okay, so I understood the question, how does this compare to genetic algorithms? And I have to simply admit I do not know. We have not tested it. To my general knowledge, deep learning outperforms genetic algorithms in tasks which require large datasets usually. But we have not looked at it or tried it. I'm sorry, I can't comment more than that. Okay, so there is this fast but inaccurate way to do it, and then there is again a very accurate but slow way to do it, and that's called smooth particle hydrodynamics, SPH, a very expensive way to make these simulations. And our goal was to tackle the problem with machine learning, in particular RESTNETs, and achieve a good trade-off. So you invest some time in SPH simulations, and then on that dataset, you have a good network that can make good predictions and fast for new data. Yeah, and here you see some collision here where both planets, so to say, survive, and here they actually merge, the blue and the red one, they collide and merge. So there's this hit-and-run scenarios and mergers. And this takes a couple of hours, such planetary collisions. So previously in the astrophysical community, such things were tried already, but previous datasets did not take into account the water of planets, which is quite important for good simulations, and they did not take into account initial rotations of the two planetary systems. And we did all this. We created a dataset of about 10,000 SPH simulations. We did consider water fractions and rotations of the target and the projectile. We randomly sampled over a wide variety of initial conditions. So what is the initial rotation? What is the impact angle? What is the impact velocity? What is the impact ratio? Something between 5% and 100% from the smaller to the larger target. The total masses are also between Ceres and Earth. And we also had some algorithms that identified the fragment. So after these collisions, all hell breaks loose, but sometimes small fragments can be identified, and sometimes they are not even touching each other, but only gravitationally bound. We also were able to have that treated correctly. In general, there are three major outcome regimes. So there can be erosion that the target loses mass, so that the projectile hits the target and takes away some mass. This is how probably our moon was created, I guess. And then there's accretion. That means that the target gains mass from the projectile and in hit and run there are two remnants. And it depends on the impact velocity, the impact angle, and the mass ratios on where you are. And previous approaches in this community just took classical machine learning algorithms, like support vector machines, SVMs, KNNs, K nearest neighbors, RF is random forests. Some feed-forward neural networks have been tried and in regression gradient boosting has been yes. Angular momentum, the rotation is important. Yeah? Yeah? Yeah, here, it's hard to draw three, four, five-dimensional graphs. Here it's only the impact velocity and the impact angle over the masses. And our approach was a genuine deep learning approach. So we built a residual net architecture with an autoregressive model that ticked through time steps. So we had multiple iterative steps. So we have an inductive bias due to the physics here. And we have a weight-tied residual net with actually three shared feed-forward networks. And we do these adaptive updates, which effectively is something like an Euler integration. So this is one of few examples, actually, where we have a very physics-inspired way how the network works and therefore also an interpretability of the result, which is quite nice. So let me quickly report here on the results. So this rest net really significantly outperformed feed-forward all other classical approaches like support vector machines and so on on most of the tasks, including an out-of-distribution set. This is always particularly interesting to the researchers how well does a trained neural net generalize to data out-of-distribution so that it really has never seen. Our rest nets are slightly less parameter efficient. That means they need more parameters than the compared FNN, but they are significantly more data efficient. So we can train them much better with a small amount of data. Yeah, and this advantage is due to the algorithmic bias. It's just the network itself that has an architecture that is close to the problem at hand. And in the immediate states of the rest nets, they are interpretable because they store the physical values themselves. This is not at all typically the case in a neural network. It does whatever it wants, and in the end converges to some output, but whatever is in between typically is a black box. And here, this is not the case. Okay, that brings me to the last part of the talk. The last application is we want to use classical shadow measurements to learn ground state properties of quantum Hamiltonians. So predicting ground state properties of large scale quantum systems is very interesting from a fundamental research perspective, but also very important for developing quantum technologies. And as you all know here, of course, the exact description requires exponentially many measurements, which would be a full tomography. So this is a big overarching problem, but of course there are some solutions or ways to tackle this. And one is the so-called classical shadows that have been developed in the last five years ago, so by Aronson and then Juan King Presquil. Presquil won the bell prize half now for the classical shadows, I think. So these classical shadows are a way to randomize the measurements and then with the only logarithmically number of... So logarithmically in the number of qubits with only a logarithmic number of measurements to have an approximation to the density matrix which allows to predict certain properties, like for instance two point correlations. Obtaining data is still expensive, so we were interested and motivated by the question can we have a sample efficient machine learning model? And for some tasks it is proven that there should be machine learning superiority, so to predict for instance these correlations of lattice Hamiltonians. So our goal was to leverage the graph structure of these Hamiltonians. We were really motivated by taking the physical situation at hand and using graph neural networks and we wanted to show that these GNNs, these graph neural networks exhibit better sample efficiency so that they need less data presented to them to already make good predictions. Our problem Hamiltonian at hand was the 2D antiferromagnetic random Heisenberg Hamiltonian. So random means the coupling JIJs are randomly chosen between 0 and 2, so they are all positive, so this is antiferromagnetic and it's Heisenberg because it's really x, y and z. In principle this has couplings between all spins, but in the end we took nearest neighbors only. And the ML task here is by observing that the JIJs are an implicit representation of the ground state, implicit in the sense that the JIJ make up the Hamiltonian, the Hamiltonian determines the ground state. And based on the couplings JIJ they are really given to us, this is so to say our input to the graph neural net, we want to predict expectation values of the ground state two point correlators. So not of the ground state itself, not even of the ground state energy, only about the two point correlators. So trace c row where c is this xx plus yy plus zz over 3. Yeah, you're right. The structure of the Hamiltonians is so to say given and then the JIJs tell you, yeah, you're very right. If I only give you chase and don't tell you whether it was Heisenberg or using Hamiltonian or whatever, then you know nothing. You're right. So on the left here we see the illustration of such a 2D Heisenberg Hamiltonian with 20 spins and only nearest neighbor couplings now. This is quite important for the next slide. So we now restrict ourselves to nearest neighbors only and choose some random couplings and the strength are written here as numbers and are illustrated by the width of these gray bars. And of course for such a Hamiltonian you can compute the so-called ground truth. This is machine learning language, so the actual values of the two point correlators are computed from DMRG for one Hamiltonian here. And also then you can think of what is the average correlation over all possible pairs, so 0 with 1, 0 with 2, 0 with 3, 1 with 6, 1 with 7, so on. So all pair combinations that have only distance 1, sorry, that have distance 0, well that's the spin with itself. It's of course perfectly correlated. Whenever you look at next neighbor correlations you get a negative value because it's an anti-ferromagnetic system and for d equal 2 they are then positively correlated and so on. So you can look at these average correlations over distance d and this is what we will later also look at. Yeah, we compare the following methods but actually I'm only interested now in showing you the difference between the first two. One is a fully connected graph neural network where the prediction of the correlation is directly calculated from the edge embedding. So we have a graph that has these 20 nodes and every node, so to say, corresponds to a spin of the real system but now in reality only next neighbor spins are coupled but in the graph we want to predict also a correlation between spin 5 and 10. So there's a distance of 5 maybe and so we need to have an edge in the graph where in the edge we can learn the correlation. So the graph doesn't look at all like the system because in the graph everybody is connected with everybody and we have an edge learning task. You might think that this is not ideal and indeed it is not ideal but so this requires edges between all the nodes in the edge you learn a hidden vector and that is fed into a feed forward network to obtain the correlation. Then we took the very physically motivated so-called pair-wise GNN where the edges only are present for the nearest neighbors so now our graph neural network looks exactly like the spin net. It looks exactly like that, 5x4 and only the nearest neighbors are coupled but now how do we get correlations? How do I get the correlation from node 5 to 10 if this is not connected in the graph and the answer is in the following way. You compute the correlation prediction based on the inner product of this node embedding. So in every node there's a big embedding space there's a huge spin, so larger than the real physical spin and then the inner and then the spins are learned such that the inner products lead to the correlation to the two-point correlation and we compare this to an MLP which has done before already and to a neural tangent kernel which was also done before already in the last two and our quality measure is the mean squared error on the test set so it's the squared difference between the prediction C hat so this is the prediction of the correlation value between spins i and j and the ground truth, the true value is C ij and in the test set of course you know all these values this is how you learn and this is the result here so on the left we can see the error the mean squared error on the test set versus the number of training Hamiltonians 10 to 300 and we can clearly see here that this pair-wise GNN the graph neural network that respects the physical graph structure is the best and similarly we can look at the error versus the system size so how many... we now take 80 Hamiltonian we take 80 different Hamiltonians and now we vary the system size 4 by 5, 5 by 5, 6 by 5 and 7 by 5, so up to 35 qubits and also there we see that the error is best and then we calculate the average correlation for next neighbors and again the pair-wise GNN is winning and then we do it for d equal 5 so for the distance 5 steps to the next, next, next neighbor so to say so the conclusion here is that the graph neural networks outperform all other methods and it has the same graph structure as the Hamiltonian so this inductive bias seems to have a good effect here yes in the pair-wise GNN we learn the nodes in every spin there would be an embedding space with a big vector and to compute the correlation between two nodes doesn't matter whether they are neighboring or not neighboring so we take the scalar product plus some processing of that it's not directly this but almost and the outlook is here to use GNN in larger experimental setups if I have one minute left I want to show you a follow-up work which was done by the same consortium but I was in parental leave so now I'm talking about work not done by me but direct follow-up work on what I just talked about and this is basically theoretical work but also partially experimentally so it was already previously demonstrated that learnability of observables can be done when the number of data scales polynomially in relation to the system size N this is a result for PAC learning from a couple of years ago and this work now improved this bound to a logarithmic scaling by introducing a locality inductive bias so the number of training data to learn properties should scale only logarithmically in the number of qubits which is extremely powerful and the trick here is that the feature vector is formed not for one node itself but from a locality region so in this hexagonal net this for instance it would be always a full placate that we would write into an embedding vector and just very briefly it turns really out that this new model requires less shadow measurements so less measurements of these classical shadows I have talked earlier it has a better sample complexity and better system size scaling so indeed the theoretical results somehow now merge with the experimental endeavors and the holy grail of course would be to have only a logarithmic number of measurements and samples from some maybe classical shadow techniques and then have good predictions for certain properties for instance two point correlators of physical systems and that in time brings me to the conclusion of this talk so I have tried in the first 10 or 15 minutes to show you that deep learning is an extremely powerful technique and that there exist many different neural network architectures specific problems require specific architectures there is no one size fits all solution in deep learning and very often depending of course on your problem at hand but especially in deep learning you do need good and large data sets and also large computational power otherwise classical machine learning techniques like support vector machines gradient boosting and random forest and so on much better so if you have only small or medium size data sets deep learning might not be the way to go forward but deep learning can be applied in a plethora of fields and subfields of physics and science in general namely whenever you have enough data to train your models and the drawback is that typically I showed you this one exception in our astrophysics project but typically you get the black box solution and you get the credibility that is a drawback we have to live with somehow and with this I thank you very much for your attention thanks so much for this very nice talk we already had a lot of questions but there are more thank you for the very nice talk I have two quick questions so the first one is I mean the result that you had with graph neural network where you basically are on your Eisenberg disorder Hamiltonian where you had this locality between the interactions so did you try to see if you have a fully connected graph neural network and you have an all to all interaction are the results better or the previous one you are referring to here the local regions of parameters no the previous sites okay you are referring so here the pair-wise GNN is outperforming the fully connected GNN so if we have a graph where everybody is connected with everybody such that on the edges we can try to learn the correlations this is poorer than only the pair-wise GNN where we respect the physical coupling structure what happens if you have a fully connected Hamiltonian not pair-wise on a fully connected graph sorry now I got the question what if the Hamiltonian is fully connected we have not tried that if I remember correctly I reckon if the Hamiltonian is fully connected then also the fully connected graph neural network will probably then I guess the pair-wise connected graph will perform poorer as it will not respect the structure anymore we have not done it but that would be my guess for the fully connected Hamiltonian I expect the fully connected graph network so you need some knowledge of the Hamiltonian in your answer so my second question is for the other slides where you have this embedding of the placets and you see this logitemic dependence on the system size so what happens if you it seems to me that your system has translational invariance suppose that you have like disorder between the nodes do you think that you still have this very nice logitemic scaling that's also a good question that has not been looked at yet I guess that even then some advantages should remain because this really is about the efficiency of the embedding space in machine learning so I don't know this is a good question it has not been looked at yet but my guess would be that even without the translational symmetry it would get a bit messier in the implementation but I believe that the advantage should remain because I feel that this is part of the embedding in the machine learning protocol one more question about the correlations in the Heisenberg model so if I understood correctly you're basically learning the pairwise correlations here so is it possible to use your setup to learn pairwise correlations maybe three-body or higher-body correlations? Yeah, good question well, the thing is that for the pairwise correlations we know that they behave well for data that has been acquired from the classical shadows but it is known that the higher the correlations are the less good the classical shadow techniques is in predicting them so it's our first step generation where we have this big information that cannot be saved anymore by the later machine learning of course but yeah, these higher-order correlations are an open research question Maybe you can use this to denoise the shadow in some sense Thanks Thank you One more question do you have any sort of estimate how resilient your predictions are for the setup if you would have small terms in the Hamiltonian that you did not know there would be Oh, you mean if in the Hamiltonian itself there is some some other term Like a small magnetic field in the lab No, we have not made any attempts in that direction or any studies also very good question I can't say anything to that Thank the speaker again


Okay, so one thing before we do the next one that I should tell you is that you can figure out if you've got what we call saturated hydrocarbons, okay, so these are hydrocarbons, right, because they're only composed of carbon and hydrogen, okay, so that's a hydrocarbon. If you ever hear of that in the news or whatever, you know, that's what they're talking about. So the things that you make gas up, you know, gasoline and fuels and stuff, not these molecules in particular, but hydrocarbons in general. But anyways, if you wanted to know what, if you've got a saturated hydrocarbon, the molecular formula is always going to be this, CnH2n plus 2. So you can see that if we've got C4, right, this is going to be H2 times 4 plus 2 if it's a saturated hydrocarbon, right, so that's C4H2 times 4 is 10, or 8 plus 2 is 10, right. So, is everybody okay with figuring out the formula of a saturated hydrocarbon, okay? So if you notice, that other formula that we were going to do is C4H8, so is that a saturated hydrocarbon? No, you already know that just because of the formula, okay? So I'm going to erase everything else, except for that C4H8, yeah. These are structural isomers as well, right, these are structural isomers as well, okay? They are like, in that sense, dimethyl ether and ethanol, okay, that they're different molecules, okay? So this molecule, he's got the same atoms as that molecule, right?  Butane, right, okay, so that's, so remember, how do we know if it's the same molecule or a different molecule? If we have to break bonds and then make up, okay? So if you need to build these models, do it, okay, in fact, I'll let you bring the model kit on the test, okay? But anyways, so can we twist this around anyway to make it, no, right, the other thing? So let's try breaking bonds and reforming them, break, reform, reform, huh, that's the other molecule, huh? Do you guys see that? So let's just point it out. So that's the bond line formula or the ball and stick model of the other bond line formula, right? So notice, this molecule we can twist it around and do stuff like that, right? Those are called conformational isomers, and it's much easier to see if we do it with butane instead of isobutane, which is the other thing. So like, look, so we can do conformate, here, let's actually do that in a different video, okay? So are there any more questions on the structural isomers, these two? In fact, I wanted to get to the C488, but we turned it into a different video, so we'll do that in a different video, okay? That's totally cool. That was a good question. Any questions on that stuff?


This paper proposes a new approach for remote-sensing image scene classification called CNN CapsNet. It combines the advantages of convolutional neural networks, CNN, and capsule networks, CapsNet. First, a CNN is used as an initial feature map extractor. This CNN was pre-trained on the ImageNet dataset and is then used to extract features from the input remote-sensing image. Next, the extracted features are passed through a CapsNet to further enhance the classification accuracy. The CapsNet is able to capture the spatial information of the features in the image, while also preserving the hierarchical structure of the features. Finally, the output of the CapsNet is fed into a classifier to produce the final classification result. Experimental results show that the proposed CNN CapsNet outperforms other state-of-the-art methods on three publicly available remote-sensing image datasets. This article was authored by Wei Zhang, Ping Tang, and Li Junzhao. We are article.tv, links in the description below.