My name is Craig, I'm a security researcher with the Tripwire VRT group. I write phone content for our IP 360 scanner and I also do a lot of vulnerability research on my free time and at work. But today I'm actually going to be talking to you not about vulnerabilities but rather about how to work with some of the tools that are out there for RFID hacking. So I've accumulated with VRT a number of tools for working with high frequency and low frequency RFID and I've noticed that there's not really a lot of consistent documentation out there for some of these tools. People release patches for adding new functionality for various things but doesn't often get a lot of documentation for it. I mean you can definitely read through the source code and figure things out but I wanted to make things easier for people and then I also wanted to look at ways that I could use the 3D printer that we have to try and enhance some of the practical attacks that you have with RFID tools and also look at some of the different options that you have or opportunities for research through that. So what we're going to go through here, I'm going to overview the tools that I'm working with for this project. Go through some of the basics of how RFID functions, some of the basics of how 3D printing functions and then we'll get into firmware changes that I've made to the Proxmark 3 and how these changes were made. This is actually documented in a lot more detail on a white paper that's going to be available on the DEF CON site and then we'll get into the 3D printing stuff and look at how you can make antennas and how you can conceal different tools with 3D printed objects. So these are the tools that we have going on for this. You've got the RFID tools over here. The Proxmark 3 is kind of your Swiss Army knife for all things RFID in the low frequency and high frequency bands at least. I've got below that the PN533 USB stick. That's an NXP chipset on a USB stick and finally on the bottom there is the RFIDler V22 board which I obtained at DEF CON last year. Moving over to the embedded computing tools. We have the USB armory which is a little thumb stick sized computer and also a Raspberry Pi board that I was looking at. Finally for the 3D printing I was using primarily a cube pro 3D printer which we acquired in the office. So getting right into it the Proxmark 3 for anybody who doesn't know this is a board that was developed as part of a PhD thesis by someone who is analyzing the transit cards in their city. As I said it does low frequency and high frequency support and at the heart of it you've got a Xilin Spartan 2 FPGA. This is what's used for doing the precise timings that are needed to do effective NFC communication or high frequency RFID communication. It also is providing spy interface and an SSP interface for transferring data and commands. The heart of that is the APMEL microcontroller so something very similar to what you might find on our Arduino board. This is actually what's handling the communication from the computer and relaying commands over the FPGA as needed and doing demodulation and decoding. Most of the heavy lifting actually does happen in C code in there so you don't really have to worry so much about the verilog that goes on in the FPGA unless you really want to do some low level stuff with changing around the NFC modulations. You've got 8-bit ADC on there, an analog digital converter which gives you a 40 mega sample per second bit rate on there and it receives its commands over a spy interface. The connectors on the board, you've got mini USB that you're using for powering it and also for data transfer and then you've got a high rose connector. This has four wires on it, two for a low frequency coil, two for a high frequency coil and then you've got a number of different human interfaces for it. You've got a push button which gives you input to it and four LEDs for taking output from it of different colors so this tells you the status that the device is in while you're working with it. Some of the commands that are very helpful for this have been added recently like the LF search and HF search. These give you opportunities for identifying a tag that you don't know what kind of tag it is. I've also got the commands here for reading in a wave form for low frequency tag, reading NFC tags, the ISO 14A standard and also some commands here for cracking the encryption on my fair classic cards. The USB stick that we're talking about here, this is one of many LibNFC compatible USB sticks. It supports a wide range of NFC and not a whole lot to say about that. Some commands here that you might find helpful for working with LibNFC for doing various emulation, reading tags, relay attacks and also you can use LibNFC to do more advanced functionality through scripting of course, sending out NDF messages if you want to do fuzz testing on Android. The RFidler board finally on the RFID tools. This is a low frequency tool. You've got a lot of LEDs for output. A PIC32 is the heart of it and then you've got two banks or a bank of digital potentiometers which use for adjusting thresholds on the reads. Here's some of the common commands that you would find on that. This is of course in the white paper as well. A late comer to this project though was the chameleon mini. This is a platform for working with and emulating contactless smart cards. It was developed at the Royal University in Bochum and recently was started shipping from RISC Corp. But you can also grab the schematics and fab the sport yourself looks like this. USB is for power and data and then also you've got a reprogramming port PDI headers but you don't generally need to use that so often. On the embedded devices, the Raspberry Pi, I think everybody in this room is probably familiar with the USB armory as the little thumb stick which also has an OTG host adapter. So you can use this to act as a client or a host in the USB mode. Now the 3D printer that we had the Cube Pro, this has got one of the larger build areas that I see in the consumer market and it has the ability to go down to 70 micron layers and supports several different materials. When you're working with this printer, the first step is going to be you want to draw up your model on a CAD program just like you would with any CNC or 3D printing process but then unlike maybe the maker bot for example which many people are familiar with, on this one you're going to put down a coating of water soluble glue. You then have, you put that into the printer and you have an extruder that's going back and forth depositing one layer of whatever thickness you wanted onto the board as the build plate moves down so that your project moves off from it. In the end you can just pull it off with some hot water that dissolves the glue. There are some problems with 3D printing. You can't just draw anything you want and print it and you do have to worry about machine maintenance. Up here on the screen that's a picture of the extruder from, or the drive gears for the extruder on RQ Pro. There's some debris in there. When you get debris in there it can lead to clogs within the print jet which are not fun. But RFID that's what we're here to talk about so to build a little basis of this there's low frequency tags. These run around 125 kilohertz. Primarily access controls also pet tags. Some vehicle immobilizer technologies I think work in this range. And then you've got the high frequency class. This is where NFC falls in. Access control, contactless payment cards, the German identification card, US passports, these all have high frequency tags in them. When you would open up one of these what you're going to find is a coil of wires that is actually tuned to work and make an LC circuit with, or an LC tank with the little chip that you see in the top corner there. What this means is that when you put this into a field that it's tuned for like the 125 kilohertz it's going to draw some power, power up the chip and the chip is going to be able to dampen, undampen or kind of open and close the circuit so that the reader on the other side is going to be able to see modulations in the waveform. These are common modulations for many RF applications not going to get into that. When you want to clone an RFID tag you're going to work with the T55XX tags. These are like you can buy these over in the vendor area. I think you get 10 cards for $30 so they're not too expensive. You've got to support for a couple different modulation schemes and then what you also have is reprogrammable EEPROMs. So you're able to throw on a configuration for this to tell it what type of demodulate or data rate you want, what type of modulation scheme you want and also how many blocks you're going to use for the demodulated buffer. If you want to work with one of these using the Proxmark to clone a tag that you don't know you can start by doing the LFsearch command. It's going to read in samples and try and demodulate until it finds the tag. When it does find it you'll see a report letting you know what the tag format is, the ID. You can see here we have an HID tag with dead beef on it and then we can use this printDModBuffer command which was recently added in the past year to Proxmark. This will actually give you the bytes that the tag is sending out. You can break these down and split them up into the blocks that you need to put on the T55 tag and then you still need the block zero, the configuration block. So you could obtain this by going through the data sheets for your T55 card as well as the tag that you're working with or you could try and decipher those values but you generally don't need to do this because the Proxmark forums have lots of detailed information telling you the blocks that you need to set up for these. So you can see the configuration for the HID tag right there and then now that we have that information you just do a series of write commands and you've got a cloned tag. We can read it back in and we see that it is in fact cloned. So moving on quickly to the NFC end of things, one of the popular formats here is MyFair which we're going to be working on today. These are tags that have some UID 4 bytes or 7 bytes and then some amount of data, possibly some security features. They get used in all sorts of places. Hotel key cards which we'll look at cloning today. Payment cards, lots of places. For cracking these cards, one of the earliest formats of them were MyFair Classic and it was discovered that you could launch an attack whereby you power up the card and you could repeatedly get the same nonce and weaken the perform a cryptographic attack on it. Once you've recovered one of the keys you could then launch a nested attack and actually in ideal circumstances you can recover all of the keys for one of these cards like what you might find on a train pass in under a minute. And then you can use what are called magic cards which are fully reprogrammable to write the data that you found and make for all intents and purposes a clone of that original card. This is really why you don't want to ever use an application like that lock sitting on the table over there that's only going to use a UID for validation. If you want to clone for example MyFair Ultralight card, something that you might see in hotel key cards for example, you can use LibNFC is a very effective tool, commands up here using NFC, MF Ultralight. You can also of course scan a tag with your phone and you have the bytes from it you can then enter those bytes into a file, write that onto a tag that you want. I was using tags from clonemykey.com. So as I said some of the things you can clone, you've got hotel cards, you've also got as we learned this week at Black Hat the possibility of being able to use some of these tools to clone transactions on EMV credit cards because they have some legacy support in there. Also Android smart unlock tags and as I mentioned the Samsung NFC locks, these are only validating UID so it's very easy to break those. You can see here what it looks like if we're using a cloned key card. It's unmarked but clearly it opens the door. So now getting into the firmware hacking aspects of this project, one of the things that I wanted to do was work with both high frequency and low frequency and I figured that one of the useful applications for me would be working with NFC tags in the standalone offline mode. So I went ahead and worked on writing code for capturing the tags and working doing a clone to a magic card of the UID, also replaying of the UID. This can be extended fairly easily towards data sections on the card as well. So the initial low frequency mode of the Proxmark standalone, this is flow charted out by the Proxbroot white paper. Basically you hold down a button, flashes some lights and then through either holding the button or pushing the button you can manipulate whether you're in a playback mode or if you are in a clone mode, cloning to a T5-5 tag. Now when you look through the source code, this is all in the white paper of course, but you're going to find that everything runs on the ARM processor of course for this. There's a function SAMI run which makes use of various functions that are available to the ARM processor there for the HID functionality. When we want to go over to high frequency, however, I wanted to try and reproduce as much of that functionality as possible with the focus on my fair cards, not just my fair classic of course though. The ability to clone the UID onto my fair classic only was implemented though. Here are some of the setup functions that you need for working with high frequency tags, selecting them, getting yourself in reader mode, then for simulating. But the most interesting part of this I think was I decided to go ahead with a different work flow for how this was going to happen. In the low frequency mode for Proxmark, for anybody who has used it, you might have noticed that you can jump right into the playback mode with uninitialized data. So you're dropping into the stand alone mode, but then you have to hit a button again before you're going to get into read mode. I decided to get rid of that and instead jump right into the record mode and then when you read a tag, jump into playback mode, use this as the hopping point for your other functions through either a button press or a button hold. And also add in some sanity checks so you would never use an uninitialized value and you also wouldn't inadvertently fill your banks with the same card. I do have a demo here of this, but I'm going to hold off on doing that because I have a lot of slides and not as much time. So if there's time at the end, I'll come back to that. Otherwise, if anybody would like to see this, you can hit me up Twitter or tap me on the shoulder, whatever it might be. So the second component of my firmware hacking, I wanted to add support for a tag format that wasn't in Proxmark. I wanted to learn how to do that and be able to document this with really a tutorial-like example so that other people would be able to go back and add support for other tag formats that they might be interested in. So in order to do this, I decided I would make the LFAWid context for Proxmark. This means cloning most of the functionality that you would have in the HID mode. Writing things to T55 tags based on the numbers printed on the tag, because if you see here on the printout of that tag, it's actually screen printed with all the information that you need for being able to clone it or simulate it. Whereas most tags you're going to have an ID number, but it's not going to disclose all of the information that you need, like the facility code and the card number, rather. So since there was no support in this when I started looking, I decided it would be a great place to work on. So the AWID 26-bit format, that's what I specifically targeted. This is comprised of an 8-bit facility code and a 16-bit card number. Now, that's only 24 bits, but then the other two bits come from parity. And the card is going to work with the same parameters actually as an HID tag. It's got this FSK2A, which means RF50 data rate, and it's specifying a certain number of cycles that you're going to go through with the higher frequency to indicate a logical low versus a logical high. And then when we want to start adding the commands into the Proxmark, you have to understand that there is a command table structure, things are hierarchical, so you just need to add a definition in your new file that you're creating. And then within the functionality that's actually going to be called, you're creating a USB command structure and sending that off to the FPGA. So you can see here how it looks. This code is all, by the way, in GitHub already in the master branch for Proxmark. You can see here the FSKD mod functionality, and up on the slide here are some of the functions that you need for working with that. In order to move beyond this into the clone and simulate functions, I needed to develop a function to take those numbers printed on the card and convert that into the Wagon. For this, I decided it should stay within the client code. It doesn't make sense to have this down on the arm chip, where it's going to take up space, and the arm is never going to know anything about these numbers anyway. So I was then able to go right ahead into doing the LFAWid clone functionality, which is piggybacking on the T55XX commands under the covers, and also showing to you as you see the blocks that it's calculated that you need to program onto the card. The simulate function also has no purpose within the arm, so this lives in the client code, and you can see up here the parameters that you need to specify in the commands. There's a lot more detail about that in the white paper. But now the antenna construction. This is where we get into 3D printing and the applications there. So when I started, I was always interested in making an antenna for my Proxmark and for other tools, but I noticed that the DIY projects out there, they felt a little bit too artsy-craftsy for me maybe. There's a bit of trial and error going on there, so I thought I might be able to do better with making 3D printed forms for that, and that's what I did exactly.  So you've got some functions that you can find for going between the frequency and the inductance that you need based on the capacitance in your circuit. You've also got some functions or some equations from white papers out there explaining the relationship between the number of turns and the dimensions of your form to the inductance. With all of that, I went and looked and I saw that there was a nice design for an LF badge on the Proxmark website, but it was using cutting out CD cases, gluing them together, stuff that I didn't really want to go through. So I took that design and I basically just drew it in some CAD software, printed it out, and I found that it worked really quite well. In the end, it took around 87 turns for the Proxmark, about 57 turns for the R Fiddler, but you would find if you wanted to reproduce that it's going to vary a little bit due to the nature of the system. But all those equations, fortunately, don't really matter all that much. As long as you've got enough turns, you plug it in, you try and tune your thing so you see what frequency it's optimal at, what voltages you're getting out of it, and then you simply unwind it one by one until you get to the frequency that you want. So you can see here the antenna that came out from this, it was, I tuned it exactly for 125 kilohertz, and it actually worked better than the commercial antenna that I had already paid $60 for. So whereas that antenna was picking up at its optimal voltage 29.43, I was getting 31.21, I know you can get a lot higher than that even, but this badge worked very effectively for, or this card worked very effectively for me. So I then went ahead and added a lanyard clip to it and made something that looks like this. This actually broke on me. Word to the wise that those, that 40 gauge wire, very, very thin, you want to do everything you can to reinforce that hot glue, higher gauge wire, these are your friends in this. But in the end, this was a very inexpensive build. Even going out through shapeways, you can use the model that's on github now and have this printed for $7.68 before your shipping and handling. With the cube pro printer, it costs maybe four dollars to print out the form for it, and if you actually had like a maker bot that you can feed engineer filament to, you can do this for well under a dollar. And then I cut up a lanyard and RCA cable and made it so that I could have the cable going down through my shirt into the Proxmark in my pocket out of sight and just kind of looking like an actual badge with a sticker with my picture on it. The next project that I looked at here was the clippone. So a lot of people talk about using clipboards in pen testing situations for RFID. So I decided to see what I could do with the fact that I now have 3D printable antennas that could be kind of fit into my thing.  You can hide this with some papers inside there to, in case somebody opens it up, it won't look so suspicious. And in general, if you want to make one of these, there are all sorts of storage clipboards out there on Amazon, you just need to make sure that you find something that has enough depth to fit whatever it is that you're trying to hide in there. So you can also enhance this if you wanted to by adding in something like the USB armory or an RFIDler board, I'm sorry, a Raspberry Pi board, and if say you had an RFIDler in there, all you need to do is connect something that's going to be listening with USB-0 and logging that data very simply and you can walk around all day and then come back and have a log of all the UIDs you've captured. With the Proxmark, you can of course take the client code and you can build this. It's already got an arm build for Android, so that shouldn't be too much work. And then you can also move on to making fake readers and doing other things to hide your tools in the field. So one of the valuable resources for this, you have building information models out there. So if you have AutoCAD, they have their seek environment, which allows you to search for lots of things that you'd find in different buildings. I found for example this HID reader enclosure which if my 3D printer hadn't failed on me, I would have printed out to bring here, but you could very easily hide inside of that a custom circuit board or a tool with an antenna and maybe put it in an unexpected place, maybe conceal a legitimate reader and use that to try and capture badge swipes. So really what we're looking at here is the fact that you can make realistic prints of things and you've got models out there, but even if you don't, you can use something like a Kinect or even an Android phone or an iPhone with the 123D Sketch app and actually just take pictures and get a very reliable 3D form out of this. The next thing that I was looking at was using a phone case to actually hide an antenna inside of that and some of the things that I would envision doing with this, you could of course do exploits on the Android Beam type functionality, say somebody thinks it's just a phone, but really you've got some more sophisticated hardware on there, but also the initial intent that I had for this was being able to eavesdrop on the NFC communications for something like Apple Pay or Google Wallet and there are so many phone cases out there that it really should be trivial to just take one of those and actually merge it with the design for a coil in such a way that it's not going to be obvious. On the embedded side of things, some of the other things that I was looking at were adding in the support for the USB armory to be able to log keys, doing scriptable responses and also being able to do something like if you use the OTG adapter on here with a little passive USB hub, you could have a Wi-Fi adapter in there and actually have a two-man team where one person is going to be using the device but not being able to see it, not knowing when it's read something, but somebody else, maybe a couple hundred feet away, is going to be able to access it, monitor what's being scanned, trip it into simulate mode whenever it needs to be. So there are a lot of possibilities with that for pen testing and actually at this point I do have some time that I can try and do the demo of how the standalone mode works for NFC. So I'm going to take the Proxmark out of the clipboard and plug it into a little battery here and I need my high frequency antenna for this, just right here. So now parallel that on, also add the extra battery back into this so it's let us know it's happy and we got to lock it. It's not so happy. Try that again. It thinks it's locked. It's not the smartest smart lock. Alright, so now it is in its locked mode and if I take this card, we can see this one does not open it. This one however will open it and we'll relock it. This however is a magic tag. So if I now power up the Proxmark by holding on this button, it flashes its lights for me, which I'm sure nobody in this room can see, but I can then go to the tag that works for it. When I touch it down, it is lighting up to let me know that it has scanned in a tag. Now it's in playback mode. Now one of the problems with playback mode, either the shape of the antenna or possibly the implementation of NFC on here, isn't always so conducive for this. So what we're going to do is try it out, but this did work once earlier today, but generally that's not happening. So we're going to clone it to a tag by holding down the button now, putting it on the field of the tag and then when I release it, it flashes to let me know it's written it and with any luck now, we can go ahead and oops, maybe not. Let's try that again. Oh, my antenna was loose. The high-res connector on this is not always so great for holding in cables. So let's just put it in stand alone mode again. Go ahead and scan in this tag and then we will clone it onto this one. Hopefully it hasn't been pricked. One of the risks with working with the Magic Cards is that if you don't have a strong connection to the antenna while you're writing to it, you can actually break it, which is why I stuck to just my fair classic where the tags are cheaper. But now we do have it so that's pretty much what I've got for you. If anybody wants to see any of this stuff up close, you can definitely come up here or actually meet me outside afterwards and just I want to say thanks for the Proxmark development team, Marshmallow and Iceman. Very, very great for being helpful with working with this stuff, being very patient and also to my family for putting up with all the crazy hours leading up to DEF CON. So, yeah, thank you. Oh, and I guess there are a few minutes for questions if anybody has questions.


Diagnosing your performance issues with your apps that are already in production may not be the greatest thing in the world, but have no fear because Azure has got you covered with the Azure Application Insights tool. So learn more on this episode of Visual Studio Toolbox. Hey everybody, welcome to another episode of Visual Studio Toolbox. I'm your host, Leslie Richardson, and today I am joined by Principal Software Engineering Manager Chuck Weiner, who is a member of the DevDiv Azure Services team. Welcome, Chuck. Hello Leslie. Hello Leslie, thank you. Thanks for coming. So today we're going to be talking specifically about Azure Application Insights, right? So can you tell us more about what that is? Yeah, Application Insights is an application monitoring features within Azure that allow you to collect data about your application while it's running. And the profiler is an advanced feature of Application Insights that lets you collect performance data about your application while it's running in production. As you know, with apps running in the cloud, it can be really hard to get debugging data about those applications. You can't just put a debugger on them. You can't just run a profiler on them whenever you want, unless you're using our service, which we run the profiler for you and collect that data and upload it to our service and make it available for you through Application Insights. Cool. So that's really interesting. Just the whole space of profiling. I feel like that's a subject that a lot of people are aware of, but don't quite use or learn up on until it's imperative. So something's going wrong while the app's already out in production, right? Yeah, it's very true.  What do I do? How do I look into this? You know, it happened yesterday. How do I find out why and what I can do to fix it? And that's the cool thing about our profilers. You can turn it on today and just let it run. And it'll be collecting data in the background and it'll have that data ready for you when you have a problem that you want to investigate. And it does so in a way that's not intrusive to your app. It doesn't change your app code at all. It runs out of proc. It's collecting for Windows services. It's collecting ETW traces, which is the Windows eventing, and getting call stacks from that data. And so it only runs by default. It runs a couple minutes an hour. So it's not running all the time. It's a sampling profiler. So we're hoping that over a period of time, it'll collect the relevant data that you need to debug your application. That's cool. And I like that you can kind of leave it on in the background and go about the rest of your dev work that you need to do. Yeah. So why don't I show you how to enable it and start collecting this data? Yeah, sounds good. So this is an application that we have that demonstrates the profiler. It's an Azure app service, which is the easiest way to use the profiler. We have a real simple switch to turn it on and off, which I'll show here in a minute. But for this app service, you can go to the application insights tab. And if you have application insights enabled for your app, it'll look like this. It shows application insights is turned on. And if it's not enabled, you can enable it here. But then you'll see this profiler section. And you can turn the profiler on and off. And as you can see for this app, we have the profiler on. So when the profiler's on, like I said, it's collecting data a little bit every hour. And the hope is that over a period of time, you'll get all kinds of requests will be coming into your application during those couple minutes an hour. And you'll hopefully get some long-running requests than you can see what's happening, why those are taking a long time. So once you've got it on, you can go to your app insights resource by clicking the view data at the top. And this takes you to the actual application insights resource. I'm in a different resource now than before. I've gone from my app service to app insights. And the profiler is part of the performance features of app insights. And so this is the performance blade for application insights. And there's a lot of information on this blade that I'll get to in a little bit. But the first thing you can do is hit the profiler button at the top, which this is our kind of configuration page or kind of profiler home page. This allows you to set some different properties about the profiler. We have triggers, which I'll show you here. And this is a list of profiling sessions that have happened. This app's been running for a long time and been collecting traces, profiling traces, you know, every day for a long time. And it lists those here. But at the top is some information about the profiler or settings about the profiler that you can set. The first one is profile now. If I click that button, it will start profiling right now in my service. This is really cool if you're doing some testing. You might have a perf test or a stress test that you're running and you want to get a profile from that test. You can click that button and it will start profiling now. And then there's triggers that you can set. If CPU goes above a certain threshold on the machines that are running your service, it will trigger the profiler to start. And we have that by default set at 80%. And it will run for 120 seconds. That's really cool. So you don't even have to play the trial and error game or trying to figure out where to start? No. Yeah, it will start for you when CPU gets to a certain point. And we also have a similar thing with memory. If your memory threshold gets above a certain point, we'll start the profiler. And, again, that's set at 80% as the default. And then there's this setting here. This isn't released yet. I'm showing you kind of testing bits here. Like I said, the profiler runs by default like a couple minutes an hour. And up until now, we haven't had a way to change that setting. It's just that's the way it is and that's what you get. We've had a lot of people ask us, can I turn that off? I only want to have the other triggers or I only want to have profile now. So we've given you ability to turn default profiling on and off. And then you can also set it normal, high or max. Instead of running two minutes an hour, you might want to run it more than that. So that's kind of high or max is like run it as much as you can possibly run it. I wouldn't recommend doing that in a production service for very long because you could start affecting the performance of your app because we're collecting some very low-level data and that it could and then uploading that data, it could take up resources that your app would need to run. So I recommend keeping it on normal but there are cases where you might want to increase it. And then this shows you recent profiling sessions that we've done and on the left here you can see it tells you how those sessions were triggered. Most of these are default sampling which is that random we're just going to profile and hopefully catch something cool. And then we have a few here that are triggered by the CPU trigger. That's so helpful. Just filtering out my problem. Yep, yep. And it's nice too because the dates here, if you knew you had a problem at a certain point in time, you can come in here and look, okay, on Tuesday at 9 o'clock in the morning, we had a high CPU. Let me click on that and see. I don't want to click on this right now because it takes a bit of time to load. But if you do click on that, it loads all the events that happened during that profiling session and lets you look at traces for that session, which I'll show you that in a minute in a slightly different way. Cool. So that's kind of the profiler home page. Now let's go back to the performance blade. This is the page that people are going to use to investigate performance problems. By default on this page, it shows you the average length of time for a request. But I would recommend if you're looking into a problem, you want to switch this to the 99th percentile. Now this is showing you the longest request. And this chart down here shows you all the requests that your application serves and how long those requests took for the 99th percentile. So you can see we have some requests here. Now this is a contrived example. We've done this on purpose to make it have long-running requests. But you can see this one here takes 23 seconds. That's a long time. So that's quite a long time for requests. Yeah, you want to look into that. Why is that taking so long? And this chart over here is now not very interesting for this service because all of my requests, because it's an example, take this kind of the same amount of time. But this is a histogram that will show you how many requests, for each time slot, how many requests you had in that time slot. And then the little triangles at the top indicate for that time slice, or that time slot, you have a profile. So I can drill into that if I want and see, okay, I have profiles or request to take 20 seconds. I request to take 22 seconds. And on a production service, hopefully you have a broader range of times. It looks kind of cool. To see all the different time slots that your requests are taking. But once you drill into this, you can click down here to this profiler traces button. And this, you can see it says 41. It means that for the filters that I have on this page, we have 41 examples of a request that took that amount of time that we can show you detailed data about. So I'll click that. And this will load the list of requests on the left. And you could probably count them. These should be close to 41, or if not exactly on 41. And then this is the detailed data that we have about that request. And you should be able to see your code in here. This is a call stack of what happened during that request and what took the most time. We try to highlight the hot path for you, which is the pieces of that request that took the most CPU time. And for this one, you can see we're doing an array sort. And that sort is taking a long time. There's CPU time scattered throughout here. And there's waiting time scattered. So the waiting time is because this is taking so long for the sort that this thread is getting swapped off the CPU and back on. Because the CPU can't allow one thread to just take over. So it keeps sending it off the CPU. Okay, I'll get back to you in a minute coming back on. That's what the waiting is. So you'd probably want to look into why is my sort taking so long? Am I doing it some weird way? Am I sorting too much data? There might be something you could do to speed that up. So I noticed, for instance, I noticed the download trace button. And in Visual Studio, there's also that built-in profiler. So could you theoretically download this trace, upload it into VS, and then track down what line of code is causing that giant performance spike? Yes, you can. Very good question. That's exactly what you can do. And the really cool thing about the download trace button is it gives you that whole two-minute trace. The view we have here is only what happened during your specific request that you're looking at. But this download trace button downloads the whole two-minute ETW file, which you can open in many different tools, like PerfView, Windows Performance Analyzer, or Visual Studio. I can show what that looks like in Visual Studio. This is one I've opened in Visual Studio. And one thing to note about that when you download that file, it downloads it as a zip file. Visual Studio doesn't recognize a zip file. You have to rename the file to end in .diaxession, D-I-A-G-S-E-S-S-I-O-N. If that's the file extension, Visual Studio can open it. PerfView and Windows Performance Analyzer can open the zip files. Sweet. Yeah. And then you can use the Visual Studio Diaxession tools to dig into this and see what's happening with your code at that point. That's awesome that you can work together harmoniously if you want to. Yeah. And the really cool part about it is you're getting this from a production service. You didn't have to do anything to collect this data. We've collected it for you. Hopefully, it has interesting things in it, and you can find something to fix in your app. Yeah. That's awesome. Yeah. And let me just show you a view of Windows Performance Analyzer, because it has amazing analysis that you can do. This is from a Diaxession that I collected from one of our actual production services, and you can see just the charts and graphs are just, you know, unbelievable data that you can get from this. Yeah. It takes a little bit of time to figure out what all it's telling you, but... I'm a little overwhelmed. Yeah. Yeah. But it has really nice, you know, documentation, and I know users that have got some really valuable information from this tool. So it works very well. It's just nice that users have options either way. Yeah. Yeah. And this being a Visual Studio show, we hope you use Visual Studio, but there's other tools, too. Sweet. So if I wanted to go learn more, since I'm very much new to the profiling space, especially with application insights, where can I go to learn more? Yeah. You can go to our Help page, which is here. We have pretty extensive documentation. We're looking to improve it, so leave us a comment if you have questions, and we can edit this documentation anytime. But this tells you how to enable it for different types of services, and then some other options that you have. And then there's some troubleshooting at the bottom if you have problems. Great. Yeah. Seems like that's a lot of good stuff to get started with. Yeah. We hope so. Sweet. And I mean, what's next for the profiler? Well, thanks for asking. We are looking at expanding what we show. In the browser, we have this view of just one request, but we'd like to be able to show, and we've recently added this flame graph. I can't remember if I showed that earlier. But we want to be able to show you information from the whole trace file. This is a very request-oriented view, and we've gotten a lot of feedback from users that they'd like to see. Without downloading the trace, can I see information about the whole trace? And so that's what we're looking at doing now, is allowing you to see maybe a flame graph for the entire ETW session. We want to give you some more insights, too, as to where things might be slowing down and why. And so we're working on some detection of patterns that we see in people's code, and we can give you hints about what might be slowing your code down. Cool. Yeah. Well, I can't wait to see that stuff. Yeah. We'll have to come back and do another show when we have some of that ready. Yeah, definitely. Speaking of which, thanks for coming now. I think that's really cool info that I'm sure tons of people would find incredibly useful whenever they run into those performance issues. Yeah. You're welcome. Great. And then at the top here, there's the help and send feedback. If you use these links, it'll contact our team also. So if you need help with things, we're there ready to help people. Awesome. Yeah. We actually do pay attention to the feedback that people send. Yeah. Yeah. Don't hesitate to reach out. And with that, yeah, once again, thank you so much. And hope everyone goes and tries out the cool profiling tools that we just talked about. And with that, happy coding.